101527 FINAL READ TRUST FUND REPORT 2008 –2015 2 FINAL READ Trust Fund Report 2008–2015 MESSAGE FROM THE READ TRUST FUND PROGRAM MANAGER Over the past seven years, the READ Trust Fund program has provided financial and technical support to low-income countries to help them improve their students’ learning outcomes through the design, implementation, and use of more effective student assessment systems. The results of this groundbreaking partnership between the Russian Federation and the World Bank are impressive. At the global level, the program has produced many high-quality knowledge products and tools. At the country level, local teams have significantly increased their capacity to carry out and use the data from assessments to improve education quality and student learning. These results are due to the efforts of many dedicated individuals and teams. We are particularly grateful to the Russian Government for their generosity and foresight in supporting this critical aspect of the global education agenda; to the country teams who worked tirelessly to design and implement activities on the ground; to the World Bank staff and international experts who assisted them; and to the READ Council members for providing guidance and strategic advice along the way. As we head into a new era of Sustainable Development Goals that will require countries to measure and report on student learning, the work supported under the READ Trust Fund program will be an important resource upon which countries can draw. On behalf of the READ Trust Fund program team, I am pleased to present this final report on the key achievements and results of this pioneering program. Marguerite Clarke READ Trust Fund Program Manager, World Bank This report was prepared by Emily Gardner under the guidance and direction of Marguerite Clarke, the READ Trust Fund Program Manager. Valuable inputs and comments were provided by members of the READ Trust Fund Global and Country Teams, including Cristian Aedo, Saodat Bazarova, Fadila Caillaud, Michael Crawford, Dingyong Hou, Ana Jurema, Julia Liberman, Diana Manevskaya, Nino Kutateladze, Jem Heinzel Nelson, Vidya Putcha, Cristina Panasco Santos, Gulmira Sultanova, An Thi My Tran, and Girma Woldetsadik. Design: Phoenix Design Aid A/S, Denmark TABLE OF CONTENTS EXECUTIVE SUMMARY.....................................................................................................3 FINANCIAL SUMMARY....................................................................................................10 GLOBAL-LEVEL RESULTS..............................................................................................13 SABER-Student Assessment.................................................................................... 13 Conceptual Framework and Diagnostic Tools................................................. 14 Country and Regional Reports........................................................................ 17 Case Studies.................................................................................................. 18 Reviews and Research on Key Assessment Topics........................................ 18 Training Sessions and Learning Events...........................................................20 SABER-Learning Standards......................................................................................21 Conceptual Framework and Diagnostic Tools.................................................21 Country Reports.............................................................................................21 Case Studies..................................................................................................21 Global Knowledge Sharing........................................................................................22 READ Trust Fund Conferences and Regional Workshops...............................22 Co-Sponsored Events.....................................................................................25 Partnerships...................................................................................................26 Global Knowledge Products...........................................................................29 Knowledge Dissemination...............................................................................31 Program Coordination...............................................................................................32 READ Council Meetings..................................................................................32 READ Trust Fund Technical Group Retreats....................................................33 Collaboration with READ RAS and CICED......................................................35 Program Monitoring and Reporting.................................................................38 COUNTRY-LEVEL RESULTS...........................................................................................39 Angola......................................................................................................................43 Armenia....................................................................................................................52 Ethiopia.....................................................................................................................59 Kyrgyz Republic........................................................................................................68 Mozambique.............................................................................................................78 Tajikistan...................................................................................................................87 Vietnam.....................................................................................................................96 Zambia....................................................................................................................108 APPENDICES.................................................................................................................. 119 Glossary of Assessment Terms............................................................................... 119 SABER-Student Assessment Learning Events and Training Sessions..................... 120 Summaries of READ Trust Fund-Sponsored Events................................................ 123 Global Knowledge Products Supported by READ Trust Fund................................. 135 LIST OF ACRONYMS AEAA Association for Educational Assessment in Africa AIR American Institutes for Research ALECSO Arab League Educational, Cultural, and Scientific Organization ATC Assessment and Testing Center CAT Computer-Adaptive Testing CICED Center for International Cooperation in Education Development CEATM Center for Educational Assessment and Teaching Methods CNECE National Board of Examinations, Certification, and Equity DFID Department for International Development (United Kingdom) EAOKO Eurasian Association for Educational Assessment ECZ Examinations Council of Zambia EDI Early Development Instrument EGRA Early Grade Reading Assessment ETS Educational Testing Service GEQIP General Education Quality Improvement Program GPE Global Partnership for Education IAEA International Association for Educational Assessment ICT Information and Communications Technology ILSA International Large-Scale Assessment INDE National Institute for the Development of Education INIDE National Institute for Research and Development of Education IT Information Technology KAE Kyrgyz Academy of Education MENA Middle East and North Africa MOET Ministry of Education and Training NEAEA National Educational Assessment and Examinations Agency NASTG National Assessment System Technical Group NLSA National Large-Scale Assessment NTC National Testing Center OECD Organization for Economic Cooperation and Development PISA Program for International Student Assessment RAS Reimbursable Advisory Services READ Russia Education Aid for Development SABER Systems Approach for Better Education Results SACMEQ Southern Africa Consortium for Monitoring Education Quality SAM School Achievements Monitoring SIEF Strategic Impact Evaluation Fund TIMSS Trends in International Mathematics and Science Study UEE University Entrance Examination UIS UNESCO Institute for Statistics UNESCO United Nations Educational, Scientific, and Cultural Organization USAID United States Agency for International Development EXECUTIVE SUMMARY 3 EXECUTIVE SUMMARY Established in October 2008, the Russia Education Aid for Development (READ) program has been a collaboration of the Government of the Russian Federation and the World Bank that focuses on improving education quality in low-income countries. The READ Trust Fund was a part of this program, with an amount of US$32 million to be executed over a seven-year period, 2008 to 2015. Better Assessment for Better Learning The READ Trust Fund’s main objective was to help low-income countries improve their student learning outcomes through the design, implementation, and use of robust systems for student assessment. It supported analytical work and technical assistance to help countries: establish systems or institutions – or strengthen existing ones – that formulate learning goals and carry out assessments of student learning; improve existing or develop new instruments to measure student learning outcomes; and strengthen existing or develop new mechanisms (policies) to use learning outcomes data to improve teaching and learning. READ Trust Fund Organization and Structure Under the guidance of the READ Council, a group comprised of key Russian and World Bank officials, the READ Trust Fund supported work at both the global and country levels. At the global level, the focus was on generating and sharing knowledge and good practices in the form of tools, analytical reports, and case studies. At the country level (in Angola, Armenia, Ethiopia, the Kyrgyz Republic, Mozambique, Tajikistan, Vietnam, and Zambia), World Bank teams and country stakeholders worked together to implement a set of READ Trust Fund–supported activities that addressed gaps in the country’s existing learning assessment system. 4 FINAL READ Trust Fund Report 2008–2015 Figure 1. Main Areas of Program Activity under the READ Trust Fund READ Trust Fund Global-Level Work Country-Level Work SABER- SABER- READ Learning READ READ READ Student Kyrgyz Standards Angola Armenia Ethiopia Assessment Republic Program Knowledge READ READ READ READ Coordination Sharing Mozambique Tajikistan Vietnam Zambia The READ Trust Fund was organized around 12 main areas of program activity (see Figure 1). This report contains a detailed account of program achievements in each of these areas from 2008 to 2015. Together, the different sections provide a complete picture of program results and demonstrate the significant impact of the program. Key Program Achievements The World Bank’s Education Sector Strategy 2020 highlights the importance of “Learning for All.” The proven economic gains and poverty reduction tied to education are only obtainable when children actually learn. All actors in an education system need to know whether or not learning is taking place so that they can use this information to improve education quality. The main outcome of the READ Trust Fund program has been increased institutional capacity of countries to develop, carry out, and effectively use data from student assessments to improve education quality and student learning. Armed with information on how well students are performing, teachers, policy makers, and international donors alike are now better able to determine where to focus their energy and resources for the greatest improvement in learning outcomes. The activities supported by the READ Trust Fund program have had a significant impact on the work being carried out worldwide to strengthen EXECUTIVE SUMMARY 5 systems that measure learning outcomes. The activities and outcomes achieved under the READ Trust Fund have been referenced in many high-level conversations on defining the post-2015 development agenda in education and on identifying effective ways to strengthen countries’ capacity to measure learning outcomes. For both the Russian Federation and the World Bank, the partnership through the READ Program was groundbreaking. As a result of this innovative program, many developing countries received valuable support in the area of student assessment and several highly useful knowledge tools have been created. The following summary captures the main program achievements at both the global and country levels, broken down by each area of program activity. Highlights of Global-Level Results At the global level, the READ Trust Fund contributed an impressive body of knowledge products on student assessment systems. There is now much greater understanding among education policymakers and practitioners of what it takes to develop an effective assessment system. By the end of the program, nearly 60 countries had used the SABER-Student Assessment tools developed under READ to analyze the status of their assessment systems; more than 30 different knowledge products on assessment topics had been published; and over 2,800 individuals had benefited from trainings and learning events on the SABER-Student Assessment tools. Key partnerships had also been established with donors and education practitioners working in the field of student assessment, such as the OECD, UIS, Brookings Institution, and STEP Skills Measuresment program. These strategic partnerships enabled READ to have a key role in several international initiatives to develop new tools to monitor education quality and learning outcomes. Through joint publications, co-sponsored events, and other means of collaboration, the program partnered with more than 15 other donors and organizations and READ was widely recognized as a leader in pushing forward the education quality and learning outcomes measurement agenda. The main program achievements in each activity area at the global-level include: SABER-Student Assessment A conceptual framework that identifies the key indicator areas for tracking the development of an effective system and 4 questionnaires and rubrics to collect and evaluate data on each of the assessment types – classroom assessment, examinations, NLSA, and ILSA – found in most assessment systems 6 FINAL READ Trust Fund Report 2008–2015 SABER-Student Assessment tools applied in nearly 60 countries,  resulting in 2 regional reports, 37 full country reports, and a final benchmarking report evaluating the impact of the READ program in all 8 READ-beneficiary countries 11 case studies highlighting various countries’ best practices and lessons  learned in developing student assessment systems 36 different learning events and more than 2,800 development  practitioners, partners, and other donors trained on the tools and products developed under the SABER-Student Assessment program SABER-Learning Standards A conceptual framework that identifies the key characteristics of effective learning standards Questionnaires and rubrics to collect and evaluate data on content, process, and performance standards in the areas of Language Arts, Mathematics, and Science 4 case studies highlighting various countries’ best practices and lessons learned in developing learning standards Global Knowledge Sharing 4 READ Global Conferences and 2 READ Regional Workshops – each conference had more than 120 participants, including education leaders from all 8 READ Trust Fund-supported countries Co-sponsorship and participation in 4 AEAA and IAEA conferences to share READ program results with more than 1400 assessment experts from over 60 different countries Key partnerships established to support the development of international assessment initiatives (i.e., PISA for Development and STEP Skills Measurement), leading to the publication of capacity building plans for all 6 PISA for Development pilot countries, 2 high-level reports, and a tool-kit on the development of socio-emotional skills A 5 volume series on key concepts in national assessments of student achievement levels and the dissemination of more than 12,000 copies Program Coordination 12 READ Council and READ Partnership Meetings to provide guidance on the strategic direction of program activities and identify areas for collaboration across the READ Trust Fund, READ RAS, and CICED programs A READ Technical Group, comprised of 6-8 international assessment experts, which supported the development of global knowledge products and advised country teams during the development of their action plans More than 8 events and training sessions to promote the SAM and ICT Literacy tools developed under CICED A READ Results Framework and 5 READ Trust Fund Annual Reports that capture program achievements and track country progress EXECUTIVE SUMMARY 7 Highlights of Country-Level Results At the country level, a final evaluation of each country program determined that all of the countries were able to use READ support to develop their assessment systems in significant ways and lay a strong foundation for future work in this area. By the end of the READ program, 4 new agencies or units responsible for assessment activities had been established, 6 new policies governing assessment activities had been adopted, 8 new testing instruments had been developed, and more than 20,000 individuals had received some level of training on assessment. To sustain these capacity-building activities, 13 formal training courses on assessment for teachers, school leaders, and education officials were also developed. The main achievements for each country program include: Angola Nearly 20 specialized training activities, national workshops, international conferences, and study visits on assessment topics, benefiting more than 300 Ministry of Education staff, provincial education officials, and teacher trainers A new unit within the Ministry of Education responsible for assessment activities – the National Assessment Technical Group An early grade reading assessment – EGRA – implemented in 144 schools throughout Angola’s 18 provinces, and training for more than 100 education authorities in the skills needed to ensure the successful implementation of EGRA Participation of the country in its first ILSA – SACMEQ Armenia Training for over 100 key education stakeholders through participation in 15 different international conferences and specialized courses on assessment topics A new Master’s course in education that incorporates courses on student assessment 2 new assessment tools – an NLSA to assess grade 8 students in the areas of Armenian, Literature, and History and an ICT Literacy Test for grade 9 students 2 training courses on classroom assessment and examinations for pre- and in-service teachers piloted with 70 university lecturers and Armenian teachers 8 FINAL READ Trust Fund Report 2008–2015 Ethiopia A new agency – the National Educational Assessment and Examinations Agency – responsible for national examinations and NLSA and 2 new national policies and guidelines on national examinations and NLSA Capacity building through study visits, international conferences, workshops, and specialized courses on assessment topics for more than 400 education leaders and practitioners A new Directorate for School Inspection, a new school inspection framework and guidelines, training for more than 60 Ministry of Education staff, and the participation of nearly 28,000 schools in school inspection activities A manual on classroom assessment for teachers and training for 70 Ministry of Education and university staff on its use Kyrgyz Republic A new national assessment strategy to support the government’s goal of improved learning outcomes More than 60 Ministry of Education and education experts participated in 12 different learning events on assessment topics A new manual and guidelines on classroom assessment, training sessions for more than 6,200 primary school teachers, and 2 formal training courses on classroom assessment added to the curriculum for in-service and pre-service teachers A new school leaving examination for grade 11 and training on item development and the new examination for 145 NTC staff, subject experts, and school administrators Mozambique 8 Ministry of Education staff and 1 University staff earned their Masters degrees in Education Management and Evaluation. More than 50 Ministry of Education staff participated in several international training events and gained a greater understanding of key issues and best practices in assessment A new classroom assessment tool, Provina, developed to assess early grade reading skills in grade 3, which was piloted and then scaled-up to include 206 schools in 5 districts A new policy to ensure regular NLSAs to assess grade 3 students in Portuguese and Mathematics Tajikistan A new system, policies, and testing instruments for a Unified University Entrance Examination (UEE) A new NTC building and 60 full-time staff members to implement the new system Training sessions for more than 5,100 NTC staff, district education staff, registrars, administrators, and secondary school directors to ensure successful implementation of the new UEE EXECUTIVE SUMMARY 9 In addition to training specifically targeted to the new UEE, more than 75 Ministry of Education and NTC staff benefited from study visits, international conferences, and other specialized workshops on assessment topics Vietnam 5 training modules on assessment for key stakeholder groups and 26 training sessions delivered to more than 2,800 participants More than 20 workshops with nearly 1,000 representatives from the Ministry of Education, universities, and school leaders to discuss various policies and reform efforts to improve assessment activities New policies and guidelines to support both classroom assessment and national examinations and new assessment frameworks and examinations for 7 different subject areas Successful participation in the country’s first ILSA – PISA 2012 Zambia Through participation in 35 different workshops, international conferences, and a study visit, more than 1,600 Ministry of Education staff and other key education stakeholders received targeted training on assessment topics Based on an in-depth quality review, an improved version of the grade 5 NLSA and testing instruments for a new grade 9 NLSA developed A broad-reaching communications plan to share the findings of the 2012 National Assessment in grade 5, which included the creation of a 10-minute film entitled, “Are Zambia’s Children Learning?”, and a DVD with a teacher discussion guide and 2 case studies on effective teaching strategies 2 courses on assessment – 1 for School Inspectors and 1 for pre-service teachers Education Quality and Assessment in the Post-2015 Development Agenda Through the joint efforts of Russia and the World Bank, the READ program has made significant and recognized contributions to the Learning for All agenda and increased global attention on the importance of education quality and learning outcomes. In the global discourse, the attention given to the importance of learning and monitoring education quality continues to grow. In the post-2015 era, work to establish strong systems for assessing learning outcomes is likely to expand. In this regard, both the READ program and the eight countries supported under the program are well positioned to play an important leadership role. Having already experienced a six-year journey of activities and outputs at global and country levels, the READ program offers an exciting and evidence- based model to those who wish to learn how exactly to go about improving and institutionalizing the way they monitor and use information on student learning outcomes. 10 FINAL READ Trust Fund Report 2008–2015 FINANCIAL SUMMARY The READ Trust Fund was designed to support activities at both the global and country levels. Initially, the fund was set up to be executed only by World Bank staff. However, mid-way through the program, it was agreed to make it possible for some of the funds to be executed by recipient country governments. (Ultimately, the Kyrgyz Republic and Tajikistan were the only two countries with recipient-executed programs.) Global-Level vs. Country-Level Activities The US$32 million under the READ Trust Fund was distributed so that 35 percent of the funds were set aside for global-level activities and 60 percent for country-specific work programs (see Figure 2). To allow sufficient time to complete a final evaluation of each country’s progress, all country-level grants were closed by October 2014. At that time, the balance of unspent funds (approximately US$950,000) was reallocated to fund key global-level activities and partnerships. The final percentages account for these additional funds at the global-level as the program nears completion. During the program’s first few years of implementation, the majority of funds were spent on developing global knowledge tools, capacity building activities, and agreeing on country-level action plans. While disbursements were initially greatest at the global level, rates of disbursements at the country level significantly increased in 2013 and 2014 (see Figure 3). Final Program Activities Over the course of the program, the Russian Ministry of Finance granted three different extensions to the original closing date – a one-year extension in 2011, a seven-month extension in 2014, and a six-month extension in 2015. The first extension accounted for initial delays at the country-level; the second extension allowed sufficient time to evaluate the impact of country-level activities; and the final extension is allowing the program to finalize a number of global-level knowledge products and fund some strategic partnerships. Table 1 includes a financial summary of all program activity through September 1, 2015. FINANCIAL SUMMARY 11 Figure 2. READ Trust Fund Allocations World Bank Trust Fund Management Fee 5.1% World Bank Trust Fund Management Fee 5.1% Global-level Country-level 36.4% 58.5% Global-level Country-level 36.4% 58.5% 35 30 3. Breakdown of Total Disbursements by Global vs. Country Figure 25 35 US$ MILLIONS US$ MILLIONS 20 30 15 25 10 20 5 15 0 10 2009 2010 2011 2012 2013 2014 2015 5 Country-level 0 Global-level 2009 2010 2011 2012 2013 2014 2015 Country-level Global-level 12 FINAL READ Trust Fund Report 2008–2015 Table 1. READ Trust Fund Financial Summary as of 9/1/15 (in U.S. dollars) Grant Total Disbursements Commitments Balance   MANAGEMENT FEE World Bank Trust Fund Management Fee 1,635,000 1,635,000 0 0 SUBTOTAL 1,635,000 1,635,000 0 0 GLOBAL PROGRAM SABER-Student Assessment 4,574,783 4,539,248 31,705 3,831 SABER-Learning Standards 805,724 805,724 0 0 Knowledge Sharing/Program Coordination 6,260,785 5,781,085 141,457 338,244 SUBTOTAL 11,641,292 11,126,056 173,162 342,075 COUNTRY PROGRAMS* READ Angola 2,141,721 2,141,721 0 0 READ Armenia 1,370,359 669,041 0 0 READ Ethiopia 1,056,013 1,056,013 0 0 READ Kyrgyz Republic 2,029,737 2,029,737 0 0 READ Mozambique 3,021,894 3,021,894 0 0 READ Tajikistan 4,009,175 4,009,175 0 0 READ Vietnam 2,905,644 2,905,644 0 0 READ Zambia 2,189,165 2,189,165 0 0 SUBTOTAL 18,723,708 18,723,708 0 0 TOTAL OVERALL $32,000,000 $31,484,764 $173,162 $342,075 GLOBAL-LEVEL RESULTS 13 GLOBAL-LEVEL RESULTS READ Trust Fund activities at the global level fell into four main categories – SABER-Student Assessment, SABER- Learning Standards, Global Knowledge Sharing, and Program Coordination. These global-level activities centered on: developing new knowledge products to expand the current understanding of what makes for an effective student assessment system; partnering and knowledge sharing with other donors and education development practitioners; and providing opportunities to share lessons across READ Trust Fund-supported countries and good practices in the field of student assessment more generally. The work on SABER-Student Assessment and SABER- Learning Standards is part of a larger World Bank initiative called Systems Approach for Better Education Results (SABER) (see Box 1). SABER-Student Assessment SABER-Student Assessment makes available information and tools based on the latest evidence for “what matters most” in creating effective assessment systems. It involves: classifying and analyzing student assessment systems around the world according to key criteria; collecting information on these key criteria by administering a set of questionnaires in a particular country; producing country and regional reports to inform policy dialogue; and  developing other products to add to the current knowledge base on how to develop and strengthen student assessment systems. 14 FINAL READ Trust Fund Report 2008–2015  ollecting and Analyzing Data on Education Policies around the Globe Box 1. C The World Bank has created an open data tool that provides in-depth, comparative, and easily accessible data on education policies around the world. This tool, SABER, helps countries collect and analyze information on their education policies, benchmark themselves against other countries, and prioritize areas for reform, with the goal of ensuring that all children and youth go to school and learn. SABER aims to improve education quality by supplying policymakers, civil society, school administrators, teachers, parents, and students with meaningful data about key education policy areas, including early childhood development, student assessment, teachers, school autonomy and accountability, and workforce development, among others. SABER helps countries improve their education systems in three ways: Providing new data. SABER collects comparable country data on education policies and  institutions to allow governments, researchers, and other stakeholders to measure and monitor progress. Benchmarking education policies and institutions. Each policy area is rated on a four-point  scale, from “Latent” to “Emerging” to “Established” and “Advanced.” These ratings highlight a country’s areas of strength and weakness while promoting cross-country learning. Highlighting key policy choices. SABER data collection and analysis produce an objective  snapshot of how well a country’s education system is performing in relation to global good practice. This helps highlight the most important policy choices to spur learning. The World Bank, through SABER, has analyzed data from more than 100 countries to guide more effective reforms and investments in education at all levels, from pre-primary to tertiary education and workforce development. (Source: World Bank, www.saber.worldbank.org.) Conceptual Framework and Diagnostic Tools Work on the SABER-Student Assessment knowledge products and tools began in 2010. Initially, the focus was on creating a conceptual framework and identifying the policies, institutional arrangements, and practices that matter most in establishing effective assessment systems (see Box 2). The results of this work are summarized in the 2012 publication, “What Matters Most for Student Assessment Systems: A Framework Paper”, which provides policy makers, education ministry officials, development organization staff, and other stakeholders with an evidence-based framework and key indicator areas for building an effective student assessment system. Once established, the framework was used to develop questionnaires and rubrics for collecting and evaluating data on key characteristics of a country’s assessment system. The questionnaires and rubrics were organized around four types of assessment activity – classroom assessment, examinations, national large-scale assessments (NLSA), and international large-scale GLOBAL-LEVEL RESULTS 15 assessments (ILSA). Early versions of the questionnaires and rubrics were piloted in Chile, New Zealand, and Uganda. Based on lessons learned from this experience, the tools underwent minor revisions and fine tuning. Since 2010, the questionnaires and rubrics have undergone three further revisions. Each revision aims to improve the utility and validity of the data generated by these diagnostic tools based on experience gained from applying them, feedback from stakeholders and other end users, and the latest research on what makes for an effective assessment system. In general, the revisions have involved adding questions or expanding portions of the rubrics to emphasize key criteria. The conceptual framework for SABER-Student Assessment underpins all of the work carried out under the READ Trust Fund program and has provided program participants with a common language for discussing key issues, requirements, and plans for building a more effective assessment system. The key indicator areas identified by SABER-Student Assessment provided the basis for the READ Trust Fund Results Framework, which was used to track each READ Trust Fund–supported country’s progress in strengthening its assessment system. 16 FINAL READ Trust Fund Report 2008–2015 Box 2. What Matters Most in a Student Assessment System? The SABER-Student Assessment framework is built on the available evidence base for what an effective assessment system looks like. The framework is structured around two main dimensions of assessment systems: the types of assessment activities, and the quality of those activities. Assessment types Classroom assessment provides real-time information to support ongoing teaching and learning  in individual classrooms. Classroom assessments use a variety of formats – including observation, questioning, and paper-and-pencil tests – to evaluate student learning, generally on a daily basis. Examinations provide a basis for selecting or certifying students as they move from one level of  the education system to the next. Examinations cover the main subject areas in the curriculum and usually involve essays and multiple-choice questions. Large-scale, system-level assessments provide feedback on the overall performance of the  education system at particular grades or age levels. These assessments typically cover a few subjects on a regular basis (such as every 3 to 5 years), are often sample-based, and use multiple- choice and short-answer formats. They may be national or international in scope. Quality drivers Enabling context refers to the broader setting in which the assessment activity takes place and  the extent to which that setting is conducive. It includes the policy framework for assessment activities, institutional structures, sources of funding, and the presence of trained assessment staff. System alignment refers to the extent to which the assessment is aligned with the rest of the  education system. It includes the degree of congruence between assessment activities and system learning goals, standards, curriculum, and pre- and in-service teacher training. Assessment quality refers to the psychometric quality of the instruments, processes, and  procedures for the assessment activity. It covers such issues as the design and implementation of assessment activities, analysis and interpretation of student responses, and the appropriateness of how assessment results are reported and used. Crossing the quality drivers with the different assessment types provides the framework and broad indicator areas shown in Figure 4. This framework is a starting point for identifying indicators that can be used to review assessment systems and plan for their improvement. Figure 4. Framework and Key Indicator Areas for Student Assessment Systems ASSESSMENT TYPES/PURPOSES Classroom assessment Examinations Large-scale, system- level assessments Enabling context Policies Leadership and public engagement Funding QUALITY DRIVERS Institutional arrangements Human resources System alignment Learning/quality goals Curriculum Pre- and in-service teacher training opportunities Assessment quality Ensuring quality (design, administration, analysis) Ensuring effectiveness (Source: Clarke, M. 2012. “What Matters Most for Student Assessment Systems: A Framework Paper.” World Bank. Washington, DC.) GLOBAL-LEVEL RESULTS 17 Country and Regional Reports Using the data obtained from applying the SABER-Student Assessment tools to a particular country, a report is produced that summarizes how well the country’s student assessment system is performing and offers recommendations for improvement. In 2011, the SABER-Student Assessment tools were retroactively applied to 2009 data for READ Trust Fund–supported countries to produce country reports and establish baselines against which to measure these countries’ progress in developing their student assessment systems. In 2014, upon completion of the READ Trust Fund program, the benchmarking tools were applied again to the eight countries to determine the progress made in developing their assessment systems and to identify specific aspects of their systems that had improved as a result of the activities supported under READ. The final developmental ratings for each country are described in the country-level section of this report. A full summary of the findings from the benchmarking exercise are available on the READ website in the report, “Russia Education Aid for Development Trust Fund Program: Results of the Country Benchmarking Exercise, 2009-2014.” In 2011, the SABER-Student Assessment team used an abbreviated version of the questionnaires and rubrics to benchmark countries in East Asia and the Middle East & North Africa (MENA). Summary findings for countries in East Asia were made available in the report, “Strengthening Education Quality in East Asia”, which was published in partnership with the United Nations Educational, Scientific, and Cultural Organization (UNESCO). In the case of the MENA countries, the application of abbreviated versions of the tools in a few countries was quickly followed by application of full versions of the tools in 17 countries, which allowed for the production of a detailed regional report as well as individual country reports. This work was carried out in cooperation with the Arab League Educational, Cultural, and Scientific Organization (ALECSO) and UNESCO. The regional findings were published in the 2014 report, “Regional Mapping Report on Assessment in the Arab States”, which is available in Arabic and English on ALECSO’s website. From 2012 through 2015, there continued to be strong demand from countries to use the SABER-Student Assessment tools to benchmark their assessment systems. By the end of the READ Trust Fund program, in addition to the work carried out on the eight READ Trust Fund-supported countries, the team had collected data and produced country reports for 29 countries. In total, data for student assessment systems in 51 countries is now publically available on the SABER-Student Assessment website. (See Figure 5 for a list of all countries and Appendix 4 for all country reports.) 18 FINAL READ Trust Fund Report 2008–2015 Case Studies As a way to build the evidence base on good practices in reforming and strengthening assessment systems, a series of case studies was commissioned early on in the READ program with the goal of finding out what countries around the world have done to improve their assessment systems. The case studies focused on different aspects of the quality drivers outlined in the SABER-Student Assessment conceptual framework. The first set of case studies looked at the enabling context in six countries – Australia, Brazil, Chile, New Zealand, Russia, and Uganda – and identified key drivers and choices that led to positive changes over time. A second set of six case studies focused on the uses of assessment information in Chile, The Gambia, Germany, Jordan, Russia, and Singapore. Each of the 12 case studies was presented at one of the READ Global Conferences, and most were further developed and published under the READ Working Paper series. Given some general interest expressed in Armenia’s efforts to strengthen their student assessment system, an additional case study was commissioned to capture the lessons learned from this READ-sponsored country’s experiences. By the end of the READ Trust Fund program, 11 case studies had been published, including two on Chile, two on Russia, and one each on Armenia, Australia (Queensland), Brazil, The Gambia, Jordan, New Zealand, and Uganda. Reviews and Research on Key Assessment Topics A portion of the global work program was dedicated to reviews and research on key topics and issues in the area of student assessment. As part of this work stream, the World Bank Education Note, “Measuring Learning – How Effective Student Assessment Systems Can Help Achieve Learning for All,” and the READ Working Paper, “Review of World Bank Support for Student Assessment Activities in Client Countries, 1998–2009,” were published. Other work included a review by the Inter-American Dialogue of learning assessment systems in Latin America. The findings were summarized in the 2015 report, “What has Happened with Learning Assessment in Latin America? Lessons learned over the last decade”. The report examines changes in assessment systems in Latin America through an analysis of 1) institutional and organizational frameworks; 2) the dissemination and uses of assessment information by authorities, school officials and teachers, families and civil society; and 3) participation in international large-scale assessments. Two other reports were published in partnership with the Organization for Economic Cooperation and Development (OECD) as part of their PISA (Program for International Student Assessment) for Development initiative. The first of these reports, “Review of Component Skills Assessed and Contextual Data Collection Used in Relevant International Assessments”, looks at the similarities and differences among existing international, regional, GLOBAL-LEVEL RESULTS 19 and household-based large-scale assessment programs. The second report, “The Experience of Middle-Income Countries Participating in PISA, 2000-2015”, analyzes the experiences of non-OECD countries, particularly low- and lower-middle income countries, in PISA. (See page 26 for further details.) Figure 5. Countries Participating in SABER-Student Assessment Activities (as of 9/1/2015) Case Studies Country Reports Regional Reports IBRD 40127 JULY 2013 This map was produced by the Map Design Unit of The World Bank. The boundaries, colors, denominations and any other information shown on this map do not imply, on the part of The World Bank Group, any judgment on the legal status of any territory, or any endorsement or acceptance of such boundaries. Case Studies Country Reports Regional Reports Australia Angola Oman (2) Cambodia Bahrain (1) Armenia (1) Armenia (3) Pakistan (Punjab) China Egypt (1) Brazil Bahrain (2) Pakistan (Sindh) Hong Kong SAR, China Iraq (1) Chile Bangladesh Papua New Guinea Indonesia Jordan (6) The Gambia Brunei Qatar (2) Japan Kuwait (1) Germany Democratic Republic of Samoa Rep. of Korea Lebanon (1) Jordan (4) the Congo Saudi Arabia (2) Lao PDR Libya (1) New Zealand Egypt (2) Serbia Malaysia Mauritania (1) Uganda (1) Ethiopia Solomon Islands Mongolia Oman (1) Singapore (2) Ghana Sri Lanka The Philippines Qatar (1) Russian Federation Iraq (2) Sudan (2) Singapore Saudi Arabia (1) Jordan (5) Syria (2) Thailand Sudan (1) Kazakhstan Tajikistan Vietnam (1) Syria (1) Kuwait (2) Tunisia (2) Tunisia (1) Kyrgyz Republic Uganda United Arab Emirates (1) Lebanon (2) United Arab Emirates (2) West Bank & Gaza (1) Libya (2) Vietnam (2) Yemen (1) Macedonia, FYR West Bank & Gaza (2) Mauritania (2) Yemen (2) Mozambique Zambia Nepal (1) Also a Country Report (2) Also a Regional Report (3) Also a Case Study (4) Also a Country Report and Regional Report (5) Also a Case Study and Regional Report (6) Also a Case Study and Country Report (Source: World Bank, www.saber.worldbank.org.) 20 FINAL READ Trust Fund Report 2008–2015 Training Sessions and Learning Events Since 2010, at least 36 different events were organized to train development practitioners, partners, and other donors on the tools and products developed under the SABER-Student Assessment program. (See Appendix 2 for a complete list of training sessions and learning events.) These events provided a platform for showcasing the work carried out and tools developed under the READ program, and for soliciting suggestions for improvement. They also produced greater visibility for the READ program on the international stage. Through these events, the SABER-Student Assessment team was able to reach more than 2,800 individuals. Add to this the general workshops and trainings on SABER that took place during this time period, and which included student assessment, and the total number of people reached surpasses 5,000. As a way to further expand the reach of the SABER-Student Assessment program, the training materials developed for face-to-face sessions were turned into an e-learning course for education policymakers and practitioners. It is expected that this e-learning course will be launched globally in 2016. GLOBAL-LEVEL RESULTS 21 SABER-Learning Standards In 2011, READ Trust Fund support was used to initiate background work for a possible SABER domain on learning standards. The work was put out for competitive bid and ultimately awarded to the American Institutes for Research (AIR). Similar to the work carried out on SABER-Student Assessment, the aim was to develop an evidence-based framework and set of tools to benchmark learning standards policies and systems in countries around the world. In 2013, given the close conceptual links between the topics of assessment and learning standards, it was decided to incorporate the work on the latter into the SABER-Student Assessment domain as part of the broader suite of tools and products available in this area. Conceptual Framework and Diagnostic Tools In 2011, the AIR team completed a report on key characteristics of effective learning standards based on their review of the existing global evidence base and best practices. Subsequently, a set of questionnaires and rubrics were developed that could be used to collect and evaluate data on these aspects of a country’s education system. The tools allow for the collection of data on content, process, and performance standards in the areas of Language Arts, Mathematics, and Science. In 2012, the diagnostic tools were piloted, reviewed, and revised and a user’s guide was created to guide the data collection process. These products are being made available as part of the suite of tools under SABER-Student Assessment. Country Reports The piloting of the tools involved the collection of data and production of country reports for six countries: Armenia, Jordan, Mozambique, Nepal, Peru, and Vietnam. The country reports summarize key findings and recommendations derived from application of the diagnostic tools. These reports are available as background material on the SABER website (www. saber.worldbank.org). Case Studies To build the evidence base in this topic area, and to capture lessons learned, a series of case studies were commissioned that describe what various countries have done to develop and implement effective learning standards. The six case study countries are Chile, India, Jordan, Namibia, New Zealand, and Singapore. These case studies are being published as part of the READ Working Paper series. 22 FINAL READ Trust Fund Report 2008–2015 Global Knowledge Sharing At the global level, the program focused on identifying opportunities to share lessons and best practices in student assessment with READ Trust Fund-supported and other developing countries. These activities included global conferences and regional workshops, partnerships and collaborations with others working on the topic of student assessment, and the development and dissemination of knowledge products on assessment. These activities enhanced the program’s work at the country level and increased overall awareness of good practices in assessment. READ Trust Fund Conferences and Regional Workshops The READ Council decided that the READ Trust Fund would hold an annual event to bring together everyone working directly on the READ program (including education officials from READ-supported countries, Russian officials and education experts, World Bank staff, and international assessment experts) to share experiences, discuss good practices, and learn more about Russia’s role as a donor in the area of education quality. (See Table 2 for additional details and Appendix 3 for summaries of each of these events.) The READ Global Conferences held in 2009, 2010, 2011, and 2014 served as important milestones for the program. All four conferences were highly successful and greatly valued, in particular by READ-country participants. In 2012 and 2013, in lieu of a stand-alone READ Global Conference, the READ Trust Fund co-sponsored annual conferences held by international assessment associations (the International Association for Educational Assessment [IAEA] and the Association for Educational Assessment in Africa [AEAA]) and organized READ Regional Workshops. This was done in order to forge closer connections with the broader assessment community and to provide opportunities for READ Trust Fund-supported countries to build networks with their regional neighbors. At all of these co-sponsored events, there were several panel sessions on READ Trust Fund and READ RAS activities. GLOBAL-LEVEL RESULTS 23 Table 2: READ Trust Fund Conferences and Regional Workshops First READ Global Conference Location/Date: Moscow, Russia – September 30-October 2, 2009 Theme: “Developing a Vision for Assessment Systems” Key messages: When children learn, nations prosper – countries cannot afford to postpone making learning a national priority. Evidence – obtained through regular assessments of student learning – and use of that evidence, is vital. Developing a system of assessment is important, i.e., creating an enabling environment for assessment, ensuring alignment with the rest of the education system, and paying attention to the quality of the instruments used to measure student learning. Second READ Global Conference Location/Date: Nairobi, Kenya – October 4-6, 2010 Theme: “Strengthening the Enabling Environment for Assessment Systems” Key messages: Leadership plays a vital role in building effective assessment systems. Establishing a strong assessment system requires focusing on the policies, institutional arrangements, and financial and human resources for assessment activities. Strengthening the enabling environment is fundamental to ensuring that learning outcomes improve. Third READ Global Conference* Location/Date: Eschborn, Germany – October 23-26, 2011 Theme: “Using Student Assessment Results for Education Quality and Systems Strengthening”. Key messages:  Assessments provide a way to gather information on what students know, understand, and can do and identify areas in need of quality improvements.  Assessment activities are only effective in improving education quality when they are used in ways that inform policymaker decisions, help teachers improve their teaching practices, and empower parents with the knowledge of how well their children are actually learning.  (*This event was combined with the World Bank’s Education Conference for Eastern Europe and Central Asia, and co-hosted by the German Development Agency – GIZ.) 24 FINAL READ Trust Fund Report 2008–2015 READ Trust Fund Regional Workshops Locations/Dates: Astana, Kazakhstan – September 19, 2012; Maputo, Mozambique – December 3-5, 2012 Objectives: The workshops were designed to encourage South-South knowledge sharing across READ-supported countries. Specific objectives included: supporting the READ country teams to implement their action plans; providing READ country teams with an opportunity to discuss key challenges faced during implementation of their action plans; and providing opportunities to share lessons learned and expertise across READ country teams. 31st AEAA Conference/ 39th IAEA Conference Locations/Dates: Arusha, Tanzania – August 12-16, 2013; Tel Aviv, Israel – October 20-25, 2013 Objective: Based on the success of the partnership with the International Association for Educational Assessment (IAEA) Conference in 2012, the READ Trust Fund followed a similar model in 2013, partnering again with the IAEA as well as with the Association for Educational Assessment in Africa (AEAA). Panel sessions on READ-supported country achievements were held at both organizations’ annual conferences. Fourth READ Global Conference Location/Date: St. Petersburg, Russia – May 12-16, 2014 Theme: “Measuring for Success: The Role of Assessment in Achieving Learning Goals” Key messages: Measurement helps identify which initiatives are successful at achieving impact – spending choices should be guided by the relative impact on learning. High quality national and international assessments allow countries to identify gaps in performance and target resources where most needed. Classroom assessment by teachers is key – they need comprehensive training on this topic which includes a focus on learning goals and how to collect evidence of student learning. GLOBAL-LEVEL RESULTS 25 Co-Sponsored Events To achieve greater visibility for the READ program and build connections with others working on assessment, READ occasionally partnered with other organizations on assessment-related events. These collaborations provided another platform for showcasing the READ work, generated additional support from assessment experts and donors, and created greater visibility for the program on the international stage. All Children Reading by 2015 – From Assessment to Action  Workshop: Co-sponsored by the READ Trust Fund, the Global Partnership for Education (GPE) (formerly known as the Education for All – Fast Track Initiative) and the United States Agency for International Development (USAID), this three-day event brought together country leaders, experts, and donors to discuss how countries can ensure that all children are able to read by the end of Grade 2. The workshop aimed to help the countries in attendance (Egypt, Ethiopia, The Gambia, Guatemala, Kenya, Liberia, Mali, Nicaragua, and South Africa), most of which had recently conducted an early-grade reading assessment, to devise clear strategies on how to use the assessment results to identify actions to improve learning outcomes in reading. This event took place in Washington, DC from April 12-14, 2010. Users and Uses of National Assessment Results Workshop:  Another partnered event was the workshop, “Users and Uses of National Assessment Results”, held on April 13–15, 2010 in Capetown, South Africa. The Africa Alliance for Learning and Education Results (Africa ALERT) brought together representatives from international organizations and senior policy makers and technical staff in charge of national assessments from nine African countries (Ethiopia, Kenya, Lesotho, Mauritius, Mozambique, Nigeria, South Africa, Uganda, and Zambia) for a three-day workshop. Participants analyzed ways in which assessment results have been used for policy design, teacher professional development, enhancing school management, and informing parents and increasing public awareness, and then developed country-specific plans for ways to better use national assessment results. Benchmarking Education Systems for Results – East Asia  Regional Conference: The World Bank’s SABER Program partnered with organizations in East Asia to benchmark the region’s education systems. READ (which was the primary source of support for the benchmarking work on assessment) was invited to partner with the Korean government, the Nanyang Technological University, the UK Department for International Development (DFID), the Education Program Development Fund, and UNESCO to organize a conference in Singapore on June 21–23, 2010 at which the results of the regional benchmarking were presented. Bringing together international experts, representatives from donor organizations, and education policy makers and school 26 FINAL READ Trust Fund Report 2008–2015 administrators from 20 countries throughout the East Asia Region, this event provided an opportunity to build partnerships, promote assessment of education programs, and help countries define a framework for benchmarking. World Bank Symposium – Assessment for Global Learning: As the  development community comes closer to finalizing the post-2015 development agenda in education, there have been serious conversations about the possibility of a universal learning goal. In order to contribute to these discussions, and with the support of the READ Trust Fund, the World Bank hosted a symposium in November 2013 on the topic of “Assessment for Global Learning.” This two-day symposium brought together international assessment experts and policy researchers to review and discuss the existing menu of tools and approaches for monitoring and supporting student learning, and to explore how best they can be used or improved to meet the needs of the post-2015 development agenda. Partnerships Over the course of the program, the READ Trust Fund established a number of partnerships with other key players working in the area of student assessment. Partnerships with OECD on the PISA for Development initiative, UNESCO Institute for Statistics (UIS) and Brookings Institution on the Learning Metrics Task Force, and the World Bank team leading the STEP Skills Measurement Program allowed the program to leverage additional funds and play an influential role in several key international initiatives that also promote education quality and improved learning outcomes. PISA: Beginning in 2009, and with the ongoing support of the READ  Trust Fund, the World Bank and the Governing Board of OECD’s PISA have been working together to find ways to adapt the PISA tests so that low-income countries can more readily benefit from them. As a result of this collaboration, a new reading component skills assessment was developed and made available as an international option for the PISA 2012 survey, of which both Poland and Peru were able to avail. PISA for Development: In 2013, OECD announced a new initiative –  PISA for Development. Through this initiative, OECD is piloting a modified version of the current PISA, one in which certain enhancements have been made for the developing country context. After initial discussions, six countries were selected to participate in the pilot: Cambodia, Ecuador, Guatemala, Paraguay, Senegal, and Zambia. READ Trust Fund support was used to finance a number of a key activities to launch the initiative, including hosting technical workshops, conducting capacity needs analyses and developing capacity building plans for participating countries, and producing two high-level reports. The READ Trust Fund GLOBAL-LEVEL RESULTS 27 program manager also serves on the project’s International Steering Group and is part of the group of technical experts leading this work. (Contributions from the READ Trust Fund to the PISA for Development pilot have directly supported Zambia’s international participation costs.) •  Technical Workshops: In 2013 and 2014, READ provided support for three different PISA for Development technical workshops – one in Paris at OECD, one in Washington, DC at the World Bank, and one in Montreal at UIS. READ funds were used to support the participation of various country representatives at these events. During the workshops, country representatives, assessment experts, and international donors discussed the development of the testing instruments and possible ways to reach out-of-school 15-year olds. Capacity Building Plans: As part of the PISA for Development initiative, •  participating countries are meant to benefit from activities aimed at building their national capacity in assessment design, data analysis, and use of results for monitoring and improvement. During the initial phase of the project, an analysis was carried out of each country’s capacity and needs in these areas. The SABER–Student Assessment tools and PISA technical standards were used as guidelines for these analyses. •  Publications: To help build the evidence base on ways to make PISA more relevant for a wider range of countries, READ support was used to commission two background reports. The first of these, “The Experiences of Middle-Income Countries Participating in PISA, 2000-2015,” provides a systematic review of the experiences of 40 middle-income countries that have participated in PISA over the years. The second report, “Review of Component Skills Assessed and Contextual Data Collection Used in Relevant International Assessments,” gives greater insight into possible ways of collaborating across different international assessments, including PISA. 28 FINAL READ Trust Fund Report 2008–2015 Learning Metrics Task Force: Another means of collaboration has  been through participation in the Learning Metrics Task Force. UIS and the Center for Universal Education at Brookings joined efforts to convene this task force, which is investigating the feasibility of global learning competencies and measures for the post-2015 development agenda. Through the READ Trust Fund program manager’s participation in this task force, the knowledge, tools, and experiences accumulated under the READ program have been able to inform these discussions. STEP Skills Measurement Program: Due to technology and other  advances, 21st century labor market requirements have shifted from needing workers with educational qualifications for “life-time” jobs to needing workers with strong cognitive and socio-emotional skills who are able to develop the skills needed to do specific jobs that are constantly changing. In 2010, the World Bank launched the STEP Skills Measurement Program to support countries in implementing surveys of the skills levels of their adult populations. Through the use of household and employer surveys, the STEP program collects information on three types of skills: cognitive, socio-emotional, and job-relevant. By 2015, three rounds of STEP surveys had taken place with the participation of 17 different countries across the globe. In 2014, the READ Trust Fund team partnered with the STEP Skills Measurement team in several ways to advance this work. •  STEP-Skills Measurement Statistics Pocketbook and Infographics: The STEP Skills Measurement surveys have generated a wealth of data on cognitive, socio-emotional, and job-relevant skills in the 17 countries that have participated thus far. To make this data more accessible and disseminate it further, READ support was used to create a statistics pocketbook and various infographics. •  Redesign of Socio-Emotional Skills Module: In order to incorporate lessons from the first two rounds of surveys as well as keep up with methodological developments, the STEP team embarked on a significant revision of the STEP survey instruments. READ support was used to redesign the module on socio-emotional skills. The support also covered production of a technical report summarizing findings from an analysis of the existing STEP data on socio-emotional skills. •  Toolkit on Socio-Emotional Skills: The World Bank has been working with the Ministry of Education in Peru to develop and implement a unique toolkit, based on international good practices, to teach socio- emotional skills throughout the entire cycle of basic education. The toolkit includes grade-specific curricula and instructional guides, hands-on interactive activities, and audio-visual material. With READ Trust Fund support, this toolkit was adapted and translated into English to make it readily accessible to other countries worldwide. GLOBAL-LEVEL RESULTS 29 •  STEP-Skills Measurement Reading Assessment: READ support was used to conduct additional analysis of the STEP Reading Assessment data for 12 countries. These analyses allowed for comparison of the countries’ results with the OECD’s PIAAC surveys. Global Knowledge Products With the support of the READ Trust Fund, several new global knowledge products on student assessment were published. These products enhanced the program’s work at the country level and increased overall awareness of good practices in assessment. National Assessments of Educational Achievement Series: The  READ Trust Fund supported the publication of a five-volume series on key concepts in national assessments of student achievement levels. This series introduces key concepts in national assessments of student achievement levels, from policy issues to address when designing and carrying out assessments through test development, sampling, data cleaning, statistics, report writing, and the use of results to improve education quality. The entire series was translated into French, Portuguese, and Russian to facilitate its use in READ Trust Fund- and other World Bank-recipient countries. (Note: Russia’s Center for International Cooperation in Education Development [CICED] oversaw the translations into Russian.) Table 3 includes a brief description of each volume in this flagship series. READ Working Paper Series: To meet the increasing demand for  materials on student assessment, the READ Trust Fund created its own working paper series. Learning and assessment experts from developing and industrialized countries were invited to write papers that addressed key aspects of the design and use of assessments and standards to support improved education quality and learning outcomes. These papers and case studies were published as part of the READ Working Paper series. By the end of the READ Trust Fund Program, a total of 17 READ Working Papers had been published. (See Appendix 4 for a list of all READ Working Paper Series publications.) 30 FINAL READ Trust Fund Report 2008–2015 Table 3. Volumes in the National Assessments of Educational Achievement Series Volume 1: Assessing National Achievement Levels in Education This introductory book describes key national assessment concepts and steps, and is intended primarily for policy makers and decision makers in education. It includes a description of the purposes and main features of national assessments, reasons for carrying out these assessments, and key decisions that have to be made. Volume 2: Developing Tests and Questionnaires for a National Assessment of Educational Achievement The second volume is aimed at national assessment teams and introduces them to the activities involved in the development of achievement tests. Some of the topics covered include: developing an assessment framework, writing multiple-choice and constructed-response type items, designing questionnaires, linking questionnaire and test score data, and the development of a test administration manual. Volume 3: Implementing a National Assessment of Educational Achievement This book focuses on practical issues to be addressed in implementing a large- scale national assessment program. It covers areas such as planning, budgeting, staffing, selecting test administrators, test security, and sampling. It also includes a CD with sampling software and a training dataset to be used in conjunction with the guide.  Volume 4: Analyzing Data from a National Assessment of Educational Achievement  This book provides step-by-step details on how to analyze data collected in a national assessment. Steps in the analysis of pilot and final test administrations are described in detail. An accompanying CD contains specifically designed exercises and supporting data files. Volume 5: Using the Results of a National Assessment of Educational Achievement The final book in the series focuses on writing reports in a way that will influence policy. Among other things, it introduces a methodology for designing a dissemination and communication strategy for a national assessment program and describes the preparation of the main report. It also highlights ways that countries have actually used the results of national assessments for policy making, curriculum reform, and monitoring of changes in achievement and other variables over time. (Source: World Bank, www.worldbank.org/readtf.) GLOBAL-LEVEL RESULTS 31 Knowledge Dissemination The READ Trust Fund engaged in a wide range of activities to disseminate the key messages of the program, highlight the role of Russia as a donor, and READ ANNUAL REPORT 2012 n 17 share the knowledge products developed. These activities included: 1) producing a video, highlighting the importance of assessment and education quality; 2) creating a website dedicated to READ Trust Fund program activities (www.worldbank.org/readtf); 3) disseminating stories on READ Trust Fund GLOBAL KNOWLEDGE SHARING At the Bank activities through internal/external World websites; global level, translating 4)works the program to identify global opportunities to share lessons and best practices in the knowledge products into Russian, field Portuguese, of learning assessmentand with READ Trust and French; 5) Fund–supported and other developing countries. These activities are meant to enhance the program’s work at the country level and increase overall awareness of best practices inpublications disseminating READ Trust Fund–supported assessment. at international events (i.e., World Bank education events, AEAA and IAEA READ Trust Fund Sponsored Events Conferences, and the Eurasian Association for Educational Assessment In 2012, (EAOKO) the READ Trust Fund Conferences). was a co-sponsor of the 38th IAEA Conference. This conference, which took place from September 17–21 in Astana, Kazakhstan, drew more than 320 participants from 40 different countries. The majority of the participants were practitioners and researchers involved in various assess- Documentary – “When Children  Learn, ment projects. Nations The READ Trust FundProsper”: It is the assistance of READ Fee-Based Services— and CICED—through organized several panel sessions on such topics as: (i) assessment-related tools being developed with READ why learning critical that countries understandsupport, (ii) activitiesmatters and what being implemented they by READ can Trust Fund–supported countries, and (iii) the creation of a new assessment association, the Eurasian Association for Educational Assessment (EAOKO) (see page 18). do to ensure that learning is taking place. In this documentary (24 and 7 The event provided an excellent opportunity to share the work being carried out under the READ program, and to connect Singapore, minute versions), stories from Colombia, and with leaders in the South global Africa assessment community. highlight various ways governments, As agreedschools, by the READ and communities Council, in lieu of a READ have Global Conference, two READ Trust Fund Regional focused on learning and measuring Workshops were student in the Eastern With performance. held in 2012—one Europe and Central Asia region (Astana, Kazakhstan, September 19) and one in the Africa region (Maputo, Mozambique, December 3–5). The target audiences for READ-supported countries as the primary each workshop target were audience, delegations and World the Bank video Task Teams from the eight countries receiving targeted support at the country level under the READ Trust Fund. The first workshop in Astana brought together was made available in English, Russian, and Portuguese. teams from Armenia, the Kyrgyz Republic, Tajikistan, and Vietnam; the second workshop in Maputo brought together teams from Angola, Ethiopia, Mozambique, and Zambia (see Appendix 2). Website: A website was created  to 1) highlight the role of Russia as a Knowledge Dissemination messages donor and 2) disseminate the keyThe hasREAD of the READ Trust Fund engagedprogram, READ in a wide range of activities to dis- country achievements, and theseminate knowledgethe key messages of the program, highlight the role of Russia products developed. This as a donor, capture READ Trust Fund–supported country achievements, website was a major communication and sharetool for the program the knowledge and included products developed at the global level. This has been done through various channels—producing a video, which regular stories on program results. Through highlights internal/external the importance World of assessment and Bank education quality, creating a website dedicated to READ Trust Fund program activities (www.world- websites, newsletters, and blogs, more than 25 web-based articles were bank.org/readtf), disseminating stories on READ Trust Fund activities published on READ program achievements. through internal/external World Bank websites, establishing a working paper series on topics related to student assessment, and disseminating READ Trust Fund–supported publications at international events. During Translations: To facilitate the 2012,  use more than 2,275 print copies of READ Working Papers were dis- of global products by countries seminated, in addition to almost 1,000 copies of volumes in the National receiving READ Trust Fund support, many Assessments of were translated Educational Achievementinto Russian, series. Portuguese, and French. By the end the To facilitate READ, of use of global translated materials products by countries receiving READ Trust Fund support, as well as by the included most of the READ Working Papers and the entire National Russian donor, considerable effort has been made to translate a significant amount of material (including a video, conference materials, and publications) into both Russian and Portuguese. In 2012, the “What Mat- Assessments of Educational Achievement series. ters Most for Student (See Appendix Assessment 4 for aPaper was translated, as were several working Systems” Framework complete list of all translations.) Materials translated into Russian were also widely disseminated in Russia through key partners such as CICED, the Russian Training Center in Moscow, and the Moscow School of Social and Economic Sciences. By the end of the READ Trust Fund program, more than 5,000 print copies and 2,700 electronic copies of the READ Working Papers had been disseminated, in addition to more than 4,200 print copies and 8,000 electronic copies of volumes in the National Assessments of Educational Achievement series. 32 FINAL READ Trust Fund Report 2008–2015 PROGRAM COORDINATION A variety of mechanisms were used to coordinate and monitor READ program implementation and ensure program quality. A READ Council was established; a READ Technical Group comprised of international assessment experts was formed; partnership meetings were held between the READ Trust Fund team, Russian authorities, the READ RAS team, and CICED; and regular progress reports were published. (See Table 4 for a list of all meetings.) READ Council Meetings The READ Council, comprised of Russian authorities and World Bank education sector leaders, was established to provide guidance on the overall strategic direction and priorities of the READ program. The Council met nine times between 2009 and 2015. Initially, the meetings occurred on an annual basis, but as the program headed into the final phases of implementation, the frequency increased to accommodate discussions on next steps and the future of READ. These meetings were highly beneficial and resulted in a number of key decisions such as agreement on the READ Results Framework, reallocations of funds, approval of global-level work programs, an amendment to the administrative agreement to allow recipient-executed grants, preparations for the READ Global Conferences, and agreement on future areas of work for a possible second READ Trust Fund. While these meetings typically took place in either Moscow or Washington, DC, in 2012, a READ Council meeting was held in Hanoi, Vietnam to give Council members the opportunity to see READ Trust Fund-supported activities on the ground. READ Council members presided over each of the READ Global Conferences and also had the opportunity to hear directly from READ country beneficiaries during these events. GLOBAL-LEVEL RESULTS 33 READ Trust Fund Technical Group Retreats In the early stages of the READ Trust Fund, a Technical Group was formed to help conceptualize the overall work program at the global and country levels. This group, comprised of international assessment experts, supported the development of global knowledge products, provided quality assurance for program activities, and advised country teams during the diagnostic phase and development of their action plans. While membership of the Technical Group varied over time, it typically consisted of six to eight members, including both World Bank staff and international assessment experts. Two Technical Group retreats were held in 2009 and another was held in early 2011. These retreats were instrumental in providing opportunities to brainstorm ideas, get advice on various global products, establish the overall program for the READ Global Conferences, and create tools for use at the country-level, such as the Self-Diagnosis tool. 34 FINAL READ Trust Fund Report 2008–2015 Table 4. READ Council, READ Technical Group, and READ Partnership Meetings, 2009-2015 DATE EVENT LOCATION 2009 January 29 READ Council Meeting Moscow, Russia April 29-30 READ Technical Group Retreat Washington, DC, USA August READ Technical Group Retreat St. Petersburg, Russia 30-September 1 December 16-17 READ Partnership Meeting Paris, France 2010 April 2 READ Council Meeting Moscow, Russia 2011 January 16 READ Partnership Meeting London, UK February 24-25 READ Trust Fund Program and Technical Group Retreat Washington, DC, USA April 21 READ Council Meeting Moscow, Russia 2012 January 31- READ Council Meeting Hanoi, Vietnam February 1 September 18 READ Partnership Meeting Astana, Kazakhstan November 9 READ Council Meeting Moscow, Russia 2013 March 19 READ Council Meeting Paris, France November 9 READ Council Meeting Washington/Moscow October 22 READ Partnership Meeting Tel Aviv, Israel November 15 READ Council Meeting Washington, DC, USA 2014 May 14 READ Council Meeting St. Petersburg, Russia GLOBAL-LEVEL RESULTS 35 Collaboration with READ RAS and CICED From 2009-2015, the READ Trust Fund team organized a number of partnership meetings with Russian authorities, READ RAS, and CICED to identify program synergies and areas for collaboration. These opportunities for information sharing across programs and frequent communication among the different teams resulted in several collaborative efforts, including jointly- organized international conferences, training sessions in Moscow on global knowledge products developed by the READ Trust Fund, workshops and sessions to promote the School Achievements Monitoring (SAM) and Information and Communications Technology (ICT) Literacy tools developed by CICED, and opportunities to disseminate READ Trust Fund knowledge products and incorporate them into READ RAS capacity-building activities in Russia. Annual READ Global Conferences: The annual READ Global  Conferences led by the READ Trust Fund team brought together representatives across the entire READ program, including key education officials from all eight READ Trust Fund-sponsored countries, READ Council members, the READ Trust Fund Technical Group, international assessment experts working on READ activities, World Bank staff, and the READ RAS and CICED teams. Each year, a conference session was dedicated to the READ RAS and CICED teams so that they could present their achievements and share their expertise with other conference participants. 2012 and 2013 IAEA Conferences: The IAEA Conferences provided  valuable opportunities to share the work being carried out under READ and connect this work to the broader global assessment community. In 2012, both READ Trust Fund and CICED served as co-sponsors of the IAEA conference and jointly organized three panel sessions on READ activities. Training Sessions on SABER-Student Assessment Tools: In  collaboration with READ RAS, a series of workshops and talks on student assessment by members of the READ Trust Fund team were organized in Russia. On March 5-7, 2012, presentations on student assessment and the SABER tools were delivered to around 100 participants, including students, researchers, and faculty, at the Moscow Higher School of Economics and the Russian Training Center. On May 14-17, 2013, another member of the READ Trust Fund team delivered a series of presentations and trainings at the Russian Academy of Education’s Russian Training Center, the Moscow School of Higher Economics, and the Moscow School of Social and Economic Sciences. This time, the 142 participants included students, researchers, faculty, and education officials from Russia, Armenia, Belarus, Kazakhstan, and the Kyrgyz Republic. 36 FINAL READ Trust Fund Report 2008–2015 Events and Training Sessions on SAM: The READ Trust Fund  financed a number of initiatives to support and generate further interest in CICED’s newly-developed SAM tool. •  Assessment for Global Learning Symposium: At the World Bank’s high-level symposium, “Assessment for Global Learning”, a presentation on SAM was included during the session, “How Can National Large-Scale, Citizen-led, and Other Assessments Inform Learning?” This event brought together world-renowned international assessment experts and policy researchers to review and discuss existing tools, like SAM, and explore how best they can be used to meet the post-2015 development agenda. READ Global Conference Technical Workshop: Directly following the •  main proceedings of the final READ Global Conference, two technical workshops were held for READ Country Team participants. One of these workshops was on large-scale assessments, with a particular focus on SAM and how countries can use this tool to improve the learning process in primary schools. SAM Workshop: On April 13-14, 2015, CICED conducted another •  workshop on the SAM tool for teachers, curriculum developers, and policymakers. The workshop, entitled “Use of a three-level assessment system for knowledge acquisition and education quality improvement”, took place in Moscow but was broadcast online to include a wider audience. It included lectures, interactive sessions, and discussions about the tool. Event on ICT Literacy Tool: Through the support of the READ Trust  Fund, Armenia was one of the first countries to administer the ICT Literacy Tool. On May 26-27, 2015, an international workshop on the tool was held in Yerevan, with the support of the Ministry of Education in Armenia. This workshop brought together assessment experts and representatives from Armenia, Belarus, Russia, Kazakhstan, the United Kingdom, and Jamaica – some of the other countries that have applied the ICT Literacy Tool – to discuss the results, share experiences, and identify good practices. Workshop participants confirmed the value of the tool and its importance in helping to improve the quality of education in their respective countries. GLOBAL-LEVEL RESULTS 37 READ Trust Fund Global Knowledge Products: Significant effort  was made to translate the global knowledge products developed under the READ Trust Fund into Russian and disseminate them among Russian partner organizations, such as the Russian Training Center and EAOKO. The READ Trust Fund encouraged broad participation by READ Countries in the EAOKO launch event in 2012 and made available knowledge products for dissemination at future EAOKO Conferences. Additionally, some of the research developed under the READ Trust Fund was further disseminated by being published in EAOKO’s peer-reviewed journal. Box 3. Russia’s Center for International Cooperation in Education Development (CICED) The READ Program consisted of two main components: the READ Trust Fund and READ Reimbursable Advisory Services (RAS). The primary objective of READ RAS was to develop the capacity of Russians, and CICED in particular, to engage in international development activities in the area of education. Since its inception in 2008, CICED has served as a network organization to bring together Russian expertise in assessment and quality management in education. Two of the key education quality assessment tools developed with CICED support were the School Achievements Monitoring (SAM) and the ICT Literacy tools. SAM is designed to measure primary school students’ proficiency in the areas of mathematics, language, and science. The instrument is based on Vygotsky’s theory of learning and evaluates students’ competencies at three different levels – formal, reflexive, and functional. The SAM toolkit includes tests in the three subject areas, questionnaires for collecting background information, and recommendations on how to interpret and use the test results. The ICT Literacy tool measures the ability of secondary school students to work effectively with information and use digital communication technologies to learn and solve problems. The test assesses students’ abilities in seven cognitive areas: definition, control, access, integration, assessment, development, and information transfer. Test takers must respond to a number of interactive tasks, based around real life situations that represent different levels of difficulty in the seven cognitive areas. (Source: CICED, www.ciced.org.) 38 FINAL READ Trust Fund Report 2008–2015 Program Monitoring and Reporting Multiple actors worked on both the global and country-level activities under READ: eight READ Country Task Teams, READ Technical Group experts, the Student Assessment Team, and the READ Management Team. Regular video conferences, conference calls, and occasional retreats were organized to facilitate communication across these groups and to monitor the quality of program implementation. In addition to regular monitoring activities, program results and achievements were reported through published annual reports, bi-annual grant monitoring reports, and country presentations during the annual READ Global Conferences. In 2011, in an effort to demonstrate clear, measureable results at the country level, a READ Results Framework was established. This framework draws heavily on the work done under SABER-Student Assessment and identifies key indicators (e.g., clear policies, effective human resources) against which to measure progress in developing a country’s assessment system. COUNTRY-LEVEL RESULTS 39 COUNTRY-LEVEL RESULTS Under the direction of the READ Council, eight countries were selected to receive targeted READ Trust Fund support – Angola, Armenia, Ethiopia, the Kyrgyz Republic, Mozambique, Tajikistan, Vietnam, and Zambia. Through a combination of Bank-executed and Recipient-executed grants, each of these countries received approximately US$1 to $4 million to support the further development and strengthening of their assessment systems. In each country, World Bank Task Team Leaders and Country Teams worked closely with their respective government counterparts to develop action plans and implement a set of agreed-upon activities. Kyrgyz Republic Armenia Tajikistan Vietnam Ethiopia Angola Zambia Mozambique 40 FINAL READ Trust Fund Report 2008–2015 Table 5. READ Trust Fund Results Framework and Key Indicators Enabling Context (EC) EC1 – Setting clear policies There is a formal document that provides guidelines about assessment activities. The formal document is available to key stakeholders and the public. EC2 – Having strong leadership Key stakeholders support the assessment activities. There is key stakeholder support for continuous improvement of assessment activities. EC3 – Having regular budget/funds for assessment activities There is a line item in the government education budget for assessment activities. The budget provides adequate funding in major areas, including design, administration, data processing, and reporting. EC4 – Having strong organizational structures There is an agency, institution, or unit with the mandate to carry out assessment activities. The assessment agency, institution, or unit is accountable to a clearly recognizable body. EC5 – Having effective human resources There is a team of people with the requisite skills/capacity to carry out assessment activities. There are opportunities available to build assessment capacity – for example, courses/training on test development, sampling, and so forth. System Alignment (SA) SA1 – Aligning the assessment with learning goals There is a clear, common understanding among key stakeholders of what the assessment activities measure. Assessment activities are aligned with an official curriculum/learning standards that outlines what students are expected to learn. SA2 – Providing opportunities to learn about assessment activities There are training sessions/courses for teachers to learn about the assessment activities. Teachers are involved in some aspect(s) of assessment-related activities. Assessment Quality (AQ) AQ1 – Ensuring quality There is formal documentation about the technical aspects of the assessment activity. Assessment results are deemed valid and reliable by key stakeholders. AQ2 – Ensuring effective use of assessment results Assessment results are disseminated in meaningful ways to key stakeholders. Assessment results are used to promote and inform students’ learning. Source: READ Trust Fund. COUNTRY-LEVEL RESULTS 41 Creating a Results Framework Establishing a sound assessment system is not easily accomplished within a few years. Even the best-performing countries have taken decades to firmly establish their systems and effectively use results in ways that improve overall education quality. However, the development of the SABER-Student Assessment tools made it possible to break down various aspects of this task into manageable steps by identifying key indicators, defining levels of development on each indicator, and describing what it takes to reach each level. In order for the READ Trust Fund to be able to demonstrate measureable results, a results framework and key indicators against which to measure progress were established. This framework draws heavily on the benchmarking tools created under SABER-Student Assessment and summarizes key indicators common across all four assessment types (see Table 5). Monitoring and Tracking Progress The SABER-Student Assessment tools were applied in all eight READ Trust Fund-supported countries to establish baseline ratings against which to measure progress. During the first phase of project implementation, the team conducted a forecasting exercise to get an idea of the results one could expect by the end of the READ Trust Fund program. When the country programs reached completion in 2014, the SABER-Student Assessment tools were applied again to determine the actual level of progress each country had achieved in further developing their assessment system. Table 6 summarizes the findings of this exercise. (A detailed discussion of the findings from this benchmarking exercise are available on the READ website in the report, “Russia Education Aid for Development Trust Fund Program: Results of the Country Benchmarking Exercise, 2009-2014.”) The SABER-Student Assessment tools evaluate the overall level of development of each assessment type (i.e., classroom assessment, examinations, NLSA, and ILSA) and assign it a developmental rating (i.e., latent, emerging, established, or advanced). Within these developmental ratings, there are a range of possible scores determined by the results in different key indicators areas (e.g., clear policies, effective human resources, assessment quality). While the overall developmental rating may not change, the scores make it possible to track progress and capture movement within a developmental rating. 42 FINAL READ Trust Fund Report 2008–2015 Many of the READ Trust Fund-supported countries began with very latent assessment systems or systems that were just emerging. A large portion of activities were concentrated on laying important groundwork to systematize and institutionalize assessment efforts. During the six years of program implementation, many countries piloted new assessment tools – once these pilots are brought to scale and operating regularly, the developmental rating will be able to advance to the next level. In the meantime, examining the changes in each country’s scores over time helps us to see the significant impact of the READ Trust Fund program. In the following summaries of the country programs, it is clear that all eight beneficiary countries were able to develop parts of their assessment systems in significant ways and lay a strong foundation for future work. Table 6. Summary of the Benchmarking Results for Student Assessment Activities in READ Countries, 2009-2014 Classroom Examinations NLSA ILSA Assessment 2009 2014 2009 2014 2009 2014 2009 2014 Angola Emerging Emerging Emerging Emerging Latent Emerging Latent Latent Armenia* Established Established Established Established Emerging Emerging Established Established Ethiopia Latent Emerging Established Established Emerging Established Latent Latent Kyrgyz Rep. Latent Emerging Emerging Emerging Emerging Emerging Emerging Latent Mozambique Emerging Emerging Emerging Established Emerging Emerging Emerging Emerging Tajikistan Emerging Emerging Emerging Emerging Latent Latent Latent Latent Vietnam Emerging Established Emerging Established Emerging Emerging Latent Emerging Zambia Emerging Emerging Established Established Emerging Established Emerging Emerging *The initial benchmarking exercise in Armenia took place in 2011. COUNTRY-LEVEL RESULTS – ANGOLA 43 ANGOLA Total Grant Amount: US$2,265,000 Total Disbursements: US$2,141,721 COUNTRY CONTEXT When Angola ended its 25-year civil war in 2002, the country needed to rebuild its education system from the ground up. During the long civil war, many children did not attend school. The government has been dedicated to improving its education system in three key areas: 1) expanding school access; 2) improving the quality of education; and 3) strengthening the efficiency and equity of the education system at all levels. Since 2002, Angola’s primary education system has improved dramatically, but much work remains to establish a strong overall system that ensures quality learning outcomes. Similar to many other countries, although more children now have access to school in Angola, many are not necessarily acquiring the expected basic literacy and numeracy skills. During the READ Trust Fund program, there was strong commitment from the highest levels of government in Angola to improve the capacity of the education system to monitor quality and measure student learning outcomes. In 2009, at the beginning of the program, Angola’s education system had a weak assessment culture, no full-time staff trained in assessment, and very few resources for assessment activities. However, as a result of strong support from the Ministry of Education, and with the help of READ funds, by the end of 2014, the country was able to complete its first-ever national assessment of early-grade reading and lay a strong foundation for future work in this area. 44 FINAL READ Trust Fund Report 2008–2015 PROGRAM OBJECTIVES, RATINGS, AND PROGRESS (2009-2014) Program Objectives and Developmental Ratings The dual objective of the READ Trust Fund program in Angola was 1) to improve the capacity of the Ministry of Education to assess student learning outcomes, and 2) to develop a culture of evidence-based decision making. At the beginning of the program, the baseline diagnostic showed that most aspects of Angola’s assessment system were at a latent stage of development. With READ Trust Fund support, Angola chose to focus first on laying a foundation for NLSA. A key priority was to create a new Assessment Unit within the Ministry of Education capable of carrying out NLSA activities. After successfully implementing an early-grade reading assessment (EGRA) exercise, the Assessment Unit team used the results to inform the design of various interventions to improve outcomes in reading. By the end of READ, as demonstrated in Table 7, Angola had made significant progress in laying a strong foundation for future NLSA exercises. Table 7. Developmental Ratings for Angola’s Student Assessment System Classroom Examinations NLSA ILSA Assessment 2009 2014 2009 2014 2009 2014 2009 2014 Emerging Emerging Emerging Emerging Latent Emerging Latent Latent COUNTRY-LEVEL RESULTS – ANGOLA 45 Program Implementation The READ Trust Fund program in Angola was implemented in three phases with the support of three Bank-executed grants. A first grant for US$300,000 supported the diagnostic phase and the creation of an action plan; a second grant for US$1,005,000 supported capacity-building activities and the implementation of an EGRA; and a third grant for US$900,000 supported various EGRA follow-on activities and foundational work in classroom assessment. Due to the foundational work accomplished under READ, an IDA credit for Angola’s Learning for All Project was able to leverage an additional US$9,000,000 for further work on strengthening Angola’s student assessment system. Progress by Assessment Type and Key Indicator Areas Angola’s progress under the READ Trust Fund program has been carefully monitored using the READ Results Framework and SABER-Student Assessment benchmarking tools. Table 8 indicates the baseline and end-of- program scores for Angola based on the results of benchmarking exercises carried out at the beginning and end of the program. The progress seen is based on achievements in the key indicator areas being tracked under the READ Trust Fund program. The activities that Angola carried out under each key indicator area are described in the next section. Activities that benefited more than one type of assessment activity are described under a general category. Activities that benefited a specific type of assessment activity are described under assessment-specific headings. Table 8. Developing Angola’s Assessment System: Progress Achieved from 2009 to 2014 Latent Emerging Established Advanced 1 2 3 4 Classroom Assessment Examinations NLSA ILSA 2009 2014 Progress from 2009-2014 46 FINAL READ Trust Fund Report 2008–2015 General Clear Policies Due to the success of EGRA, a decision was made to implement regular EGRA and NLSA exercises. Additionally, greater attention is being given to a policy mandating national examinations in grade 6, which was never implemented due to logistical issues. A five-year national assessment strategy for work in each of these areas is under development as part of the Learning for All Project. Strong Leadership High-level education officials, including Vice Ministers of Education and the National Director, consistently provided strong support for developing Angola’s assessment system and for the activities implemented with READ Trust Fund support. In 2010, a workshop led by three Vice Minsters was held to discuss and validate the findings and proposed action plan from the baseline diagnostic. This workshop included donor representatives and 100 provincial-level education staff. The Vice Minister responsible for overseeing assessment activities met regularly with the Assessment Unit and led the Angolan delegations for several international events, including the 2011 study visit to Brazil and the READ Global Conferences. Additionally, a report using the findings from EGRA was produced by this Vice Minister for discussion with the Minister and Presidency. Regular Budget In 2010, a line item for assessment activities was created in the education budget. Since then, a regular annual budget of around US$1 million has been allocated to support the development of Angola’s assessment system. Strong Organizational Structures In 2010, as a result of the baseline diagnostic, the Ministry of Education issued a Ministerial decree mandating the creation of a new Assessment Unit. This unit, called the National Assessment System Technical Group (NASTG), included a dedicated team of 14 staff from various departments within the Ministry and was given the mandate to establish and develop the national assessment system. The five core team members from the National Institute for Research and Development of Education (INIDE) have been responsible for leading the group’s work. Under READ, the NASTG benefited from a large number of hands-on trainings and capacity-building activities. Initially, the majority of these trainings centered on EGRA. This training was later expanded to include examinations and classroom assessment. As a result of this “A National Assessment System is fundamental to ensure quality primary education to all Angolan children.” – Narciso Benedito, Education Secretary of State in Angola COUNTRY-LEVEL RESULTS – ANGOLA 47 training, the NASTG was able to lead the preparations and implementation of an EGRA, with the support of international consultants and World Bank staff. The NASTG was allocated space within the Ministry, and government funds have been used to provide them with the necessary equipment. Under the Learning for All Project, this team has been given overall responsibility for overseeing all assessment-related activities. Effective Human Resources The bulk of effort and funds under the READ program in Angola were spent on training local staff and building capacity to carry out assessment activities. These capacity-building activities were in the form of study visits to Brazil, international conferences on assessment topics, national workshops, and hands-on trainings. In total, more than 300 key education stakeholders now have increased capacity to understand and implement assessment activities. Study Visits: In March 2011, 21 Ministry of Education staff traveled to  Pernambuco, Brazil to learn from the country’s assessment practices and quality improvement efforts. In order to share these lessons back home, a three-day conference, which included more than 150 participants from the Ministry of Education, provincial governments, and teacher training institutes, was held on the group’s return. This south-south collaboration with Brazil gave Angola some key insights on ways to develop their assessment system. 48 FINAL READ Trust Fund Report 2008–2015 Conferences: Representatives from Angola’s Ministry of Education  gained a better understanding of key issues and best practices in assessment as a result of participating in several international learning events, such as the annual READ Global Conferences and the 2012 READ Regional Workshop in Mozambique. At each of these READ events, one representative from the five- to six-member Angolan delegation delivered a presentation on the development of Angola’s assessment system and progress achieved to date under READ. At the final READ Conference in St. Petersburg, four members of the NASTG and the Director of INIDE participated in a panel session on the overall impact of the READ program in Angola. Additionally, in an effort to connect with the broader assessment community in Africa, four representatives from the NASTG participated in their first AEAA Conference in Arusha, Tanzania in 2013 and contributed to the panel session, “Improving education quality through stronger student assessment systems.” Classroom Assessment Effective Human Resources In 2013 and 2014, the Carlos Chagas Foundation from Brazil began to work closely with the NASTG and other Ministry of Education staff to build their capacity in classroom assessment. In May 2013, the NASTG hosted a five-day workshop on classroom assessment and how it can be used to improve learning outcomes. The workshop was attended by two representatives from 18 of Angola’s teacher training institutions throughout the country. As a result of the workshop, participants gained a better understanding of effective classroom assessment practices and tools. The workshop also provided the foundation for beginning the preparation of pre- and in-service teacher training modules on classroom assessment. “Our EGRA work showed us that it is extremely important for us to work with the teachers. Teachers should know how they can use the results and other materials so they can improve their work with individual students.” – Angolan Delegation, Fourth READ Global Conference COUNTRY-LEVEL RESULTS – ANGOLA 49 Between April and October 2014, work on classroom assessment focused on the development of learning standards and rubrics for Portuguese and Mathematics and a training module on classroom assessment in these subject areas for grades 4 and 6 teachers. Workshops to develop the tools and build capacity were attended by the 14 NASTG members and 16 Ministry of Education Portuguese and Mathematics Specialists from both the central and provincial levels. The final workshop also included representatives from the 18 teacher-training institutions in order to share the tools developed and validate the grades 4 and 6 learning standards rubrics. In total, more than 50 people benefitted from these workshops. Examinations Effective Human Resources In October 2013, a nine-day training workshop was conducted for the 14 NASTG members and 20 provincial education staff responsible for overseeing provincial-level examinations. The purpose of the training was to further develop capacity in the area of student assessment, with a focus on national examinations. The workshop targeted the 20 provincial representatives involved in the production of provincial-level examinations. The objectives of the training were to help participants: 1) develop knowledge about external assessments, how they are used, and how they can be implemented; 2) deepen their understanding of how to design and implement assessments at a national level; 3) learn about the development of other countries’ assessment systems, with a focus on the types of assessments used and lessons learned; and 4) learn how to develop and design assessment tools and items. In October 2014, six members of the NASTG and six Ministry of Education specialists in Portuguese and Mathematics traveled to Brazil to learn about that country’s experience in implementing national examinations. NLSA Effective Human Resources In 2010 and 2011, a number of workshops were organized in Washington, DC and Luanda to train the NASTG in the skills necessary to implement an EGRA. In 2010, staff participated in three different workshops – on sampling, data entry, and data cleaning and analysis – in order to prepare for the pilot exercise. Following the pilot, in 2011, the team participated in another three workshops to prepare for the full-scale EGRA. During these workshops, the staff reviewed the EGRA instruments, created the final sample design, and received additional training on data collection and software. A total of 75 teachers and teacher trainees were also trained in how to administer the assessment. During the preparations for EGRA, more than 100 people, including education authorities and members of teacher-training schools from the 18 provinces of Angola, received training and started to build 50 FINAL READ Trust Fund Report 2008–2015 capacity in carrying out assessment activities. After completing the full-scale implementation of EGRA, members of the NASTG traveled to Washington, DC to present their experience and share lessons learned. Assessment Quality As part of their capacity-building activities, the Assessment Unit team was responsible for implementing EGRA on a national scale. In 2011, the team adapted the tools, created the sample, recruited and trained teacher trainees to implement the test, oversaw the pilot exercise in three provinces, and actively participated in the full-scale administration of EGRA in 144 schools throughout Angola’s 18 provinces. In 2012, the team worked on data cleaning, analyzed the data, and prepared a report with the final results. This report established the first baseline data on learning outcomes in Angola. A workshop was held with 100 provincial education staff members to share and discuss the results and identify actions for moving forward. Use of Results The EGRA results in Angola highlighted the need to improve the teaching and learning of basic literacy skills and to promote the importance of reading to parents and the society at large. In order to address some of the gaps detected by EGRA, the NASTG developed targeted materials for teacher- training institutions, teachers, and parents. Materials produced included 1) a “Caderno de Apoio” which emphasizes the importance of teaching literacy techniques to pre-service teachers; 2) a “Cartilha” of reading activities for primary school teachers to use in their classrooms to reinforce reading; and 3) a flyer on the importance of reading with useful information for parents and caregivers. These materials are being used during teacher-training workshops and other events to address the issue of low literacy outcomes and to provide specific guidance on what can be done to improve. Work is also underway to create a series of short, public-awareness videos to be broadcast on national television as part of a national literacy campaign. COUNTRY-LEVEL RESULTS – ANGOLA 51 ILSA Clear Policies In 2013, Angola’s Ministry of Education decided to initiate participation in its first ILSA by joining the Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ). A formal agreement to participate was signed, although a national policy on participation in this or any other ILSA has yet to be created. Participation in SACMEQ is being carried out with funds from the government’s own budget. Upon completion of a full cycle of SACMEQ, it is expected that Angola’s overall developmental rating and score for ILSA activity will advance. Effective Human Resources Twelve individuals from the NASTG and Ministry of Education have participated in international meetings and three workshops organized by SACMEQ. These workshops have covered general issues related to SACMEQ participation as well as more focused technical discussion on topics such as item construction, translation, and testing software. Assessment Quality Three members of the NASTG have overall responsibility for coordinating SACMEQ in Angola. Under their guidance, testing instruments have been developed in Portuguese for the three subject areas to be assessed – Mathematics, Portuguese, and Life Sciences. SACMEQ is creating the sample design and administration will take place in 2015. CONCLUSION The READ Trust Fund played a critical role in strengthening assessment activity in Angola. The dual objective of the READ program in Angola was to improve the capacity of the Ministry of Education to assess student learning outcomes and to develop a culture of evidence-based decision making. This dual objective was met. In addition, the success of READ directly led to an IDA credit of US$75 million for the Angola Learning for All Project of which US$9 million is set aside to finance further work on establishing Angola’s assessment system. 52 FINAL READ Trust Fund Report 2008–2015 ARMENIA Total Grant Amount: US$1,400,000 Total Disbursements: US$1,370,359 COUNTRY CONTEXT Education has been a priority sector for public investment in Armenia since the late 1990s. The government has implemented a number of reforms to increase access and improve quality in the education system. However, despite significant improvements, the country’s education system still faces challenges in overall learning levels and in the size of equity gaps between rich and poor students. Armenia is committed to ensuring an efficient and effective allocation of resources in education. As a part of this commitment, serious effort has been given to creating new policies for Armenia’s system of measuring and monitoring learning outcomes. In order to effectively implement these policies, the capacity of key stakeholder groups must be improved so that they can carry out assessment activities in an appropriate manner and link the results to education policies, practices, and decision making. Through support from the READ Trust Fund, an extensive training program was carried out in Armenia to equip these key stakeholder groups with greater capacity in measuring and using data on learning outcomes. COUNTRY-LEVEL RESULTS – ARMENIA 53 PROGRAM OBJECTIVES, RATINGS, AND PROGRESS (2011-2014) Program Objectives and Developmental Ratings The objective of the READ Trust Fund program in Armenia was to enhance the country’s capacity to design, implement, and use robust national and international assessment and examination systems; strengthen classroom assessment practices; and provide channels for feedback, policy analysis, and recommendations for actions to improve student learning outcomes. For Armenia, the 2011 baseline results showed that while many of the necessary policies were in place, there was a need to focus on capacity building and improving implementation. With READ Trust Fund support, Armenia chose to primarily concentrate on building capacity across all four assessment types. By the end of READ, while the overall developmental ratings for Armenia’s student assessment system remained the same, significant progress had been made in strengthening the capacity of key stakeholders to effectively implement assessment activities across the different assessment types. Table 9. Developmental Ratings for Armenia’s Student Assessment System Classroom Examinations NLSA ILSA Assessment 2011 2014 2011 2014 2011 2014 2011 2014 Established* Established Established Established Emerging Emerging Established Established *The benchmarking tool used to establish the baseline for classroom assessment measured policies, but did not capture the degree to which those policies were implemented. Questions related to policy implementation were added to a later version of the tool. The original baseline rating for classroom assessment in Armenia was retroactively adjusted (from Advanced to Established) to account for this change. Program Implementation Armenia was invited to become a READ-recipient country in 2011. Given the shortened timeline compared to other READ-recipient countries, the READ program in Armenia was implemented with the support of a single Bank-executed grant. This grant for US$1 million supported the diagnostic phase, capacity-building activities, designing and piloting assessment instruments, and activities related to Armenia’s participation in TIMSS. In 2013, a decision was made by the READ Council to allocate an additional US$400,000 under the grant to allow Armenia to carry out more of the activities outlined in their action plan. In addition to the work funded by the READ Trust Fund and regular support from the government, the Ministry of Education and Science’s State Committee of Science funded a number of initiatives during this period to strengthen the work of the Assessment and Testing Center (ATC). Armenia also was a recipient of CICED’s small-grants program, receiving US$20,000 to make improvements to Russian Language testing instruments. 54 FINAL READ Trust Fund Report 2008–2015 Table 10. Developing Armenia’s Assessment System: Progress Achieved from 2011 to 2014 Latent Emerging Established Advanced 1 2 3 4 Classroom Assessment Examinations NLSA ILSA 2009 2014 Progress from 2009-2014 Progress by Assessment Type and Key Indicator Areas Armenia’s’ progress under the READ Trust Fund program has been carefully monitored using the READ Results Framework and SABER-Student Assessment benchmarking tools. Table 10 indicates the baseline and end-of- program scores for Armenia based on the results of benchmarking exercises carried out at the beginning and end of the program. The progress seen is based on achievements in the key indicator areas being tracked under the READ Trust Fund program. The activities that Armenia carried out under each key indicator area are described in the next section. Activities that benefited more than one type of assessment activity are described under a general category. Activities that benefited a specific type of assessment activity are described under assessment-specific headings. General Effective Human Resources In Armenia, the READ Trust Fund program supported a number of capacity- building activities. Through participation in international conferences on assessment topics and specialized training courses, more than 100 key education stakeholders have increased their capacity to carry out assessment activities. COUNTRY-LEVEL RESULTS – ARMENIA 55 Conferences: Delegations from Armenia’s Ministry of Education and  other key agencies participated in a number of international events on assessment topics. These events included the 2011 and 2014 READ Global Conferences, the 2012 READ Regional Workshop, the 2012 and 2013 IAEA Conferences, and the 2012 and 2013 EAOKO Conferences. At each of these events, delegations of key education officials and representatives from Armenia’s ATC delivered presentations on the READ Trust Fund program in Armenia and the progress achieved. Also, in 2012, the Head of the Training Department of the ATC attended courses as part of the Master’s program in Educational Evaluation and Assessment at the Moscow School of Social and Economic Sciences in Russia. READ funds were also used to support the participation of one education official at a seminar on “Contemporary issues in planning and formation of rankings in education.” All of these events have enhanced Armenia’s understanding of key issues in assessment and further developed its capacity to carry out assessment activities. Master’s Course: To further institutionalize efforts to build assessment  capacity in Armenia, work was completed on developing a Master’s program in education that incorporates courses on student assessment. In 2013, a report on international best practices and an implementation plan with program specifications and course content for such a Master’s program was prepared and discussed with universities in Armenia. In 2014, the content of the program was agreed upon by the Deputy Minister, and the State Linguistic University was selected through a competitive process to be a partner to Durham University in the United Kingdom in implementing the program. Classroom Assessment Effective Human Resources The READ Trust Fund program supported efforts to improve classroom assessment practices in Armenia, including the development of training courses to ensure that teachers develop skills and expertise in classroom assessment. In 2012, two reports were produced on this topic. The first report identified and evaluated current opportunities for pre- and in-service teacher training on classroom assessment in Armenia. The second report outlined different options for revising/designing pre- and in-service teacher training on classroom assessment in Armenia in order to address the country’s needs and incorporate international best practices. Drawing on the information in these reports, training courses on classroom assessment and examinations were developed for both pre-service and in-service teachers. 56 FINAL READ Trust Fund Report 2008–2015 Pre-Service Teachers: Four workshops were held with 45 university  lecturers to deliver the training developed for pre-service teachers and enable them to lead the implementation of these new modules in their pre-service training institutions. The Ministry of Education and Science provided the training material to 10 universities that conduct pre-service teacher training, and two of these have agreed to pilot this new training program in their universities. In-Service Teachers: To pilot and further develop the training modules  for in-service teachers, three workshops were held for 25 Armenian teachers. These 25 lead teachers represented the 25 different high schools and general education schools included in the pilot. During these workshops, the lead teachers were trained as trainers for the in-service modules and then made responsible for delivering the training in their respective schools. Based on the success of the pilot, additional in-service trainings are expected to be implemented throughout the entire school system. Examinations Quality Assurance In an attempt to ensure quality, reduce the perceived level of malpractice associated with the current system, and increase public confidence in the integrity of the school leaving and university entrance examinations, a feasibility study was conducted on the possibility of introducing computer- adaptive testing (CAT) and computer-based testing for the Unified Examination in Armenia. As a part of this study, a number of reports were produced, including a report outlining the priorities of the Government of Armenia for the Unified Examination system and the goals the Government would like to achieve by introducing CAT to this system; a report on the definition, use, and implementation of CAT; and a report that outlines different scenarios and cost estimates for introducing CAT to the Unified Examination system. Additionally, COUNTRY-LEVEL RESULTS – ARMENIA 57 a background paper on the Georgian experience with CAT was completed to inform Armenia’s possible adoption of this technology. Based on the information provided, the Ministry of Education and Science and ATC are discussing what actions to take to continue to improve the current system. NLSA Effective Human Resources In the area of NLSA, an intensive training program on large-scale assessment was developed and delivered to 45 participants from the ATC, universities, and other units within the Ministry of Education and Science. There were six different workshops in total, covering the following topics – general statistics, item analysis, test and questionnaire development, and national assessment for educational policymakers. Additionally, in 2013, a delegation of officials responsible for large-scale assessment traveled to New Jersey to participate in a tailored training provided by ETS. As a result of these trainings, the capacity of existing specialists responsible for overseeing large-scale assessment activities was enhanced. Alignment with Learning Goals The READ Trust Fund supported the application of the CICED-sponsored ICT Literacy Test to grade 9 students in Armenia as a way to assess their competencies in using modern information and computer technologies. In 2013, after adapting the tools to the Armenian context, a pilot of the test took place. The results of the pilot were analyzed and adjustments made in order to create the final version of the assessment for Armenia. In 2014, the ICT Literacy Test was administered to a nationally-representative sample of 2,400 students from all regions of Armenia. The results of this survey are being used to inform policy choices in the area of integrating ICT policies in education. “We think that the ICT testing will help us adjust our curriculum in that area. We saw that the schools and children most successful on those tests worked with ICT not only one hour a week but where ICT was integrated into other subjects and disciplines.” – Armenian Delegation, Fourth READ Global Conference 58 FINAL READ Trust Fund Report 2008–2015 Assessment Quality In order to help Armenia expand its ongoing NLSA program, READ funds were used to support the design and development of a questionnaire and testing instrument to assess grade 8 students in the areas of Armenian, Literature, and History. This program, known as the Trends in Armenistic Subject Studies, underwent a pre-pilot in 2014. For the pre-pilot, the data were cleaned, a statistical analysis of the results was completed, and a final report prepared. The full pilot and main survey will be prepared after analysis and refinement of the instrument is complete. It is expected that the ATC will implement the main survey in 2015. ILSA Effective Human Resources With READ Trust Fund support, Armenia’s participation in the Trends in International Mathematics and Science Study (TIMSS) 2011 was finalized and preparations were made for TIMSS 2015. The preparatory work for TIMSS 2015 included tailored training for ATC staff and other assessment specialists by international consultants and experts from the Data Processing and Research Center of the International Association for the Evaluation of Education. The training covered topics such as using TIMSS data for secondary analysis and writing thematic reports with TIMSS data. The READ grant also supported the National Research Coordinator’s attendance at three planning meetings for TIMSS 2015. Use of Results TIMSS 2011 country results were discussed with the Ministry of Education and Science and ATC staff. An article with the results that was tailored to key stakeholders was published and disseminated to inform education policy making, curriculum, and teacher training. CONCLUSION Through support from the READ Trust Fund, an extensive training program is taking place in Armenia to equip various stakeholders with greater capacity in measuring and using data on learning outcomes. Through the training courses developed, both pre- and in-service teachers are acquiring the skills necessary to carry out formative and summative assessments in their classrooms and education officials tasked with implementing NLSA have greater capacity in areas such as such as statistics and item analysis. The effectiveness of Armenia’s assessment system will continue to improve as it develops a strong cadre of people across the entire education system with the requisite skills to implement assessment activities and use the results to inform their education practices. COUNTRY-LEVEL RESULTS – ETHIOPIA 59 ETHIOPIA Total Grant Amount: US$1,091,000 Total Disbursements: US$1,056,013 COUNTRY CONTEXT Ethiopia is the second most populous country in Africa, with 80 percent of its population living in rural areas. Although the education system in Ethiopia has undergone impressive expansion, it still faces serious challenges, particularly in the area of quality. The Government of Ethiopia sees education as a means to eradicate poverty, but realizes that this will only occur if the education system delivers a quality educational experience that promotes learning. Since 2009, the main source of funding for general education in Ethiopia has come from the multi-donor-financed General Education Quality Improvement Program (GEQIP). Total funds available through GEQIP I and II reach nearly US$1 billion. This program is expansive and covers nearly all aspects of the education system, including the area of assessment. Some of the main challenges for Ethiopia to overcome in the area of assessment have been the lack of policies, unclear lines of responsibility, irregular funding, and low technical capacity. The funds made available through READ were targeted to support activities to strengthen existing institutions responsible for measuring learning outcomes and to establish new ones where necessary. Through a combination of these activities and work done by other donors, by the end of 2014, Ethiopia was able to significantly advance the development of its assessment system. 60 FINAL READ Trust Fund Report 2008–2015 photo: petteriik Wiggers/unESCo (Ethiopia) PROGRAM OBJECTIVES, RATINGS, AND 8 PROGRESS (2009-2014) R E A D A n n uA l R E p o R t 2009 Program Objectives and Developmental Ratings The objective of the READ Trust Fund program in Ethiopia was to help strengthen existing or develop new institutions for measuring and assessing student achievement in order to improve education quality and student learning outcomes. At the beginning of the program, the baseline results indicated that for Ethiopia, both classroom assessment and ILSA activities were at a latent stage of development, NLSA was emerging, and examinations were established. Coordinating with other donors working on education, READ Trust Fund–supported activities focused on further strengthening the areas of classroom assessment, examinations, NLSA, and school inspection. (School inspection contributes directly to education quality, but was not one of the areas formally tracked under the READ Results Framework.) By the end of READ, as demonstrated in Table 11, Ethiopia had made significant progress in laying a strong foundation for classroom assessment and NLSA, and also improved its system for examinations. Table 11. Developmental Ratings for Ethiopia’s Student Assessment System Classroom Examinations NLSA ILSA Assessment 2009 2014 2009 2014 2009 2014 2009 2014 Latent Emerging Established Established Emerging Established Latent Latent COUNTRY-LEVEL RESULTS – ETHIOPIA 61 Program Implementation The READ Trust Fund program in Ethiopia was implemented with the support of two Bank-executed grants. The first grant of US$291,000 supported the diagnostic phase. In addition to conducting an overall benchmarking exercise, in-depth studies were conducted in the areas of examinations, NLSA, and school inspection. The findings from these studies were used to develop detailed action plans. A second grant for US$800,000 supported implementation of the action plans, including developing formal policy frameworks, establishing clear institutional arrangements, and building greater capacity to carry out assessment activities. Although not included initially, over time the Ministry came to recognize the importance of classroom assessment and teacher guides, and training was subsequently provided in this area as well. All of the activities completed with READ Trust Fund support were designed to complement activities taking place under GEQIP I and II. Progress by Assessment Type and Key Indicator Areas Ethiopia’s progress under the READ Trust Fund program has been carefully monitored using the READ Results Framework and SABER-Student Assessment benchmarking tools. Table 12 indicates the baseline and end-of- program scores for Ethiopia based on the results of benchmarking exercises carried out at the beginning and end of the program. The progress seen is based on achievements in the key indicator areas being tracked under the READ Trust Fund program. The activities that Ethiopia carried out under each key indicator area are described in the next section. Activities that benefited more than one type of assessment activity are described under a general category. Activities that benefited a specific type of assessment activity are described under assessment-specific headings. Activities related to school inspection also have been included. Table 12. Developing Ethiopia’s Assessment System: Progress Achieved from 2009 to 2014 Latent Emerging Established Advanced 1 2 3 4 Classroom Assessment Examinations NLSA ILSA 2009 2014 2009 and 2014 Progress from 2009-2014 62 FINAL READ Trust Fund Report 2008–2015 General Strong Leadership A number of workshops were held in Ethiopia with the purpose of drafting and endorsing new policy and procedural frameworks for each target area. In August 2010, a workshop attended by approximately 60 key education officials was held to discuss the findings of the NLSA in-depth review. In February 2011, a similar workshop was held to discuss the findings of the school inspection report. Once new policies and frameworks had been drafted in each area, a series of validation workshops were held to review and endorse them. The first of these workshops, conducted in October 2012, focused on reviewing the school inspection framework and guidelines. More than 20 representatives from the Ministry of Education and Regional Education Bureaus were in attendance. A joint review of the draft policy frameworks for examinations and NLSA was held in December 2013. Given the presence of the Minister of Education at this workshop, along with at least 35 other education department heads and officials from the Regional Education Bureaus, action was also taken to seek endorsement of a draft manual on classroom assessment for primary and secondary teachers. In March 2014, two final validation workshops took place – one to validate the school inspection framework and data collection checklist for Alternative Basic Education, and the other to validate the school inspection framework and data collection checklist for Early Childhood Care and Education. Again, each of these workshops was attended by at least 30 key education officials from the Ministry of Education and Regional Education Bureaus. As a result of these six workshops led by the Minister of Education and other high-level education officials, there has been greater awareness, leadership, and support to develop Ethiopia’s systems for assessment and school inspection. In total, more than 150 education leaders in Ethiopia have been actively engaged in carrying this work forward. “In Ethiopia, READ played a paramount role, especially in bringing about unified and [clearly] defined institutional arrangements for assessment.” – Ethiopian Delegation, Fourth READ Global Conference COUNTRY-LEVEL RESULTS – ETHIOPIA 63 Strong Organizational Structures In order to establish clear institutional arrangements for assessment activities, a new agency was created in 2012. This agency, the National Educational Assessment and Examinations Agency (NEAEA), is autonomous, led by a board, and responsible for both national examinations and NLSA. These new institutional arrangements greatly improved the enabling context for carrying out assessment activities in Ethiopia. Effective Human Resources Through participating in international conferences and workshops, study visits, and specialized trainings, more than 400 education leaders and practitioners in Ethiopia are now more aware of key issues in assessment and able to implement assessment-related activities. Capacity-building activities that benefited more than one assessment type include: Conferences: With READ Trust Fund support, nearly 100 staff members  working on NLSA, national examinations, and school inspection were able to participate in international events on assessment topics. Delegations of up to six staff members working in these areas attended each of the annual READ Global Conferences and the 2012 READ Regional Workshop in Mozambique. READ funds were also used to support Ethiopia’s participation in three other assessment-related events in Africa. The first of these was the 2010 Users and Uses of Assessment Information Workshop in South Africa. Then, in 2013, a small delegation represented Ethiopia at its first ever AEAA Conference, which was held in Arusha, Tanzania. Following this conference, a workshop (“The Role of Assessment in Quality of Education in Ethiopia: Challenges and the Way Forward” ) was held to share lessons learned from the event with more than 50 of Ethiopia’s education leaders. During this workshop, the decision was made for Ethiopia to become a full-time member of the AEAA. In 2014, Ethiopia attended its second AEAA Conference in Livingstone, Zambia with an even larger delegation of education officials and as full members of the association. These events were effective in connecting Ethiopia with the broader assessment community in Africa and creating additional professional networks for NEAEA and school inspection staff. Study Visit: In May 2012, five NEAEA staff participated in a study visit to  South Africa as a way to learn from that country’s assessment practices and identify lessons for Ethiopia. This south-south collaboration with South Africa gave Ethiopia some key insights on ways to operationalize their newly-created assessment agency, NEAEA. 64 FINAL READ Trust Fund Report 2008–2015 Training for NEAEA staff: To further build the capacity of NEAEA staff,  international firms and consultants were hired to provide targeted technical assistance and training in key areas. In the last half of 2012, 60 NEAEA staff benefited from a month-long, in-depth training in areas such as item writing, sampling, test administration, data analysis, and reporting assessment results. This training was provided by ETS and primarily delivered long distance. Towards the end of the training, three senior staff from NEAEA traveled to ETS headquarters in Princeton, New Jersey to participate in an additional week-long leadership and technical training session. In 2014, another series of training events was provided by the American Institutes for Research. In April, a targeted training on sampling, data analysis, test equating, scaling, standard setting, and reporting was delivered to seven key technical staff working on NLSA; in August, 22 NEAEA staff received training on item bank development and data analysis; and in September, a training on standard setting was attended by more than 150 education staff from NEAEA, the Ministry of Education, and the Regional Education Bureaus. NEAEA’s increased capacity in these areas has improved the overall quality of existing assessment activities. Assessment Quality In 2013, work began to determine what was needed to develop sustainable item banks for both the NLSA and national examinations. Having item banks in place would greatly enhance future activities in these areas through the provision of a secure source of pre-screened test items with known technical characteristics. With READ Trust Fund support, a scoping study on item bank development was completed in 2014. The work to develop an item bank is now continuing with the support of GEQIP. COUNTRY-LEVEL RESULTS – ETHIOPIA 65 Classroom Assessment Clear Policies In Ethiopia, the Curriculum Framework for K-12 outlines what students are expected to learn at different grade levels. Under GEQIP, textbooks and teacher guides for nearly all primary and secondary subject areas have been made available. In 2014, a new manual with guidelines on classroom assessment for teachers was created by NEAEA staff. This “Classroom Assessment Manual for Primary and Secondary School Teachers” includes guidelines such as scoring criteria and rubrics for grading student work. The manual was validated in late 2013 during a workshop supported by the READ Trust Fund. Effective Human Resources In January and February 2014, two rounds of a “training of trainers” were conducted for 70 NEAEA, Ministry of Education, Regional Education Bureau, University, and Teacher Training College staff on the manual for classroom assessment. Later in 2014, workshops on the manual were also provided for some teachers. Assessment Quality Classroom assessment was included as one of the five domains covered by the School Inspection Framework, which was developed with the support of the READ Trust Fund. Additionally, classroom assessment has been included as a required component of teacher supervision and performance evaluations. With the support of GEQIP, a manual with information on classroom assessment and a related training module was developed for teacher supervisors. Examinations Clear Policies With the support of the READ program, a national policy framework for national examinations was developed. An in-depth review of Ethiopia’s system for national examinations was completed in 2011. Based on the findings and recommendations of this review, a policy framework with guidelines for the grades 8, 10, and 12 examinations was created and then adopted by the Ministry in 2013. 66 FINAL READ Trust Fund Report 2008–2015 NLSA Clear Policies An in-depth review of NLSA activities was conducted in 2010. Based on the findings of this review, work began to establish a clear policy for activity in this area moving forward. In 2013, a policy and procedural framework for NLSA was authorized and adopted by the Ministry of Education and NEAEA. The policy framework includes guidelines on the governance of the NLSA, who and what should be assessed, the frequency of administration, and how results should be used and communicated to stakeholders. Regular Budget In the past, funding for NLSA activities was irregular and came on an ad-hoc basis from either the Ministry of Education or international donors. With the creation of the NEAEA in 2012, a dedicated line item for NLSA activities was added to the budget and this area now receives a regular annual budget. School Inspection Clear Policies At the beginning of the READ program, activities to monitor education quality in Ethiopian schools were mostly undertaken on an ad hoc basis and not institutionalized in the education system. Education leaders recognized the need to develop a more coherent national system for inspecting schools and monitoring education quality. In 2010, an in-depth review of school inspection practices in Ethiopia was completed. Drawing on the findings of this report, a policy framework, guidelines, and school classification standards were created to govern work in this area. Strong Organizational Structures With the support of the READ Trust Fund, a new directorate dedicated to school inspection was established within Ethiopia’s Ministry of Education in 2012. This directorate includes 10 staff members fully dedicated to school inspection activities. Effective Human Resources In 2011, before the Directorate for School Inspection was established, three Ministry of Education staff went on a study visit to Ghana to learn how this country conducts its school inspections and to incorporate the lessons learned into the design of Ethiopia’s new directorate. In 2012 and 2013, after the Directorate for School Inspections was established, more than 60 Ministry of Education staff at the central and regional levels received training in how to implement and use the newly-created school inspection framework and guidelines. COUNTRY-LEVEL RESULTS – ETHIOPIA 67 School Inspection Schedule Aspect Standards Indicators Evidence The specific The minimum Statements that Examples of the aspects of the leves of illustrate the evidence that school’s work perfomance that standards and inspectors can all schools in provide a basis gather to judge Ethiopia should for assessing whether the achieve whether the standards are school is being met achieving them For each standard, inspectors assess whether the school is not achieving the standard (grade 1) the school is achieving the standard, and is performing in line with the standard (grade 2) the school is performing above the standard (grade 3) the school is performing well above the standard (grade 4) Inspectors also make an assessment of the school as a whole, using the same scale Assessment Quality As part of the pilot of the new school inspection activities, nearly 28,000 schools completed a self-assessment exercise that measured school quality against the standards in the school inspection framework. On-going school inspection activities are now taking place with the support of GEQIP. CONCLUSION The objective of the READ Trust Fund program in Ethiopia was to help strengthen existing or develop new institutions for measuring and assessing student achievement in order to improve education quality and student learning outcomes. As a direct result of the work accomplished under READ, Ethiopia now has a new agency and a new Directorate within the Ministry of Education, both of which are completely dedicated to monitoring education quality and assessing student achievement. The policy frameworks, stable funding, strong organizational structures, and trained staff together create a strong foundation for work in the area of student assessment. 68 FINAL READ Trust Fund Report 2008–2015 KYRGYZ REPUBLIC Total Grant Amount: US$2,350,000 Total Disbursements: US$2,029,737 COUNTRY CONTEXT The quality of education in the Kyrgyz Republic was dramatically impacted by the collapse of the Soviet Union. To reverse the decline in education quality, the government and development partners began working together to modernize the Kyrgyz education system. A number of reforms were implemented by the Ministry of Education and Science in an effort to strengthen the system and improve quality. Given all of these targeted efforts to improve education quality, the country was shocked when it received the results from the 2006 and 2009 PISA exercises and discovered that, in each instance, the Kyrgyz Republic ranked last out of all participating countries. The government realized that, in addition to collecting system-level data on learning outcomes, there was a serious need to actively monitor and support the education process in classrooms. With the support of the READ Trust Fund, the Kyrgyz Republic has been placing an even greater emphasis on regular, high-quality, formative assessment in the classroom. A new national assessment strategy highlights the importance of formative, summative, and system-level assessments all working together to reach the government’s goal of quality learning for all Kyrgyz children. Table 13. Developmental Ratings for the Kyrgyz Republic’s Student Assessment System Classroom Examinations NLSA ILSA Assessment 2009 2014 2009 2014 2009 2014 2009 2014 Latent Emerging Emerging Emerging Emerging Emerging Emerging Latent COUNTRY-LEVEL RESULTS – KYRGYZ REPUBLIC 69 PROGRAM OBJECTIVES, RATINGS, AND PROGRESS (2009-2014) Program Objectives and Developmental Ratings The main objective of the READ Trust Fund program in the Kyrgyz Republic was to strengthen the capacity of institutions responsible for measuring student learning outcomes, and improve the use of information from those assessments to improve teaching and learning. At the beginning of the program, the baseline diagnostic showed that examinations, NLSA, and ILSA were all emerging, while the weakest area was classroom assessment. With READ Trust Fund support, the Kyrgyz Republic chose to focus primarily on classroom assessment, while making concerted efforts to continue strengthening the other assessment areas as well. The main activities carried out under READ included capacity building in classroom assessment, developing a new school leaving examination for grade 11, administering an NLSA in grade 4, and developing a new national assessment strategy. By the end of READ, the Kyrgyz Republic had made the most progress in classroom assessment. As demonstrated in Table 13, the level of development for this area went from latent to emerging. While small improvements were also made in the areas of examinations and NLSA, they were insufficient to change the developmental ratings for these areas. The Kyrgyz Republic put on hold its participation in ILSAs in 2009 (linked to its performance on PISA 2006 and 2009), and hence its progress in this area retreated from emerging to latent. However, a recent decision was made by the Minister of Education and Science to resume work in this area and the country will participate in PISA 2018. 70 FINAL READ Trust Fund Report 2008–2015 Program Implementation READ Trust Fund-sponsored activities in the Kyrgyz Republic were implemented with the support of two Bank-executed and one Recipient- executed grant. A first grant for US$350,000 supported the diagnostic phase and the creation of an action plan. The other two grants were a combination of a Recipient-executed grant for $1,650,000 and a Bank-executed grant for US$350,000. The Recipient-executed grant primarily supported capacity- building activities for classroom assessment, the development of a new school leaving examination, and the implementation of an NLSA in grade 4; the Bank-executed grant primarily supported supervision activities and participation in international learning events. Implementation of the READ Trust Fund program in the Kyrgyz Republic was seriously delayed due to the April 7 revolution in 2010. In 2011, the program began to get back on track and steps were taken to finalize the action plan and other activities started earlier. In early 2013, the action plan received the necessary parliamentary ratification and the program went into full-scale implementation. Despite this shortened timeline for implementation, all activities were successfully completed before the end of the READ Trust Fund program. Table 14. Developing the Kyrgyz Republic’s Assessment System: Progress Achieved from 2009 to 2014 Latent Emerging Established Advanced 1 2 3 4 Classroom Assessment Examinations NLSA ILSA 2009 2014 Progress from 2009-2014 COUNTRY-LEVEL RESULTS – KYRGYZ REPUBLIC 71 Progress by Assessment Type and Key Indicator Areas The Kyrgyz Republic’s progress under the READ Trust Fund program has been carefully monitored using the READ Results Framework and SABER- Student Assessment benchmarking tools. Table 14 indicates the baseline and end-of-program scores for the Kyrgyz Republic based on the results of benchmarking exercises carried out at the beginning and end of the program. The progress seen is based on achievements in the key indicator areas being tracked under the READ Trust Fund program. The activities that the Kyrgyz Republic carried out under each key indicator area are described in the next section. Activities that benefited more than one type of assessment activity are described under a general category. Activities that benefited a specific type of assessment activity are described under assessment-specific headings. General Clear Policies In 2011, more than 40 representatives from all major stakeholder groups, including the Ministry of Education and Science, the Kyrgyz Academy of Education (KAE), the Educational Committee of the Kyrgyz Parliament, the Educational Assessment Unit, education NGOs, and development partners, participated in two workshops. These workshops were used to validate the findings of the self-diagnosis exercise and agree on an overall vision for further strengthening the country’s student assessment system. Then, in 2014, a working group created under the Ministry of Education and Science to oversee READ-funded activities, along with two international assessment experts, came together to develop a new national assessment strategy for the Kyrgyz Republic. This strategy document outlines a new vision and roadmap for an enhanced national assessment system – one that supports the government’s goal of improving learning for all Kyrgyz children. The proposed strategy has been endorsed by the Ministry of Education and Science for further discussion, along with the development of related policy documents for each of the different types of assessment activities. Strong Organizational Structures In 2009, international experts conducted a detailed analysis of the two institutions formally charged with the responsibility of carrying out assessment activities – the National Testing Center (NTC) and KAE’s Education Assessment Unit. The analysis assessed the strengths and weaknesses of these two institutions and provided recommendations for further strengthening and alignment. This report was a key input to the development of the final action plan. 72 FINAL READ Trust Fund Report 2008–2015 Effective Human Resources Through READ Trust Fund support, representatives from the Kyrgyz Republic’s Ministry of Education and Science and key assessment institutions participated in a series of learning events aimed at deepening their understanding of key issues in assessment and the importance of monitoring education quality. These learning events included attendance at international conferences on assessment topics, a study visit, and specific training courses. As a result, a critical mass of education officials and assessment practitioners have increased capacity to carry out assessment-related activities. Conferences: READ funds enabled delegations of four to six key  education officials to participate in each of the annual READ Global Conferences, the READ Regional Workshop in 2012, two IAEA Conferences in 2012 and 2013, and two EAOKO Conferences in 2012 and 2013. During each of these events, delegates delivered presentations on the progress achieved to date in further developing the Kyrgyz Republic’s assessment system. At the final READ Global Conference in St. Petersburg, the entire delegation participated in a panel session on the overall impact of the READ Trust Fund program and presented a short video demonstrating key achievements under the program in the Kyrgyz Republic. COUNTRY-LEVEL RESULTS – KYRGYZ REPUBLIC 73 Study Visit: In April 2011, a group of seven key education leaders went  on a study visit to the Central Institute for Test Development in the Netherlands where they learned how to use large-scale assessments to improve learning, and how to conduct independent, summative classroom assessments. Training Courses: From 2009–2014, between five and ten (depending  on the training session in question) assessment specialists from KAE and the Ministry of Education and Science participated in a series of trainings offered by the Russian Academy of Education’s Training Center and the Higher School of Economics in Moscow. Through these trainings, participants gained a better understanding of how to design an effective assessment system at both the national and subnational levels, and how to use computers to conduct psychometric analysis of test results. Representatives from the Kyrgyz Republic also traveled to Armenia to participate in a training on education quality and the use of assessment results. Use of Results In 2011, an in-depth secondary analysis of the Kyrgyz Republic’s results from PISA 2006 and 2009 as well as its 2007 and 2009 NLSA exercises was conducted by two international assessment specialists. This analytical work aimed to identify the main factors underpinning the scores in Reading, Mathematics, and Science and to provide recommendations for policy action. Classroom Assessment Clear Policies In addition to system-level curriculum and learning standards documents that outline what students are expected to learn, a manual with guidelines for teachers on classroom assessment was developed in 2014. This manual, created by KAE, includes guidelines for what should be assessed, uses of assessment information, and formats for assessment questions and tasks. More than 90,000 copies of these manuals were distributed during in-service training sessions and provided to national and regional teacher-training institutions to be incorporated into their training for pre-service teachers. This manual and related training materials were also made available online to support teachers in their use of classroom assessments. 74 FINAL READ Trust Fund Report 2008–2015 Effective Human Resources A key focus of READ Trust Fund-supported activities in the Kyrgyz Republic was to improve teachers’ capacity to carry out classroom-based assessments. In 2013, KAE developed a training program on classroom assessment for lead teachers and primary teachers. Two rounds of the five-day training for lead teachers were delivered to 53 lead teachers selected from all seven provinces. Then, in late 2013 and 2014, the lead teachers delivered 158 different three-day training sessions for primary school teachers. By August 2014, this training to ensure that teachers develop skills and expertise in classroom assessment had been delivered to more than 6,200 primary school teachers. Based on feedback from teachers, the training course materials were revised. In 2014, KAE authorized these materials to be formalized into a 10-hour training course for in-service teachers and a 60-hour training module for pre-service teachers. The teacher education curriculum was updated and these courses are now part of the regular curriculum for in-service and pre-service training institutes. “ [READ] helped us to develop tests [for] the new generation that allow us to identify students’ competencies and how well they can apply their knowledge in the practical world.” – Dogdurkul Kendirbaeva, Deputy Minister of Education and Science COUNTRY-LEVEL RESULTS – KYRGYZ REPUBLIC 75 Assessment Quality With the support of the READ Trust Fund, an innovative program developed by ETS to support teachers in conducting classroom-based assessments was translated into Russian. This program, called CBAL, which stands for “Cognitively Based Assessment of, for, and as Learning,” was adapted for training on classroom-based assessment in both Mathematics and Reading. Training was provided to NTC staff and a select group of teachers, with a view to future adoption across the education system. Examinations Effective Human Resources In order to develop a new school leaving examination for grade 11, international and national experts were hired to train 15 NTC staff and 30 subject experts on test theory, statistical analysis, item development, and test administration. As a result of this training, a new school leaving examination was developed for mandatory (Kyrgyz Language and Literature, Uzbek Language and Literature, Russian, Mathematics, and National History) and optional (Biology, Chemistry, Geography, Physics, and Foreign Languages – English, French, and German) subjects in the national curriculum. Additionally, 100 school administrators received training on this newly-developed school leaving examination. System Alignment In the past, the school leaving examination was well-aligned with the official curriculum. That curriculum was more knowledge-based and the examinations focused primarily on memorization and the reproduction of knowledge. However, in 2011, a new competency-based curriculum was introduced. This curriculum requires a new approach to developing testing instruments that will allow for the assessment of students’ competencies. With READ Trust Fund support, the NTC worked to develop a new examination that is better aligned with these ongoing curriculum changes. Assessment Quality In 2014, after a field trial to validate the new testing instrument, the examination was piloted in 200 schools in the Talas Region. The NTC analyzed the results of this pilot exercise and prepared a final report to share with schools, teachers, and other stakeholders. Discussions are ongoing about these potential changes and improvements to the existing school leaving examination. 76 FINAL READ Trust Fund Report 2008–2015 NLSA Assessment Quality The Center for Educational Assessment and Teaching Methods (CEATM), an independent firm not linked to the Ministry of Education and Science, is the institution responsible for carrying out NLSAs in the Kyrgyz Republic. Building on previous NLSA rounds conducted in 2007 and 2009, and with the support of the READ Trust Fund, CEATM implemented another NLSA of Reading, Mathematics, and Science in grade 4. In 2013, CEATM worked together with primary school teachers and assessment experts in the Ministry of Education and Science to develop updated versions of the testing instruments and questionnaires. A pilot to validate the new tools was conducted in 20 schools from two different regions. Based on the results, the main survey instruments were finalized. In April 2014, the NLSA was implemented in a randomly- selected sample of 204 primary schools across the country. In August 2014, CEATM finalized the analysis of the results and prepared the final report for dissemination to key stakeholders. During previous NLSA rounds, few documents were available to the public on the more technical aspects of the NLSA exercise. The report produced for 2014 provides information on these technical aspects, including test specifications, item construction, analysis of the pilot results, and scoring criteria. This enhanced reporting improves the ability to ensure the quality of NLSA activities. Use of Results Although the Ministry of Education and Science supported the 2007 and 2009 NLSA activities, the results were not optimally disseminated to the different stakeholder groups and rarely used in decision-making. In 2014, CEATM’s analysis of the results included disaggregation by subject, languages of instruction (Kyrgyz, Russian, and Uzbek), gender, school location, and region. In addition, secondary analysis of the questionnaire data for students, teachers, and school administrators was conducted to identify the key determinants that had the greatest effect on learning outcomes. This breakdown of results was used to determine concrete actions for the Ministry of Education and Science to take moving forward. Reports on these results were prepared in both Kyrgyz and Russian and delivered to regional education administrators, schools, and teacher training institutes. In addition to the official report, presentations on the results were delivered to schools and regional and local education authorities. COUNTRY-LEVEL RESULTS – KYRGYZ REPUBLIC 77 CONCLUSION The READ Trust Fund program in the Kyrgyz Republic played a catalytic role in promoting the importance of assessment and monitoring student learning outcomes. The new strategy document provides a clearer vision and roadmap for developing a national assessment system and KAE, the NTC, CEATM, and teacher training institutions all have more clearly defined mandates. Due to the training provided in classroom assessment, more than 6,200 primary school teachers are better equipped to incorporate formative assessment into their teaching practices. The new guidelines for teachers, other materials, and training courses for both in-service and pre-service teachers better ensure that teachers will develop the requisite skills and provide more regular and systematic monitoring of learning in the classroom. 78 FINAL READ Trust Fund Report 2008–2015 MOZAMBIQUE Total Grant Amount: US$3,091,937 Total Disbursements: US$3,021,894 COUNTRY CONTEXT Since 2000, Mozambique’s education system has undergone rapid expansion and the number of students participating in primary education has more than doubled. The education system has struggled to keep up with this demand, resulting in shortages of teachers and other resources that have negatively impacted education quality. The government’s vision for its education system includes access, equity, and quality goals and efforts have been made to establish a system to monitor learning outcomes. At the beginning of the READ Trust Fund, national examinations, NLSA, and ILSA were taking place in Mozambique, but they were irregular and failed to interconnect as a cohesive system. In addition, there was little evidence that assessment results were being used to improve education quality. With the support of the READ Trust Fund, the technical capacity of those responsible for designing and implementing assessment activities was greatly enhanced and a new classroom-based assessment tool for teachers was developed. Overall, the quality of assessment activities in Mozambique greatly improved during the lifetime of the READ program and there is now a culture within the Ministry of Education where assessment results are valued and used to inform policy dialogue and education reforms. Table 15. Developmental Ratings for Mozambique’s Student Assessment System Classroom Examinations NLSA ILSA Assessment 2009 2014 2009 2014 2009 2014 2009 2014 Emerging Emerging Emerging Established Emerging Emerging Emerging Emerging COUNTRY-LEVEL RESULTS – MOZAMBIQUE 79 PROGRAM OBJECTIVES, RATINGS, AND PROGRESS (2009-2014) Program Objectives and Developmental Ratings The objective of the READ Trust Fund program in Mozambique was to improve the Ministry of Education’s technical and institutional capacity to assess student learning outcomes. This included improving the quality of assessment work, aligning assessment-related activities with national learning goals, and helping to create a culture where assessment results are valued by education stakeholders at all levels. At the beginning of the program, the baseline diagnostic indicated that most aspects of Mozambique’s assessment system were at an emerging level of development. READ Trust Fund– supported activities primarily focused on building capacity within the Ministry of Education for work in all assessment areas and developing a new classroom-based reading assessment tool in grade 3. By the end of READ, significant progress had been made in all assessment areas and, as demonstrated in Table 15 above, the level of development of Mozambique’s examinations program had even shifted to established. 80 FINAL READ Trust Fund Report 2008–2015 Program Implementation The READ Trust Fund program in Mozambique was implemented with the support of three Bank-executed grants. A first grant for US$460,000 supported the diagnostic phase and the creation of an action plan; a second grant for US$2,031,937 supported capacity-building activities and the development and implementation of an early-grade reading assessment called Provinha; and a third grant for US$600,000 provided additional financing for Provinha-related activities as well as an impact evaluation to assess the utility of Provinha as a tool to monitor education quality. As a direct result of the capacity-building activities financed under READ, additional resources from the Education Sector Support Fund and other donors were made available to support assessment-related activities, including the implementation of a NLSA in 2013, and the purchase of new buildings and equipment for the examinations program. Progress by Assessment Type and Key Indicator Areas Mozambique’s progress under the READ Trust Fund program has been carefully monitored using the READ Results Framework and SABER-Student Assessment benchmarking tools. Table 16 indicates the baseline and end-of- program scores for Mozambique based on the results of benchmarking exercises carried out at the beginning and end of the program. The progress seen is based on achievements in the key indicator areas being tracked under the READ Trust Fund program. The activities that Mozambique carried out under each key indicator area are described in the next section. Activities that benefited more than one type of assessment activity are described under a general category. Activities that benefited a specific type of assessment activity are described under assessment-specific headings. Table 16. Developing Mozambique’s Assessment System: Progress Achieved from 2009 to 2014 Latent Emerging Established Advanced 1 2 3 4 Classroom Assessment Examinations NLSA ILSA 2009 2014 Progress from 2009-2014 COUNTRY-LEVEL RESULTS – MOZAMBIQUE 81 General Strong Leadership Leaders within Mozambique’s Ministry of Education have demonstrated strong support for developing the country’s assessment system. In October 2009, a workshop was held with key Ministry of Education officials and international partners to discuss and validate the findings from the baseline diagnostic and agree on the way forward. Then in 2010, a second national workshop was organized to bring together 50 Ministry of Education officials from various departments, with a particular emphasis on those responsible for planning, curriculum development, and examinations. This workshop highlighted the need to establish a national assessment system with clear linkages to the overall work of the Ministry, and to focus greater attention on the institutional arrangements for various assessment-related activities. Effective Human Resources In Mozambique, a significant amount of support under READ was targeted to building capacity within the Ministry of Education to carry out assessment activities. These capacity-building activities took place in the form of international conferences on assessment topics, study visits to Brazil, short courses on assessment topics, and participation in a Master’s degree program on Educational Management and Evaluation. This capacity building and technical assistance provided to the Ministry of Education, and the National Institute for the Development of Education (INDE) in particular, directly contributed to being able to make a NLSA program technically possible for the first time in Mozambique. Conferences: As a result of participating in several international learning  events, more than 50 representatives from Mozambique’s Ministry of Education, and INDE and the National Board of Examinations, Certification and Equity (CNECE) in particular, have a greater understanding of key issues and best practices in assessment. Delegations of five to six representatives from the Ministry of Education participated in each of the annual READ Global Conferences. At the final READ Conference in St. Petersburg, an eight-member delegation participated in a panel session on the overall impact of the READ program in Mozambique. In 2012, the READ Regional Workshop for Africa was held in Maputo and the team had the chance to directly share their experiences implementing Provinha with other African countries. “The Government’s vision puts quality and equity in basic education at the center of its education policy. Improvement in access cannot happen without a substantial improvement in quality.” – Mozambique’s Strategic Plan of Education and Culture 82 FINAL READ Trust Fund Report 2008–2015 READ funds were also used to support Mozambique’s participation in four other assessment-related events in Africa. In April 2010, several staff from the Ministry of Education participated in the Users and Uses of Assessment Information Workshop in South Africa and received training in how to effectively disseminate NLSA results. Then, in 2011 and 2013, representatives from INDE and CNECE traveled to Kenya and Tanzania to attend two AEAA Conferences. In both Kenya and Tanzania, the team had the opportunity to present papers on assessment topics, discuss regional challenges, and, in Tanzania, participate in a panel session on READ country-level results. In Kenya, the team also had the opportunity to visit the Kenya National Examinations Council to learn about its operations and processes for administering national examinations. Finally, in preparation for the impact evaluation supported under READ, a team traveled to Ethiopia to take part in the workshop, “Evaluating the Impact of Development Programs: Turning Promises into Evidence.” This workshop provided the six-member team in attendance with a general overview of impact evaluations and practical guidance on how to design the impact evaluation for Provinha. Following this workshop, a training program for 25 Ministry of Education staff directly involved with Provinha was held in Maputo in October 2012. Study Visits: In June and August 2010, 12 representatives from INDE  and CNECE traveled to Brazil to learn from the country’s experience in conducting assessments and making effective use of the results. During these visits, the participants were given opportunities to interact with assessment experts, school directors, and policymakers, and to make connections with potential providers of future capacity-building activities. COUNTRY-LEVEL RESULTS – MOZAMBIQUE 83 Short Courses: In 2011 and 2012, a team of around 15 staff from INDE, CNECE, and key universities participated in three short courses on assessment topics. The first two courses took place in Brazil and covered the topics of test design, item analysis, statistical analysis, and proficiency scale development. The third course took place in Maputo and covered the topics of data analysis, report writing, and dissemination strategies. These courses were intended to enhance the capacity of participants to the degree that they could carry out assessment activities with limited external support. As a result of their participation, team members were able to apply their newly-acquired knowledge to the development of a new classroom-based assessment tool called Provinha. Master’s Program in Educational Management and Evaluation:  With READ Trust Fund support, eight staff from the Ministry of Education (six from INDE and two from CNECE) and one staff member from the University of Eduardo Monlande completed a two-year Master’s program in Educational Management and Evaluation through the Centro de Politicas Publicas e Avaliação da Educação based in Brazil. This program included both distance and face-to-face learning. Between the start of the program in 2011 and its completion in 2013, the participants travelled to Brazil five times in order to participate in the face-to-face part of the course. As a result of this program, there are now several staff formally qualified in the design and management of national assessments. Alignment with Learning Goals In 2011 and 2012, the government worked with directors, teachers, and development partners in six provinces to create and field test 70 education quality indicators. Since then, the Ministry has been in the process of finalizing these indicators and incorporating them into documents that summarize their uses for a range of users. Additionally, in 2011, a blueprint for writing test items for different topics in the national curriculum at grades 3, 5, and 7 was created. A bank of more than 600 test items is now available to draw on for various assessments, such as Provinha and the recent NLSA. Classroom Assessment Assessment Quality During the study visit to Brazil, staff from INDE learned about an innovative classroom assessment program that was being used to monitor early grade reading skills. They decided to create something similar for Mozambique. Applying the knowledge they acquired through various training activities, INDE led the design and implementation of a new classroom assessment tool, an early grade reading assessment for grade 3 called Provinha. 84 FINAL READ Trust Fund Report 2008–2015 The materials created for Provinha included a test kit with administration and interpretation manuals; a test booklet; a scoring guide; communication materials for parents, teachers, and school leaders; and results posters. After an initial pilot phase in two Maputo districts during 2011 and 2012, the program was scaled up to include an additional two districts in northern Mozambique in the Tete and Niassa Provinces. In 2013, 188 schools in these four districts participated in Provinha. In 2014, a fifth district from the Cabo Delgado Province was added and a total of 206 schools participated. Including the first pilot in 2011, Provinha was administered seven times – once at the beginning of each school year and again at the end in order to track student progress. Each time, the assessment was administered and scored by teachers and the results were intended to be used at the school level to improve teaching practices and student learning outcomes in reading. Follow-up workshops were organized with pedagogical directors to discuss the results and options for adjusting teaching practices accordingly. “Provinha as a tool proved to be very effective. It changed the attitudes of the teachers – the attitude toward their work and the attitude toward the subjects they needed to teach.” – Mozambican Delegation, Fourth READ Global Conference COUNTRY-LEVEL RESULTS – MOZAMBIQUE 85 In 2013, through support from READ and the Strategic Impact Evaluation Fund (SIEF), an impact evaluation was set up to ascertain whether Provinha had a significant impact on student achievement. Specifically, the evaluation aimed to assess the utility of Provinha as a tool for 1) increasing teachers’ awareness about the strengths and weaknesses of their students; 2) helping teachers adjust their teaching practices accordingly; and 3) improving their students’ learning outcomes. In March and October 2013, Provinha was administered in a total of 110 schools – 53 treatment schools and 57 control schools across two districts. Based on the final results of this impact evaluation, a decision will be made about whether or not to further scale up the assessment. Use of Results Results from Provinha have been shared at the national level in the context of the overall Education Sector’s Annual Meetings. The results and experiences from Provinha are contributing to the overall debate on strengthening the quality of students’ learning in Mozambique. Based on its success, INDE is continuing to implement Provinha and contemplating the creation of a similar instrument to support numeracy. Examinations Strong Organizational Structures Through a combination of the READ Trust Fund and the Education Sector Support Fund, targeted efforts were made to improve the system for examinations. CNECE, previously just a unit within the Ministry of Education, is now considered a “semi-autonomous,” permanent institution which is accountable to an external board. In 2014, CNECE staff moved into new and secure buildings equipped with up-to-date computers, software, and security. Data-scanning technology was also made available in almost all provinces to allow them to automatically capture examination results. NLSA Clear Policies Prior to 2009, there was no policy guiding NLSA activities in Mozambique. As a result of the efforts to strengthen Mozambique’s assessment system, in 2013, the Ministry of Education issued a new formal document that authorizes NLSA activities. This Education Sector Strategic Plan and the accompanying operation plan specify that NLSAs will assess grade 3 students in the areas of Portuguese and Mathematics every three years. 86 FINAL READ Trust Fund Report 2008–2015 Regular Budget In 2010, following on a national workshop to discuss NLSA activities, a decision was made to create a line item for NLSA in the annual education budget. Since then, stable funding has been provided in the area of NLSA. Strong Organizational Structures INDE staff were responsible for NLSA exercises in 2000, 2006, and 2009. However, there was no unit within INDE exclusively dedicated to assessment. In an effort to further establish and institutionalize a system for NLSA, in 2010, a decision was made to reorganize INDE to include a unit with clear responsibility for overseeing national assessments. Assessment Quality Between 2009 and 2013, major revisions were made to previous NLSA activities, including the test structure, items, administration, and logistics. The quality of the NLSA benefitted significantly from the large number of capacity- building activities made available to INDE staff under the READ Trust Fund. The NLSA implemented in 2013 was considered the first “official” NLSA in Mozambique. Data for this assessment were collected in 2013 and the initial findings were shared with all relevant stakeholders during a Joint Sector Review in 2014. NLSA results now feature prominently in the overall policy debate on the quality of education and possible reforms. CONCLUSION As a result of the READ program in Mozambique, the Ministry of Education’s technical and institutional capacity to assess student learning outcomes has vastly improved and assessment results are valued by education stakeholders at all levels. Between 2009 and 2014, the READ Trust Fund program contributed to a significant culture shift in how assessment data are collected and used to inform both 1) teaching practices at the classroom level and 2) the overall policy debate at the national level. The availability of nationally representative data on learning outcomes in grade 3 is helping to inform the overall policy debate on the quality of education and the types of interventions that are most likely to produce change. This type of information now features prominently in discussions between the Ministry of Education, development partners, and civil society organizations and has directly impacted the expansion of the current Education Sector Strategic Plan beyond 2016. COUNTRY-LEVEL RESULTS – TAJIKISTAN 87 TAJIKISTAN Total Grant Amount: US$4,100,000 Total Disbursements: US$4,009,175 COUNTRY CONTEXT The collapse of the Soviet Union and the ensuing civil war had a serious impact on education quality in Tajikistan. Since the signing of a peace accord in the late 1990s, the Government of Tajikistan has focused on rebuilding its education system and ensuring the effective delivery of, and equal access to, a high-quality education. While access to basic education has become nearly universal, access to secondary and tertiary education has continued to show significant disparities by both gender and income. Part of Tajikistan’s mission to improve education quality and access has included firmly establishing a NTC tasked with the responsibility of measuring student learning outcomes, particularly at the interface between secondary and university education. Although a NTC and university entrance examinations previously existed, the system was deemed by most to be flawed and the test results unreliable. The main objective of READ Trust Fund support in Tajikistan was to increase the capacity of the NTC to implement assessment activities, including a new Unified University Entrance Examination (UEE) system. Through the creation of a centralized examination for admission to all universities in Tajikistan, the government aimed to standardize testing practices and procedures, reduce corruption, and give students more equitable access to higher education. Based on the success of the pilot, the new UEE was adopted into law in late 2014. 88 FINAL READ Trust Fund Report 2008–2015 PROGRAM OBJECTIVES, RATINGS, AND PROGRESS (2009-2014) Program Objectives and Developmental Ratings The objective of the READ Trust Fund program in Tajikistan was to increase the capacity of the NTC to oversee the sustainable introduction of the new UEE and eventually national grade-level assessments. At the beginning of the program, the baseline diagnostic showed Tajikistan’s overall assessment system to be at a latent to emerging stage in its development. Given the government’s strong interest in having a UEE, an additional, more in-depth diagnostic was conducted to evaluate the existing examinations system and better understand its strengths and weaknesses. Based on the results, a decision was made to use READ funds to provide tailored capacity building and support for the operationalization of the newly-established NTC and the development and introduction of the new UEE system. By the end of the READ Trust Fund program, the NTC was fully operationalized and the new UEE had been successfully piloted, adopted into law, and implemented for the first time on a national scale. While the developmental rating for examinations remains emerging, it is expected to move to established once additional rounds of the new UEE have been successfully conducted. COUNTRY-LEVEL RESULTS – TAJIKISTAN 89 Table 17. Developmental Ratings for Tajikistan’s Student Assessment System Classroom Examinations NLSA ILSA Assessment 2009 2014 2009 2014 2009 2014 2009 2014 Emerging Emerging Emerging Emerging Latent Latent Latent Latent Program Implementation The Government of Tajikistan’s plan to strengthen and operationalize the NTC was supported through various funding streams, including the READ Trust Fund (US$4.1 million), the World Bank (US$2.3 million), the Open Society Institute (US$1.5 million), and the government’s own budget (US$.5 million). The READ Trust Fund primarily supported the development of the new UEE system, capacity-building activities, recurrent cost co-financing for NTC staff, and the provision of equipment; the World Bank supported the design and construction of the new NTC, in addition to the provision of furniture and equipment; the Open Society Institute supported the initial preparations of the UEE design and capacity-building activities for national experts and NTC staff; and the government’s own funds were used to allocate land, provide temporary space for the NTC, and rehabilitate regional offices. From 2009-2014, a total of US$8.4 million was provided to strengthen and operationalize the NTC and develop the new UEE system. Table 18. Developing Tajikistan’s Assessment System: Progress Achieved from 2009 to 2014 Latent Emerging Established Advanced 1 2 3 4 Classroom Assessment Examinations NLSA ILSA 2009 2014 2009 and 2014 Progress from 2009-2014 90 FINAL READ Trust Fund Report 2008–2015 READ Trust Fund-sponsored activities in Tajikistan were implemented with the support of two Bank-executed and one Recipient-executed grant. A first grant for US$400,000 supported the diagnostic phase and the creation of an operations manual for the NTC; the other two grants were a combination of a Recipient-executed grant for $3,400,000 and a Bank-executed grant for US$300,000. The Recipient-executed grant primarily supported capacity- building activities for NTC staff; the development, pilot, and full-scale implementation of the new UEE; and the provision of equipment. The Bank-executed grant supported supervision activities and an evaluation of the new UEE system. In addition to their strong support and coordinated work to establish the new NTC, the Ministry of Education and Science and donors such as the World Bank (through funds from GPE) and USAID have been working to modernize the school curriculum. The government’s 2012 National Strategy for Educational Development indicated that the more traditional, knowledge- based system would be reformed to be more competency-based. As a part of this curriculum reform work, progress was also made in the area of classroom assessment. Progress by Assessment Type and Key Indicator Areas Tajikistan’s progress under the READ Trust Fund program has been carefully monitored using the READ Results Framework and SABER-Student Assessment benchmarking tools. Table 18 indicates the baseline and end-of- program scores for Tajikistan based on the results of benchmarking exercises carried out at the beginning and end of the program. The progress seen is based on achievements in the key indicator areas being tracked under the READ Trust Fund program. The activities that Tajikistan carried out under each key indicator area are described in the next section. Activities that benefited more than one type of assessment activity are described under a general category. Activities that benefited a specific type of assessment activity are described under assessment-specific headings. “The Government of Tajikistan has embarked on key reforms to improve transparency and fairness in the delivery of education, beginning with a highly symbolic and publicized reform of the university entrance exam (UEE) to improve transparency and equity in student selection and access to university.” – Abla Safir, “High Hopes, Moderate Expectations: Barriers to Access and Complete Higher Education in Tajikistan” COUNTRY-LEVEL RESULTS – TAJIKISTAN 91 General Effective Human Resources In addition to targeted activities to build the capacity of NTC staff to develop the new UEE, several Ministry of Education and NTC officials had the opportunity to participate in a number of international events on assessment- related topics. These events included the annual READ Global Conferences, a READ Regional Workshop, an IAEA Conference, and two EAOKO Conferences. At the annual READ Global Conferences, delegations of four to five key education officials each year delivered a presentation on the progress achieved to date in establishing the NTC. At the final READ Global Conference in St. Petersburg, the Deputy Minister of Education joined the delegation and participated in a panel session on the overall impact of the READ Trust Fund program in Tajikistan. In 2012, four NTC staff traveled to Kazakhstan to participate in the IAEA Conference and share experiences with other READ-sponsored countries during the READ Regional Workshop for ECA. In 2012 and 2013, a representative from the NTC also participated in the EAOKO Conferences organized by CICED. As a result of their participation in these international learning events, a critical mass of key individuals from Tajikistan’s Ministry of Education and NTC have a greater understanding of key issues and best practices in assessment. 92 FINAL READ Trust Fund Report 2008–2015 Classroom Assessment Alignment with Learning Goals Based on the 2012 National Strategy for Educational Development in Tajikistan, the Ministry of Education has been developing a new curriculum that moves away from the more traditional, knowledge-based approach to a more competency-based one. As part of this work, learning standards are being developed for all primary subjects in grades 1 to 4, including Tajik, Russian, English, Mathematics, Nature, Arts and Crafts, Physical Education, and Music and Singing. The standards clearly outline what students at different grades and age levels are expected to learn, and the desired performance levels. Based on these standards, guidelines for classroom assessment by teachers are now being developed. Examinations (UEE) Clear Policies At the beginning of the READ Trust Fund program, a NTC Master Plan of Action and UEE Operations Manual were drafted to ensure the appropriate regulatory and legal frameworks were in place. Through technical assistance provided under READ, the three-year Master Plan (which covered staff hiring, capacity building, procurement of equipment and materials, UEE piloting and final delivery, among other things) was periodically updated. In 2011, the NTC collected information from various universities and government institutions in order to finalize the UEE model and prepare the requisite regulatory and legal documents for the introduction and supervision of the UEE. These documents were subsequently adopted into law by the Government. Strong Leadership The NTC and the new UEE system are fully supported by the government. Two large workshops for leaders from the government, universities, civil society, and the development partners were held early on in the process – one for the launch of the UEE program in 2011, and another after completion of the full-scale pilot in 2013. Then, in December 2013, the President of Tajikistan officially inaugurated the new system. In 2014, the President and the President’s Office continued to demonstrate their strong commitment to the new UEE system by facilitating a mass-media campaign throughout the entire country. A series of workshops were also used to orient all municipal and district-level education officials to the new UEE system and help facilitate preparations for the first national implementation in 2014. COUNTRY-LEVEL RESULTS – TAJIKISTAN 93 Regular Budget Beginning in 2011, the government has provided an annual allocation of US$100,000 in state funds to support the work of the NTC. With the approval of the Ministry of Finance, the student enrollment fee for the UEE system was piloted in 2013 and finalized in 2014. The fee was set at a level that would ensure the financial stability of the NTC to conduct the UEE, including all research and development costs. The fee mechanism worked smoothly during the first national implementation in 2014. Starting in 2015, the NTC will receive an increased state budget allocation to further ensure the sustainability of the NTC and UEE operations. Strong Organizational Structures At the beginning of the READ program, the NTC had limited staff and no office space. By the end of the READ program, a new NTC building and four regional offices had been secured, the NTC had recruited 60 full-time staff, and the NTC offices had been equipped with all essential Information Technology (IT) and printing equipment. In 2014, a significant change was made in the status of the NTC. The NTC went from being an institution of the Ministry of Education and Science to coming under the direct authority of the President of the country. As such, changes were made to the NTC’s Senior Management, which further strengthened the NTC team in time for the delivery of the UEE in 2014. Effective Human Resources NTC staff and Ministry of Education and Science officials participated in a variety of activities aimed at building their capacity to carry out assessment activities and develop the new UEE system. With the support of other donor funds, staff members participated in: 1) study trips to National Testing Centers in Azerbaijan, Russia, Ukraine, and Turkey (5 to 8 staff on each trip); 2) a series of trainings on assessment topics provided by the Russian Testing Center (10 NTC staff); and 3) training in test development and data analysis (40 NTC staff and national experts). Additionally, with READ Trust Fund support, two prospective NTC staff traveled to the Kyrgyz Republic and one NTC staff member traveled to Kazakhstan to receive training in procurement and financial management. In 2014, concentrated training was provided to ensure the successful implementation of the UEE. Fifteen NTC staff received specific training on security procedures; 60 district education staff received training on overall UEE procedures; and nearly 1,600 registrars, administrators, and verifiers received training on their respective tasks. This training resulted in the successful registration of over 52,000 students at 21 different locations and the equally successful administration of the UEE to over 46,000 students who sat for the examination at 40 different locations. 94 FINAL READ Trust Fund Report 2008–2015 Alignment with Learning Goals To ensure that the new UEE aligned well with established university admission requirements, the NTC developed a database with information from 30 universities (covering more than 400 specialties) about entrance rules, procedures, requirements, and admission plans. These data were analyzed and used to design the UEE. In January 2014, an official admissions plan, which included all universities and specialties covered by the UEE, the number of places available, language of instruction, and so on was published. Additionally, the UEE papers were carefully designed to correspond to the curriculum and learning standards established by the Ministry of Education and Science. Opportunities to Learn about Assessment Activities A public relations and communications strategy to introduce the new UEE was developed with READ Trust Fund support. At the request of the President of Tajikistan, this strategy was augmented in 2014 to use mass media as a way to broadly disseminate information about the NTC and UEE. By the end of READ, 221 articles had been published in local newspapers and posted on local and international websites, 45 films for different target audiences had been produced and broadcasted on national television, 247 television messages and 168 televised speeches had also been broadcasted on national television, and 12 press releases had been shared with 28 different national and international mass media agencies. At the end of READ, the NTC organized a conference to disseminate information on the 2014 UEE, discuss lessons learned, and clarify changes needed for the 2015 UEE. In 2013, a series of meetings took place with schools, parents, and students to inform them about the new UEE. Then, in 2014, 136 municipal authorities and district education officials were trained on the UEE by NTC staff using the Student Handbook and other materials. These officials then trained over 3,500 secondary school directors who provided 36-hour orientation sessions to over 100,000 grade 11 students. The Student Handbook used during the training sessions was also made available on the NTC website. COUNTRY-LEVEL RESULTS – TAJIKISTAN 95 Quality Assurance In 2012 and 2013, the NTC carried out several quality assurance procedures as part of the design and implementation of the new UEE. In May 2012, the NTC designed a pilot version of the new UEE and conducted a series of mini-pilots to test various aspects of the system, including the testing instrument, registration, and administration. Then, in 2013, 7,000 grade 10 students participated in a full-scale pilot, which included the revised tests, administrative, and security procedures. The pilot was deemed successful and a decision was made to implement the new UEE system on a national scale in 2014. In 2014, all UEE test items were approved by an expert committee; the training provided to, and the performance of, the registrars and administrators were carefully monitored for quality by NTC staff; and the IT system was independently evaluated by a team of experts. In addition, at the end of the pilot stage, the World Bank conducted an independent evaluation of the procedures of the new UEE system to ensure their quality and confirm that the new system was ready for full-scale implementation. CONCLUSION The READ Trust Fund-supported activities in Tajikistan revolutionized the country’s university admissions system. Over the past six years, the new model for the UEE system was carefully designed and piloted. Learning from other countries’ experiences and international best practices, the NTC followed a detailed, step-by-step process to introduce the new system. Careful attention was given to such issues as creating the necessary legal frameworks; hiring staff with the right skills and continuing to build their capacity; constructing a secure building, equipped with an adequate IT system; and ensuring that UEE test items were well designed and aligned with university requirements. Based on the success of the pilot, the President of Tajikistan formally adopted the new system into law in late 2013. The NTC is now fully staffed with trained individuals who are capable of overseeing assessment activities, and is financially stable due to examination fees and contributions from the government. The new UEE system is standardized, transparent, and merit based – and as a result, qualified students have a more equal chance at obtaining entrance to university. 96 FINAL READ Trust Fund Report 2008–2015 VIETNAM Total Grant Amount: US$3,000,000 Total Disbursements: US$2,905,644 COUNTRY CONTEXT Similar to its economy, Vietnam’s education system has improved at a remarkable pace. Having almost met the goal of universal primary education, government officials are now paying even greater attention to ensuring education quality. Vietnam’s vision for its education system is one that enables the country to become a globally competitive, knowledge-based economy. Vietnam’s education system is vast, including nearly 40,000 schools, one million teachers, and 20 million students. During the READ Trust Fund program, the government prioritized the measurement of learning outcomes and the monitoring of education quality in order to ensure that investments in education produced quality learning outcomes. Prior to READ, various assessment activities were already in place and departments tasked with assessment responsibilities were well-staffed. The challenges that Vietnam faced related more to 1) the size of its education system and the large number of key stakeholders that needed training in order to be able to effectively design and implement assessment activities, and 2) the need to transform its curriculum and assessments from knowledge- to competency-based. By the end of READ, a new competency-based assessment framework had been adopted, training programs had been developed for all key stakeholders, and the groundwork was laid for a new, competency-based curriculum. Table 19. Developmental Ratings for Vietnam’s Student Assessment System Classroom Examinations NLSA ILSA Assessment 2009 2014 2009 2014 2009 2014 2009 2014 Emerging Established Emerging Established Emerging Emerging Latent Emerging COUNTRY-LEVEL RESULTS – VIETNAM 97 PROGRAM OBJECTIVES, RATINGS, AND PROGRESS (2009-2014) Program Objectives and Developmental Ratings The objective of the READ Trust Fund program in Vietnam was to strengthen and improve the effectiveness of the country’s assessment system. The program supported Vietnam to enhance its capacity to conduct systemic assessments of learning outcomes and provide recommendations for actions to improve student learning. At the beginning of the program, the baseline diagnostic showed that the most developed part of Vietnam’s assessment system was classroom assessment, which was only emerging. With READ funds, Vietnam focused significant efforts on capacity building in a number of assessment areas, and among a wide range of stakeholders, i.e., teachers, school leaders, provincial departments, Ministry of Education and Training (MOET) staff, teacher training colleges, and universities. Additionally, Vietnam concentrated its READ support on developing and institutionalizing competency-based assessment practices into their new general education curriculum. By the end of READ, as demonstrated in Table 19, Vietnam had made significant progress in all assessment areas and had built a strong foundation for future activities. 98 FINAL READ Trust Fund Report 2008–2015 Program Implementation The READ Trust Fund program in Vietnam was implemented in two phases with the support of two Bank-executed grants. A first grant for US$350,000 supported the diagnostic phase and the creation of an action plan; a second grant for US$2,650,000 supported the development of training programs and capacity-building activities for different stakeholders, the implementation of various surveys (i.e., Early Development Instrument (EDI)), improvements to classroom assessments and national examinations, and the design of a new curriculum and textbooks that incorporate competency-based assessment practices. In conjunction with these READ-funded activities, a number of other World Bank-funded projects and donors were working to advance Vietnam’s education system. Vietnam’s decision to participate in PISA 2012, and the team’s attendance at the initial meetings, were supported under READ, while the full cost of Vietnam’s participation in this exercise was supported by the Asian Development Bank. Also, additional rounds of NLSA activities were supported by the World Bank-financed School Quality Assurance project, and additional efforts to improve classroom assessment practices were supported by the Global Partnership for Education-financed Vietnam New School project. In 2013, after conducting a financial review of the entire READ Trust Fund program, the READ Council decided to reallocate US$1 million of the funds originally awarded to Vietnam under their second grant (US$3.5 million) to other READ countries. This was an effort to better balance the amount of money allocated to each country under the program, taking into account the time remaining and various countries’ absorptive capacity. In 2014, given Vietnam’s significant progress, and available funds remaining under the READ program, the country was reallocated US$150,000 to finalize remaining activities. Hence the total amount of Vietnam’s second grant ultimately came to US$2,650,000. Table 20. Developing Vietnam’s Assessment System: Progress Achieved from 2009 to 2014 Latent Emerging Established Advanced 1 2 3 4 Classroom Assessment Examinations NLSA ILSA 2009 2014 Progress from 2009-2014 COUNTRY-LEVEL RESULTS – VIETNAM 99 Progress by Assessment Type and Key Indicator Areas Vietnam’s progress under the READ Trust Fund program was carefully monitored using the READ Results Framework and SABER-Student Assessment benchmarking tools. Table 20 indicates the baseline and end-of- program scores for Vietnam based on the results of benchmarking exercises carried out at the beginning and end of the program. The progress seen is based on achievements in the key indicator areas being tracked under the READ Trust Fund program. The activities that Vietnam carried out under each key indicator area are described in the next section. Activities that benefited more than one type of assessment activity are described under a general category. Activities that benefited a specific type of assessment activity are described under assessment-specific headings. General Clear Policies With READ Trust Fund support, a study and series of workshops took place in 2010 to review Vietnam’s existing assessment policies, regulatory framework, and practices in examinations and classroom assessment. The results of this review indicated a lack of coherence and consistency across the system and a need for new assessment frameworks to help improve the quality of national examinations and assist teachers in conducting classroom assessment activities. In a second series of workshops in 2010 and 2011, the Department of Education Testing and Accreditation worked with the Primary and Secondary Education Departments, primary and secondary schools, and assessment experts to develop new assessment frameworks aligned with the curriculum, along with first-semester, end-of-year, and high school entrance and graduation tests. Assessment frameworks and tests were developed for Vietnamese and Mathematics in grades 1–5; Science in grade 3; History and Geography in grades 4–5; Vietnamese and Mathematics for high school entrance; and Vietnamese, Mathematics, History, Geography, Physics, and Chemistry for high school graduation. Guidelines for teachers and a policy and regulatory framework to support the new frameworks and tests were developed and officially adopted by MOET in 2011. During 2012-2014, the implementation of these new policies, assessment frameworks, and first-semester and end-of-year tests were monitored and reported on. 100 FINAL READ Trust Fund Report 2008–2015 Strong Leadership Leaders within MOET demonstrated strong support for further developing the country’s assessment system. In December 2009, at the beginning of the READ program, a workshop was held with all key education stakeholders to discuss and validate the findings from the baseline diagnostic and agree on an action plan. Key leaders remained actively engaged throughout the entire program by organizing and participating in yearly READ-Vietnam Conferences, leading workshops to review and endorse new assessment policies, and overseeing activities to develop assessment training programs. This demonstration of strong leadership and support for assessment activities contributed to significant improvements in Vietnam’s student assessment system. Effective Human Resources A significant amount of the support provided to Vietnam under READ was focused on building capacity within MOET to carry out assessment activities. Several of these capacity-building activities took the form of international conferences on assessment topics, study visits, and training courses. COUNTRY-LEVEL RESULTS – VIETNAM 101 Conferences: Representatives from MOET gained a better understanding  of key issues and best practices in assessment as a result of participating in several international learning events, such as the annual READ Global Conferences, the 2012 READ Regional Workshop in Kazakhstan, and the 2012 and 2013 IAEA Conferences. At each of these events, one representative from the five- to six-member Vietnamese delegation delivered a presentation on the development of Vietnam’s assessment system and progress achieved to date under READ. At the final READ Conference in St. Petersburg, all six members of the delegation participated in a panel session on the overall impact of the READ program in Vietnam. Immediately following these international events, Vietnam typically organized its own internal READ-Vietnam Conferences – each involving around 50 participants from universities, colleges, research and management institutes, local education departments, and different departments of MOET – to share lessons learned, discuss the overall work program, and share the findings of any recent activities. Study Visits: To assist Vietnam’s ongoing efforts to institutionalize  competency-based assessment practices into their new general education curriculum, two study visits were organized. In 2013, 16 MOET and university faculty leaders traveled to South Korea to meet with Curriculum Department staff and learn how assessment practices had been included in that country’s new curriculum. The entire South Korean curriculum was also translated into Vietnamese. In 2014, another delegation of 16 MOET and university faculty leaders traveled to the United States to learn about the newly-developed common core standards and related assessment activities. As a result of these study visits, Vietnamese policymakers and curriculum developers have a better understanding of international best practices in curriculum reform and are better equipped to incorporate lessons learned into their own ongoing reforms. 102 FINAL READ Trust Fund Report 2008–2015 Training Courses: With READ Trust Fund support, Vietnam focused a  significant amount of its efforts on developing an extensive capacity- building program in assessment to ensure that each stakeholder had the requisite knowledge to perform their expected role in assessment activities. According to a 2012 review of Vietnam’s existing education training programs, teachers and education officials received almost no systematic training in the area of assessment. To address this, a general competency framework for educational assessment was developed and a variety of training programs were created. In •  2012, a general competency framework was developed that outlines expected skills and knowledge in assessment for five key stakeholder groups (pre-service teachers, in-service teachers, Masters students, education managers, and MOET assessment specialists). In 2013, nine workshops, with a total of nearly 300 representatives from MOET, universities, and school leaders, were held to discuss and validate the overall framework and the proposed training modules. • Based on the skills and knowledge outlined in the general competency framework, five training modules were developed for the different stakeholder groups. Given the large number of individuals that needed to be trained in certain groups, a two-day workshop was organized with nearly 100 education officials to explore the possibility of eventually delivering one or more of these training modules through e-learning. In 2014, face-to-face versions of the training programs began to be piloted across all five stakeholder groups. By the end of the READ program, and including both the training-of-trainers professional development courses and sessions delivered to pre-service and in-service teachers, more than 2,800 participants had benefited from 26 different training workshops that were delivered by assessment experts, the General Department of Testing and Accreditation, and Hanoi National University of Education. COUNTRY-LEVEL RESULTS – VIETNAM 103 System Alignment The overarching framework that embraces all assessment reforms in Vietnam is the new competency-based general education curriculum under development by MOET. A taskforce assigned by MOET to prepare the new curriculum includes representatives from the National Institute of Educational Sciences and education experts. With the support of READ funds, an International Advisory Board, comprised of six leading international education experts, was established to help MOET 1) review its current curriculum and identify how good assessment practices could be integrated into the new curriculum; 2) review its current textbooks and benchmark Vietnam’s Mathematics and Language curricula to those of top-achieving countries; and 3) improve the capacity and skills of curriculum developers. In July and September 2014, a series of technical assistance workshops on curriculum and assessment were conducted for Vietnam’s education leaders, curriculum developers, and textbook writers. The International Advisory Board members reviewed different types of standards, including learning standards; how teachers should communicate assessment results and use this information to improve their teaching; and how assessments should be aligned with curriculum. These workshops provided excellent opportunities to discuss the substance of the ongoing education reforms in Vietnam, including the implications of competency-based education, problem-solving, self-directed learning, teamwork, etc. Classroom Assessment Clear Policies As a direct result of READ support, MOET now has formal, system-level documents that provide guidelines for classroom assessment at both the primary and secondary levels. These documents include guidelines for what should be assessed and the uses of assessment information. In addition to these guidelines, a variety of system-wide resources are available to teachers in the area of classroom assessment. For example, at the primary and secondary levels, documents that outline what students are expected to learn in different assessment domains at different grade levels and the performance levels that they are expected to reach at different grade levels are available to all teachers. In addition, student textbooks as well as teacher guides and test frameworks are available at the primary and secondary levels to support classroom assessment activities. Use of Results To assist in efforts to monitor education quality, MOET recently developed an Education Quality Management System for primary education. The database was designed to collect learning results and school-level data. 104 FINAL READ Trust Fund Report 2008–2015 Examinations Effective Human Resources/Quality Assurance In 2012, two workshops were organized with the intended purposes of increasing capacity to develop test items and improving the quality of university entrance examinations. As a result, nearly 200 representatives from high schools, universities, and MOET have increased their capacity to develop high-quality tests. From May 3–5, the Department of Testing and Accreditation hosted a  workshop for 140 participants from high schools and universities across the country. The purpose of the workshop was to 1) train participants in how to write and analyze test items, and 2) review the current assessment frameworks and validate proposed multiple-choice items. As a result, item writers are better equipped to write test questions that align with the intended purpose of the examination. The second workshop to discuss issues related to the reliability and  validity of the results of university and college entrance tests was held from May 25–27. Over 50 participants were in attendance, including representatives from various departments within MOET, universities, and high schools. The purpose of this workshop was 1) to enhance participants’ understanding of testing theories, use of assessment frameworks, item writing, and item analysis skills; and 2) to identify ways to improve the current process for developing university and college entrance tests. Quality Assurance The Department of Testing and Accreditation collected test items and results for the 2008-2014 high school graduation and university entrance examinations in Biology, Chemistry, Physics, and Foreign Language in order to conduct analysis of the data using IRT. The purpose was to help determine the reliability of the examination results and identify item-level characteristics such as difficulty, discrimination, and fit. The database of test items collected for this analysis will possibly be used in the future to create an item bank. In May 2014, a workshop was organized to share initial findings and concerns about the validity and reliability of the high school graduation and university entrance examination results. As a result of these discussions, a decision was made by MOET that these two examinations will be merged into one in the near future. COUNTRY-LEVEL RESULTS – VIETNAM 105 NLSA Effective Human Resources Drawing on the experience of the assessment expert on the International Advisory Board, a series of in-depth trainings in psychometrics were conducted to further train six technical experts in the areas of test design and data analysis. Additionally, training in test design was provided to around eight different groups of subject experts. This training was provided in order to improve the quality of Vietnam’s NLSA activities. Assessment Quality With READ support, a series of research activities was conducted to fill existing information gaps on learning outcomes in different areas of the education system and to increase the ability to use evidence in decision-making. Trends in Assessment: Following the self-diagnosis of Vietnam’s overall  assessment system in 2009, it was recommended that further in-depth analysis be conducted to determine the fiscal and human resources allocated for assessment activities, and to better understand the use of assessment data and the feedback loops among students, teachers, and education managers. Reports on the findings of these studies were completed in 2011 and used to identify potential constraints in the current system and ways to improve the use of assessment information moving forward. EDI Survey: With READ Trust Fund support, an EDI survey was  conducted in 2012 to measure the school readiness of preschool children in Vietnam. The EDI tool measures various developmental domains in children, including physical development and well-being, language and cognitive development, and socio-emotional development. To carry out the survey, ten early childhood education experts and education officials were trained in how to adapt the tool to Vietnam and how to conduct trainings for the 154 provincial and district officers who then trained preschool teachers to complete the questionnaires. After a pilot exercise, the instrument was rolled out in April 2012 to a population of 8,400 five-year-old children in 400 preschools in 54 of the 63 provinces in Vietnam. In 2013, a complete report of the findings in English and Vietnamese was prepared and disseminated during a launch event for the World Bank-financed School Readiness Promotion Project. Drawing on the experience and capacity built during the initial EDI survey, MOET used its own funds to implement another round of the survey in 2014. 106 FINAL READ Trust Fund Report 2008–2015 Bias in Curriculum and Teaching Materials: The Vietnam National  Institute of Education Sciences conducted a review of potential biases in the development of the new curriculum that might further marginalize vulnerable groups, such as ethnic minorities, females, and children with disabilities. The study reviewed international practices in this area as well as the school experiences of ethnic minority children in Vietnam. The findings were used to develop a bias-avoidance framework for curriculum developers and other stakeholders. ILSA Clear Policies With support from the READ Trust Fund, Vietnam registered to participate in its very first ILSA, PISA 2012. A team of Vietnamese officials participated in an introductory course in 2009, and then a second training in Hong Kong SAR, China in 2010. (Funding to cover all other costs of PISA implementation was provided by the Asian Development Bank.) A formal agreement to participate in PISA 2012 was signed by the Vietnamese government and, although a national policy on continued participation in ILSAs has yet to be created, preparations have already been made for Vietnam to participate in the next round of PISA in 2015. Strong Organizational Structures The Center for Education Quality Evaluation, the same unit responsible for NLSA activities, was designated by MOET as the unit responsible for managing Vietnam’s participation in ILSA activities. This unit has permanent staff with prior training in assessment and the skills necessary to carry out ILSA activities. Additionally, a National Steering Committee was created to provide the requisite leadership for Vietnam’s participation in PISA. Assessment Quality The Center for Education Quality Evaluation ensured that Vietnam’s participation in PISA 2012 was successful and that the country met all technical standards required to have its data published in the final report. Vietnam’s results were released in early December 2013. Use of Results Vietnam’s performance on PISA 2012 was literally astounding. Results showed that, although Vietnam is a lower-middle income country, its students are performing at a level similar to peers in Germany and Austria, and above the OECD average in all three subjects – Mathematics, Sciences, and Reading. With support from the READ Trust Fund, Vietnam conducted further analysis of its PISA results to identify areas in which curricular reforms would be necessary to ensure that students graduate from secondary school with 21st century skills. PISA 2012 results were disseminated in the country and the results have been used to monitor education quality, hold the government accountable, and inform education policy reforms. COUNTRY-LEVEL RESULTS – VIETNAM 107 CONCLUSION The READ Trust Fund played a critical role in strengthening Vietnam’s overall student assessment system. By the end of READ, the ground work had been laid for its new competency-based curriculum, a new competency-based assessment framework had been fully adopted, and training programs had been developed for all key stakeholders. In addition, Vietnam’s education leaders had access to a wealth of new information on learning outcomes that would allow them to make more evidence-based decisions about ongoing education reforms. As demonstrated by the PISA 2012 results, Vietnam’s education system is on track to becoming a strong system that produces high-quality education results. 108 FINAL READ Trust Fund Report 2008–2015 ZAMBIA Total Grant Amount: US$2,350,242 Total Disbursements: US$2,189,165 COUNTRY CONTEXT Over the past few decades, Zambia has undertaken numerous education reforms with an eye to attaining universal primary education and improving education quality. Enormous progress has been achieved in school access and Zambia has nearly met the goal of universal primary education. At the same time, serious concerns remain about education quality and learning outcomes. According to the 2012 NLSA, less than 40 percent of learners in grade 5 are acquiring even the most basic literacy and numeracy skills. Zambia’s Ministry of Education has been committed to monitoring the quality of learning and to using assessment results to make evidence-based reforms. The activities that comprised the READ Trust Fund program in Zambia were focused on strengthening the capacity of institutions responsible for measuring student learning outcomes and on ensuring that findings from assessment activities are used to improve teaching and learning. As a result of strong support from the Ministry of Education and READ funding, by the end of 2014, the country had strengthened and successfully expanded its NLSA program, improved the quality of its examinations, and been selected by the OECD to be one of six pilot countries for the PISA for Development initiative. Table 21. Developmental Ratings for Zambia’s Student Assessment System Classroom Examinations NLSA ILSA Assessment 2009 2014 2009 2014 2009 2014 2009 2014 Emerging Emerging Established Established Emerging Established Emerging Emerging COUNTRY-LEVEL RESULTS – ZAMBIA 109 PROGRAM OBJECTIVES, RATINGS, AND PROGRESS (2009-2014) Program Objectives and Developmental Ratings The objective of the READ Trust Fund program in Zambia was to help strengthen the capacity of the institutions responsible for measuring student learning outcomes and to ensure that findings from assessments are used to improve teaching and learning. At the beginning of the program, the baseline diagnostic showed Zambia’s assessment system to be established in the area of examinations and at an emerging level of development for the other three assessment types. With READ Trust Fund support, Zambia opted to target specific areas within each assessment type that would address key weaknesses and create a foundation for future development. These included quality reviews of current assessment activities, further capacity building of staff, and better use of assessment results to improve overall education quality. By the end of READ, as demonstrated in Table 21, Zambia had made significant progress in the area of NLSA. Progress was also seen in building teacher capacity to conduct classroom assessments, the further strengthening of the examinations system, and the laying of a foundation for ILSA activities. 110 FINAL READ Trust Fund Report 2008–2015 Program Implementation The READ Trust Fund program in Zambia was implemented in three phases with the support of two Bank-executed grants at the country-level as well as global-level funds to support its involvement in the PISA for Development initiative. A first grant for US$300,000 supported the diagnostic phase and the creation of an action plan; a second grant for US$2,113,500 supported capacity- building activities, in-depth studies of current assessment practices, efforts to effectively disseminate the results of the 2012 NLSA, and the development of assessment frameworks and examinations syllabi. In 2013, with the support of the READ Trust Fund, Zambia was selected as one of six pilot countries in the PISA for Development Initiative. During this third phase, nearly US$575,000 of global-level funds were used to cover a large share of Zambia’s participation costs, including the participation of Zambian representatives in international meetings. Other costs associated with Zambia’s participation are being covered by the government’s own budget and direct budget support provided by DFID. Progress by Assessment Type and Key Indicator Areas Zambia’s progress under the READ Trust Fund program has been carefully monitored using the READ Results Framework and SABER-Student Assessment benchmarking tools. Table 22 indicates the baseline and end-of- program scores for Zambia based on the results of benchmarking exercises carried out at the beginning and end of the program. The progress seen is based on achievements in the key indicator areas being tracked under the READ Trust Fund program. The activities that Zambia carried out under each key indicator area are described in the next section. Activities that benefited more than one type of assessment activity are described under a general category. Activities that benefited a specific type of assessment activity are described under assessment-specific headings. Table 22. Developing Zambia’s Assessment System: Progress Achieved from 2009 to 2014 Latent Emerging Established Advanced 1 2 3 4 Classroom Assessment Examinations NLSA ILSA 2009 2014 Progress from 2009-2014 COUNTRY-LEVEL RESULTS – ZAMBIA 111 General Strong Leadership High-level education officials, including the Minister, Permanent Secretary, and representatives on the National Assessment Steering Committee consistently provided strong support for developing Zambia’s assessment system and the activities implemented with READ Trust Fund support. In 2010, a workshop led by key education officials was held to discuss and validate the self-diagnosis findings and proposed action plan. In 2013, the Permanent Secretary played a key role in the film produced for national television on the 2012 NLSA results and the Minister led the event to launch the film. Additionally, these key education leaders and Heads of Department all participated in the media event organized to launch PISA for Development in Zambia. Strong Organizational Structures The institution responsible for the majority of assessment-related activities in Zambia is the Examinations Council of Zambia (ECZ). The baseline diagnostic found that the lack of an adequate IT system at ECZ hampered its ability to effectively store and retrieve data from previous assessments and to conduct further analyses. With the support of READ, an audit of ECZ’s IT systems was conducted in 2012 and actions identified that ECZ could take to improve its IT policies, practices, hardware, and software. Subsequently, due to the improvements made, ECZ was able to conduct secondary analysis of the data available in the areas of examinations and NLSA. Effective Human Resources The majority of READ funds in Zambia were spent on training ECZ staff, Education Standards Officers, other representatives from the Ministry of Education, and teachers to build their capacity to carry out assessment activities. Taking into account both general capacity-building activities (such as international conferences on assessment topics and national workshops) and also capacity-building activities targeted to specific assessment types (classroom assessment, examinations, and NLSA), more than 1,600 key education stakeholders now have increased capacity to understand and implement assessment activities. “The READ program helped us to redefine our horizon . . . we are able to think of better ways of doing things, improve the quality, disseminate information well, and move as a team. “ – Michael Chilala, Director of the Examinations Council of Zambia 112 FINAL READ Trust Fund Report 2008–2015 Conferences: READ supported the participation of more than 35 ECZ  and Ministry of Education staff in various international conferences on assessment topics. These events included the annual READ Global Conferences, AEAA Conferences in 2011 and 2013, IAEA Conferences in 2011 and 2012, and the READ Regional Workshop in 2013. Delegations of five to six government representatives participated in each of the READ Global Conferences. At each of these events, the Zambian delegation delivered a presentation on the development of Zambia’s assessment system and progress achieved to date under READ. At the final READ Global Conference in St. Petersburg, the entire delegation participated in a panel session on the overall impact of the READ program in Zambia. In an effort to connect with the broader assessment community,  representatives from ECZ also participated in a number of IAEA and AEAA Conferences. In 2014, Zambia served as hosts of the AEAA Conference and a panel session was organized to discuss the SABER- Student Assessment tools and the results of the READ program in Africa. Additionally, a representative from Zambia was invited to deliver a presentation at the World Bank’s high-level symposium, “Assessment for Global Learning,” which took place in Washington, DC. As a result of their participation in these events, education officials in Zambia have a greater awareness of best practices in student assessment and how to address common assessment challenges. COUNTRY-LEVEL RESULTS – ZAMBIA 113 Training for Education Standards Officers: According to a study  conducted in 2011, over 75% of Standards Officers in Zambia had never received training related to their job as school inspectors and had a very limited understanding of assessment practices. To build capacity in this area, in early 2013, ECZ organized several three-day workshops on the principles of assessment and school inspection to train all 360 Standards Officers in the country. ECZ intends to offer similar workshops every two years. Classroom Assessment Effective Human Resources At the beginning of the READ Trust Fund, a study was conducted on existing teacher capacity in assessment. This study clearly identified a need for teachers to receive more systematic training in this area. In 2013, through a series of consultations with representatives from the Teacher Education Unit and Teacher Training Institutions, a decision was reached to develop and incorporate a course on assessment into the existing curriculum at all teacher training institutions. The course outline has been agreed upon and work is underway to further develop the course materials. The course is designed to ensure that pre-service teachers learn the requisite skills and are capable of using assessment results to support teaching and learning in their classrooms. System Alignment In 2014, the Ministry of Education developed a teacher’s guide on school- based assessments in reading for grades 1-4. In addition, another teacher’s guide, which outlines what students are expected to learn and the level of performance they are expected to reach in reading at each grade level was developed. These guidelines are meant to help teachers carry out classroom assessments and ensure students are learning the literacy skills outlined in the national curriculum. Examinations Effective Human Resources In the area of examinations, capacity-building activities primarily focused on strengthening ECZ’s current assessment practices. In order to ensure quality and meet the growing demand for additional examination setters and markers, ECZ delivered a series of workshops to train more than 1,200 teachers and other Ministry of Education staff. Additionally, a study visit to Barbados was organized to introduce ECZ staff to some innovative ideas for using ICT that could be used to improve their existing system. 114 FINAL READ Trust Fund Report 2008–2015 Training for Examination Setters and Markers: Over the past several  years, Zambia’s education system has expanded dramatically. Due to an increasing number of students enrolled in the system and registering to take the examinations in grades 7, 9, and 12, there was an immediate need for ECZ to train many additional question setters and markers. With the support of the READ Trust Fund, in 2011, a series of trainings was conducted. A total of 161 question setters received training in test design, construction, and item writing. Then, another training session on how to mark examinations and detect and report cheating was delivered to over 900 markers. After the completion of the examinations cycle in early 2012, it was evident that the quality of the examinations was much improved as a result of having many more trained staff. Based on the success of the first series of trainings, an additional training took place in 2012 to train more than 100 question setters for Zambian languages. Study Visit: To learn from the Caribbean Examinations Board, a team of  ECZ staff traveled to Barbados in 2011. During this study visit, the team gained a greater understanding of how to use ICT in assessment, create item banks, improve candidate registration procedures, and use social media to distribute study materials and mitigate examinations malpractice. As a result, ECZ now has a very active Facebook page, communicates regularly with the public, and makes materials available to students online via its website. COUNTRY-LEVEL RESULTS – ZAMBIA 115 System Alignment Zambia’s inherited its examinations system from the United Kingdom. In 2014, in order to improve the alignment of their examinations with Zambia’s current curriculum and national learning goals, 20 ECZ and Ministry of Education staff received a week-long training in how to develop an assessment framework and syllabi for key subject areas – English, Mathematics, and Science. This training enabled ECZ staff to more closely examine what their examinations are intended to measure and to identify possible improvements to the current system. The new assessment framework and syllabi for the key subject areas are now available on ECZ’s website for students to use to prepare for their examinations. NLSA Effective Human Resources Drawing on the results of the in-depth technical review of the NLSA program, a number of capacity-building activities were organized to enhance ECZ’s ability to conduct NLSA exercises. These targeted trainings allowed ECZ to 1) improve the quality of its ongoing NLSA program which measures learning outcomes in Literacy, Mathematics, Life Skills, and Local Languages for grade 5; and 2) develop a new NLSA for grade 9 in Mathematics, Life Skills, Local Languages, and Science. University Course: In 2011, two ECZ staff responsible for carrying out  NLSA activities attended a statistics course at the University of Michigan. This course covered the topics of sampling, basic and advanced statistical analysis, and report writing. Prior to the course, ECZ lacked the capacity to conduct its own sampling. As a result of the training, ECZ is less reliant on other government departments and can carry out their work more efficiently. Trainings: A series of trainings were designed to fill knowledge gaps  among ECZ staff responsible for NLSA activities. With the support of an international assessment expert, 30 ECZ staff received training in key areas, such as how to use SPSS and IATA software packages to perform data analysis, Item Response Theory, and report writing. ECZ’s increased capacity in these areas greatly enhanced their analyses of the 2012 NLSA grade 5 results and the design of the new NLSA for grade 9. “We used to share the [NLSA] findings in one thick report which was not effective. But under READ, we produced a video that was played on national television . . . and we also produced brochures with specific messages to [key] stakeholders.” – Charity Kombe, Examinations Council of Zambia 116 FINAL READ Trust Fund Report 2008–2015 Assessment Quality To improve the quality of the NLSA in grade 5, an in-depth technical review was conducted by an international assessment expert in 2012. This review helped identify possible areas for improvement to the NLSA and in which ECZ staff would benefit from additional training. It also highlighted improvements that could be incorporated into the design of the new NLSA in grade 9. The review was completed in early 2013 and followed by a series of training workshops for ECZ staff in the area of NLSA. Use of Results Although ECZ had been conducting regular NLSA activities, it struggled to disseminate the results in ways that optimized their use to inform policy and teaching practices. Prior to 2013, ECZ primarily relied on the issuance of one large technical report to communicate NLSA results. With the support of the READ Trust Fund, ECZ and key education stakeholders came together to devise a broad-reaching communications plan for disseminating the results of the 2012 NLSA in ways that would raise awareness about learning outcomes across the country and motivate people to take action. Each activity in the communications plan was designed to reach specific audiences. These communication materials included: Film: A 10-minute film about the NLSA results, entitled “Are Zambia’s  Children Learning?,” was created for parents and other concerned citizens and aired on broadcast television. Brochure: In addition to the more traditional technical report, 15,000  copies of an easy-to-read brochure targeting teachers, school administrators, and policymakers were printed. This brochure highlighted the main NLSA findings and recommendations for action by key stakeholders. Posters: For schools and district central offices, 7,000 posters  highlighting the main messages of the NLSA results for teachers and school administrators were created. DVD and Teacher Discussion Guide: A DVD, including a teacher  discussion guide and case studies on effective teaching strategies for fractions and phonics (two of the weakest areas, according to the NLSA results), was developed for use in teacher in-service trainings. In September 2013, an event was organized to launch these materials and engage in a national dialogue on the actions needed to strengthen Zambia’s education results. This event was led by the Minister and included nearly 100 representatives from all education departments, international donors, and several media outlets. As a result, greater attention is being paid by all key education stakeholders to the introduction of reforms that will improve the quality of education in Zambia. COUNTRY-LEVEL RESULTS – ZAMBIA 117 The Teacher Education Unit worked closely with ECZ on identifying the teachers and key messages to highlight in the case study videos. Following the official launch event, all of the Teacher Resource Centre Coordinators and Provincial Officers came together to participate in a training on how to use the DVD and accompanying discussion guide for in-service teacher training. All stakeholders agreed that accompanying the NLSA results with a list of concrete actions that teachers can incorporate into their teaching practices was a good strategy and should be continued in the future. ILSA Effective Human Resources In 2013, through the support of the READ Trust Fund, Zambia was selected to be one of six pilot countries to participate in the PISA for Development initiative. In order to draw upon and further strengthen the organizational and human resource capacity developed under READ, it was decided that this initiative would be led by ECZ. The PISA for Development initiative is being designed to include a strong capacity-building component and several peer-to-peer learning activities. With READ support, one to two ECZ representatives have been able to participate in all three of the international workshops organized to discuss the development of the PISA for Development tools with the other pilot countries, donor groups, and the OECD. 118 FINAL READ Trust Fund Report 2008–2015 Opportunities to Learn about Assessment Activities In February 2014, with READ Trust Fund support, an event was organized to officially launch Zambia’s participation in PISA for Development. This event was led by the Permanent Secretary for Education and included nearly 100 representatives from various departments within the Ministry of Education, ECZ staff, and donor partners. OECD representatives were in attendance to deliver presentations on the overall goals of the program, the expected benefits of country participation, and examples of lesson learned from other countries that have participated in PISA. This event successfully engaged a range of key stakeholders and began a national dialogue on how the PISA results might be used to informed education policies in Zambia. CONCLUSION The objective of the READ Trust Fund program in Zambia was to help strengthen the capacity of its institutions responsible for measuring student learning outcomes and to ensure that findings from assessments are used to improve teaching and learning. As a direct result of the work accomplished under READ, Zambia has strengthened the capacity of more than 1,600 ECZ and Ministry of Education staff to carry out assessment activities. The national campaign to disseminate the 2012 NLSA results greatly improved the use of assessment results to inform education practices and more attention is being given to improving the quality of education in Zambia. APPENDIX 119 APPENDIX 1. GLOSSARY OF ASSESSMENT TERMS Classroom A variety of tools and procedures employed by teachers and students in the course Assessment: of daily classroom activity to collect and interpret written, oral, and other forms of evidence on student learning or achievement (for example, oral questioning and feedback, homework assignments, student presentations, diagnostic tests, and end-of-unit quizzes). Provides “real time” information to support the teaching- learning process. Educational A group of interconnected policies, structures, and practices for the generation Assessment System: and use of information on student learning or achievement. Formative Ongoing assessments, reviews, and observations in a classroom as part of the Assessment: instructional process. Used to modify and validate instruction. National/ Also referred to as “large-scale assessments” or “large-scale surveys.” Designed International to provide data on the achievements of students in a curriculum area, aggregated Assessments: to provide an estimate of achievement levels in the education system as a whole at a particular age or grade level. Involves the administration of achievement tests either to a sample or a population of students. Teachers and others may be asked to provide background information which, when related to student achievement, can provide insights on how achievement is related to factors such as household characteristics and levels of teacher training. “Public” or Tests that measure individual student achievement in relation to an official “External” curriculum or other formal body of knowledge and skills. Results are used, alone Examinations: or in combination with other information, to make highly consequential decisions about individual examinees, such as promotion to the next grade or admission to higher education. Also referred to as “high-stakes” tests. Standards: Content standards are broadly stated expectations of what students should know or be able to do in particular subjects and grade levels. Performance standards are explicit descriptions of what students must know or be able do in order to demonstrate a particular level of proficiency on the content standards. Student The process of gathering and appraising information on what students know, Assessment: understand, and can do in order to make an informed decision about what to do next in the educational process. Also referred to as “learning assessment,” “educational assessment,” or simply “assessment.” Summative Typically used to evaluate the effectiveness of instructional programs and services Assessment: at the end of an academic year or other predetermined time, or to make a judgment of student competency after an instructional phase is complete. 120 FINAL READ Trust Fund Report 2008–2015 APPENDIX 2. SABER–STUDENT ASSESSMENT LEARNING EVENTS AND TRAINING SESSIONS DATES EVENT LOCATION 2009 September First READ Global Conference Moscow, Russia 30–October 2 2010 April 30–May 3 American Educational Research Association Conference Colorado, USA June 21–23 Benchmarking Education Systems: East Asia Regional Singapore Conference September 9–10 OECD Group of National Experts on Evaluation and Assessment Paris, France Meeting September Second READ Global Conference Nairobi, Kenya 30–October 2 November 29–30 International Conference on Educational Monitoring and Beijing, China Evaluation 2011 January 27 U.S. Agency for International Development Education Staff Washington, DC, USA Training March 8 World Bank Education Staff Learning Week Washington, DC, USA March 16 World Bank Institute’s Strategic Choices for Education Reform Washington, DC, USA Course March 16–18 U.K. Department for International Development Education Staff Scotland, UK Training May 1–5 55th Comparative and International Education Society Symposium Montreal, Canada APPENDIX 121 DATES EVENT LOCATION June 6–8 East Asia Regional Conference on SABER Bali, Indonesia July 18 World Bank Institute’s Strategic Choices for Education Reform Washington, DC, USA Course September 30 UNESCO Institute for Statistics Meeting Montreal, Canada October 23–26 Third READ Global Conference and Sixth ECA Education Eschborn, Germany Conference 2012 March 5–7 Workshops on SABER-Student Assessment Tools Moscow, Russia April 22–27 56th Comparative and International Education Society Symposium San Juan, Puerto Rico May 7–8 World Bank Education Staff Development Program Pilot Washington, DC, USA July 11–12 World Bank Education Staff Development Program Pilot II Brisbane, Australia September 17–21 International Association for Educational Assessment Conference Astana, Kazakhstan September 19 READ Trust Fund Regional Workshop – ECA Astana, Kazakhstan December 3–5 READ Trust Fund Regional Workshop – Africa Maputo, Mozambique 2013 February 6–7 World Bank Education Staff Learning Week Washington, DC, USA February 11–15 U.K. Department for International Development Education Staff Scotland, UK Training May 14–17 Workshops on SABER-Student Assessment Tools Moscow, Russia August 12–16 31st Association for Educational Assessment in Africa Conference Arusha, Tanzania October 21–25 39th International Association for Educational Assessment Tel Aviv, Israel Conference November 7–8 World Bank Symposium: Assessment for Global Learning Washington, DC, USA December 11–13 SABER-Student Assessment Training and Adaptation Workshop Nadi, Fiji 122 FINAL READ Trust Fund Report 2008–2015 DATES EVENT LOCATION 2014 March 11–12 58th Comparative and International Education Society Symposium Toronto, Canada April 5–6 National Council on Measurement in Education Annual Pennsylvania, USA Conference May 11–16 Fourth READ Global Conference St. Petersburg, Russia August 8–14 32nd Association for Educational Assessment in Africa Conference Livingstone, Zambia November 16–18 WB-IMF Regional Workshop on Student Assessment Kuwait City, Kuwait November 19–20 Arab League Educational, Cultural and Scientific Organization/ Kuwait City, Kuwait UNESCO-Beirut Regional Policy Seminar on Student Assessment 2015 March 8–13 59th Comparative and International Education Society Symposium Washington, DC, USA APPENDIX 123 APPENDIX 3. Summaries of READ Trust Fund-Sponsored Events FIRST READ GLOBAL CONFERENCE – 2009 The First READ Global Conference took place September 30-October 2, 2009 in Moscow, Russia. This event brought together nearly 130 participants comprised of delegations and World Bank education task teams from all seven READ member countries (Angola, Ethiopia, the Kyrgyz Republic, Mozambique, Tajikistan, Vietnam, and Zambia); education assessment experts from the international community, including Russia; and other experts from across Russia. Objectives The three-day agenda was designed to: Highlight the fundamental message that access to and even completion of primary school are not enough, it is learning that is important Share the experiences of other countries Hear from READ countries as to their aims, plans, and challenges on the road to higher learning outcomes Draw attention to Russia’s role as an important emerging donor in education 124 FINAL READ Trust Fund Report 2008–2015 Key Messages The theme for the conference was “Developing a Vision for Assessment Systems”. The key messages were: When children learn, nations prosper – countries cannot afford to  postpone national priority for learning Evidence – obtained through regular assessments of student learning  – and use of that evidence, is vital Developing a system of assessment is important, i.e., creating an  enabling environment for assessment, ensuring alignment between assessment and other components of the education system, and paying attention to the technical quality of the instruments used to measure student learning Structure READ Country delegates were the primary audience for the conference. Country delegations, World Bank education task teams, and international assessment experts were seated at roundtables to encourage active participation and interaction. The materials were presented in English, Russian, and Portuguese and slides were projected in English and Russian. Each presentation was followed by ample time for discussion. The first two days concluded with table exercises aimed at engaging country delegates in identifying key takeaway messages for their countries. On the final day, each country delegation delivered a presentation on their country’s goals for assessments, followed by a panel discussion. Outcome The conference marked an important milestone for READ, as it was the first occasion for the seven READ countries to interact with each other, the donor, and international experts. The main objectives for the conference were achieved. It consolidated READ countries’ commitment to the assessment agenda, spurred their desire for more knowledge and interaction with each other and with international experts, created visibility for Russia as an important donor in education, developed a strong reputation for the READ program, and positioned the World Bank and Russia in a strong leadership role in the area of assessment. According to feedback from the READ countries and the Russian government, the conference was highly successful and made significant strides in moving forward the READ agenda. All conference materials are available on the READ Trust Fund website (www. worldbank.org/readtf). APPENDIX 125 READ ANNUAL REPORT 2010 n 33 SECOND GLOBAL SECOND READ CONFERENCE READ GLOBAL CONFERENCE SUMMARY – 2010 The Second READ Global Conference took place October 4–6, 2010 in Nairobi, Kenya. This event brought together nearly 130 participants comprising high-level education officials from the seven READ country governments (Angola, Ethiopia, the Kyrgyz The Second READ Global Conference took place October 4–6, 2010 in Republic, Mozambique, Tajikistan, Vietnam, and Zambia), the Kenyan assessment community, Russian officials and education experts, World Bank staff, and international assessment experts. Nairobi, Kenya. This event brought together nearly 130 participants comprising oBjECtivEs high-level The was designedofficials education three-day agenda to: from the seven READ country governments n Support READ countries in improving learning outcomes through strong, sustainable assessment systems. (Angola, n Provide READ countries with an the Ethiopia, Kyrgyz opportunity Republic, to collectively, and with supportMozambique, Tajikistan, from assessment experts, think through ways in which to improve the enabling environment for assessment systems. Vietnam, n Share lessons and Zambia from other ), the inKenyan countries’ experiences strengthening assessment community, Russian their enabling environments. n Offer a venue for those engaged in assessment work to interact, network, and consult with each other. officials n andways Communicate education experts, the READ Program World is working to helpBank countriesstaff, developand international their assessment systems.assessment experts. kEy mEssAgEs The theme for the conference was “Strengthening the Enabling Environment for Assessment Systems”. Key messages included: n Leadership plays a vital role in building effective assessment systems. Objectives n Establishing a strong assessment system requires focusing on the policies, institutional arrangements, and financial and human resources in which assessment activities take place. n Strengthening the enabling environment is fundamental to ensure that learning outcomes improve and that money spent on The three-day assessment agenda produces results. was designed to: stRuCtuRE Delegations of senior education officials and assessment experts from the seven READ countries were the primary target  Support READ countries in improving learning outcomes through strong, audience for the conference. READ country delegations sat together at roundtables with World Bank education task teams and international assessment experts to encourage interaction among participants. The conference aimed to provide ample time for sustainable assessment systems. countries to present on their progress under READ and discuss challenges in building their own assessment systems. To facilitate in-depth discussions on challenges in each country, on the second day, World Bank staff and international experts  Provide READ countries with an opportunity to collectively, and with conducted individual country clinics with each delegation. Upon completion, delegates reported back on what future actions they would take to strengthen their assessment systems moving forward. On the last day, assessment specialists presented support from assessment experts, think through ways in which to improve lessons learned from several commissioned case studies on other countries’ experience in strengthening their enabling environment and a couple of guidance notes developed to motivate countries and give them a few simple ideas to take back to the enabling environment for assessment systems. their countries.  outComEShare lessons from other countries’ experiences in strengthening their The annual conference marks an important milestone for READ, as it is the only occasion when all seven READ countries enabling interact environments. with each other, the donor, and international experts. The conference successfully brought together a large number of stakeholders working on the READ agenda and participants capitalized on the opportunity to network and learn from each other.  Offer a venue for It broadened READ countries’ those engaged understanding in assessment elements required to work of the fundamental build sound to assessment interact, deepened their resolve to find concrete ways to improve their own enabling environments for assessments, and made network, systems, and consult with each other. connections as to how tools being developed at the global level can guide them in their efforts. According to feedback READ countries and World Bank country teams, the conference was highly valued by participants and successful in motivating from  READ Communicate ways the READ Program is helping countries to develop countries to continue to advance their READ Action Plans. their assessment systems. Key Messages The theme for the conference was “Strengthening the Enabling Environment for Assessment Systems”. Key messages included: Leadership plays a vital role in building effective assessment systems. Establishing a strong assessment system requires focusing on the policies, institutional arrangements, and financial and human resources for assessment activities. Strengthening the enabling environment is fundamental to ensure that learning outcomes improve and that money spent on assessment produces results. 126 FINAL READ Trust Fund Report 2008–2015 Structure Delegations of senior education officials and assessment experts from the seven READ countries were the primary target audience for the conference. READ country delegations sat together at roundtables with World Bank education task teams and international assessment experts to encourage interaction among participants. The conference aimed to provide ample time for countries to present on their progress under READ and discuss the challenges in building their assessment systems. To facilitate in-depth discussions on the challenges in each country, on the second day, World Bank staff and international experts conducted individual country clinics with each delegation. Upon completion, delegates reported back on the actions they would take to strengthen their assessment systems moving forward. On the last day, assessment specialists presented lessons learned from commissioned case studies on other countries’ experiences in strengthening their enabling environments and provided practical guidelines to motivate countries and give them ideas to implement on their return home. Outcome The conference successfully brought together a large number of stakeholders working on the READ agenda, providing them with valuable opportunity to network and learn from each other. It broadened READ countries’ understanding of the fundamental elements required to build sound assessment systems, deepened their resolve to find concrete ways to improve their own enabling environments for assessment, and allowed them to better understand how tools being developed at the global level can guide them in their efforts. All conference materials are available on the READ Trust Fund website (www.worldbank.org/readtf). APPENDIX 127 THIRD READ GLOBAL CONFERENCE – 2011 The World Bank held a joint Third READ Global Conference and Sixth World Bank ECA Education Conference in Eschborn, Germany on October 23–26, 2011. The event was hosted at the headquarters of the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) and was sponsored by READ, the World Bank’s education division for the Europe and Central Asia Region (ECA), and the German Federal Ministry for Economic Cooperation and Development (BMZ). This event brought together over 160 participants comprising delegations of high-level education officials from 21 countries (AFR – Angola, Ethiopia, Mozambique, and Zambia; EAS – Vietnam; and ECA – Albania, Armenia, Bulgaria, Croatia, Estonia, Georgia, Kosovo, the Kyrgyz Republic, Macedonia, Moldova, Montenegro, Poland, Romania, Serbia, Slovenia, and Tajikistan), Russian officials and education experts, World Bank staff, GIZ staff, and international assessment experts. All in all, there were more than 38 countries represented at this international event. Objectives The key objective of the conference was to help READ-supported and ECA countries to strengthen their ability to use assessment results in meaningful ways that ultimately leads to improvements in education quality. The conference helped participants develop a better understanding of how to: use assessment results to improve education policy and practice at national/sub-national and school levels; support efforts to help teachers and other educators understand how to interpret assessment data; and involve students, parents, and public stakeholders in monitoring learning outcomes in their schools and communities. 128 FINAL READ Trust Fund Report 2008–2015 Additionally, the conference provided participants with an opportunity to learn lessons from other countries’ experiences with the use of student assessment results. Site visits with local stakeholders that use student assessment results highlighted the practical, concrete ways that Germany is using assessment data to improve its student learning outcomes. Key Messages Under the theme, “Using Student Assessment Results for Education Quality and Systems Strengthening,” the 2011 conference focused on ways to use data from various assessment activities (international and national large-scale assessments, examinations, and classroom assessments) to strengthen education systems and improve learning outcomes. Assessment activities are only effective in improving education quality when they are used in ways that inform policymakers’ decisions, help teachers improve their teaching practices, and empower parents with information on how well their children are learning. Key overall messages that emerged during the conference included the following: Classroom assessment has a high potential for increasing achievement and reducing learning disparities but is underdeveloped, poorly understood, and inadequately utilized in most education systems. Different kinds of assessments provide critical information for different purposes, including supporting pedagogy, monitoring learning levels, benchmarking school or district performance, certifying competencies, ensuring accountability, and strengthening the education system, but few countries examine, use, and disseminate the results adequately. One of the biggest challenges countries face is weak capacity and expertise to analyze assessment results, communicate results, and use results to drive improved and equitable learning. Structure As a combined event, the conference was structured to meet different donor needs. The first half of the conference was intended for a broad audience, open to all conference participants. These plenary sessions covered universal themes such as using assessment results to improve overall education quality; reporting large-scale assessment results in ways to ensure equity; good practices and lessons learned from a series of case studies on uses of assessment results; and techniques to effectively use classroom assessment results. On the afternoon of day two, the conference divided into READ-only and ECA-only sessions. Participation in the READ-only sessions was limited to those directly associated with the READ program. These sessions were intended to provide an opportunity for Russian representatives, READ country government delegations, and World Bank staff to report back on the progress achieved thus far and discuss any challenges impeding their progress. After a series of site visits on day three, READ participants were brought back to participate in country clinics in which they examined their country’s benchmarking report and reflected on ways they could improve the use of assessment results. APPENDIX 129 Outcome This year’s conference provided a unique opportunity to expand the reach of the previous two conferences to include a much broader audience and expose many new countries to the important work being carried out under the READ program. The conference successfully brought together a large number of assessment experts and high-level education leaders and focused their attention on the critical importance of improving education quality. READ countries reported their early successes and left the conference motivated to press forward and ramp up their efforts. According to evaluation forms from conference participants, the conference was deemed a success by all who participated. All conference materials are available on the READ Trust Fund website (www.worldbank.org/readtf). 130 FINAL READ Trust Fund Report 2008–2015 READ REGIONAL WORKSHOPS – 2012 READ Regional Workshop for ECA The READ Regional Workshop for ECA was a one-day side event during the IAEA Conference. The READ Trust Fund global team invited assessment practitioners and others implementing READ-supported activities in four of the eight READ countries to participate in the workshop. The 34 participants at the September 19 workshop included delegations from Armenia, the Kyrgyz Republic, Tajikistan, and Vietnam, in addition to World Bank staff and representatives from CICED. The primary objective of the workshop was to create an interactive environment that allowed the country teams to engage in South-South knowledge sharing. The objectives of the workshop were to i) support the four participating READ country teams in moving forward with the activities outlined in their READ action plans; ii) provide participating READ country teams with the opportunity to discuss and resolve key challenges in relation to those plans; and iii) provide an opportunity for the sharing of lessons learned and expertise across the four READ country teams. To encourage active participation, each team came prepared to discuss their experiences in a variety of assessment areas. Topics discussed included: Large-Scale Assessment, with a dual focus on (i) successful participation in international large-scale assessment exercises and (ii) reporting and using assessment information from large-scale assessments Classroom Assessment, with a focus on teacher training models Examinations, with a focus on ensuring quality and combating malpractice Higher-Order Thinking Skills, with a focus on approaches to assessing these skills The country teams were eager to discuss and share experiences with each other. For example, based on their own prior experiences, Armenia and the Kyrgyz Republic were able to provide valuable advice to the Vietnamese team, who were participating in their very first PISA exercise. It was encouraging to see the countries build off each other’s experiences and share lessons learned. All presentations and workshop materials have been posted on the READ Trust Fund website (www.worldbank.org/readtf). APPENDIX 131 READ Regional Workshop for Africa The READ Trust Fund hosted a three-day READ Regional Workshop for African countries in Maputo, Mozambique from December 3–5. The workshop focused on supporting four country teams – Angola, Ethiopia, Mozambique, and Zambia – in moving forward with activities being funded under the READ Trust Fund program, providing them with the opportunity to discuss and resolve key challenges/issues in relation to those plans as well as share lessons learned and expertise. The 37 participants included country teams of between 5 to 10 assessment practitioners implementing READ activities, World Bank staff, and representatives from the Russian embassy. The workshop was designed for country delegates to take on a very active role, driven by their specific assessment interests and needs as they pertained to READ. The workshop sessions included the following focal areas: Examinations, with a focus on ensuring quality and combating malpractice  National Large-Scale Assessment,  with a focus on disseminating and using assessment information National Large-Scale Assessment, with a focus on how to train staff to improve assessment design and implementation All assessment types, with a focus on planning and monitoring. To encourage knowledge sharing across countries, each delegation was asked to select one of the topics to be discussed during the workshop and prepare a presentation, which highlighted their experiences, lessons learned, and challenges in this area. Workshop participants were also given the chance to visit schools in Maputo to discuss Provinha – a written, early-grade reading assessment that was designed as part of the program of READ Trust Fund–supported activities in Mozambique. The workshop proved to be a valuable opportunity for READ-supported countries to present their work, solicit feedback, and share lessons. All materials from the workshop are available on the READ Trust Fund website (www.worldbank.org/readtf). 132 FINAL READ Trust Fund Report 2008–2015 AEAA AND IAEA CONFERENCES – 2013 In lieu of a stand-alone READ Global Conference in 2013, the READ Trust Fund combined efforts with two large assessment associations – the AEAA and IAEA. Each of these associations organizes an annual conference to bring together assessment experts from across the globe to discuss key issues and the latest developments in the area of measuring student learning. The READ Trust Fund sponsored the participation of the four READ countries from Africa (Angola, Ethiopia, Mozambique, and Zambia) at the AEAA Conference in Tanzania and the other four countries (Armenia, the Kyrgyz Republic, Tajikistan, and Vietnam) at the IAEA Conference in Israel. Sharing READ Trust Fund Results At both of these events, panel sessions were organized to present the work being carried out under the READ Trust Fund program. These sessions began with some introductory remarks on the key role that assessment plays in the “Learning for All” agenda and how the SABER-Student Assessment tools can help countries further develop their systems to monitor learning outcomes. Then, a representative from each READ-sponsored country shared specific examples of activities that have led to measurable improvements in their assessment systems. Through these sessions, the program was able to communicate and link READ’s key messages and results to the bigger community of international assessment experts. Disseminating Knowledge Products on Student Assessment These two events drew many of the world’s leading experts in assessment and provided a tremendous opportunity to disseminate and share the products developed under the READ Trust Fund. Expo stands with more than 20 different READ-supported products on student assessment were organized for each event, and more than 1,000 copies of various READ publications were distributed. Recognizing Russia’s Contributions in the Area of Education Quality At the AEAA Conference, there were over 400 participants from more than 22 African countries. At the IAEA Conference, there were nearly 370 participants from 45 countries across the globe. By partnering with assessment organizations like AEAA and IAEA and sharing the innovative work being accomplished under the READ Trust Fund, Russia has become widely recognized as a key player in the area of student assessment. APPENDIX 133 FOURTH READ GLOBAL CONFERENCE – 2014 Overview The Fourth Russia Education Aid for Development (READ) Global Conference took place from May 12-16, 2014 in St. Petersburg, Russia and focused on the theme of “Measuring for Success: The role of assessment in achieving learning goals.” The conference provided an opportunity to explore a variety of learning goals – for further study, work, and life – and the role that assessment plays in monitoring and achieving them. The conference also allowed the countries participating in READ to discuss their achievements under the program and how these serve as a foundation for attaining learning goals and ensuring success for their children. Each country team led a session at the conference to highlight their successes under READ. Teams also showcased their results at a READ exposition event. In total, 122 participants from 19 countries attended the 2014 conference, including representatives from the eight READ countries (Angola, Armenia, Ethiopia, the Kyrgyz Republic, Mozambique, Tajikistan, Vietnam, and Zambia); global experts in the areas of assessment and learning; World Bank and other international agency staff; and government officials. In the opening session, the Ministry of Finance of the Russian Federation, the World Bank, and the country teams recognized the efforts of the many people who conceived, launched, and implemented the READ program and its activities. Since its inception, the program has provided an important link between assessment theory and practice, and helped inform policy dialogue on education quality and learning. 134 FINAL READ Trust Fund Report 2008–2015 Key Messages During the conference, international experts and country representatives shared their experiences in “measuring for success” at the classroom, national, and global levels. There were three main takeaways from this discussion: Measuring for Success: Resources should be allocated to initiatives that make the most impact on learning. Measurement helps identify which initiatives are successful at achieving impact. Spending choices should be guided by the relative impact of initiatives on learning as opposed to just the volume of spending involved. Role of Assessment: Assessment plays an important role in achieving learning goals. Classroom assessment by teachers is key in this regard. For teachers to effectively use classroom assessment as a tool for improving student learning, they need comprehensive pre- and in-service training on this topic that includes a focus on learning goals and how to collect evidence of student learning Achieving Learning Goals: Large-scale assessments are effective tools for measuring progress toward learning goals at the system level. High-quality national and international assessments allow countries to identify gaps in performance or the acquisition of key skills so that policy makers can target resources where most needed. Outcome The conference concluded with the Russian Federation, World Bank, and READ country delegates reiterating their appreciation for the opportunity to come together to learn and share. The Russian Federation and the World Bank emphasized their commitment to supporting countries in building their capacity to measure learning; the country delegates expressed their gratitude for this support, and also praised the conference program, speakers, and discussions, which they found to be very informative and relevant. APPENDIX 135 APPENDIX 4. Knowledge Products on Student Assessment Supported by the READ Trust Fund READ Working Paper Series 136 FINAL READ Trust Fund Report 2008–2015 READ Working Papers available in Russian APPENDIX 137 SABER–Student Assessment Country Reports 138 FINAL READ Trust Fund Report 2008–2015 APPENDIX 139 National Assessments of Educational Achievement Series (Also available in French, Portuguese, and Russian) Russia Education Aid for Development Trust Fund Education, Human Development Network World Bank 1818 H Street, NW Washington, DC 20433 USA Website: www.worldbank.org/readtf