Document of The World Bank FOR OFFICIAL USE ONLY Report No: PAD1645 INTERNATIONAL BANK FOR RECONSTRUCTION AND DEVELOPMENT PROJECT APPRAISAL DOCUMENT ON A PROPOSED LOAN IN THE AMOUNT OF US$67 MILLION TO THE REPUBLIC OF KAZAKHSTAN FOR AN EDUCATION MODERNIZATION PROJECT FEBRUARY 6, 2017 Education Global Practice EUROPE AND CENTRAL ASIA REGION This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective December 1, 2016) Currency Unit = Kazakhstan tenge KZT 360 = US$1 FISCAL YEAR January 1 – December 31 ABBREVIATIONS CEP Center for Education Programs OECD Organisation for Economic Co- CoE Center of Excellence operation and Development CQS Selection Based on Consultants’ PDO Project Development Objective Qualifications PISA Programme for International DC Direct Contract Student Assessment EALA External Assessment of PIU Project Implementation Unit Learning Achievements POM Project Operations Manual EMIS Educational Management PIRLS Progress in International Information System Reading Literacy Study FM Financial Management SIP School Improvement Plan FMM Financial Management Manual SPESD State Program of Education and FRM Feedback and Resolution Science Development Mechanism SSS Single-Source Selection GDP Gross Domestic Product TA Technical Assistance GoK Government of Kazakhstan TIMMS Trends in International GRS Grievance Redress Service Mathematics and Science Study ICB International Competitive UNDB United Nations Development Bidding Business IFR Interim Financial Report UNT Unified National Test MOES Ministry of Education and Science NAE National Academy of Education NCB National Competitive Bidding NIS Nazarbayev Intellectual Schools NTC National Testing Center Regional Vice President: Cyril E. Muller Country Director: Lilia Burunciuc Acting Senior Global Practice Director: Amit Dar Practice Manager: Mario Cristian Aedo Inostroza Task Team Leader: Dingyong Hou Republic of Kazakhstan Education Modernization Project TABLE OF CONTENTS Page I. STRATEGIC CONTEXT ....................................................................................................1 Country Context ............................................................................................................ 1 Sectoral and Institutional Context ................................................................................. 1 Higher-Level Objectives to which the Project Contributes .......................................... 5 II. PROJECT DEVELOPMENT OBJECTIVES......................................................................5 PDO............................................................................................................................... 5 Project Beneficiaries ..................................................................................................... 6 PDO-Level Results Indicators ...................................................................................... 6 III. PROJECT DESCRIPTION ..................................................................................................6 Project Components ...................................................................................................... 6 Project Financing .......................................................................................................... 9 Lessons Learned and Reflected in the Project Design ................................................ 11 IV. IMPLEMENTATION ........................................................................................................12 Institutional and Implementation Arrangements ........................................................ 12 Results Monitoring and Evaluation ............................................................................ 13 Sustainability............................................................................................................... 13 V. KEY RISKS .......................................................................................................................13 Overall Risk Rating and Explanation of Key Risks.................................................... 13 VI. APPRAISAL SUMMARY ................................................................................................15 Economic and Financial Analysis ............................................................................... 15 Technical ..................................................................................................................... 16 Financial Management ................................................................................................ 16 Procurement ................................................................................................................ 17 Social (including Safeguards) ..................................................................................... 17 Environment (including Safeguards) .......................................................................... 18 Other Safeguards Policies Triggered (if required)...................................................... 18 World Bank Grievance Redress .................................................................................. 18 Annex 1: Results Framework and Monitoring...............................................................................19 Annex 2: Detailed Project Description ..........................................................................................28 Annex 3: Implementation Arrangements .......................................................................................47 Annex 4: Implementation Support Plan.........................................................................................56 Annex 5: Economic and Financial Analysis ..................................................................................58 map: IBRD 33425R1 ....................................................................................................................63 PAD DATA SHEET Kazakhstan Education Modernization Project (P153496) PROJECT APPRAISAL DOCUMENT EUROPE AND CENTRAL ASIA Education Global Practice Report No.: PAD1645 Basic Information Project ID EA Category Team Leader(s) P153496 C - Not Required Dingyong Hou Lending Instrument Fragile and/or Capacity Constraints [ ] Investment Project Financing Financial Intermediaries [ ] Series of Projects [ ] Project Implementation Start Date Project Implementation End Date 03-Jan-2018 31-Dec-2022 Expected Effectiveness Date Expected Closing Date 03-Jan-2018 31-Dec-2022 Joint IFC No Practice Senior Global Practice Country Director Regional Vice President Manager/Manager Director Mario Cristian Aedo Amit Dar Lilia Burunciuc Cyril E Muller Inostroza Borrower: Republic of Kazakhstan Responsible Agency: Ministry of Education and Science Contact: Yerlan Sagadiyev Title: Minister Telephone No.: 77172742414 Email: pressa@edu.gov.kz Project Financing Data (in USD Million) [X] Loan [ ] IDA Grant [ ] Guarantee [ ] Credit [ ] Grant [ ] Other Total Project Cost: 77.00 Total Bank Financing: 67.00 Financing Gap: 0.00 i Financing Source Amount Borrower 10.00 International Bank for Reconstruction and 67.00 Development Total 77.00 Expected Disbursements (in USD Million) Fiscal Year 2017 2018 2019 2020 2021 2022 Annual 0.00 5.00 25.00 25.00 10.00 2.00 Cumulative 0.00 5.00 30.00 55.00 65.00 67.00 Institutional Data Practice Area (Lead) Education Contributing Practice Areas Proposed Development Objective(s) The project development objective (PDO) is to improve quality and equity in primary and secondary education, particularly in rural and disadvantaged schools Components Component Name Cost (USD Millions) Supporting system-wide improvement in primary and 9.72 secondary Supporting rural and disadvantaged schools to reduce 62.04 disparities in learning results Supporting citizen engagement, monitoring and evaluation, 5.24 and implementation Systematic Operations Risk- Rating Tool (SORT) Risk Category Rating 1. Political and Governance Moderate 2. Macroeconomic Substantial 3. Sector Strategies and Policies Moderate 4. Technical Design of Project or Program Substantial 5. Institutional Capacity for Implementation and Sustainability Substantial 6. Fiduciary Substantial 7. Environment and Social Low 8. Stakeholders Substantial ii 9. Other OVERALL Substantial Compliance Policy Does the project depart from the CAS in content or in other significant Yes [ ] No [ X ] respects? Does the project require any waivers of Bank policies? Yes [ ] No [ X ] Have these been approved by Bank management? Yes [ ] No [ ] Is approval for any policy waiver sought from the Board? Yes [ ] No [ X ] Does the project meet the Regional criteria for readiness for implementation? Yes [ X ] No [ ] Safeguard Policies Triggered by the Project Yes No Environmental Assessment OP/BP 4.01 X Natural Habitats OP/BP 4.04 X Forests OP/BP 4.36 X Pest Management OP 4.09 X Physical Cultural Resources OP/BP 4.11 X Indigenous Peoples OP/BP 4.10 X Involuntary Resettlement OP/BP 4.12 X Safety of Dams OP/BP 4.37 X Projects on International Waterways OP/BP 7.50 X Projects in Disputed Areas OP/BP 7.60 X Legal Covenants Name Recurrent Due Date Frequency PIU 15-Feb-2018 Description of Covenant No later than thirty (30) days after the Effectiveness Date of the Loan Agreement, the Borrower, through MOES, shall establish and operate and maintain throughout project implementation, the PIU, with a composition, resources and terms of reference, and functions, all acceptable to the Bank. Name Recurrent Due Date Frequency Technical Advisory Panel 11-May-2018 Description of Covenant No later than ninety (90) days from the Effective Date of the Loan Agreement, the Borrower, through MOES, shall establish, and thereafter maintain throughout Project Implementation, a Technical Advisory Panel, with composition, resources and terms of reference all satisfactory to the Bank. iii Name Recurrent Due Date Frequency MOES 09-Mar-2018 Description of Covenant No later than forty-five (45) days from the Effective Date of the Loan Agreement, the Borrower shall, through MOES, develop and launch, within the Borrower's existing automated accounting software, a module acceptable to the Bank to generate interim unaudited financial reports and capture the Project's accounts. Conditions Source of Fund Name Type IBRD POM Effectiveness Description of Condition The Borrower, through MOES has finalized and adopted a Project Operational Manual detailing the policies, procedures and arrangements for project implementation with terms of references, satisfactory to the Bank. Team Composition Bank Staff Name Role Title Specialization Unit Dingyong Hou Team Leader Senior Education GED03 (ADM Specialist Responsible) Nurbek Kurmanaliev Procurement Senior Procurement GGO03 Specialist (ADM Specialist Responsible) Aliya Kim Financial Financial GGO21 Management Management Specialist Specialist Aliya Bigarinova Team Member Consultant GED03 Aliya Bizhanova Team Member Operations Officer GED03 Anar Sheshmukhanova Team Member Consultant GED03 Cristina Marosan Ling Team Member Senior Evaluation GEDDR Officer Dorsati H. Madani Team Member Senior Economist OPSPQ Ekaterina Romanova Safeguards Social GSU03 Specialist Development Specialist Gerard Peart Team Member Consultant GED06 Janelle Plummer Team Member Senior Social GSU03 Development Specialist iv Jasna Mestnik Team Member Finance Officer WFALN Katia Marina Herrera Team Member Senior Economist GED03 Sosa Kirill Vasiliev Team Member Senior Education GED03 Specialist Lisa Lui Counsel Lead Counsel LEGLE Naveed Hassan Naqvi Team Member Country Manager ECCAZ Ruxandra Costache Counsel Senior Counsel LEGLE Sujani Eli Team Member Program Assistant GED03 Extended Team Name Title Office Phone Location Locations Country First Location Planned Actual Comments Administrative Division Consultants (Will be disclosed in the Monthly Operational Summary) Consultants Required? Consultants will be required v I. STRATEGIC CONTEXT Country Context 1. Kazakhstan is a large, ethnically and culturally diverse country with a low population density. It is the ninth-largest country in the world by area and the largest landlocked country. It has a territory of 2,727,300 square kilometers and a population of about 17.3 million. Kazakhs account for 63 percent of the population with other ethnic groups present, including Russians, Uighurs, Ukrainians, Koreans, Uzbeks, and Tatars. 2. The economy grew rapidly between 2000 and 2013. Gross domestic product (GDP) growth averaged 8 percent between 2000 and 2013, driven in part by growth and job creation in the services sector. Unemployment halved between 2001 and 2013, and Kazakhstan’s poverty rate (defined as income under US$5 per day) dropped from 54 percent in 2006 to 17.8 percent in 2014, due in part to a doubling of real wages between 2003 and 2013. Household income of the poorest 40 percent of the population increased by 19 percent from 2008 to 2012 compared to 15.2 percent for the top 60 percent. The middle class increased from 8 percent to 28 percent of the population. 3. Growth has slowed since 2014. Economic growth has slowed since 2014, due to a sharp devaluation in the tenge, the oil price shock, and lower external demand. GDP growth slowed from 6 percent in 2013 to 4.4 percent in 2014, to an estimated 1 percent in 2015. Kazakhstan’s economy has also been affected by China’s economic slowdown and Russia’s recession, two of Kazakhstan’s main trade partners. 4. The country has made strong strides in policy reforms; however, the economy remains highly natural-resource dependent. Minerals, oil, and natural gas account for 80 percent of total exports and 37 percent of GDP. The Government of Kazakhstan (GoK) continues to pursue social, economic, and structural reforms, and is making efforts to improve the education, skills, and health of the population. 5. Kazakhstan’s development objective of joining the top 30 most developed countries by 2050 depends on its ability to sustain balanced and inclusive growth. In the near to medium term, economic prospects depend on a continuation of stability-oriented macroeconomic policies. Medium-to-long-term development depends on creating skilled human capital for a diversifying and competitive economy and inclusive growth. Sectoral and Institutional Context 6. Education has historically been a priority in Kazakhstan. There are 7,511 schools enrolling 2.7 million students. Approximately 25 percent of schools are in urban areas and serve 48 percent of the student population; 75 percent of schools are in rural areas and serve 52 percent. The country has made significant strides toward universal access and enrollment, with gender parity. The net enrollment rate for primary and lower secondary education (ages 5–14) is 99 percent, and for upper secondary education (ages 15–19) it is 86 percent. The difference in enrollment rates between boys and girls is less than 1 percentage point. The challenge for Kazakhstan today is to provide quality education for all. 1 7. The 2012 Programme for International Student Assessment (PISA) results show marked improvements and narrowing gaps in achievement compared to 2009. However, the results placed students of Kazakhstan significantly behind their counterparts from countries of similar income level.1 In 2009, 59 percent and 55 percent of students scored below the basic competency level in math and science, respectively; in 2012, these shares had fallen to 45 percent and 42 percent, respectively. In reading, 58 percent of students did not achieve the basic level of competency in 2009, and this essentially remained unchanged in 2012 (57 percent). The results also show wide but narrowing disparities in learning by income level; there were 91 points (equivalent to over two years of schooling) between the top and bottom quintiles in 2009, declining to 73 points in 2012. However, while the lowest quintile improved its performance by 8 points between 2009 and 2012 (and the second to fourth quintiles also experienced improvements, though smaller), the scores in the top quintile fell by 10 points. Urban schools performed better than rural schools; the score differences between urban and rural students in math and reading in 2012 were 13 and 32 points, respectively. PISA scores were higher in schools with Russian as the language of instruction than schools with Kazakh as the language of instruction. However, between 2009 and 2012, the difference in scores fell. In 2011, Kazakhstan participated in the Trends in International Mathematics and Science Study (TIMMS). Grade 4 Kazakhstani students scored 501 points in math, right at the mean of 500 points, while grade 8 students scored 487 points, just below the mean. In science, grade 4 and grade 8 Kazakhstani students scored 495 and 490 points, just below the mean. 8. Kazakhstan supports the professional development of teachers, and has made progress in addressing teacher shortages in hard-to-staff schools. However, challenges remain in attracting the best into teaching and motivating teachers to perform, in part because a teacher’s starting salary is one-third less than a health worker’s and one-sixth of a banking sector professional’s. 9. Pre-service training and qualifications are not adequately addressed. An emphasis on theory that has little to do with practice has limited Kazakhstan’s potential to reach the performance benchmarks of high-performing education systems. While teaching is shifting toward a student-centered approach, emphasis should be placed on inquiry-based problem solving and critical thinking. Weak school leadership is also an obstacle to raising teacher effectiveness. 10. Kazakhstan has a strong student learning assessment system. Classroom assessments are conducted regularly, including the abovementioned international assessments. The Unified National Test (UNT) is used to certify learning at the end of the secondary cycle and for admission to higher education. However, formative assessments have not been used in classrooms to assess student progress and inform the teaching strategy and professional 1 Kazakhstan students scored 432 points in math compared to the OECD average of 494 points in PISA, a difference equivalent to 1.6 years of schooling. The difference between Kazakhstan and the OECD average in reading is starker—the gap is equivalent to 2.5 years of schooling. Forty points is equivalent to one year of schooling In general, a change of 10 points in a score is considered statistically significant. (See “Strengthening Kazakhstan’s Education. An Analysis of PISA 2009 and 2012,” Education Global Practice, Europe and Central Asia Region, World Bank Group, Washington, DC, 2014; and PISA 2012 Results: What Students Know and Can Do (Volume1), OECD, Paris, 2014.) 2 development of teachers, and there is a debate about whether the UNT accurately assesses actual learning, and whether it reflects curriculum revisions. 11. The education system continues to devolve to the local level. School principals prepare budget requests, but local authorities make the decisions over budget approval and execution. Principals also hire and fire staff. Stakeholders like parents participate in school activities through school councils, which have no legal authority, and school accountability is hampered by parents’ lack of power over the budget and personnel management, and weak links between student and teacher performance and school accountability. 12. Education spending has been declining and resource distribution is unequal. Education expenditures decreased from 6 percent of GDP in the 1990s to around 3.6 percent in 2012, substantially below comparator countries and the Organisation for Economic Co-operation and Development (OECD) average of 5 to 6 percent. General education is largely locally financed, but the current intergovernmental transfer system does not address regional and local disparities between rich and poor regions and rural and urban populations in financing schools. In recent years, many facilities have been built or rehabilitated to meet the needs of regions with growing student populations, but concerns exist about the equity and efficiency of the distribution of other educational resources. Learning materials and information and communication technologies tend to be more prevalent in large, urban schools, while students in rural schools, particularly “small-size” and “multigrade class”2 schools, often lack the basics, such as enough textbooks, computers, and science lab equipment. A formula-based per-student financing scheme was piloted to address equity and efficiency in allocation, but faced numerous challenges in implementation. 13. Kazakhstan has maintained gender parity in universal access to primary and secondary education. However, learning outcomes are less equal, as reflected in the PISA 2012 results. Performance in math does not vary by gender, but reading scores for boys was equivalent to one year of schooling behind that of girls. Peer comparator countries such as Turkey and Russia did much better. 14. Driven by the ambition to be the best, and building on the reforms to date, the GoK has established a strategic vision for education. The Kazakhstan 2030 Strategy provides a framework for economic and societal reforms, the first phase of which will be implemented through the State Program of Education and Science Development 2016–2020 (SPESD). The SPESD aims to improve, by 2020, the country’s economic competitiveness through improved quality of education at every level as a basis for sustainable economic growth. For primary and secondary education, new education financing mechanisms will be developed, including training highly qualified staff for the education sector and providing them with more support and incentives; developing public-private partnerships and introducing elements of corporate governance systems in schools; improving student assessment methods; transitioning to a 12- year education model and updating curriculums; addressing the challenges of small-class schools; and developing the concept of inclusive education and supporting low-performing students in schools. SPESD indicators and targets include increasing the number of students completing science and math programs, and improving the country’s ranking in international 2 Multigrade schools are schools with multigrade classrooms. 3 assessments such as PISA, TIMMS, and for Progress in International Reading Literacy Study (PIRLs). 4 15. Nazarbayev Intellectual Schools (NIS) is a government-financed project to establish 20 “intellectual schools for gifted and talented children aimed at the education and upbringing of a new generation of intellectual elites.”3 The NIS introduced many innovative practices generated through international partnerships with Kazakh institutions responsible for curriculum development, assessment, and pedagogical practice. To try to adapt NIS innovations for the mainstream system, a Ministry of Education and Science (MOES)-led partnership among NIS schools, international partners, and the Kazakhstani education research community4 produced a set of new education standards embedded in a competence-based curriculum with an aligned pedagogical approach and assessment practice and a new model of professional development of teachers. To spur adaptation and innovation, the MOES-formulated Strategic Plan 2016–20 established two priorities: (a) introduction of a new education program across all subjects from grade 1, and (b) language study revisions and use of information and communication technologies. Higher-Level Objectives to which the Project Contributes 16. The World Bank is actively and strategically engaged in Kazakhstan. The Country Partnership Strategy for FY2012–17 identifies three objectives for the Bank’s involvement in the country: (a) promoting diversification, innovation, investment in human capital, and international trade integration for employment generation; (b) promoting improved governance in public administration and service delivery; and (c) ensuring that development is environmentally sustainable. This project will strengthen human capital to enhance Kazakhstan’s competitiveness for sustained growth, productivity, and capacity for innovation. The Partnership Framework Arrangement between the GoK and the World Bank Group (reached in May 2014) aims to support the government’s efforts through investment and institutional capacity-building projects and technical assistance. The current project addresses Partnership Framework Arrangement pillar 3, “Development of human capital, promotion of science, and innovation.” The project also complements the Technical and Vocational Modernization Project (P102177) on raising literacy and closing achievement gaps in cognitive skills and modern competencies. 17. The project is aligned with the Bank’s twin goals of ending extreme poverty and boosting shared prosperity. While supporting the quality improvement of all schools, the project targets disadvantaged schools serving the most vulnerable groups. This will lay the foundation for long-term productivity, higher earnings, shared prosperity, and the reduction of intergenerational poverty. II. PROJECT DEVELOPMENT OBJECTIVES PDO 18. The project development objective (PDO) is to improve quality and equity in primary and secondary education, particularly in rural and disadvantaged schools. 3 Educational Reform and Internationalisation: The Case of School Reform in Kazakhstan , The Cambridge Education Research Series, Edited by David Bridges, Cambridge University Press, Cambridge, 2014. 4 See also Box A2.1 in Annex 2. 5 Project Beneficiaries 19. The direct beneficiaries are over 2.6 million students, as well as teachers and other stakeholders, and particularly the over 1.5 million teachers and students in 5,400 rural and disadvantaged schools with low learning achievements. PDO-Level Results Indicators PDO Indicator Assessed Aspect of PDO (1) Grade 4 EALA (revised) scores in project Enhanced quality and equity through improved schools improved, in reading and mathematics: student learning results boys/girls (2) Percentage of students from project schools Enhanced equity through access to essential benefiting from instruction materials and learning resources multimedia equipment (3) Percentage of teachers in project schools who Enhanced quality through improved teacher demonstrate improved pedagogical approaches classroom practice Note: EALA = external assessment of learning achievements. “Project schools” refer to those schools that directly benefit from inputs financed under Component 2. III. PROJECT DESCRIPTION Project Components 20. The project will contribute to producing a modern, productive, skilled workforce for an innovative, competitive, diversifying economy. The project supports system-wide reforms, targeted assistance to disadvantaged schools, enhanced monitoring and evaluation of reforms, and civic engagement and participation. The project will be implemented over five years through a phased approach with sequencing to allow the piloting, evaluation, refinement, and scaling up of select key reforms. The total project cost is US$77 million including tax. Component 1: Supporting system-wide improvement in primary and secondary education (Total component cost US$9.72 million of which US$7.71 million financed by the loan) 21. The objective of this component is to improve curricular standards, policies, and programs through a set of initiatives to implement a new curriculum for grade 1–12 that was piloted, evaluated, and scaled up, using existing national systems and resources. Subcomponent 1.1: Curriculum modernization 22. This subcomponent will support the Ministry of Education and Science (MOES) to introduce the curricular standards, plans, and programs for primary and secondary education, including the year before grade 1. Based on the 12-grade curriculum that has been developed with international support (including from the University of Cambridge in the United Kingdom) for the NIS schools and is currently piloted, the project will support the MOES to adapt and finalize the curriculum for use in all mainstream schools. The project will support, in particular, technical assistance to monitor the pilot, evaluate the results, and integrate the lessons learned from the pilot into a system-wide rollout. The project will also provide technical assistance to strengthen textbooks standards and the quality assurance mechanism for approving textbooks. 6 Subcomponent 1.2: Alignment of assessment systems with the new curriculum 23. This subcomponent will finance technical assistance to design and implement new standards in testing. In particular, it supports the alignment of the external assessment of learning achievements (EALA) and the Unified National Test (UNT) with the new curriculum. The project will finance training of MOES’s assessment specialists to build their skills to design tests that measure the content and skills of the new curriculum and analyze and communicate the results. The project will support establishment of a test bank to store and automate the test data and training of staff and test developers in managing, maintaining, and upgrading the test bank. Subcomponent 1.3: Modernizing pedagogical education and practice 24. This subcomponent will support technical assistance in the design and piloting of a new pre-service teacher education model for pedagogical universities and the development of education programs using the model upon an evaluation. Pedagogical universities will work closely with international partners to develop new programs for teaching with digital resources and math and science in English. The project will finance technical assistance to develop and/or adapt an instrument to observe the pedagogical practice of teachers in the classroom. The results of the observation will inform in-service professional development for teachers and evaluation of their teaching effectiveness. (The evaluation is included in the evaluation of specific initiatives under Component 3). Instructional materials with multimedia equipment will be provided for pedagogical universities to support their delivery of the new programs. Subcomponent 1.4: Enhancing accountability through school inspections 25. This subcomponent will enhance school accountability by improving school inspection practice and capacity building for school-based management and planning. The project would strengthen the demand for accountability by supporting community awareness-raising on the benefits of education; the rights, roles, and responsibilities of parents and students in enabling quality education services; and the national strategy and targets to improve education. The project would also finance training of staff and school leaders and community stakeholders to build their skills in school-based management, self-evaluation, and school inspections. Component 2: Supporting rural and disadvantaged schools to reduce disparities in learning results (Total component cost US$62.04 million of which US$54.78 million financed by the loan) 26. The objective of this component is to increase equitable learning outcomes in rural and disadvantaged schools, through improved access to essential instructional materials and equipment, enhanced teacher pedagogy, management capacity, and support for children with special education needs. Subcomponent 2.1: Empowering rural and disadvantaged schools with instructional materials and multimedia equipment 27. This subcomponent will provide each targeted school with a package of essential instructional materials with multimedia equipment in 5,400 rural and disadvantaged schools. The provision of materials and equipment will enable more equitable distribution of and access to 7 these essential resources to support teachers to deliver the new curriculum and help students with more practice to master new content and skills. Subcomponent 2.2: Enhanced pedagogical and management capacities 28. This subcomponent will finance the design and delivery of an integrated package of in- service teacher training and pedagogical support. The package will increase the number of hours of in-service training of teachers to target issues particularly affecting rural and disadvantaged schools with customized pedagogy to identify and remedy learning gaps, practice formative assessments, and improve student study techniques for underperforming students, for example, boy’s reading. It will also strengthen pedagogical support networks, including through the capacity building of methodologists and School Directors, providing teachers with opportunities for self-learning and post-training support. Subcomponent 2.3: Supporting inclusive education 29. This subcomponent will support the design and pilot of a new model, and educational programs for students with special education needs for social and academic integration. The project will finance technical assistance to support the MOES in designing a model in line with the national Framework on Inclusive Education. It will support Inclusive Education Resource Centers and partner schools to operationalize the model and develop methodological guidance. Component 3: Supporting citizen engagement, monitoring and evaluation, and implementation (Total component cost US$5.24 million of which US$4.51 million financed by the loan) 30. The objective of this component is to engage and support stakeholder and citizen participation, monitor and evaluate project implementation and results, and support project implementation. Subcomponent 3.1: Stakeholder participation and engagement 31. This subcomponent will support stakeholder participation and awareness-raising through consultation and communications, including the establishment of a grievance redress system and third-party monitoring of project implementation. A communications strategy will be implemented that engages and informs stakeholders about the project and key reforms including the revised curriculum, and strengthened quality assurance for textbooks.5 The project will support annual public forums designed to create transparency on the intent and progress of change, and encourage feedback from interested members of civil society. Subcomponent 3.2: Monitoring and evaluation of pilot initiatives 32. This subcomponent will finance the design and execution of the monitoring and evaluation of a specific set of pilot initiatives supported by the project. This will include observations of the quality of teaching-learning practices, and student learning assessments in project schools. The project will also finance evaluations of the new quality assurance (QA) mechanism for 5 See Annex 2 for further detail. 8 textbooks, the new model of pre-service teacher training, using English as the language of instruction, and the pilot of the new curriculum; as well as an overall evaluation of the project. The project will support the MOES in drafting new policies based on lessons learned from the evaluations, and will finance a Technical Advisory Panel to promote the maintenance of satisfactory standards as the curriculum is adapted for, and rolled out to, mainstream schools. Subcomponent 3.3: Project implementation 33. This subcomponent will finance project operating costs, including translation, interpretation, equipment, supervision costs, Project Implementation Unit (PIU) consultant salaries, and incremental operating costs at the MOES. Project Financing 34. The total project cost is estimated at US$77 million, to be financed by an IBRD loan of US$67 million and government co-financing of US$10 million to cover the taxes. The breakdown of project costs and financing by component and financier is presented in Table 1. 9 Table 1 Breakdown of Project Costs and Financing by Component, Republic of Kazakhstan Republic of Kazakhstan Education Modernization Project Components by Financiers (US$ '000) International Bank for Reconstruction Government of Local and Development Kazakhstan Total (Excl. Duties & Amount % Amount % Amount % For. Exch. Taxes) Taxes A. Supporting system-wide improvement in primary & secondary education 1. Curriculum modernization 794.1 49.63 805.80 50.37 1,599.85 2.08 79.41 714.65 805.80 2. Alignment of assessments systems with the new curriculum 3,832.2 87.13 566.00 12.87 4,398.16 5.71 383.22 3,448.94 566.00 3. Modernizing of pedagogical education & practice 2,947.9 83.96 563.00 16.04 3,510.89 4.56 294.79 2,653.10 563.00 4. Enhancing accountability through school inspections 134.2 62.44 80.70 37.56 214.88 0.28 13.42 120.76 80.70 Subtotal 7,708.3 79.27 2,015.50 20.73 9,723.78 12.63 770.83 6,937.45 2,015.50 B. Supporting rural & disadvantaged schools to reduce disparities in learning results 1. Empowering rural & disadvantaged schools with instructional materials & multimedia equipment 22,430.2 88.86 2,811.50 11.14 25,241.72 32.78 2,243.02 20,187.20 2,811.50 2. Enhanced pedagogical & management capacities 32,039.6 87.93 4,397.60 12.07 36,437.23 47.32 3,203.96 28,835.67 4,397.60 3. Supporting inclusive education 315.0 87.40 45.40 12.60 360.42 0.47 31.50 283.52 45.40 Subtotal 54,784.9 88.31 7,254.50 11.69 62,039.37 80.57 5,478.49 49,306.38 7,254.50 C. Supporting citizen engagement, monitoring and evaluation, & implementation 1. Stakeholder participation & engagement 433.3 83.18 87.60 16.82 520.93 0.68 43.33 390.00 87.60 2. Monitoring & evaluation of pilot initiatives 1,303.3 87.41 187.70 12.59 1,491.03 1.94 130.33 1,173.00 187.70 3. Project implementation 2,770.2 85.90 454.70 14.10 3,224.88 4.19 277.02 2,493.16 454.70 Subtotal 4,506.8 86.06 730.00 13.94 5,236.85 6.80 450.68 4,056.16 730.00 Total Project Costs 67,000.0 87.013 10,000.00 12.99 77,000.00 100.00 6,700.00 60,300.00 10,000.00 \a Price Contingencies is inclusive in the total cost 10 Lessons Learned and Reflected in the Project Design 35. The project design was informed by global evidence on what works in education, and by an in-depth analysis of Kazakhstani student performance on PISA 2012. 36. Teacher effectiveness matters. International evidence shows that the quality of teaching is the most important predictor of student learning. An OECD report, “PISA 2012 Results in Focus,”6 shows schools are only as good as their teachers. Teaching quality is affected by many factors, from teachers’ initial education and skills-at-entry to their classroom practices, support for continuous professional development, and being a respected profession with decent pay. The PISA 2012 results showed that Kazakhstani teachers appear to be especially challenged in how they manage assignments and relate knowledge taught to students. Students are disadvantaged by limited knowledge and application of effective learning strategies, such as relating new knowledge to other contexts and applying study methods other than memorization. While relatively low student performance indicates the need to enhance teacher effectiveness at the national level, improving the pedagogical skills of teachers in Kazakh language schools and underperforming schools and those teaching underperforming students should be a priority. Incentives to ensure that the highest-performing teachers are assigned to underperforming schools is likely an effective strategy to boost learning among underperforming students. 37. Student assessment matters. An effective assessment system is essential to collect, analyze, and provide data and information on student learning to inform system performance and support for teacher development. It should ensure that student assessments reflect the content of learning and mastery of competencies in an objective and fair manner. The instruments should be valid and reliable so that the results generated can support the system to identify areas where further research or remedial action might be needed, with strategies and investment redirected. The system also needs to ensure a clear balance, fit for the purposes of assessment learning through summative and formative assessments that can generate snapshots of system-wide learning trends, measure individual student progression on a continuum, and certify learning achievements at the end of the schooling cycle. Implementing a clear strategy and developing institutional capacity for these purposes are crucial to quality enhancement efforts at the national and school levels. 38. Learning resources matter. The PISA 2012 results show that school systems with high student performance in mathematics tend to allocate resources more equitably between advantaged and disadvantaged schools. In these systems, there are smaller differences in principals’ reports on teacher shortages, the adequacy of educational resources and physical infrastructure, and in average mathematics learning time between schools with more advantaged students and those with more disadvantaged students. For example, Estonia, Finland, and the Republic of Korea all show higher-than-OECD-average performance in mathematics. In those countries, principals in disadvantaged schools tended to report that their schools had adequate educational resources as much as, if not more than, principals in advantaged schools. 6 “PISA 2012 Results in Focus: What 15-year-olds know and what they can do with what they know,” OECD, Paris, 2014. 11 39. Reform benefits materialize over time. Experience in many countries reveals that education reforms take time to work through the system and that the process is often politicized. Therefore, the design of reforms and the timeline for planning and implementation should allow for proper sequencing and sufficient time to achieve change in tandem with proper and timely evaluation of the pilots before a full-scale rollout. The complexity, location, and stakeholders involved need to be taken into account and significant resources devoted to sustaining the effort. 40. Equipping teachers for changes. To maximize the probability of success, it is essential to equip teachers with the capacity to change, and to incentivize their efforts to do so. Traditional teacher training often focuses on the transmission of knowledge rather than on improving teaching practices and changing classroom behavior. Therefore, it is essential that project design include customized training and support for teachers to enhance their capacity to implement the reforms. The project has adopted this focus on developing teacher capacity for enhanced pedagogical practice through training and classroom observations. 41. Communication with and participation of stakeholders. For any reform to succeed, the engagement of local communities, support of teachers and administrators, and participation of key stakeholders are key. A clear communication and advocacy strategy to explain to the public the benefits of reforms and to advocate for citizen participation is indispensable. Project design will benefit from close consultation with, and continuous engagement of, stakeholders in the planning and implementation process. Experience in countries that have successfully implemented such reforms shows that public perceptions play a big part in the success or failure of such an effort. A communications strategy customized to the wide range of stakeholders should accompany the project implementation strategy. IV. IMPLEMENTATION Institutional and Implementation Arrangements 42. The MOES will be the implementing agency for the project through the Project Implementation Unit (PIU) in coordination with the organic departments to ensure ownership and coherence. The detailed responsibilities of the relevant MOES entities will be specified in the Project Operations Manual (POM). 43. A PIU will be established in the MOES and maintained throughout project implementation, with composition set forth in the POM to manage and coordinate routine implementation including coordination and monitoring, drafting technical and reporting documents, and supporting financial management (FM) and procurement work. The PIU will be led by a director and consist of a FM specialist, an accountant, one or more procurement specialists, a monitoring and evaluation specialist, coordinators for components, and an administrative staff for support and interpretation. Details of the implementation arrangements are presented in Annex 3. 44. A Technical Advisory Panel will be established drawing on international and national expertise to provide technical oversight and guidance on policy and technical issues during implementation. The panel will be led by a chairperson and consist of prestigious international and Kazakhstani experts who have in-depth knowledge of global trends in teacher policy, curriculum, assessment, and school reforms in Kazakhstan. The panel will convene twice a year 12 to review the status of implementation and advise the MOES leadership on key issues and actions. Results Monitoring and Evaluation 45. The MOES will be responsible for project monitoring through the PIU and the related functional departments. The PDO-level indicators will be tracked through (a) sample-based assessments of student learning by designated assessment institutions, to measure particularly the higher-order cognitive skills promoted by the revised curriculum; (b) sample-based surveys of teacher-student interactions using a validated and reliable instrument that measures learning- related interactions targeted by the project-supported training; and (c) project records pertaining to the school-level availability of teaching-learning resources supplied by the project. The intermediate outcome results indicators will be tracked using project records and third-party surveys of pedagogical leadership exercised by School Directors, of community consultations and functioning of school Boards of Trustees, and of project interactions with beneficiaries. These activities will contribute to the evaluation of project outcomes. The choice of project monitoring indicators was based on the indicators that the MOES uses for the SPESD program using its routine administrative data and additional survey and test data of improved instruments. The project-supported stakeholder participation, citizen engagement, and a grievance redress system also contribute to transparency and independent monitoring of implementation of the proposed reforms. Sustainability 46. The project is closely aligned to the MOES’s SPESD program. The sustainability of the project is considered high for three reasons. First, project policy objectives directly support the SPESD program, which the MOES is committed to pursuing over the near and medium term. The project will enhance and deepen SPESD reforms, which have been informed by global and in-country research evidence and lessons learned from the previous and ongoing engagements. The project also helps institutionalize the reforms by building stakeholder ownership, citizen engagement, and stakeholder participation. Second, the project complements government financing of these reforms but does not replace it. The government is expected to maintain the level of funding beyond the initial investment cost of the reforms. Third, the project strengthens institutions and the technical capacity to manage the reforms and deliver expected results so that the project-supported reforms will continue beyond the project period. V. KEY RISKS Overall Risk Rating and Explanation of Key Risks 47. The project was informed by analytics and evidence of global research and Kazakhstan- specific diagnosis. However, as with any reform that involves system-wide changes, there are risks pertaining to an overly ambitious scope and timeline, stakeholder ownership and/or resistance, and the broader political economy that can influence the process and results. 48. The overall risk for the proposed project is rated as substantial. The major risks are related to macro economic and fiscal environment, technical design, institutional capacity, stakeholder support for reforms, and fiduciary adequacy. 13 49. Macro. The deep plunge of oil price and subsequent depreciation of Kazak tengi in 2015 through 2016 caused contraction of the economy and of the government investment and spending. The government took precautionary steps to manage these shocks while reducing dependency on natural resources. The upward market trend in oil production and price would stabilize over time, thus partially mitigates the risk, but continued close monitoring would be necessary to keep this risk in check. 50. Technical design. Several aspects of the project involve innovation and reforms that are complex and require complementary measures to succeed. To minimize the risk of an overload of activities, the project seeks to balance support for system-wide reforms and key elements while keeping the design simple and realistic. This risk is mitigated by ensuring that there are (a) a select set of reforms that are essential but not overwhelming, and (b) a realistic phased approach with careful sequencing allowed for pilot, evaluation, and refinement before a system- wide rollout. 51. Institutional capacity. While the MOES has demonstrated a strong commitment to the proposed reforms, successful implementation demands significant technical capacity that is not currently available within the MOES. Key mitigation steps for this risk include capacity building through training of teachers, school leaders, and other stakeholders; providing technical assistance to the MOES and project implementers in key areas of the reforms; supporting implementation by establishing a PIU staffed with expertise and skills commensurate with the mandated functions; and close supervision and technical support from the Bank during implementation. 52. Stakeholders. Some of the proposed system-wide comprehensive reforms, such as introduction of new learning standards and curriculums and revision of the UNT are considered controversial and are likely to encounter resistance from certain stakeholders. To minimize these risks, measures will be taken to ensure (a) close and ongoing engagement with the stakeholders, complemented by advocacy and communications throughout the process; (b) that the local research community is engaged in disseminating information to the public on the reforms to minimize misunderstanding; and (c) that a phased approach is adopted that includes piloting of key initiatives while seeking stakeholder feedback before a nationwide rollout. 53. Fiduciary risks. The main fiduciary risk is the inadequate skills mix and capacity in MOES for managing the core functions of procurement, accounting, and financial reporting required under Bank-financed projects. The MOES will need to be supported by fiduciary specialists to manage the core functions that adhere to good practices of transparency, efficiency, and good governance. To minimize these fiduciary risks, key mitigating measures will include (a) equipping the PIU with a proper skills mix of procurement and FM specialists, (b) engaging stakeholders in the monitoring of the procurement processes, and (c) implementing a grievance redress system to handle complaints and feedback from the general public. 14 VI. APPRAISAL SUMMARY Economic and Financial Analysis 54. The economic analysis of the project addresses four key issues regarding the proposed investment: (a) the project’s development impact in terms of the benefits, (b) the rationale for public investment, (c) the value added of Bank assistance, and (d) measurement of project component cost-effectiveness. A cost-effectiveness analysis was conducted only for Component 2: Raising equitable learning results in disadvantaged schools, which encompasses 71 percent of total project cost. The analysis was done using PISA 2009 data for measures of effectiveness, using ordinary least squares regressions and the latest OECD Review of School Resources in Kazakhstan for calculating costs. The analysis shows that the proposed project interventions for the underserved populations are cost-effective compared with other potential interventions such as class size or early childhood development (ECD) programs. (Details of the analysis are presented in Annex 5.) Other elements of the project that will interact with the increased resources and new curriculum, although difficult to quantify, include the increased use of information to apprise, empower, and engage stakeholders, and to strengthen the assessment system. Dissemination of information can increase local monitoring and, with the right incentives, make schools more accountable. Finally, student assessments are positively correlated with student achievement, suggesting that project support in this area may increase the magnitude of the interventions. 55. The main benefits of the project are expected improvement in education quality and equity for more than 2 million primary and secondary students through the support for the SPESD. Better education enhances cognitive and foundational skills, which translate into higher future earning potential. Over five years, the proposed project is expected to reach over 2 million school-age children who will benefit from modernized learning standards, improved teacher quality, and strengthened assessment practices. While these benefits are shared with all schools, the project will target the approximately 5,400 disadvantaged schools that serve rural populations and underserved urban youth with additional pedagogical support and learning resources. These beneficiaries represent the bottom 40 percent of the socioeconomic distribution. This targeting aims to equalize the opportunities for more equitable learning results across cohorts of children of different socioeconomic status and between boys and girls. Since resources correlate with learning, the reduced gaps in resources will likely contribute to reduced achievement gaps. The benefits of these education investments will be fully realized over time, resulting in higher earning potential of individual students and higher productivity and economic returns to the economy in the medium and longer term. 56. Education is predominantly publicly financed in Kazakhstan, for two main reasons. The first reason is equity. While education is a private good, households at the bottom quintiles require state interventions to access good-quality public education, because they often cannot afford (private) alternatives. The second reason is that quality of education correlates with economic growth. Skills are an important factor for increasing a country’s productivity and growth. Good-quality education, mostly provided by public schools in Kazakhstan, helps develop foundational skills. Students with good foundational skills can acquire modern and higher-order skills that can make human capital and labor more productive. 15 Technical 57. The development objective would be achieved through its evidence-based design focusing on integrated reforms in customized pedagogy, modernized curriculum, and realigned assessment of student learning and teacher practice in the classroom, together with accountability-boosting interventions. While supporting system-wide reform in key domains to increase quality, the project also targets disadvantaged schools and children to reduce disparity in learning. 58. Project design has drawn on the latest global research on what works in education, and on specific research in Kazakhstan on a range of interventions and experiments by the University of Cambridge, Faculty of Education; the School of Education at the University of Pennsylvania; and the Bank and OECD. The meta-analysis by the Bank of 227 impact studies conducted by top researchers from around the world on what improves learning shows the following interventions most commonly produce large improvements in student learning: (a) Pedagogical interventions that match teaching to students’ learning. Studies by Conn (2014), Kremer, Brannen, and Glennerster (2013), and McEwan (2014) show this most clearly. Conn (2014) finds that pedagogical interventions (defined by her as interventions that change instructional techniques) are more effective at improving student learning than all other types of interventions combined. Within the category of pedagogical interventions, she finds that studies that employ adaptive instruction and teacher coaching techniques are particularly effective. All three studies in Conn’s sample that evaluate adaptive instruction interventions report positive, statistically significant effects on student literacy scores (Korsah et al. 2010; Piper and Korda 2011; Spratt et al. 2013). (b) Individualized continuous teacher training associated with a particular methodology. McEwan (2014) finds that teacher training produces a 0.12 standard deviation improvement in learning (significant with 99 percent confidence), and shows that examining the specific programs is crucial. Providing teachers with general guidance tends not to improve student learning, but Murnane and Ganimian (2014)7 find that detailed support tailored to the skill levels of teachers can be effective. In contrast, training that provides detailed guidance on what and how teachers should teach has proven to be effective in enhancing the skills of low- performing students (Murnane and Ganimian 2014). Financial Management 59. The MOES will oversee the financial management (FM) arrangements in the proposed project and will be supported by an FM consultant to be contracted as part of the PIU. Overall, the project will rely on the FM arrangements established for the existing projects implemented by the MOES with the support of a PIU. The project will rely on country systems for budgeting and planning, internal controls, and flow of funds. The MOES fiduciary assessment established that the existing FM systems meet the Bank’s requirements, including for budgeting and planning, accounting and financial reporting, flow of funds, internal controls, FM staffing, and 7 David K. Evens and Anna Popova, “What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews,” World Bank Policy Research Working Paper, World Bank, Washington, DC, 2015. 16 external audits. However, to bring the project’s arrangements into full compliance with the Bank’s requirements, the MOES will (a) document the FM procedures including internal controls in a Financial Management Manual (FMM) that is a part of the POM and within the timeline of preparing the POM; (b) contract an FM consultant as part of the PIU; and (c) develop a module for the accounting software currently used by the MOES, with a capacity to generate Interim Financial Reports (IFRs) and capture the project’s accounts. 60. IFRs will be prepared on a quarterly basis and will be submitted to the Bank no later than 45 days after the end of each calendar quarter. The formats of the IFRs have been agreed upon with the MOES during negotiations. Project accounts will be subject to an annual independent audit. The project audit report will be made publicly available as per the Bank’s Access to Information Policy. Procurement 61. Procurement of goods and non-consulting services for the proposed project will be carried out in accordance with the Bank’s “Guidelines: Procurement of Goods, Works and Non- Consulting Services under IBRD Loans and IDA Credits & Grants by World Bank Borrowers,” dated January 2011 and revised July 2014 (Procurement Guidelines). Procurement of consultant services will be carried out in accordance with the Bank’s “Guidelines: Selection and Employment of Consultants under IBRD Loans and IDA Credits & Grants by World Bank Borrowers,” dated January 2011 and revised July 2014 (Consultant Guidelines). All procurement will also be carried out under the provisions stipulated in the Loan Agreement. The Bank’s “Guidelines on Preventing and Combating Fraud and Corruption in Projects Financed by IBRD Loans and IDA Credits and Grants” dated October 15, 2006, and revised in January 2011 (Anti- Corruption Guidelines) will apply to this project. The overall procurement risk for the project is rated as substantial. The risk rating is based on past experience and ongoing Bank-financed projects in Kazakhstan, the general public procurement environment, and the current capacity of proposed implementing agencies to administer international procurement. (Detailed procurement arrangements are included in Annex 3.) Social (including Safeguards) 62. The social impact of the project is expected to be positive through engendered project design, citizen engagement during preparation and implementation, and results monitoring of both. Kazakhstan has achieved and maintained gender parity in access to education, but large gaps in achievement remain between boys and girls. The PISA 2012 analysis identified significant disparities in reading between boys and girls. While the project-supported interventions aim at enhancing learning outcomes of all targeted cohorts, the project will monitor and report on any change in learning results through gender-disaggregated data on student test scores to inform policy and subsequent interventions. 63. Citizen engagement. The project adopted a consultative approach during preparation, engaging citizens and other stakeholders on key reforms proposed in the SPESD and on project design, thus enabling them to express their views and voice their concerns. The project also supports specific interventions to involve stakeholders, beneficiaries, and civil society organizations in monitoring and evaluation of the project activities to ensure transparency and 17 openness. (See Annex 2 for more detail on citizen engagement.) The project will also support broad stakeholder engagement in key policy domains in the implementation process. 64. Under Component 1, the project will focus on stakeholder participation in the curriculum rollout process. Under Component 2, three areas of action are proposed. First, local communities will be engaged in the processes of improving their disadvantaged schools. This will include activities that ensure the resources distributed to schools are relevant and used. Second, to support efforts to enhance teacher capacity, a series of activities will ensure that the training subcomponent addresses their needs and knowledge and capacity gaps. Teachers will provide regular feedback and dialogue with instructors on the effectiveness of trainings, and participate in entry and exit surveys to measure before and after absorption. Third, to address low levels of understanding of the benefits of education and the performance of schools, an awareness-raising and capacity-building campaign will be launched to build both parents’ capacity and understanding of the performance and management of schools and of the importance of, and rights to, education. Citizens will also be encouraged to engage in school management through the public display of simplified budgets (revenues and expenditures). 65. A grievance redress system for the project will be incorporated within the MOES’s current complaints-handling system. In addition to strengthening the Ministry’s receipt and processing of and responsiveness to complaints, the grievance redress mechanism will be designed to be proactive. Once a year, it will request feedback, through hotlines or mobile technology, to reach out to the beneficiaries of the project or reform. 66. No resettlement or land acquisition is envisioned under the project; thus, OP 4.12 is not triggered. Environment (including Safeguards) 67. As a category C operation, the project does not finance any civil works; thus, OP 4.01 is not triggered. Other Safeguards Policies Triggered (if required) 68. None. World Bank Grievance Redress 69. Communities and individuals who believe they are adversely affected by a Bank-supported project may submit complaints to existing project-level grievance redress mechanisms or the Bank’s Grievance Redress Service (GRS). The GRS ensures that complaints received are promptly reviewed to address project-related concerns. Project-affected communities and individuals may submit their complaint to the Bank’s independent Inspection Panel, which determines whether harm occurred, or could occur, as a result of the Bank’s noncompliance with its policies and procedures. Complaints may be submitted at any time after concerns have been brought directly to the Bank’s attention and Bank Management has been given an opportunity to respond. For information on how to submit complaints to the Bank’s corporate GRS, visit http://www.worldbank.org/GRS. For information on how to submit complaints to the Bank’s Inspection Panel, visit www.inspectionpanel.org 18 ANNEX 1: RESULTS FRAMEWORK AND MONITORING Country: Kazakhstan Project Name: Education Modernization Project (P153496) Results Framework Project Development Objectives PDO Statement The project development objective (PDO) is to improve quality and equity in primary and secondary education, particularly in rural and disadvantaged schools These results are at Project Level Project Development Objective Indicators Cumulative Target Values Indicator Name Baseline YR1 YR2 YR3 YR4 YR5 End Target By subjects: a By subjects: a statistically statistically significant significant To be measured in increase in increase in Grade 4 EALA (revised) year 3 when the scores as scores as scores in project schools assessment tool (i.e., compared to compared to improved, by Reading and 0.00 0.00 TBD 0.00 EALA) is revised in baseline baseline Mathematics: boys/girls line with the new (Text) standards. Boys/girls: Boys/girls: reduction in reduction in disparity for disparity for reading only reading only Percentage of students 0.00 0.00 30.00 100.00 100.00 100.00 100.00 from project schools 19 benefiting from instruction materials and multimedia equipment (Percentage) Percentage of teachers in project schools who demonstrate improved 0.00 0.00 0.00 0.00 60.00 60.00 60.00 pedagogical approaches (Percentage) Intermediate Results Indicators Cumulative Target Values Indicator Name Baseline YR1 YR2 YR3 YR4 YR5 End Target Evaluation of the new curriculum pilot No Yes Yes Yes Yes Yes Yes completed (Text) Percentage of textbooks that are appraised by National Textbooks 0.00 33.00 41.00 66.00 100.00 100.00 100.00 Center using revised quality assurance system (Percentage) Assessment instruments are designed in line with EALA: Yes EALA: Yes EALA: Yes EALA: Yes 0.00 0.00 0.00 the new standards UNT: Yes UNT: Yes UNT: Yes UNT: Yes (Text) Number of students enrolled in the new model of pre-service teacher 0.00 0.00 0.00 280.00 280.00 280.00 280.00 training (Number) 20 Number of teachers who pass the proficiency test in 0.00 0.00 450.00 450.00 450.00 450.00 450.00 English after training (Number) Percentage of teachers from project schools trained and certified in 0.00 20.00 40.00 60.00 80.00 100.00 100.00 new pedagogical approaches, by gender (Percentage) Percentage of project school administrators who exercise pedagogical 0.00 0.00 30.00 70.00 90.00 90.00 90.00 leadership, by gender (Percentage) Number of regions that: (i) publicly display and discuss school budget in meetings with citizens, 0.00 0.00 4.00 8.00 12.00 16.00 16.00 and (ii) publicly report on meeting discussions and actions to be taken (Number) Percentage of teachers and administrators of project schools received 0.00 0.00 0.00 50.00 75.00 90.00 90.00 support post training (Percentage) Number of project schools equipped with new 0.00 0.00 5,400.00 5,400.00 5,400.00 5,400.00 5,400.00 multimedia equipment (Number) Third party and/or 0.00 0.00 Actions Actions Actions Actions Actions included 21 beneficiary monitoring included in implemented included in implemented in 2 national findings disclosed, national national action plans and discussed at open forum action plan 1 action plan implemented in and agreed actions subsequent year included in national action plans (Text) Percentage of schools that formulate School Improvement Plans (SIPs) reflecting feedback 0.00 0.00 0.00 35.00 0.00 75.00 75.00 provided by parents and beneficiaries in project schools (Percentage) Percentage of project schools in which parents report an improvement in the openness and 0.00 0.00 0.00 35.00 0.00 75.00 75.00 responsiveness of school teachers and management (Percentage) 22 Indicator Description Project Development Objective Indicators Responsibility for Data Indicator Name Description (indicator definition, etc.) Frequency Data Source / Methodology Collection Grade 4 EALA (revised) Measures the extent to which students in Twice, at The External Assessment of NTC/PIU scores in project schools project schools have mastered the Grade 4 baseline and Learning Achievements improved, by Reading and curriculum (revised) in Reading and in Years 3 (EALA) is administered by Mathematics: boys/girls Mathematics. The EALA will be revised and 5. the National Testing Centre to measure the knowledge and skills, (NTC). Prior to establishing particularly non-traditional and higher- a baseline, the instrument order cognitive skills, targeted by the will be revised (with project revised curriculum. Higher grades are not support under selected for the indicator because the roll- Subcomponent 1.2) to be out schedule is such that no higher grade aligned with the revised will have been exposed long enough to the curriculum. The new curriculum for one reasonably to instruments will be expect an impact on learning outcomes, designed to discriminate at within the project's time frame. (See the lower end of Annex 2, Box 1). The indicator measures achievement range, so as to the effectiveness of the new curriculum pick up on improvements and its delivery in bringing about higher achieved by lower- learning outcomes particularly in relation achieving students. Sample- to the new, nontraditional cognitive skills based, with a disaggregated targeted by the new curriculum. That is, structure to determine an using instruments constructed with overall value and a value equated items, the baseline assessment for rural and disadvantaged will test students following the old schools supported under curriculum while the follow-up assessment Component 2. will test students who have been following the new curriculum. Given the new curriculum’s rollout schedule, it is not possible within the Project’s time frame to test two different Grade Four cohorts who follow the revised curriculum, with one 23 cohort being tested prior to project interventions and the other being tested after project interventions have been going on sufficiently long to expect learning impacts. The target will be measured by incremental values in language and mathematics in national assessments rather than targets. This is because there is no reliable national student assessment trend data that provides information on language and mathematics to inform the setting of targets, particularly given that the indicator measures performance against a new curriculum. Percentage of students from Measures access to learning resources by Twice: Years Administrative data PIU project schools benefiting project schools before and after the 2 and 3 from instruction materials project. To be measured prior to the start and multimedia equipment of the school year. All the resources will be supplied by the end of Year 3. Percentage of teachers in Measures the number of teachers observed Twice: Years Sample-based. Contracted evaluation project schools who to follow revised curriculum and have 2 and 3 Observations will be firm/PIU demonstrate improved good-quality teaching-learning practices, conducted by a party pedagogical approaches as a percentage of all teachers observed. independent of agencies The criteria that constitute a satisfactory delivering any of the level of pedagogical skills will be defined teacher training supported by the teacher training program by the project. The development TA. instrument to be used will be validated and internationally recognized (e.g., class or similar), and will be designed to capture the quality teaching- learning practices targeted by the teacher training. 24 Intermediate Results Indicators Responsibility for Data Indicator Name Description (indicator definition etc.) Frequency Data Source / Methodology Collection Evaluation of the new Evaluation report submitted including Year 1 Based on formal resolutions MOES/PIU curriculum pilot completed recommendations for system-wide of NAE and MOES implementation new curriculum based on pilot results Percentage of textbooks Based on textbooks review records kept by Years 3, 4, Independent, sample-based Contracted third party that are appraised by the National Textbooks Center, which will and 5 review of textbooks National Textbooks Center be analyzed to determine if the textbook evaluation reports using revised quality was reviewed in compliance with the assurance system revised QA system designed by the TA contracted under this component. Denominator: all the textbooks submitted for consideration by the National Textbooks Center. The analysis will be undertaken by an expert and independent third party familiar with the new QA system. Assessment instruments are Two assessment instruments will be Years 3, 4, Project records PIU designed in line with the designed in line with the new standards: and 5 new standards EALA and UNT Number of students Refers to the number of students enrolled Years 3, 4, Project records PIU enrolled in the new model in the programs under the new model and 5 of pre-service teacher developed). training Number of teachers who Refers to the number of teachers who Years 3,4, Project records PIU pass the proficiency test in complete the course by passing the and 5 English after training proficiency test. Percentage of teachers from Denominator is all teachers working in a Years 2, 3, 4, Orleu (the network of PIU project schools trained and project school. Numerator is number of and 5 national in-service teacher certified in new teachers certified in new pedagogical training colleges), based on pedagogical approaches, by approaches training records 25 gender Percentage of project Denominator is all general secondary Years 2, 3, 4, Contracted TA. TA will be PIU school administrators who School Directors in a representative, and 5 responsible for sampling, exercise pedagogical randomly selected sample. Numerator is design of data collection leadership, by gender number of project School Directors in that (instruments and sample who are observed to practice mechanism), and reporting improved pedagogical leadership. The disaggregation refers to those School Directors already working in schools, e.g., “the percentage of female School Directors from project schools that exercise pedagogical leadership” refers to the percentage of those female School Directors already working in the schools. The indicator does not set a target that 90% of School Directors exercising pedagogical leadership will be female (or male); rather it refers to gender parity in terms of results. The criteria that constitute improved leadership will be defined by the TA, contracted under the project. Number of regions that (i) Numerator is the number of schools Years 3 and Survey PIU publicly display and discuss disclosing both budgets and meeting 5 school budget in meetings minutes (containing both discussions and with citizens, and (ii) decisions), denominator is total number of publicly report on meeting schools. Data derived survey-sample- discussions and actions to based will be conducted by an independent be taken third party. Percentage of teachers and Teachers and administrators of project Years 3, 4, Resource centers, NAE PIU administrators of project schools who receive support post training. and 5. schools received support post-training Number of project schools No description provided. No No description provided. No description provided. equipped with new description 26 multimedia equipment provided Third party and/or A third-party report will be completed and Years 3 and Third party survey PIU beneficiary monitoring publicly disclosed in Year 3 and Year 5, 5 findings disclosed, discussed at a public forum and a national discussed at open forum action plan produced. The national action and agreed actions included plan is implemented in the subsequent in national action plans year. All results where appropriate will be gender disaggregated by gender and disadvantaged schools. Percentage of schools that Sample based evaluation. Denominator is Years 3 and Third party survey PIU formulate School all project schools. Numerator is the 5 Improvement Plans (SIPs) project schools which have improvement reflecting feedback plan approved by the Board and show provided by parents and evidence of including community beneficiaries in project feedback. schools Percentage of project Sample based evaluation. Denominator is Years 2, 3, 4, Contracted TA. TA to PIU schools in which parents project schools surveyed. Numerator is the and 5 mobilize and train school report an improvement in number of project schools where parents community members. TA the openness and surveyed reported both openness and will be responsible for responsiveness of school responsiveness of (i) teachers and (ii) sampling, design of data teachers and management management. Openness and collection (instruments and responsiveness will be defined through a mechanism), and reporting. basket of actions by teachers and management. 27 ANNEX 2: DETAILED PROJECT DESCRIPTION Republic of Kazakhstan: Education Modernization Project (P153496) 1. The project is designed to support implementation of the Government of Kazakhstan’s State Program for Education and Science Development (SPESD). It will introduce a new curriculum based on a carefully selected set of complementary policy initiatives and institutional building in critical areas the government is not financing. Specifically, the project supports three complementary components. Component 1 focuses on ensuring quality in the design of key components of the reform, including monitoring and evaluation of the pilot of the new curriculum and its adaptation, the alignment of the assessment system, improved quality assurance of textbook-enhanced teacher practice and pre-service education, and strengthening accountability through school inspections. This would contribute to a strong foundation for the MOES to scale up the reforms using its own resources. Component 2 targets the delivery of performance-related resources to rural and disadvantaged schools, thus contributing to the government’s goal to improve equity of results. Component 3 evaluates reform initiatives supported by the project and finances communication and outreach activities and incremental operating costs. Table A2.1 summarizes how project interventions are aligned with the MOES’s SPESD and with global evidence on effective interventions to improve learning. 2. Table A2.1 How Project Interventions Improve Education Quality and Outcomes Alignment of Project with National Strategy and Global Evidence on What Works in Education State Program for Education and Global Evidence on What Works in Education Modernization Project Science Development (SPESD) Education (Interventions) (selected key actions) (Evidence) Phased introduction of revised Establishing ambitious, focused, Monitoring and evaluation of the curriculum and coherent education standards pilot of the new curriculum in that are shared across the system which new learning standards were and aligned with high-stakes adopted gateways and instructional systems. Address deficiencies of Unified Aligning assessment systems with National Test (UNT) new curriculum (C1.2) Enhance quality assurance Revision of textbook QA mechanism for textbooks and mechanism (C1.1) learning materials Increase teacher qualification at all Developing more capacity at the Enhanced teacher capacities and levels point of delivery; institutionalizing practice in disadvantaged schools improved instructional practice; through in-service training and pedagogical interventions that classroom observation. match teaching to student learning; individualized continuous teacher Focus on new standards and training associated with a pedagogy, meeting diverse learning particularly methodology; and needs, and customized training and ensuring classroom learning mentoring (C2.2) strategies. Increase the prestige of the teaching Attract, support, and retain quality Piloting a new model of pre-service profession, teacher pay, and quality teachers. teacher training and observation of at entry teaching practice (C1.3) 28 Transition to multilingual training Piloting novel approach to English language instruction (C1.3) Introduction of improved school Strengthen and support school Enhancing accountability through management and increased leadership; enhance links with school inspections (C1.4) proportion of schools with school parents and communities; boards complementing accountability to agents outside schools with accountability by professional colleagues and parents. Address disparities between high- Availability of quality learning C2.1 Providing (essential and low-performing schools resources, effectively used. instructional materials and multimedia equipment), and C2.2 Make multigrade schools better Investing resources where they can (building teacher capacities) functioning by addressing low- make the most difference. combined and targeting quality teachers, and poor provision disadvantaged schools (C2) of materials and teaching equipment Sources: a. David Evans, “OECD: Lessons from PISA for the United States; Meta-analysis on what works to improve learning,” World Bank, Washington, DC, 2015; “Overcoming School Failure: Policies that Work,” OECD, Paris, 2010; and PISA 2012 Results: What Makes Schools Successful? Resources, Policies and Practices (Volume IV), PISA, OECD Publishing, Paris, 2013. Component 1: Supporting system-wide improvement in primary and secondary education (Total component cost US$9.72 of which US$7.71 million financed by the loan) The objective of this component is to improve curricular standards, policies, and programs. 3. The MOES is in the initial stages of adapting and introducing the Nazarbayev Intellectual Schools (NIS) curriculum into mainstream schools, and plans to complete the process by 2021 (see box A2.1). The curriculum was designed in part to address shortcomings in the current curriculum that were reflected in the country’s performance on international assessments, and includes deepened content and a greater focus on higher-order cognitive skills. It was initially designed for the NIS, a relatively small number of well-funded schools for the talented and gifted. Three education institutions were created to support the NIS, including the Center for Education Programs (CEP), which focuses on curriculum; the Center of Excellence (CoE), which focuses on building teacher capacities; and the Center for Pedagogical Measurements. These institutions have worked closely with international technical partners to design the curriculum and build NIS capacities for its delivery, including the University of Cambridge (Faculty of Education, the University of Cambridge Press, and Cambridge International Examinations) and the University of Pennsylvania (Graduate School of Education). They have also been working with national counterpart institutions and pilot mainstream schools to adapt the national curriculum and prepare teachers for its introduction. 4. Successfully adapting such a curriculum for the mainstream public schools and rolling it out nationwide will need to address several challenges. These include ensuring that the adapted curriculum keeps the positive aspects of the NIS curriculum while catering to the wide range of abilities encountered in mainstream schools; working with teachers who are unfamiliar with teaching the new cognitive skills targeted by the curriculum, and are in some cases not highly motivated; schools that do not have all the necessary teaching and learning resources; and aligning the system of assessment with the revised curriculum. Further, the schedule of 29 introducing the revised curriculum is such that, in the year of introduction, students in grades 5, 7, and 10 will be seeing it for the first time; they will therefore require some form of supplementary preparation and support. Maintaining the introduction process at a satisfactory level of quality on this schedule will require strong technical oversight and support to ensure that these various challenges are met on a timely and adequate basis. 5. Some national capacity has already been created to adapt, introduce, and start the pilot of the new curriculum, as noted above, but continued technical support is essential to ensure that reforms are well designed, tested, and scaled up. Therefore, the first component is largely devoted to providing technical assistance to support the design, introduction, and evaluation of reforms in four areas: monitoring and evaluation of the pilot of the new curriculum; aligning assessment systems; building teacher capacities; and managing the education system in a manner that promotes equity, efficiency, and performance. To ensure support for the policy change, the project would engage stakeholders through (a) awareness-raising activities engaging the research community and interested stakeholders in proactive explanation of project interventions; (b) annual multistakeholder forums designed to create transparency on the intent and progress of the reform and encourage feedback from civil society; and (c) building skills of Boards of Trustees. 30 Box A2.1 Introducing the Reformed Curriculum Working closely with the University of Cambridge and other international partners, the Center for Education Programs (CEP), a pedagogical institute attached to the NIS, created a 12-year curriculum for the NIS schools, which have been using the curriculum since 2010. Innovative aspects of the curriculum include a competency- based design, focusing on cognitive skills that have not been traditionally targeted (for example, that analytically explain why something happened, compare and contrast, interpret data, evaluate the quality of something); deeper-level content; active learning pedagogy; the use of English as a language of instruction for some subjects in the highest grades; clarifying indicators to measure mastery of competencies; strengthening formative assessment; and using revised summative instruments to reflect the curriculum, including testing for nontraditional cognitive skills. The MOES is adapting this curriculum for mainstream public schools and plans to introduce the adapted curriculum nationwide over the next six years. For any given grade, the process of introduction is divided into four main steps, with each step lasting roughly one year: adaptation of the curricular standards, plans and programs, and teaching-learning resources; piloting of the revised curriculum in a sample of schools; further revision and finalization of the curriculum, including teaching-learning resources, based on the results of the pilot; and introduction into mainstream schools. The sample of pilot schools is sized and distributed to be representative of the different profiles of school and to enable local education authorities across the country to observe the new curriculum in practice before its introduction. Pre-primary Year/Grade year 1 2 3 4 5 6 7 8 9 10 11 2016-2017 I 2017-2018 I I I 2018-2019 I I I I 2019-2020 I The current 11-year-old program I Introduction of new curriculum Subcomponent 1.1: Curriculum modernization 6. The project will finance technical assistance to evaluate the pilot of the new curriculum and derive lessons learned from the evaluation and incorporate them into the strategy for introduction in mainstream schools before its national rollout. The evaluation would ensure rigor and strike a balance between the more challenging environments in which mainstream schools operate, and the high standards and the new cognitive skills targeted by the NIS curriculum. It will also closely coordinate with the work that will be done with project support (under Subcomponent 1.2) to align national assessment systems with the adapted curriculum, to ensure that the curricular and assessment reforms are mutually reinforcing. The high-level technical assistance to be contracted under Component 3.2 will also monitor these facets of the adaptation work. 7. The technical assistance will also support monitoring of the pilot including ensuring that all collected data are properly analyzed and discussed to distill and incorporate findings and recommendations. The assistance will support an assessment of the adapted curriculum in representative classroom settings, including any revisions made in light of the pilot findings prior 31 to the national rollout, ensuring that the process is properly planned, budgeted, coordinated, and overseen. 8. The project will finance certain operating costs associated with monitoring visits to schools and stakeholder discussions. 9. Textbooks are an integral part of the curriculum, and textbook publishers will be kept abreast of revisions to content, competencies, and pedagogy so that they can produce on a timely basis revised manuscripts for consideration by the approvals and selection process. The process of textbook production and selection is advanced for the region, in that it includes elements of competition among publishers, independent quality review, approval of a list of eligible textbooks (rather than mandating one book), and local-level choice. Nonetheless, there is a national debate on how the system might be improved in light of perceived shortcomings in the quality of some textbooks, particularly in the Kazakh language. One area of weakness that has been identified is the methodology for appraising and selecting textbooks. 10. Therefore, this subcomponent will finance technical assistance to update textbook standards to be in line with the new curriculum, and to reinforce the QA mechanism. This includes the elaboration of new standards and specifications for print and electronic formats for textbooks of science, mathematics, social sciences, humanities, and culture by primary, lower, and upper-secondary education levels. Further, the evaluative criteria, and the method of their use, will be redesigned, and there will be capacity building for the technical experts for assessing the quality of textbooks. Subcomponent 1.2: Alignment assessment systems with the new curriculum 11. The assessment system includes “internal” and “external” assessments. The external assessments are summative in character and implemented through two main instruments: the EALA and the UNT. The EALA is administered on a sample basis at grades 4 and 9 to provide decision makers and other stakeholders with valid and reliable data on how well schools are imparting knowledge to students. The UNT is currently taken by all students at grade 11 and serves to provide a statement of achievement at school leaving and as the basis on which entrance to universities is decided and financial assistance allocated. 12. The design and administration of external assessments need reform. With respect to design, both the EALA and the UNT rely heavily on multiple choice and closed questions, and emphasize rote reproduction of facts. They do not test higher-order cognitive skills emphasized by the reformed curriculum, or the kinds of abilities required by universities and sought by employers. Given the need for tests that measure mastery of the new curriculum, and that teachers teach to the test, these instruments will need to be realigned. 13. With respect to test development and administration, a recent audit recommended various improvements, including the elaboration of new standards of testing, and the formal qualification of individuals as test administrators. One underlying difficulty with the external assessments is the logistical stress put on the system by continuing to rely on paper-based instruments. 14. Thus, the MOES is exploring options to revise the external instruments. These include the use of multiple, differently formatted tests at school leaving and the incorporation of more 32 items testing the newly targeted cognitive skills. To improve test development and administration, the NTC plans to clarify standards, build administrative capacities, and draw a clearer institutional division of responsibility between test design and administration. 15. In support of these efforts, the project will finance technical assistance in three main areas. 16. First, the technical assistance will build the skills of MOES assessment specialists to design items and tests with appropriate technology that measures new curricular content and skills. The project assistance would work closely with international partners who have been working with the MOES. 17. The project will support the design of new testing standards for two major examinations, namely EALA and UNT, to measure student progress and improvement and to certify learning achievements at the end of the secondary education cycle. It would also build the capacities of the MOES to more effectively analyze assessment data to produce actionable information for teachers, administrators, local government, parents, and other stakeholders. This would entail, among other things, the development and use of cut scores and performance levels; multivariate, multilevel, and subgroup analyses; and trends over time. More effective reports would be developed and tailored to different stakeholders, including the use of videos, the media, workshops, and the internet. The project will consult with policy makers and other stakeholders to develop constructive ways to communicate results. 18. Third, the project would support establishment of a test bank with test questions to measure the new curriculum content and skills and capacity building of MOES staff and test developers in the administration, maintenance, and upgrading of the test bank. Subcomponent 1.3: Modernizing pedagogical education and practice 19. All primary and secondary teachers require new knowledge and skills to deliver the revised curriculum. An analysis of PISA results also indicates that teaching practices and student learning strategies can be significantly improved. For instance, the PISA 2009 reading results showed that a substantial part of the difference in scores between Kazakhstan and the OECD countries could be explained by relatively poor learning strategies (especially the overreliance on memorization strategies) and teaching practices. 20. The MOES recognizes the need to reform pre-service training, the design of which is at odds with international practice, and that the teaching profession, and particularly teacher preparation, does not always attract good-quality candidates and inefficiently produces graduates who must then be retrained through in-service training. While pre-service training is relatively strong on subject knowledge, its theoretical components require revision and better links to practice. Further, the training needs to be redesigned to better incorporate competency-based approaches, raise the quality of entrants, provide students with experience to enable them as newly qualified teachers to be classroom ready, and build their capacities to enable them to reflect upon research and modify their practice as part of a career-long approach to improving their techniques. 33 21. Therefore, the project will support the piloting of a teacher training model at pre-service training institutions. During the pilot, teachers would receive pedagogical training, practice teaching at partner schools where experienced teachers provide mentoring support, and undertake research on their professional practice. The project will finance technical assistance to develop the curriculum for the pilot training, and build capacities at the training institutions to deliver the curriculum. Following that, a new model of cooperation between secondary schools and pre-service universities on teacher training will be developed. The project will also support coordination of the pilot to technically oversee and coordinate the pilot, working in close collaboration with international partnerships; will finance the supply of seven pre-service universities with modern multimedia equipment to enable teaching for science faculty students in English; and will also finance modernization of university facilities to deliver the new pilot program, where education digital resources can be used to improve teaching practices. 22. The reformed curriculum also foresees the use of English as the language of instruction for science subjects at the highest grades. The MOES is strongly committed to this bold yet challenging initiative. English language instruction is already being piloted in 200 schools and the NIS schools. However, many challenges must be addressed, including the need to clarify the standards of English proficiency that will be required to enable English language instruction; many upper secondary students are not sufficiently proficient in English to be able to follow English language instruction, relatively few teachers can teach in English without extensive further training, and there is limited capacity in the country to train teachers to provide English language instruction. Therefore, this component will finance technical assistance to pilot a pre- service training program for university staff to enable them to teach one subject using English as the language of instruction. The technical assistance will cover English language instruction and ongoing support to the teachers. 23. Global research evidence suggests pedagogy customized to student needs matters most in improving student learning results. While the government will finance in-service training with its budget to prepare all teachers for implementing the new curriculum, the project will support the design and or adaptation of an evaluation instrument to observe teacher pedagogical practice in classroom. The result of the observation will inform strategy for professional development of teachers and enhance their teaching effectiveness. Subcomponent 1.4: Enhancing accountability through school inspections 24. The project will enhance school accountability through improved school inspection practice and capacity building for school-based management, planning, and oversight. The technical assistance will support the design of training modules for local education authorities, school management, and Boards of Trustees; and build capacities to deliver training. The training program will cover collaboration with local communities to elaborate school development plans and budget for their implementation. 25. The quality of school performance would also be encouraged by activities that enhance the capacities of the “demand side,” that is, the students and parents as users of the school. The project would support community awareness-raising on the benefits of education; the rights, roles, and responsibilities of parents and students in enabling quality education services; and the national strategy and targets to improve education. Further, the project would build community 34 capacities to contribute to and monitor school development plans linked to the budgeting process, and to foster local debate on school performance relative to other schools and national targets. These activities would be linked to the promotion of such transparency measures as the public display of information on school budgets and performance. In support of ongoing reforms to strengthen the school inspection service, the Project would also finance the capacity building of teachers, School Directors, and community representatives in self-evaluation; and of the Committee for Control in Education and Science on how to use evaluative criteria during the attestation process. Training of methodology developers and school principals on self-evaluation methods will also be part of the overall process of improvement of transparency and integrity of the school inspection system. Component 2: Supporting rural and disadvantaged schools to reduce disparities in learning results (Total component cost US$62.04 million of which US$54.78 million financed by the loan) The objective of this component is to improve student learning outcomes in 5,400 disadvantaged schools, through improved access to essential learning resources and enhanced teacher capacity and school leadership. 26. There is a large gap in student learning between the best- and worst-performing schools. For instance, the PISA 2012 differences in math and reading scores between students in the highest and lowest quintiles of socioeconomic status were 60 points and 73 points, respectively (equivalent to 1.5 to 2 years of education). However, an analysis of changes in results between PISA 2009 and 2012 shows that improvements in school resources, governance, and management practices, and teacher and principal effectiveness can have a positive impact on the performance of low achievers. Significant disparities remain in the resources available to schools in terms of teaching-learning materials, teacher capacities, and pedagogical leadership. Therefore, remedying these disparities should work to reduce learning gaps. One category of schools that is particularly under-resourced is rural schools of multi-grades, which constitute 45 percent of all schools. 27. This component will work to remedy some of those gaps by targeting essential learning resources for rural and disadvantaged schools selected from the bottom 40 percent of districts located in demographically stable communities. The selection criteria, their weighting, and the mechanism of their application to generate the list of beneficiary schools will be further elaborated in the Project Operations Manual (POM). Subcomponent 2.1: Empowering rural and disadvantaged schools with essential instructional materials and equipment 28. This subcomponent will finance the provision of an essential package of instructional materials and multimedia equipment for about 5,400 targeted schools. The package will support the delivery of the revised curriculum to fill priority gaps in the availability of essential resources, enable access to digital learning content, and provide more practice exercises to master new content and skills. 35 29. The stages of community engagement in these efforts will include (a) participation of school users (parents/students) and local community members in the identification of local needs and gaps, prioritization, and the decision-making processes over learning resources. Participation of users and community members will be ensured through public meetings and consultations; and (b) annual beneficiary monitoring of the equipment/materials/textbooks allocated and received. A study to assess the above aspects will be included in the midterm and final independent monitoring. 36 Subcomponent 2.2: Enhanced pedagogical and management capacities 30. This subcomponent will support building the skills of School Directors for school leadership and working with Boards of Trustees to develop and monitor a Project-specific School Improvement Plan. It will also finance the design and delivery of an enhanced package of in-service teacher training and pedagogical support. The training content will focus on target issues particularly affecting rural and disadvantaged schools such as identifying and remedying learning gaps of underperforming students, with particular attention to improvement of boy’s reading, teaching in a multigrade environment, practicing classroom-based assessment, and improving student study techniques. Teachers would be consulted about designing the training content and about training delivery, and would be provided with pedagogical handbooks. To strengthen pedagogical support networks for these schools, the project will also finance the capacity building of methodologists working with these schools, provide teachers with self- learning opportunities, and build the School Director’s capacity in pedagogical leadership.8 Teachers and Directors would also receive opportunities to participate in capacity-building sessions subsequent to the trainings, focused on deepening their knowledge and skills pertaining to the delivery of the revised curriculum. Subcomponent 2.3: Supporting inclusive education 31. This subcomponent will support the design of new educational programs for students with special education needs for social and academic integration. The project will finance technical assistance to support the MOES to design a model in line with the National Framework on Inclusive Education. It will support Inclusive Education Resource Centers and partner schools to operationalize the model and develop methodological guidance. It will also provide technical assistance in capacity building of teachers and local akimat staff to implement new programs. Component 3: Supporting citizen engagement, monitoring and evaluation, and implementation (Total component cost US$5.24 million of which US$4.51 million financed by the loan) The objective of this component is to engage and support stakeholder participation, monitor and evaluate project implementation and results, and support project implementation. Subcomponent 3.1: Stakeholder participation and engagement 32. This subcomponent will support stakeholder participation and awareness-raising through consultation and communications, including the establishment of a grievance redress system and third-party monitoring of project implementation. A communications strategy will be implemented that engages and informs stakeholders about the project and key reforms including the results of the pilot of the revised curriculum and strengthened quality assurance for textbooks. The project will support annual public forums designed to create transparency on the 8 Combining and following up training with aligned methodological support has been found in the region to be effective in bringing about changes in teaching-learning practices. See, for example, “Implementation Completion and Results Report (IBRD-47260),” E-Learning Support Project in Russia, World Bank, Washington, DC, December 30, 2008. 37 intent and progress of the reform, and encourage feedback from interested members of civil society. Subcomponent 3.2: Monitoring and evaluation of pilot initiatives 33. This subcomponent will finance the design and execution of the monitoring and evaluation of reform initiatives supported by the project as well as an overall project evaluation. 34. The success of the rollout of the revised curriculum relies partly on the in-service teacher training having a positive impact on teaching-learning practices, making it imperative to monitor training effectiveness. Therefore, the project will finance technical assistance to adapt an internationally validated and reliable instrument to observe teaching-learning practices to assess the effectiveness of the in-service training.9 The instrument will be designed to measure changes in the quality of teaching-learning practices targeted by the training to support the effective delivery of the new curriculum; a section will be included to observe practices to foster reading skills, including for boys. The technical assistance will also build national capacities to administer the instrument to a sample of classrooms at baseline and subsequent to training. The sample will be structured to enable a disaggregated analysis of teachers who receive the training used as part of the national rollout and teachers from disadvantaged schools who receive the augmented training under Subcomponent 2.2. 35. The project will also finance technical assistance to design and execute evaluations for other reform initiatives and pilots supported by the project. These include the new QA mechanisms for textbook approval and selection (Subcomponent 1.1), the new testing standards (Subcomponent 1.2), using English as the language of instruction in science and mathematics (Subcomponent 1.1), the new pre-service teacher training model (Subcomponent 1.3), the new system of self-evaluation (Subcomponent 1.4), and the design and/or adaptation of an evaluation instrument to observe teacher pedagogical practice in the classroom. The project will also finance an evaluation of the curriculum rollout, and an overall evaluation of the project. In each case, the findings and recommendations will be discussed by stakeholders with a view to improving the national standards, policies, and practices. 36. Further, the project will finance the design and execution of two grade 4 learning achievement surveys in years 3 (baseline) and 5 (impact assessment) in Component 2 schools. The instruments will be aligned with the revised curriculum. Besides being of an age at which it is relatively easy to reliably measure learning achievements, grade 4 students would be the only cohort to have been exposed to the new curriculum for three or more years, a sufficient amount of time for the curriculum reforms (if successfully delivered) to show up in learning achievement measurement.10 The baseline survey will be conducted in year 3, because it will take that long for 9 For example, the Classroom Assessment Scoring System (CLASS) developed by University of Virginia,or similar. The MOES, PIU, and World Bank will review different instruments prior to selection, and select an instrument that is valid, reliable, recognized, and that captures the quality dimension of teaching-learning practices targeted for improvement by the project. 10 The indicator will measure the effectiveness of the new curriculum and its delivery in bringing about higher learning outcomes, particularly in relation to the new, nontraditional cognitive skills targeted by the new curriculum. That is, using instruments constructed with equated items, the baseline assessment will test students following the old curriculum, while the follow-up assessment will test students who have been following the new curriculum. Given the new curriculum’s rollout schedule, it is not possible within the project’s time frame to test two different 38 the activities under Subcomponent 1.2 (capacity building of NTC staff and specialists) to enable the NTC to design and administer a reformed instrument. The findings will, combined with other evaluation exercises, be used to evaluate the project and develop policy for further support to disadvantaged schools. 37. The successful rollout of the revised curriculum to mainstream schools will require that several activities be executed at a satisfactory level of quality and in a coordinated fashion, including evaluation of piloting and further adaptation, the development of teaching-learning resources, teacher professional development, and the alignment of assessment systems. Therefore, the project will finance a Technical Advisory Panel of high-level technical assistance to periodically review the quality of each of the main activities and their links with each other and advise senior officials of the MOES on the rollout. Subcomponent 3.3: Project implementation 38. This subcomponent will finance project operating costs, including translation, interpretation, equipment, and supervision costs, and PIU staff salaries to ensure successful implementation of the project. 39. Table A2.2 presents the sequence and interrelationships of the proposed activities. grade 4 cohorts that follow the revised curriculum, with one cohort being tested prior to or at the beginning of project interventions, and the other cohort being tested after project interventions have been in effect sufficiently long to expect learning impacts. 39 Table A2.2 Project Activities Rollout Comp. 1.1: Curriculum modernization 2018 2019 2020 2021 2022 System-wide improvement Curriculum revision Monitoring of the pilot Evaluation of the pilot results Finalization plans/programs Textbooks strengthening Updating textbooks standards/specifications Strengthening QA mechanism Project school level Adaptation of new education plans/programs Comp. 1.2: Alignment assessment systems with the new curriculum 2018 2019 2020 2021 2022 System-wide Strengthening capacity to design instruments in line with new curriculum Developing new standards for test development & administration Roll-out of revised instruments EALA EALA EALA/UNT Project school level Administration of revised EALA Administration of revised UNT Comp. 1.3: Modernizing pedagogical education & practice 2018 2019 2020 2021 2022 System-wide Piloting a new model of pre-service teacher training & observation of practice teaching Design Implementation Implementation Implementation Evaluation Piloting of English-language instruction Design Implementation Implementation Implementation Evaluation Strengthening the capacity of selected pedagogical universities Evaluation of regular in-service training 40 Comp. 1.4: Enhancing accountability through school inspections 2018 2019 2020 2021 2022 System-wide Capacity building of School Directors, Boards, Local Authorities Training on schools-based management and inspections Project school level School Directors, Board Members and Community stakeholders are trained in budgeting, school development planning & self-evaluation Budgets are formed and monitored by Boards of Trustees with community consultation Comp. 2.1: Instructional materials and multimedia equipment for rural & disadvantaged schools 2018 2019 2020 2021 2022 Teaching-learning resources Identification of gaps Procurement and delivery Community mobilization and oversight Resource package for schools Contact with communities and ID of project schools Comp. 2.2 Enhanced pedagogical & management capacities 2018 2019 2020 2021 2022 Training and support for improved pedagogy Identification of needs and design of training Training for teachers & directors Follow-up methodological support Assessment of teaching-learning practices Design and baseline Follow-up Learning assessments Teachers practice formative assessment Grade 4 EALA Baseline Follow-up 41 Comp. 2.3 Supporting inclusive education 2018 2019 2020 2021 2022 Design a model & methodological guidance Implementation Implementation Implementation Implementation Evaluation Comp. 3.1: Stakeholder participation & engagement 2018 2019 2020 2021 2022 Civil society evaluation of project 1st round 2nd round Awareness-raising and communications Comp. 3.2: M&E of pilot initiatives 2018 2019 2020 2021 2022 Evaluation of the new pilot curriculum Evaluation of the pilot results for a pre-service teacher education model Evaluation of textbooks QA mechanism Evaluation of the overall project Comp. 3.3: Project implementation 2018 2019 2020 2021 2022 Implementation support 42 Mechanism Demand-Side Support Supply-Side Support Responsibility Result National-Level Action Annual a. Awareness-raising b. Multistakeholder forums: PIU/MOES leads process with National campaign: An An annual event designed to consultancy support Public awareness-raising create transparency about the Consultation campaign at the start of intent and progress of the Forum project to engage the reform and encourage research community and feedback from interested For broad interested stakeholders in members of civil society Third-party and/or beneficiary monitoring stakeholder proactive explanation of (Subcomponent 3.1). findings disclosed, discussed at open forum, engagement in national education policy and agreed actions included in national key policy and reform. c. Forum report: An annual action plans. reforms. report published to create transparency about dialogue and actions. Third-Party d. Independent PIU/MOES Monitoring monitoring: Support for Contracts national NGO an independent For an organization to provide independent feedback on the progress feedback of each component of the mechanism that reform—will be compiles disclosed at Public national and Consultation Forum local feedback, (above). and provides rolling information for the annual forums. Grievance e. Grievance Redress f. Grievance Redress PIU/MOES Number of grievances received and Redress Mechanism outreach: Mechanism percentage resolved. Mechanism Outreach / awareness operationalization: (GRM) building to stakeholders Incorporated within the (Complaints received and resolved will be and communities to MOES’s existing complaints- tracked, published, and reported in the 43 To establish a understand role and handling system. annual national public consultation.) national- and expectations of GRM.  Strengthening of MOES local-level complaints mechanism complaints-  System for generating and handling system compiling local-level GRMs in the MOES,  Annual proactive feedback compiles local- request through hotlines or level using mobile technology, to complaints reach out to the beneficiaries system, and of the project/reform. reviews with a view to changes in policy and practice. 44 Mechanism/ Demand-side support Supply-side support Responsibility Result* Goal Local-Level Action Community a. Joint awareness-raising and capacity-building Akimat/Oblast organizes and Planning campaign on rights, standards, performance, contracts demand- and supply- and targets, budgets for school management and local side consultancy support. Monitoring communities/stakeholders. b. Needs identification c. Support to teacher Akimat/Oblast organizes and To improve and planning: capacity building to develop contracts demand- and supply- school Participation of school the skills and techniques to side consultancy support to performance users (parents/students) engage with communities. mobilize and train school Percentage of schools that formulate SIPs through and local community community members. that reflect feedback provided by school feedback and members in the users/beneficiaries. dialogue with identification of local teachers and needs, gaps and school prioritization, school management. development planning/ To support budgeting in the school community improvement plan. awareness- raising on the d. Input monitoring: Akimat/Oblast organizes and benefits of Annual beneficiary contracts demand- and supply- education, monitoring of the side consultancy support to rights, roles equipment/materials/ mobilize and train school and, and the textbooks allocated and community members. national received. strategy and targets to improve education. e. Performance f. Performance monitoring: Akimat/Oblast organizes and Percentage of schools in which parents monitoring (community): School report card. contracts demand- and supply- report an improvement in the openness and Beneficiary monitoring of side consultancy support to responsiveness of teachers and management national standards and mobilize and train school to beneficiary feedback. school improvement plan community members. as beneficiary input to school report card (for example, ratio of student access to science equipment/material, textbooks per student). 45 School g. Build the capacity of Akimat/Oblast. Boards of akimat officials and Boards Training firm. Number of regions that (a) publicly display Trustees of Trustees. and discuss school budget information (revenues and expenditures) in meetings To enhance with citizens, and (b) publicly report on accountability meeting discussions and actions to be taken. for education system management. Note: *Community-level data included in EMIS presentation at annual national forum. 46 Annex 3: Implementation Arrangements KAZAKHSTAN: Education Modernization Project Project Institutional and Implementation Arrangements 1. The MOES will be the implementing agency for the project through the Project Implementation Unit (PIU). 2. The project will be implemented with the participation of the relevant Ministry of Education and Science (MOES) departments and affiliated institutions. The detailed responsibility of the participating entities will be specified in the Project Operations Manual (POM). All institutions are already involved in the process of modernization of secondary education in the country and, through the project, will continue to contribute to the reforms that the project supports. The MOES will rely on the technical capacity of the existing stakeholders involved in the modernization of secondary education. 3. The MOES will have the overall responsibility for project implementation through the PIU, which will be established to take an implementation role and carry out key functions such as fiduciary, monitoring and evaluation, and reporting. The MOES will hire consultants to fill in the key positions of the PIU. The PIU will be led by a director and consist of a financial management (FM) specialist, an accountant, one or more procurement specialists as needed, a monitoring and evaluation specialist, coordinators for components, and administrative staff for support and interpretation. The main responsibilities of the PIU will be to:  Ensure achievement of the project’s intermediate results  Administer the project in accordance with the POM on a daily basis  Develop terms of reference, purchasing contracts, and other procurement-related documents, organize the related selection and procurement processes in collaboration with the relevant department of the MOES and oversee delivery of project inputs  Draft annual work plans and budgets and monitor their execution  Ensure that the project monitoring and evaluation framework is operational, and all monitoring and evaluation activities are properly undertaken  Prepare regular progress reports for submission to the MOES, and any other reports on specific project-related issues as requested by the MOES  Support FM arrangements (external audit, internal audit, accounting, and so forth)  Ensure that all fiduciary and citizen engagement requirements of the Bank are met  Report any major issues to the MOES and, if required, recommend changes to the POM  Coordinate with all involved stakeholders  Coordinate awareness-raising campaigns and support the grievance redress system 4. The relevant department of the MOES will:  Oversee implementation progress of the project, in accordance with the POM 47  Oversee progress on the results indicators and advise on and facilitate, as needed, corrective actions to ensure the project’s attainment of the PDO  Review and approve the project’s annual work plans and budgets  Endorse any major changes in the implementation arrangements in consultation with the PIU  Oversee compliance with fiduciary and citizen engagement requirements of the Bank. 5. A Technical Advisory Panel comprising selected international and national experts will be established to ensure the maintenance of satisfactory standards during project implementation. The panel will meet twice a year to review the progress made and provide advice to the MOES on remedial actions, if needed, to ensure adherence to policy reform based on best international practices. The panel will be led by a chairperson and consist of prestigious international and Kazakhstani experts who have in-depth knowledge of global trends in teacher policy, curriculum, assessment, and school reforms in Kazakhstan. 6. If deemed necessary, a Project Working Group will be established to act as a national interagency coordinating body. The Working Group, chaired by the vice-minister of the Ministry of Education, will include representatives of key stakeholder institutions with responsibilities to:  Conduct of semiannual meetings to address implementation issues and ensure achievement of project objectives  Review and recommend restructuring during project implementation  Coordinate resolution of crosscutting issues and disputes. 7. Financial Management, Disbursements, and Procurement Financial Management 8. The MOES, with support of the PIU, will be responsible for implementing the project FM function including the flow of funds, planning and budgeting, accounting, financial reporting, internal controls, and auditing. Under earlier projects, the MOES established FM arrangements that are overall acceptable and have been assessed regularly during FM implementation support visits, most recently in September 2015. However, to bring the FM arrangements into fully satisfactory status for the proposed project, the MOES will need to implement the actions outlined in Table A3.1 Table A3.1 Action Plan for Project FM Arrangements Actions Responsible Entity Completion Date 1. FM consultant is contracted by the MOES as Within 30 days from MOES part of the PIU effectiveness 2. Develop a Financial Management Manual MOES By effectiveness (FMM) as part of the POM 3. Develop and launch a module to the automated accounting software used by MOES. The Within 45 days after module has the capacity to generate Interim MOES effectiveness Financial Reports (IFRs) and capture the project’s accounts. 48 9. Implementation arrangements. The MOES Finance Department, with support from the PIU, will be responsible for preparing annual budgets for the project, based on procurement plans and in line with the FMM and the budgeting procedures of the Government of Kazakhstan. These budgets will form the basis for allocating funds to project activities. The budgets may be prepared in accordance with the IFRs format (disbursement categories, components and activities, account codes, broken down by quarter). Annual budgets should be agreed with the Bank. Approved annual budgets should then be entered in the accounting system and used for periodic comparison with actual results as part of the interim financial reporting. 10. The PIU will work closely with the staff of the Finance Department to implement the fiduciary functions of the project. Finance Department staff have experience in the Bank’s FM and disbursement procedures, gained during implementation of existing projects and from Bank training on FM and disbursement procedures. 11. Accounting. The MOES uses 1 C accounting software, which meets the Bank’s accounting and reporting requirements, as well as requirements of Kazakh legislation. However, a module will need to be developed for this software with the capacity to generate IFRs and capture the proposed project’s accounts. 12. The MOES follows the accounting policies and procedures described in the orders and regulations developed by the Ministry of Finance for budget organizations. In addition, the specific FM procedures, including internal controls, will be documented in the FMM of the POM. The FMM should document key internal control mechanisms to be followed by staff in the application and use of project funds, with specific focus on ensuring completeness of accounting transactions, reliability of accounting data, safeguarding of project assets, including safe custody of cash and other assets, proper monitoring of contracts, proper authorization and documentation of all project expenditures, and full accountability of project funds. The FMM will reflect the project structure that allows for adequate segregation of functions, job descriptions for staff with different authority levels, and the flow of funds to support project activities, including proper management of the disbursement function, contracts management, and documentation flow. The manual will also describe procedures for regular financial reporting to ensure close monitoring of project activities. 13. Reporting. Project-management-oriented unaudited IFRs will be prepared under the project. The PIU will produce a full set of IFRs every quarter throughout the life of the project. The format of IFRs will be agreed before negotiations and incorporated into the FMM. These financial reports will be submitted to the Bank within 45 days of the end of each calendar quarter. The first set of quarterly IFRs will be submitted after the end of the first full semester following the initial disbursement. 14. External audit. The audit of this project will be conducted by independent private auditors acceptable to the Bank, using International Standards on Auditing. The auditor will be engaged on standard terms of reference acceptable to the Bank. Audit of the financial statements under the project will be included in the bulk audit of the whole portfolio of donor-financed projects in Kazakhstan. Procurement of such an audit is the responsibility of the Ministry of Finance. The cost of the audit is covered by the funds of the republican budget outside the 49 project’s costs. Sample audit terms of reference will be agreed with the Bank and attached to the FMM. The annual audited project financial statements will be provided to the Bank within six months of the end of each fiscal year and for the entire project within six months of the closing of the project. If the period from the date of effectiveness of the loan to the end of the borrower’s fiscal year is no more than six months, the first audit report may cover financial statements for the period from effectiveness to the end of the second fiscal year. The borrower is required to disclose the audit reports for the project within one month of receipt from the auditors, by posting the reports on their website. Following the Bank’s formal receipt of these reports from the borrower, the Bank will make them publicly available according to the Bank’s Policy on Access to Information. Table A3.2 identifies the audit reports that will be required to be submitted by the MOES, and the due date for submission. Table A3.2 Audit Reports for the Project Audit Report Due Date Project financial statements, which include: Within six months of the end of each fiscal year, and within six months of the closing of  Sources and uses of funds the project  Uses of funds by project activity  Designated account reconciliation statement  Statements of expenses  Withdrawal schedule  Notes to the financial statements. Disbursements 15. MOES staff have knowledge of and experience with the Bank’s disbursement procedures. The MOES is using an e-Disbursement facility that will be used for the proposed project, as well. 16. The MOES will open a designated account specifically for this project, in a financial institution agreed with the Bank. The project account will be opened in the Treasury for transfer of government counterpart funding. Project funds will flow from (a) the Bank, either through the designated account, which will be replenished based on full documentation or using statements of expenses, or by using the direct payment method or the special commitment. Further details on this are provided in the Disbursement Letter; and (b) counterpart funds, which will flow through the Treasury. Both Bank and government funds will be managed by the MOES. Withdrawal applications for replenishments of the designated account will be sent to the Bank quarterly. Procurement 17. Procurement of goods and non-consulting services for the proposed project will be carried out in accordance with the Bank’s “Guidelines: Procurement of Goods, Works, and Non- Consulting Services under IBRD Loans and IDA Credits & Grants by World Bank Borrowers,” dated January 2011 and revised July 2014 (Procurement Guidelines), and procurement of consultant services will be carried out in accordance with the Bank’s “Guidelines: Selection and Employment of Consultants under IBRD Loans and IDA Credits & Grants by World Bank Borrowers,” dated January 2011 and revised July 2014 (Consultant Guidelines). All procurement will also be carried out under the provisions stipulated in the Loan Agreement. The Bank’s 50 “Guidelines on Preventing and Combating Fraud and Corruption in Projects Financed by IBRD Loans and IDA Credits and Grants,” dated October 15, 2006 and revised in January 2011 (Anti- Corruption Guidelines), will apply to this project. A General Procurement Notice shall be published for the procurements under the project by the MOES. 18. Procurement activities will be carried out by the PIU. The risk assessment rating for the entire project will be done through the Procurement Risk Assessment and Management System. Initial identified risks and proposed mitigation measures are described in Table A3.3 at the end of this annex. The procurement risk is rated high. 19. The procurement plan covering the first 18 months of the project period is under preparation by the MOES. The procurement plan will be updated at least once per calendar year, and each update will be subject to the Bank’s review and approval. The initial procurement plan together with the subsequent updates will be published on the Bank’s external website in line with the requirements of the Bank’s guidelines. A General Procurement Notice covering the project procurement activities will be prepared. Specific procurement notices will be published for all International Competitive Bidding (ICB) and National Competitive Bidding (NCB) procurement, and for all consulting services contracts as required under the respective guidelines. 20. Procurement of goods. Goods’ contracts above US$2 million equivalent will be procured under ICB procedures using the Bank’s standard bidding documents for procurement of goods. The NCB method will be applicable for procurement of goods contracts with an estimated budget of less than US$2 million equivalent. The Bank’s Europe and Central Asia (ECA) sample NCB bidding documents shall be used in accordance with the NCB conditions set forth in the Loan Agreement. Goods contracts with an estimated budget of less than US$100,000 equivalent may be procured using shopping procedures on the basis of at least three written price quotes obtained from qualified suppliers. The list of suppliers to be invited to submit quotes should be defined by an evaluation (tender) committee. 21. Selection of consultants. The methods for selection of consultants will include Quality- and Cost-Based Selection, Quality-Based Selection, Selection under a Fixed Budget, Least-Cost Selection (LCS), Selection based on Consultants’ Qualifications (up to US$300,000 equivalent) (CQS), Single-Source Selection (SSS) in compliance with paragraph 3.8 of the Bank’s Consultant Guidelines, and Individual Consultants. Contracts estimated to cost above US$300,000 equivalent will be advertised through United Nations Development Business (UNDB), the Bank’s website, local media (one newspaper of national circulation or the official gazette), and MOES’s website. Shortlists of consultants for services estimated to cost less than US$500,000 equivalent per contract may be composed entirely of national consultants under the provisions of paragraph 2.7 of the Bank’s Consultant Guidelines. 22. Operating costs. The expenses of the PIU will include communications, translation/interpretation, bank charges, office supplies, cost of advertisements, mail, and business trip expenses of government officials and other experts. Such costs will be financed by the project, based on the annual budget prior reviewed and agreed by the Bank. Purchases will be carried out in accordance with the implementing agency’s internal administrative procedures. Operating costs will not include salaries or allowances of civil servants 51 23. Training and study tours. Training and study tours will be carried out based on the annual training/study tours program. The budget is to be prepared by the PIU and reviewed and agreed by the Bank. The institutions for training/study tours will be selected considering the availability of such services, duration of training/study tour, and reasonableness of cost. 24. Governance and Anticorruption Action Plan. The project will follow the Bank Group’s anticorruption policies as set forth in the Guidelines on Preventing and Combating Fraud and Corruption in Projects Financed by IBRD Loans and IDA Credits and Grants (current edition). The Bank team will maintain close oversight and carry out prior review of all major contracts in accordance with the thresholds that will be regularly reviewed and adjusted as needed in the procurement plan. The following measures will be carried out to mitigate corruption risk. 25. Publication of advertisements and contracts. All publications for advertisements and contract awards, including the results of the awards, will be done in accordance with the Procurement Guidelines and published in the Bank Client Connection system and on external websites, that is, UNDB and Bank websites. 26. Debarred firms. Appropriate attention will be given to ensuring that debarred firms or individuals (to be verified from the Bank’s external website) are not given opportunities to compete for Bank-financed contracts. 27. Temporarily suspended firms. Appropriate attention will be given to ensuring that temporarily suspended firms or individuals (to be verified through Client Connection) are not given opportunities to compete for Bank-financed contracts. 28. Complaints. All complaints by bidders will be diligently addressed and monitored in consultation with the Bank. 29. Monitoring of contract awards. All contracts are required to be signed within the validity of the bids/proposals and, in case of prior-review contracts, promptly after the Bank’s “no objection” is issued. The procurement plan shall include information on actual dates (of “no objections” and award) and will be monitored for cases of delay, which will be examined on a case-by-case basis to identify the reasons. The PIU will maintain up-to-date procurement records and make these available to Bank staff and auditors. 30. Monitoring of payment compared to physical progress. Monitoring reports prepared for the Bank will be customized to include a form to monitor physical progress compared to payment installments to avoid front-loaded payments. 31. Timeliness of payments. The PIU will maintain a system/database to ensure payments to the suppliers and contractors are paid without delay in accordance with the provisions of the contract. Table A3.3 Summary of Procurement Risk Assessment Risk Rating Before Mitigation Rating After MOES staff lack capacity to High A qualified procurement consultant will Substantial undertake the proposed provide on-the-job training to MOES staff and 52 procurement work under the to bid evaluation committee members. The project, particularly regarding consultant will assist in the preparation of Bank procurement guidelines. bidding documents, bid evaluation reports, and contract agreements. Training in procurement under Bank guidelines will also be provided by Bank staff during the project launch workshop. In information technology High A bidders’ conference will be held with Bank High packages, collusion among participation and two-stage bidding will be the bidders is noticed despite followed. best efforts. Bid evaluation committee Substantial Consultant will assist in the preparation of Moderate members are not familiar bidding documents, bid evaluation reports, and with international contract agreements. The risk may continue to procurement procedures, and be monitored during implementation for any may obstruct or delay the change in practice because some evaluation procurement process, committee members may not agree with the especially the evaluation of consultant assessment. bids and proposals. Lack of awareness of Substantial Conduct public awareness programs using Moderate procurement opportunities various media including newspapers, brochures, available in the project for radio, television, and project website. goods and services. Average Risk High Substantial 32. Frequency of procurement supervision. Initially, procurement supervision will include prior review of contracts and procurement implementation support visits (part of project supervision visits) every six months. Once the capacity of the implementing agency is strengthened, frequency of procurement supervision missions and prior review thresholds may be revised. 33. Post-review. Twenty percent of all contracts not subject to prior review will be post- reviewed. 34. Prior review thresholds will be established in the project procurement plan and will be generally based on the following requirements:  All goods contracts awarded through ICB and NCB (>US$500,000 equivalent)  All consulting contracts for firms >US$200,000 equivalent and contracts with individual consultants estimated to cost >US$100,000 equivalent  All direct contracts (DCs) and single-source contracts. 35. The prior review thresholds will be periodically reviewed and revised as needed during project implementation based on risk assessment, procurement post-review reports, and improved capacity of the implementing agency. 36. Disclosure. The following documents shall be disclosed on the MOES website: (a) procurement plan and updates; (b) invitation for bids for goods and works for all ICB and NCB contracts; (c) request for expression of interest for selection/hiring of consulting services; (d) 53 contract awards of goods and works procured following ICB/NCB procedures; (e) list of contracts/purchase orders placed following the shopping procedure on a quarterly basis; (f) shortlist of consultants; (g) contract award of all consultancy services; (h) list of contracts following DC, CQS, or SSS on a quarterly basis; (i) monthly physical and financial progress on all contracts; and (j) action taken report on the complaints received on a quarterly basis. 37. The following details shall be sent to the Bank for publishing on the Bank’s external website and UNDB: (a) invitations for bids for procurement of goods and works using ICB procedures; (b) requests for expression of interest for consulting services with an estimated cost of more than US$300,000 equivalent; (c) contract award details of all procurement of goods and works using ICB procedure; (d) contract award details of all consultancy services with an estimated cost of more than US$300,000 equivalent; and (e) list of contracts/purchase orders placed following SSS, CQS, or DC procedures on a quarterly basis. Environmental and Social (including safeguards) 38. As a category C operation, the project does not finance any civil works; thus OP 4.01 is not triggered. 39. The social impact of the project is expected to be positive through engendered project design, citizen engagement during preparation and implementation, and results monitoring of both. Kazakhstan has achieved and maintained gender parity in access to education, but large gaps in achievement remain between boys and girls. The PISA 2012 analysis has clearly identified significant disparities in reading between boys and girls. While the project-supported interventions aim at enhancing the learning outcomes of all students, especially targeted disadvantaged cohorts, the project will monitor and report on any change in learning results through gender-disaggregated data on student test scores to inform policy and subsequent interventions. For citizen engagement, the project adopted a consultative approach during preparation, engaging stakeholders through consultation on key reforms proposed in the MOES’s state program and on project design. The consultation enabled the voice and participation of citizens and stakeholders on key policy and reforms and decisions on issues of concern to them. The project also supports specific interventions to involve stakeholders and civil society organizations in monitoring and evaluation of the project activities to ensure transparency and openness. A grievance redress system will be set up to handle public complaints on procurement and other concerns so that the demand for transparency and good governance is institutionalized through the mechanism. No resettlement or land acquisition is envisioned under the project; thus OP 4.12 is not triggered. Monitoring and Evaluation 40. The MOES will be primarily responsible for project monitoring, including the development and implementation of the baseline and follow-up surveys. The PDO-level and intermediate results indicators will be monitored through (a) baseline and follow-up surveys, as amplified in Annex 1, and (b) biannual reports prepared by the MOES. The surveys will also contribute to project evaluation. The monitoring and evaluation activities under the project will be integrated into the regular monitoring functions of the MOES. Project indicators have been selected to be compatible with those used in the government’s regular data collection. 54 41. Strong ongoing beneficiary and stakeholder participation will be integrated into the project’s monitoring and supervision activities. The Feedback and Resolution Mechanism (FRM) will provide a forum for beneficiaries and stakeholders to give feedback on the project on an ongoing basis. The FRM will be supplemented by regular public meetings in project areas, especially for Component 2. In addition to tracking beneficiary feedback on the project’s implementation, the FRM and public meetings will be used to track perceptions of the project’s governance, including its transparency, responsiveness, and relevance. 42. Monitoring of project inclusiveness and citizen engagement. Relevant PDO-level indicators and select intermediate results indicators will be disaggregated by gender and rural/disadvantaged school. Given the demand-driven design of the project, citizen engagement will be monitored over the project implementation cycle by measuring the number of beneficiaries and incorporation of their feedback. 55 Annex 4: Implementation Support Plan KAZAKHSTAN: Education Modernization Project Strategy and Approach for Implementation Support 1. The strategy for supporting implementation of the proposed project emphasizes three primary regular activities: (a) dialogue with the government, (b) joint review of project implementation, and (c) exercise of fiduciary oversight throughout the implementation period. 2. Continuing dialogue with the government will facilitate early identification of problems and obstacles that could potentially delay implementation, and enable timely provision of technical advice and support to remove such obstacles. This will contribute to a “just-in-time” identification of issues without the need to raise these during formal reviews. 3. Regular in-situ reviews, which will occur semiannually, will review the progress and achievement of agreed targets and results, as indicated in the project’s Results Framework. The Bank task team will participate in such reviews with representatives of the government and other relevant stakeholders. During each review, the necessity for and type of additional implementation support will be identified. 4. Fiduciary oversight will enable the Bank to fulfill its fiduciary obligations and verify the project’s compliance with the Bank’s fiduciary requirements, including FM and procurement arrangements and outcomes. 5. FM reviews will be performed in two ways: (a) desk reviews of the project’s quarterly IFRs and reviews of the project’s audited annual financial statements and annual auditor’s report and management letter, and (b) on-site visits to review the continued adequacy of the project’s FM and disbursement arrangements. This would include monitoring and reviewing any agreed actions, issues identified by the auditors, randomly selected transactions, and other issues related to project accounting, reporting, budgeting, internal controls, and flow of funds. Special emphasis will be placed on the adequacy of the budgetary allocations to pre-finance project expenses and internal controls framework instituted for the grants. The on-site reviews may include visits to selected beneficiaries of grants, depending on the level of risk and findings identified. 6. Procurement support and oversight will be provided through prior reviews in accordance with procurement thresholds. Procurement support visits will be carried out twice a year, through both desk and on-site reviews of procurement arrangements and results, including post-review of randomly selected contracts. As needed, on-site procurement training may be provided upon request to the relevant agency staff or project management unit. Table A4.1 provides details of the implementation support plan, and Table A4.2 lists the skills mix required for staff. 56 Table A4.1 Implementation Support Plan Time Focus Skills Needed Resource Estimate Partner Role (per year) Years 1–5 Support for the Project - World Bank task Consultation and implementation of management, team leaders (12 advisory project-supported including staff weeks) interventions and procurement, FM, - Technical monitoring and and technical specialists (15 staff evaluation of expertise in weeks) implementation curriculum - FM specialist results. development and (2 staff weeks) evaluation, teacher - Procurement policy, student specialist (2 staff assessment, school weeks) financing reforms, - Social development monitoring and specialist (2 staff evaluation, and weeks) civic engagement. Table A4.2 Skills Mix Required Skills Needed Number of Staff Weeks (SW) Number of Trips Comments Task team leader 12 staff weeks annually Field trips as required Based at HQ FM specialist 2 staff weeks annually Field trips as required Based in Kazakhstan Procurement specialist 2 staff weeks annually Field trips as required Based in Kazakhstan Social development 1 staff week annually Field trips as required Based at HQ specialist 1–2 staff weeks per specialist Technical specialists Field trips as required Based at HQ annually Monitoring and 4 staff weeks annually Field trips as required Based in Kazakhstan evaluation specialist 57 Annex 5: Economic and Financial Analysis KAZAKHSTAN: Education Modernization Project (P153496) Introduction 1. The project seeks to strengthen the quality and equity of Kazakhstan’s primary and secondary education system by (a) supporting the adaption and piloting of a new curriculum, aligning assessment systems, and building teacher capacities, (b) enhancing accountability through information to inform and empower stakeholders, and piloting reforms of education to promote performance, equity, and efficiency; (c) delivering resources to targeted disadvantaged schools; and (d) evaluating reform initiatives supported by the project and financing communication and outreach activities. Project’s Development Impact 2. The proposed project is expected to reach 2.6 million school-age children (over 5 years of age) who will benefit from modernized learning standards (curriculums), enhanced teacher quality and practice, and aligned assessments in schools with more stakeholder participation over the course of the project. While these benefits are shared with all schools, the project will target approximately 5,400 schools that serve rural populations and underserved groups with additional learning resources. These beneficiaries represent the bottom 40 percent of the socioeconomic distribution. This targeting aims to equalize the distribution of the learning resources and put the underserved schools and disadvantaged groups on an equal footing with the rest of the system. The resource distribution in schools is positively correlated with learning, and thus the reduced gaps in resources would likely contribute to improved achievement in underserved and disadvantaged schools. Like most education investments, these benefits would be fully materialized over time after modernized content, accountability, efficiency, and equity measures have been fully implemented. The project would yield, over the medium and long term, higher earning potentials for individual students and higher productivity and economic returns to the economy. Results Chain Based on Project Components 3. The results chain of the project is described in Figure A5.1. The objective of the project is to improve quality and equity in primary and secondary education, particularly in rural and disadvantaged schools. This is achieved through three components: (a) new curriculums and aligned assessment, (b) targeted schools endowed with in-kind resources and teacher training and pedagogical support, and (c) promotion of stakeholder participation and monitoring and evaluation. 58 Figure A5.1 Project Results Chain - Students are exposed to -Improved learning enhanced curriculums, outcomes - Modernized curriculums with new assessment processes, improved -Enhnaced revised student asssessments quality/teaching teaching pedagogy and - Customized pedagogy and practice, content, and better effectiveness with learning resources and school equipped schools - More equitable access acountability to learning resources & - Stakeholders are - Introduction of montoring and provided with foundation for more evaluation of policy initiatives information about equitable results of schools and opportunities students from rural and to provide feedback disadvantaged schools INPUTS OUTPUTS OUTCOME Rationale for Public Sector Provision/Financing 4. The rationale for public investment in education is twofold. First are equity considerations. While education is a private good, households from the bottom quintiles would require state interventions to access good-quality public education, since they often do not have any other (private) alternative. Analysis of the PISA 2012 data shows that students from the lowest socioeconomic status (SES) have improved the most since 2009. However, the differences between students from the top and bottom quintiles are still large—60 points for math—roughly the equivalent to 1.5 years of schooling. Further, inequalities in the education system are usually seen through an increased stratification of the education system. Higher stratification suggests that students who belong to households from similar socioeconomic status attend similar schools. Using PISA 2012 data, Figure A5.2 shows that Kazakhstan is very close to the average on social stratification, suggesting that the education system, to some extent, replicates the inequalities of society. Raising the quality of education for all children requires comprehensive reforms, particularly to offset resource inequalities, which is part of this project. 5. Second is quality of education, which correlates with economic growth. Skills are an important factor for increasing a country’s productivity and growth. Good-quality education, mostly provided by public schools in Kazakhstan, helps develop the foundational skills necessary for future learning. Students with good foundational skills will be able to acquire modern and higher-order skills that can make capital and labor more productive and the country more competitive. In the medium term, the type of firms that will grow in the country will be correlated with the quality of human capital available, and human capital is a long-term investment. In fact, firms in Kazakhstan report that the skills gap is a key constraint for doing business (World Bank 2013).11 Investment in education and, thus skills, can help remediate this situation. 11 Business Environment and Enterprise Performance Survey (BEEPS)—Kazakhstan, World Bank, Washington, DC, 2013. 59 Figure A5.2 Social Stratification in Schools and PISA Mathematics Scores, 2012 550 Above Japan OECD Netherlands Switzerland Germany Poland Finland Canada Denmark Austria Vietnam 500 Ireland Slovenia Slovak Republic Norway Iceland UK Italy Czech Republic France Sweden Spain Hungary United States Greece 450 Serbia Turkey Romania Bulgaria Below OECD Kazakhstan Thailand Chile Malaysia Montenegro Costa RicaUruguay Mexico 400 Jordan Tunisia Argentina Brazil Qatar Colombia Indonesia Peru 350 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 Low School Stratification High School Stratification Source: Bank staff estimates using PISA 2012 data. Note: PISA mathematics scores on vertical axis. Index of School Social Stratification on horizontal axis. The index ranges from 0 to 1. A higher index indicates a higher correlation between the socioeconomic status of students and schools. OECD mathematics score average is 500 points. OECD average Index of School Social Stratification is 0.525. Value Added of the Bank’s Support 6. The value added of the Bank lies in providing policy advice, technical expertise, and “convening power” and financial support that is complementary to that available from the government and other development partners. The Bank’s technical expertise stems from its global and regional experience in the proposed project’s areas: general and secondary education, particularly new learning standards, curriculum revisions, teacher practices, student assessments, and use of data for accountability and learning policies. In fact, the Bank has assisted the development of general education systems in more than 70 countries. The proposed project will leverage the Bank’s financial, operational, and analytical models. These models include a field - tested mix of monitoring and evaluation mechanisms, regulatory and financial safeguards, and overarching transparency in these processes. In particular, the Bank’s safeguards include high standards for protecting the environment and ensuring fiduciary oversight. The “convening power” of the Bank allows for knowledge on the topic areas to flow and to bridge barriers across levels of government. 7. The Bank has a strong track record of engagement in the education sector in Kazakhstan and has assisted the country through joint research, policy advice, and project financing in a broad range of key education issues that includes early childhood development (ECD), modernization of technical and vocational education and training, and QA and governance of higher education. This includes four modules of the Systems Approach for Better Education Results (SABER) policy analysis of teacher policy, school accountability, student assessment, 60 and ECD; a study on the results of Kazakhstan’s participation in PISA 2009 and 2012; and a joint study on system efficiency with the OECD. This knowledge advice was complemented by technical assistance and financing of two ongoing projects to modernize technical and vocational education and youth policy and a new project on skills and jobs that supports the Partnership Framework Arrangement pillar on jobs. The Bank’s track record of engagement has enabled the Bank to apply this extensive research and country knowledge in the design of the project to support achievement of the country’s strategic vision toward a high-performing education system. Cost-effectiveness 8. A cost-effectiveness analysis using the available PISA 2012 data and the latest OECD Review of School Resources in Kazakhstan shows that the proposed project interventions are cost-effective compared with other potential policies like class size or ECD programs. Analysis of the PISA 2009 data shows that investment in resources and less-endowed schools and in teacher practices has the potential to improve the education quality of students in Kazakhstan. The analysis of PISA 2009 data for the rural population shows that there are a number of factors that affect student learning. Many of those factors, however, cannot be easily affected by policy, such as socioeconomic conditions of the households and peer effects, which are highly correlated with student learning. 9. There are other factors, however, that can more easily be influenced by policy and that affect learning, too, particularly among students in the rural areas who tend to have access to lower-quality education services. For instance, class size seems to have a small, though negative effect, on learning. The effect of class size is statistically significant only in rural areas and may capture the effect of the multigrade classrooms. Overall, class sizes in rural areas of Kazakhstan are very small, and further reducing class size may not be the most cost-effective way to increase learning. Attending ECD programs is a variable that positively affects student learning. Expanding ECD would also be an alternative to enhance learning and is recommended for all systems, since it also promotes the development of socio-emotional skills in children and has positive health outcomes. However, ECD interventions are more costly than the ones targeted by the project, since they usually require hiring new teachers. Finally, strengthening the assessment system through the education cycle is important. An analysis of PISA 2009 data shows that it is positively correlated with student learning. 10. Overall, the analysis shows that teacher training and investment in resources in targeted schools can be cost-effective measures to increase learning (Table A5.1). For instance, some meta-cognition techniques (like summarizing the information at the end of class) are particularly important for learning and depend largely on teacher practices. Further, the analysis suggests that the lack of resources in rural areas negatively affects the learning process in schools. This may not come as a surprise since students from poorer households have access to less-endowed schools than children who belong to more affluent households, as seen in Table A5.1. This is also highlighted by the OECD analysis that shows that the lower the overall level of schools’ educational resources, the greater the gap in educational resources between advantaged and 61 disadvantaged schools.12 Scarce resources are more concentrated in advantaged schools, usually in urban areas, increasing inequality in learning. 11. Another element of the project that will interact with the increased resources and new curriculum and that has not been quantified is the increased use of information to engage different stakeholders. Dissemination of information can increase local monitoring and would make schools more accountable if the right incentives are in place. Overall, there does not seem to be a risk to the sustainability of the project interventions in terms of investment in resources in underserved populations, since the expenditure in education has increased in recent years. Despite expenditure remaining below the OECD averages, it has increased due to GDP growth. Table A5.1 Cost-Effectiveness of Education Interventions Effectiveness Costs per student Cost-Effectiveness Ratio (PISA points 2009) (In KZT) (Cost per PISA point) Teacher Practices 19 3,813 200 (Cognition strategies (US$1) Summarizing) Quality of Resources 7 35,200 5,028 (US$18) ECD – 2 years 4 115,000 28,750 (US$103) Sources: PISA 2009 and OECD Reviews of School Resources, Kazakhstan, 2015. Note: The cost of quality of resources is calculated by dividing the proposed US$60 million of Subcomponent 2.1 by 20 percent of the total primary and secondary student population (assuming that 20 percent of students are in underserved rural populations). The exchange rate was taken at US$1 per KZT 278.474 from www.oanda.com, retrieved on November 4, 2015. Similarly, the cost of teacher practices is calculated by dividing the total cost of the project spent on teacher training (US$2.5 million) by 20 percent of the total primary and secondary student population (assuming that 20 percent of students are in underserved rural populations). Figure A5.3 Quality of Resources by Quintile of Socioeconomic Status 0 Third Quintile Fifth Quintile Seocnd Quintile Lowest Quintile Fourth Quintile -0.1 -0.2 -0.3 -0.4 -0.5 -0.6 -0.7 -0.8 -0.9 -1 Source: PISA 2009. 12 PISA 2012 Results: What Makes Schools Successful? Resources, Policies and Practices (Volume IV), PISA, OECD Publishing, Paris, 2013. 62 MAP: IBRD 33425R1 63