Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005159 IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF-A9873) ON A SMALL GRANT FROM THE LOCAL SERVICE DELIVERY TRUST FUND IN THE AMOUNT OF USD 1,348,000 TO THE Bursa Pengetahuan Kawasan Timur Indonesia (BaKTI) FOR INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) PHASE 2 (P167216) Social Sustainabilty & Inclusion Global Practice East Asia And Pacific Region CURRENCY EQUIVALENTS (Exchange rate is an average throughout project’s timeline) Currency Unit = Rupiah IDR 1.00 = USD 0.000069 USD 1.00 = IDR 14,3721 FISCAL YEAR January 1 – December 31 Project end date: 31 December, 2020 Regional Vice President: Victoria Kwakwa Country Director: Satu Kristiina Jyrintytar Kahkonen Regional Director: Benoit Bosquet Practice Manager: Janamejay Singh Task Team Leader(s): Bambang Soetono ICR Main Contributor: Dorothy Lucks 1 Note: Average rate throughout the project’s timeline. All local spending is provided in USD ($) equivalent throughout the ICR. ABBREVIATIONS AND ACRONYMS App Android-based application APS Accredited Procurement Specialist ASA Advisory Services and Analytics BaKTI Bursa Pengetahuan Kawasan Timur Indonesia (Implementing Agency and Grant Recipient of this project) BAPPENAS Ministry of National Development Planning CEM Community Empowerment Mechanism COVID-19 Coronavirus pandemic 2019 CPF Country Partnership Framework FM Financial Management FMS Financial Management Specialist GoI Government of Indonesia ICR Implementation Completion and Results Report IPF Investment Project Financing IRI Intermediate Results Indicators ISR Implementation Status and Results Report JSA Joint Service Agreement KGP1 Phase 1 of the KIAT Guru Project KGP2 Phase 2 of the KIAT Guru Project (this project) KIAT Guru Kinerja dan Akuntabilitas Guru (Teacher Performance and Accountability) M&E Monitoring and Evaluation MIS Management Information System MoEC Ministry of Education and Culture MoF Ministry of Finance NCT National Coordination Team NGO Non Government Organization NMTDP National Medium-term Development Plan OHS Occupational Health and Safety PDO Project Development Objective POM Project Operations Manual PPM Pay for Performance Mechanism QPR Quarterly Progress Reports SC School Committee SDTF Single Donor Trust Fund STEP Systematic Tracking of Exchanges in Procurement TA Technical Assistance TKG Tunjangan Khusus Guru (Teacher Special Allowance) TPG Tunjangan Profesi Guru (Teacher Professional Allowance) UC User Committee USD United States of America Dollar TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 CONTEXT AT APPRAISAL ......................................................................................................4 SIGNIFICANT CHANGES DURING IMPLEMENTATION .............................................................9 II. OUTCOME .................................................................................................................... 12 RELEVANCE OF PDOs ......................................................................................................... 12 ACHIEVEMENT OF PDOS (EFFICACY) ................................................................................... 12 EFFICIENCY ........................................................................................................................ 18 JUSTIFICATION OF OVERALL OUTCOME RATING ................................................................. 19 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 20 KEY FACTORS DURING PREPARATION ................................................................................ 20 KEY FACTORS DURING IMPLEMENTATION.......................................................................... 21 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 23 QUALITY OF MONITORING AND EVALUATION (M&E) ......................................................... 23 ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE.................................................. 24 BANK PERFORMANCE ........................................................................................................ 26 RISK TO DEVELOPMENT OUTCOME .................................................................................... 26 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 28 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 31 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 38 ANNEX 3. ECONOMIC ANALYSIS ........................................................................................... 39 ANNEX 4. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 43 REFERENCES ........................................................................................................................ 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name INDONESIA: IMPROVING TEACHER PERFORMANCE AND P167216 ACCOUNTABILITY (KIAT Guru) Phase 2 Country Financing Instrument Indonesia Investment Project Financing Original EA Category Revised EA Category Organizations Borrower Implementing Agency Bursa Pengetahuan Kawasan Timur Indonesia (BaKTI) BaKTI Project Development Objective (PDO) Original PDO The Project Development Objective (PDO) is to improve teacher presence and teacher service performance in pilot schools. FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-A9873 1,348,000 1,347,762 1,347,762 Total 1,348,000 1,347,762 1,347,762 Total Project Cost 1,348,000 1,347,762 1,347,762 Page 1 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) KEY DATES Approval Effectiveness Original Closing Actual Closing 05-Mar-2019 20-Mar-2019 15-Dec-2019 31-Dec-2020 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 15-Dec-2019 1.05 Change in Results Framework Change in Loan Closing Date(s) Change in Implementation Schedule 28-Apr-2020 1.30 Change in Results Framework Change in Loan Closing Date(s) Change in Implementation Schedule 25-Jun-2020 1.34 Change in Results Framework Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 27-Jun-2019 Satisfactory Satisfactory 0.42 02 13-Jan-2020 Satisfactory Moderately Satisfactory 1.16 03 27-Jul-2020 Moderately Satisfactory Moderately Satisfactory 1.34 04 22-Dec-2020 Satisfactory Satisfactory 1.35 ADM STAFF Role At Approval At ICR Regional Vice President: Victoria Kwakwa Victoria Kwakwa Page 2 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) Satu Kristiina Jyrintytar Country Director: Rodrigo A. Chaves Kahkonen Director: Ede Jorge Ijjasz-Vasquez Benoit Bosquet Practice Manager: Nina Bhatt Janamejay Singh Task Team Leader(s): Dewi Susanti, Javier Luque Bambang Soetono ICR Contributing Author: Dorothy Lucks Page 3 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES CONTEXT AT APPRAISAL Context National Context. Indonesia had achieved drastic poverty reduction with a more than 50 percent fall in the poverty rate between 1999 and 20182 at the time of project appraisal. Over the same period, Indonesia’s inequality divide had increased disproportionately by geographical spread. The World Bank had recognized the importance of improving local service delivery as a potential driver in reducing inequality.3 Similarly, the Government of Indonesia (GoI) had identified education as an important service to improve poverty and inequality outcomes. The GoI budget had more than doubled in real terms since 2001, with an education budget that tripled during the same period, reflecting the Government’s focus on the sector. Since 2009, in line with legislation and the respective national budget allocations, 20 percent of national and district government budgets have been assigned for education.4 In 2013, the 12-year education structure was confirmed as mandatory. Education Sector and Institutional Context. Improvements in access, quality and equity of education had been recorded in Indonesia but challenges persisted. Student enrollment had increased since 2002 by more than 10 million (31 percent) at the primary and secondary education levels.5 In the ten years prior to 2018, net primary school enrollment rates remained relatively steady with a change from 93.1 percent in 2008 to 93.5 percent in 2018. There has been a greater increase in secondary enrollment rates from 64 percent in 2008 to 78.7 percent in 2018.6 However more than 1.5 million primary school aged children were not enrolled in education in 2018 including more than one million girls and less than 500,000 boys.7 Indonesia is however well below global education standards and outcomes8 and faces challenges in quality and equity of access, particularly in remote and rural areas.9 It is difficult to attract teachers to rural schools given limited connectivity, both physically and technologically. In addition, rural schools are under-resourced and lack the equipment required to provide high-quality education. A 2017 survey found that only 39 percent of rural schools had sufficient textbooks and just 54 percent had a library. Further challenges include teacher qualifications and composition as well as the necessity for multi-grade teaching.10 In addition, there can be overlapping roles and responsibilities and gaps in educational resourcing delivered due to the many ministries and devolved agencies engaged in the education sector. This creates an overcrowded political and operational 2 BPS (Indonesian Center of Statistics Agency). (2018). Profil Kemiskinan di Indonesia, Maret 2018. (Poverty Profile in Indonesia, Maret 2018). Jakarta: BPS. 3 World Bank. (2016). Indonesia's Rising Divide. Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/267671467991932516/Indonesias-rising-divide. 4 Act of the Republic of Indonesia, number 20, year 2003. 5 The World Bank. 2020. The Promise of Education in Indonesia. 6 The World bank, 2018, Databank: Education: Indonesia, https://data.worldbank.org/topic/education?locations=ID 7 The World Bank, 2018, Education Statistics: Country at a Glance: Indonesia, https://datatopics.worldbank.org/education/country/indonesia 8 World Bank. (2018). Learning to realize education’s promise. World Development Report. Washington, D.C: World Bank Group. 9 Rosser, A (2018). Beyond Access: making Indonesia’s Education System Work, https://www.lowyinstitute.org/publications/beyond- access-making-indonesia-s-education-system-work. 10 Susanti, D, Priebe, J & Bah, A (2020) The hard truth: Challenges of primary education in rural and remote Indonesia, https://blogs.worldbank.org/eastasiapacific/hard-truth-challenges-primary-education-rural-and-remote-indonesia Page 4 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) context for education provision.11 The Government of Indonesia has introduced reforms to improve education service delivery. The Ministry of Education and Culture (MoEC) has been implementing policies to address these challenges, including with assistance from the World Bank.12 Approximately half of the education budget finances teacher salaries and allowances ( Figure 1). These allowances include a Tunjangan Profesi Guru (Teacher Professional Allowance (TPG) for certified teachers and a Tunjangan Khusus Guru (Teacher Special Allowance, hereafter TKG) for teachers in what is considered special (remote, disaster, conflict) areas. Both allowances can be equal to a teacher’s salary, essentially tripling the base salary for certified teachers placed in special areas. In 2018, these payments totaled United States of America Dollars (USD) 5.78 billion equivalent. MoEC has reformed teacher management through improving teacher intake quality, teacher placement matching, tracking teacher competence and facilitating professional development along with a range of other methods. Figure 1. Teachers’ salary & allowances, compared to total education budget from 2012 -2017 Source: World Bank. (2019). KIAT Guru Phase 2 Project Paper. Teacher salary and allowances budget for the year. Teacher allowances budget for the year. Linking teacher salary payments with teaching performance. The GoI has promoted initiatives to strengthen teacher performance as a means of improving education quality. The Civil Servant Law (UU 5/2014), which recommends performance-based evaluation and performance-based incentives for civil servants, created a pathway for MoEC to improve teacher competency and associated incentives. In addition, the Village Law13 allocated resources to local villages to empower communities and support basic services delivery. Challenges in frontline education service delivery. The 2015-2019 National Medium-Term Development Plan (NMTDP) also noted that despite GoI’s increased spending on teacher salaries and allowances, this did not result in an increase in teacher performance or student learning outcomes. For example, teachers receiving the TKG 11 Rosser,A (2018). Beyond Access: making Indonesia’s Education System Work, https://www.lowyinstitute.org/publications/beyond- access-making-indonesia-s-education-system-work. 12 World Bank (2019). Implementation Completion Report for Indonesia: Improving Teacher Performance and Accountability. 13 Government of Indonesia Law No. 6/2014 on Villages. The new Village Law (No. 6/2014) dramatically increases the responsibilities and budgets of the village administration. Page 5 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) had a higher absenteeism rate compared to non-TKG recipients in the same schools.14 Similarly, the introduction of TPG did not lead to the intended increase in student learning.15 The NMTDP recognized that given the country’s diversity, customized approaches to education services were needed. Similarly, that fragmentation of reform in the education sector across different stakeholders had created challenges and that greater coordination and coherence was required. Rationale for World Bank engagement. GoI policy and law reforms and a consistent emphasis on education created an opportunity for World Bank engagement towards improved education services. The NMTDP highlighted the potential for success of local activities based on experimentation, adaptation, and knowledge sharing. As such, the GoI developed a ‘frontline’ approach which aimed to remove bottlenecks and constraints and promote mutual accountability and joint problem-solving for service delivery. The World Bank Group Country Partnership Framework (2016-2020)16 provided further rationale for engagement in the education sector, stating that: “This engagement will focus on supporting efforts to establish a performance-based fiscal transfer system, strengthening the capacity of central government agencies to support and oversee the performance of local governments, strengthening the back-office functions of local government to deliver local services and infrastructure, and developing tools for citizens to monitor local service delivery.� Figure 2. KIAT Guru Phase 2 Locations Source: KIAT Guru Phase 2 Presentation, September 2019. The Bank supported GoI to implement a proof-of-concept pilot through KIAT Guru Phase 1 (KGP1, P159191). This pilot tested differentiated mechanisms including a Community Empowerment Mechanism (CEM) that facilitated engagement of parents and teachers in a Joint Service Agreement (JSA) and Community Score Cards to facilitate teacher evaluation to identify education service improvements required. The JSA was a collaborative effort to discuss and agree upon localized strategies to improve student outcomes, e.g., for teachers to set and mark homework tasks, and for parents to assist at home. The teacher’s scorecard was a mechanism to track progress and provide feedback to teachers. It also involved a Pay for Performance Mechanism (PPM) to link the payment 14 SMERU. (2010). Teacher Absenteeism and Remote Area Allowance: Baseline Survey. Jakarta, Indonesia: SMERU Research Institute. 15 De Ree, J., Muralidharan, K., Pradhan, M., and Rogers, H., (2018). Double for Nothing? Experimental Evidence on an Unconditional Teacher Salary Increase in Indonesia. Quarterly Journal of Economics 133 (2): 993–1039. 16 The World Bank, 2015, Country Partnership Framework for The Republic of Indonesia for the period FY16-FY20, Report No. Report No. 99172, page 63-65. https://documents1.worldbank.org/curated/en/195141467986374707/pdf/99172-REVISED-World- Bank-Indonesia-Country-Partnership-Framework-2016-2020.pdf. Page 6 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) of TKG with either teacher attendance or service performance. KGP1 results showed that the combination between CEM and PPM based on teacher attendance improved learning outcomes the most. This led to consideration of how to scale up this approach through a second phase of the KIAT Guru Project (KGP2). GoI demonstrated willingness to expand the KIAT Guru model given the potential implication for national education policy. The KGP2 pilot districts remain the same as KGP1, with expanded number of schools and beneficiaries (Figure 2). Theory of Change Results Chain. The Theory of Change used in KGP2 was adapted based on the findings from KGP1 and is illustrated in Figure 3. The expected outcomes of the project would occur in stages and contribute to an overall impact of improved student learning outcomes in participating schools. The project tied TKG to teacher presence, engaged parents in their child’s education, and assessed teacher performance. Some outputs continued from KGP1 covering activities related to enabling government regulations and regular MoEC and district government meetings. In addition, a pilot of an e-KIAT Guru Android-based application (app) was implemented for photographic evidence with a date and time stamp of teacher arrival at and departure from school. The data was forwarded to an online web-based application in a management information system (MIS) that could then generate aggregated teacher attendance and service performance reports. This required specific training for school superintendents and school stakeholders on how to use the app and MIS. These interventions were expected to reduce the percentage of teachers arriving late and with unexplained absences as well as improve teacher performance scores which constituted improved teacher presence and performance. Monthly meetings between teachers and parents generated expectations for improved teacher performance and discipline in relation to JSA indicators developed in collaboration with all stakeholders at project initiation. Page 7 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) Figure 3. KIAT Guru Phase 2 Theory of Change Source: KGP2 Project Paper, 2019 (adapted) Page 8 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) Project Development Objectives (PDOs) The PDO is to improve teacher presence and teacher service performance in pilot schools. Key Expected Outcomes and Outcome Indicators Key results. The PDO-level results indicators are17: i. Reduced percentage of teacher tardiness in participating schools. ii. Reduced percentage of teachers with unexcused absence in participating schools. iii. Improved teacher performance score. Project Beneficiaries. At the time of project design, it was envisioned that the project would benefit 386 schools, 386 school principals and village heads, 548 village cadre, 1,755 village community members, and 27,000 primary school students. Performance evaluations for 3,000 teachers would be conducted. Participating Districts and Stakeholder Contribution. The districts that participated in this project were the same five pilot districts from KGP1: Sintang, Ketapang, and Landak in West Kalimantan province, and Manggarai Barat and Manggarai Timur in East Nusa Tenggara province. Within the district there was expanded scope of implementation and enhanced expectation of committed co-financing within their regions. MoEC financed trainings and workshops at the national level, while meetings, trainings and other district level activities were resourced by district governments. Village governments allocated co-financing from village budgets to support local monthly meetings as well as operational expenses of the user and school committees. The contributory co-financing at each level was designed to prepare for sustainability of KIAT Guru initiatives.18 Components The project was comprised of one component (USD 1,348,000) with the below six main activities: i. Technical Assistance (TA) for MoEC and five district governments to revise the regulatory umbrella to tie TKG with teacher presence. ii. TA for MoEC and five district governments to enhance the support for KGP1 schools and to conduct implementation of KGP2 interventions in additional schools. iii. Develop training modules and provide capacity development trainings for stakeholders from 386 schools. iv. Develop, pilot, and provide capacity development training of upgraded KIAT App and MIS. v. Testing of mechanisms to expand KIAT Guru in up to 40 secondary schools and using TPG in urban locations of the five districts. vi. Management and monitoring of project implementation. SIGNIFICANT CHANGES DURING IMPLEMENTATION PDO Indicators. There were no changes to the PDO indicators. The project experienced three restructurings, as summarized in Table 1. The first restructuring was approved in December 2019 to complete project activities and ensure alignment with new MoEC reform agendas. The second restructuring in April 2020 was in response to the 17 Please refer to ANNEX 1 Results Framework and Key outputs for further detail. 18 Please refer to ANNEX 4 for Recipient, Co-financer, and other Partner/Stakeholder Comments. Page 9 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) COVID-1919 global pandemic and the associated impact on project activities. One activity – the piloting of KIAT Guru approach in urban secondary schools – was dropped as co-financing for implementation from the MoEC and five district governments was reallocated for COVID-19 pandemic response. The third restructuring was also requested by MoEC in the context of the pandemic and the recognition of the potential of the KIAT Guru approach and tools to be scaled up and assist in the continuous delivery of education during the pandemic. Table 1. Summary of Restructuring Restructure December 2019 April 2020 June 2020 Change Loan Closing Changed from 15 December 2019 to Changed to 30 June 2020. Changed to 31 Date 30 April 2020. December 2020. Results Monthly teacher evaluation meetings Removed – Pilot KIAT Guru in 50 Amended. Finalize and Framework increased from 2,800 to 3,400. urban secondary schools – co- handover the e-KIAT financing from MoEC and five district Guru app and MIS to # of MoEC & district government governments reallocated for COVID- MoEC. officials received capacity building 19 pandemic response. increased from 200 to 1,600. Added. Provide TA to the Added. Finalize the development and five district governments # of school stakeholders received testing of e-KIAT Guru App and MIS. to develop strategies for capacity building training increased the safe reopening of from 1,200 to 4,000. Added. Continue to monitor the schools closed due to implementation activities of the five Increased scope from 386 to 410 the COVID -19 pandemic district governments. remote primary schools, and from 40 and to scale up KIAT to 50 urban secondary schools. Added. Provide TA for the five district Guru for 100% coverage governments to develop scale up in rural schools. Added “Issuance of relevant plans for coverage in all remote government regulations� for 2020, to schools and sustainability of KIAT Guru be completed by April 2020. activities. Disbursement Change in disbursement estimate No change. Change in disbursement Estimates 2019 from $739.000 to proposed estimate 2019 to $1,048,000. $440,000. Change in disbursement estimate Change in disbursement 2020 from $609,000 to proposed estimate 2020 from $300,000. $948,000 to $1,348,000. Implementati Complete the development of the App No change. No change. on Schedule (e-KIAT Guru) to align with MoEC’s new reform agenda. Provide TA to governments to issue KIAT Guru regulations and strengthen capacity for sustainability of the KIAT Guru approach. “Percentage of Tunjangan Khusus allowance for teachers are paid according to KIAT Guru formula,� to be completed by April 2020. Source: KIAT Guru Restructuring Papers, December 2019, April 2020, and June 2020. The COVID-19 global pandemic impacted the implementation of project activities. When the pandemic hit 19 Coronavirus pandemic 2019. Page 10 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) Indonesia in mid-March 2020, the project had completed most planned activities. Most of the Intermediate Results Indicators had been achieved and the PDO was rated as Satisfactory. Given social distancing measures and limits to social gatherings introduced to minimize the spread of COVID-19, implementation of key project activities such as monthly performance evaluation meetings and capacity building activities were affected. Project adjustment to align with MoEC’s priority in response to COVID-19 pandemic. Beginning in mid- March 2020, the MoEC and five district governments refocused their priorities and reallocated their budgets to handle the pandemic. Monthly evaluation meetings were halted in most schools due to closures, although meetings continued at some schools. Consequently, the remaining targets for meetings were not fully met. The pandemic also resulted in the activity to test the KIAT Guru approach in 50 urban secondary schools being cancelled. Tracking of PDO indicators was difficult due to mobility and communication challenges. In consultation with MoEC and the Bank’s team, the implementing agency adjusted the project’s remaining activities to: (a) handover e-KIAT Guru App and MIS to MoEC, (b) provide TA for the five district governments to conduct distant learning and reopen schools with new normal protocol, and (c) monitor the impact of the pandemic to activities in schools.20 Figure 4. KIAT Guru Cycle During COVID-19 Shifts in project approach to online teaching and adapted tools. While the PDO remained constant, the 20 The World Bank Group, Implementation Status and Results Report (ISR) December 2020. Page 11 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) emphasis in the project on teacher presence pivoted to supporting teachers in online teaching approaches. The student assessment process was adapted to be used as a tool to monitor student learning outcomes and individualized learning approaches in the distant teaching environment. This included the upscaling of assessment and learning packages which could be used nationally to assist in transitioning to remote education and a similar adaptation for the e-KIAT Guru app. The adaptation of these tools to a wider use in the COVID context is illustrated in Figure 4. II. OUTCOME RELEVANCE OF PDOs Assessment of Relevance of PDOs Rating: High The PDO was highly relevant at design and remained highly relevant throughout implementation. The design was in direct response to national priorities in the NMTDP to improve education service performance. KGP2 supported the key engagement area (4) of delivery of local services outlined in the World Bank’s Indonesia Country Partnership Framework 2016-2020 (CPF). In addition, the project responded to the CPF focus on strengthening the capacities of central government agencies by supporting the MoEC in policy reforms and implementation. As discussed later, the project informed MoEC’s national policies for distribution of additional school operational funds for disadvantaged schools during the pandemic, for developing student learning assessment tools, and for developing model schools for remote areas. Furthermore, in the 2021-2025 CPF, KGP2 aligns directly with engagement area 3: Nurture Human Capital, and in particular, with Objective 3.1: Strengthen the quality and equity of education and of skills.21 The project was based on a previous pilot that tested differentiated approaches to improving teaching performance. KGP1 was successful in generating lessons learned on joint teaching management approaches and localized community empowerment through engagement in the JSAs to support improved student outcomes. The establishment of school-level committees commenced in KGP1 was continued. In KGP2, two different approaches were tested: (i) continuing the KGP1 approach of a community-parent user committees (UC) comprising of locally selected parent representatives, facilitated by village cadres, and supported by the village government; and (ii) revitalizing the existing School Committee (SC), facilitated by school principals, and supported by the school superintendent. Both approaches were designed to enhance local engagement of parents and local leadership in supporting and supervising local primary education through the JSAs. The further testing and upscaling of these approaches through this project was highly relevant for improving learning outcomes in lower-than-average performance in rural primary schools. There was a potential for refining the KIAT Guru approach with a view to mainstreaming and scaling up the approach to other districts. ACHIEVEMENT OF PDOS (EFFICACY) Assessment of Achievement of Each Objective/Outcome Rating: Substantial The project objective was to improve teacher presence and teacher service performance in pilot schools . There were two expected outcomes: improved (i) teacher presence and (ii) teacher service performance. The 21 The World Bank Group, Indonesia Country Partnership Framework 2021-2025, Report No. 157221-ID. Page 12 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) project beneficiaries at appraisal were 386 schools, yet the project exceeded this number with a total of 410 primary schools (106 percent of appraisal target) reached. All school received capacity development trainings and established JSAs between teachers and parents to improve learning environments at school and at home. When the pandemic hit Indonesia in mid-March 2020, the project had completed most of the planned activities in all schools. The assessment of Efficacy is Substantial. Outcome 1 is assessed as Modest, largely due to the impact of COVID-19 affecting school attendance. Outcome 2 is assessed as Substantial based on the performance achieved prior to the onset of the pandemic where intermediate outcome indicators were largely met and exceeded. Furthermore, the effectiveness of the project’s pandemic response demonstrated continued results in line with the PDO. At the end of November 2020, the project completed offline student learning assessments, with results reported from 350 schools (Figure 5). KGP2 schools averaged 5.37 in literacy, and 5.25 in numeracy, slightly below KGP1 schools, which averaged 5.55 in literacy and 5.46 in numeracy. When results from KGP2 schools were disaggregated based on the two interventions, the schools that implemented within village governance structure (Group A) had higher averages compared to KGP1 schools, at 5.66 in literacy and 5.46 in numeracy. These results show that KGP2 Group A, which only received six months’ worth of effective implementation support, outperformed KGP1 schools, which had been supported since mid-2016. This indicates that the simplification of the facilitation process from KGP1 to KGP2 seems effective, and more importantly that village governance arrangement seems to provide stronger accountability for school providers. These learning assessments serve as an alternate indicator to gauge the project’s impact that could not be provided by the PDO indicators. Figure 5. Student learning assessment results Source: KIAT Guru online monitoring, November 2020. PDO Outcome 1: Improve Teacher Presence. Rating: Modest The PDO target for “Reduced percentage of teacher tardiness� was 2 percent and was not met by the project closing (baseline: 6.15 percent; actual: 8.32 percent). This indicator was designed to demonstrate teacher punctuality and was linked to the use of the e-KIAT Guru app to record overall presence and gain more data on Page 13 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) reasons for teacher absence. Results as illustrated in Figure 66 show that between June and September 2019 teacher tardiness increased, but there were also reported challenges in the utilization of the new e-KIAT Guru app, which could contribute to timely recording of teacher presence. Between September and October 2019 some progress was made, and teacher tardiness decreased marginally as teachers became accustomed to the process. Data from February 2020 shows a marked increase in teacher tardiness, reported by schools to be a result of the COVID-19 pandemic. However, data from March 2020 again shows a decrease in tardiness back towards the target as measures to address the pandemic advanced. Further analysis of the data shows that 65 percent of teachers were recorded as tardy in the first five months of the project, with more than half of them repeatedly recorded as tardy for two or more months.22 Discussions with teachers suggest that there was an increasing awareness of the need for punctuality and that the photograph was an impetus to be on time, but reasons for tardiness are complex and not easily addressed, often related to weather, family issues and transportation delays. Without the project, it is likely that the baseline levels would have been maintained or even declined. The increased awareness and initial shifts in trends are likely to lead to continued improvement in attendance, notwithstanding reasons beyond the control of the teachers. By the end of KGP2, final achievement was higher than the baseline level, indicating higher teacher tardiness, nonetheless, there are indications of behavior change that is likely to achieve a longer-term positive performance. Figure 6. Teacher tardiness in participating schools 12% 10% % of Teacher Tardiness 8% 6% 4% 2% 0% June 2019 September 2019 October 2019 February 2020 March 2020 Results Target Baseline Source: Project monitoring from September 2019, October 2019, February 2020, and March 2020. The PDO target for “Reduced percentage of teachers with unexcused absence� was 5 and was not met at project completion (baseline: 15; actual: 8.28). Results show that progress from the baseline was evident, with the target being achieved in September 2019 (Figure 7). However, unexcused absences surged in February and March 2020 corresponding to the onset of the pandemic. The overall teacher presence decreased from 95.13 to 90.80 percent in the same period demonstrating a link between teacher attendance and unexplained absences. Due to the pandemic constraining access to the schools for monitoring visits, it was not possible to verify the reason for the increase, but it is likely that the reversal in performance was related to the context of the pandemic rather than a decline in validity of the KGP2 approach. 22 The World Bank, June 2020. Restructuring Paper on a Proposed Project Restructuring of Improving Teacher Performance and Accountability Phase 2. Approved on March 5, 2019. Page 14 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) Figure 7. Teachers with unexplained absences in participating schools % Teachers with Unexcused 30% 25% Absence 20% 15% 10% 5% 0% June 2019 September October 2019 February 2020 March 2020 2019 Results Target Baseline Source: Project monitoring from September 2019, October 2019, February 2020, and March 2020. PDO Outcome 2: Improved Teacher Service Performance Rating: Substantial The PDO target for “Improved teacher performance� was 85 and had been met and exceeded since September 2019 and had continued to increase since this point (baseline: 0; actual: 96.5). Results related to this part of the PDO were measured through a teacher performance score. This score came from an evaluation of teacher performance by the UC and/or steering committee that was recorded in the e-KIAT Guru app. The average of these scores over the project period is displayed in Table 2. Project stakeholders met during the Implementation Completion and Results Report (ICR) mission attributed the continued growth in score to the greater involvement and capacity of the district stakeholders including school supervisors in local education management, the improved relationship between teachers and parents through the joint agreements, and recognition of the efforts to maintain educational quality for students during the pandemic. Table 2. Teacher Performance Score Results September October February March Reporting Date June 2019 2019 2019 2020 2020 Teacher Performance Score No score 89.41 94.52 93.14 96.5 (Target: 85) Level of achievement No Score 105% 111% 110% 114% Source: Project monitoring from September 2019, October 2019, February 2020, and March 2020. The appointment of a qualified Non-Government Organization (NGO) Bursa Pengetahuan Kawasan Timur Indonesia (BaKTI) as project implementing agency was effective in accomplishing operational targets . The progress towards outcomes was measured through intermediate results indicators (IRIs) and overall satisfactory progress was achieved. The extent of relevant government regulations issued demonstrated that the required policy changes occurred. Monthly teacher evaluation meetings provided a means for school committees to be active and functional for implementing the JSAs. Capacity building for MoEC, district government officials and school stakeholders in the project processes and e-KIAT Guru App enabled the project Page 15 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) process to operate smoothly. The percentage of TKG paid based on teacher presence provided an incentive for improvement in teacher attendance, although this was severely affected by the onset of the COVID-19 pandemic. KGP2 reached more schools than expected and most activities were completed prior to the onset of the COVID-19 pandemic. A total of 410 primary schools (106 percent of target) received capacity development trainings and established JSAs between teachers and parents to improve learning environments at school and at home. When the pandemic hit Indonesia, the project had completed most of the planned activities to support the establishment, capacity building and operation of the district and school committees and to facilitate the operation of the e-KIAT Guru system. Required relevant government regulations issued within required timeframe. All relevant government regulations were issued in 2019 and 2020. This included the Minister of Education's regulation which identified the KIAT Guru project as a National Priority Program for delivery of TKG. The project facilitated and provided policy recommendations to MoEC for the issuance of additional key regulations in 2020 as part of the change in leadership and in response to the COVID-19 pandemic. These were not part of the original target but were possible due to the established and positive working relationships between the project and Ministry officials. The project facilitated the issuance of government regulations at national and district level. The project supported issuance of eight national level regulations, including the Ministry decree on Teacher's Special Allowance for 2020 and Director General decrees on partnerships with five district governments; and 61 district government regulations to support the implementation of program activities in the five districts. In addition, based on KIAT Guru results and the Bank’s policy inputs, MoEC issued a national regulation in 2020 on the data-driven appointment of schools for the distribution of additional school operational funds and special allowance. These policies provided the necessary approvals for implementation, including allocation of counterpart funding at each level. Monthly teacher evaluation meetings conducted satisfactorily. The monthly meetings provided a mechanism for teachers and parents to meet in a local hall to discuss progress related to the JSA implementation. Teacher attendance as recorded through the e-KIAT Guru app would be presented by the headmaster to school stakeholders. Teachers would report on progress made on student learning and other JSA indicators. Then parents and teachers would agree a score in relation to the JSA progress achieved and identified joint solutions for unmet targets. This provided a collaborative means to monitor both teacher attendance and service performance. KGP2 included testing of two alternative approaches involving (i) the use of paid facilitators to assist in establishment and coordination of the user committee, and (ii) a mainstreamed model through the existing or revived school committee. Based on feedback from the participants, both approaches were valid, achieving similar outcomes and with potential for scale-up. The target for monthly meetings was increased from 2,800 to 3,400 monthly meetings by December 2019, due to increase in number of schools from 386 to 410. In 2020, the COVID-19 context constrained social gatherings including the monthly meetings. Achievement overall was 2,816 meetings which equated to 81 percent of the revised target and exceeded the original target of 2,800. Capacity building trainings for the School Principal, Village Head, and Village Cadre/ School Committee was largely completed as planned. KGP2 training aimed to enable all schools to carry out monthly meetings, share lessons learned and formulate improvements in the implementation of activities at the village level. Due to the pandemic conditions, there was a change in the training approach from face-to-face to online. The training duration was reduced from 15.25 hours to 3.75 hours. Training was conducted by the school supervisors, local Page 16 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) community facilitators and district governments, but this shifted to online training by the Regional KIAT Guru Team. The training was adapted to include a focus on supporting children to receive learning according to their abilities. Number of MoEC and district government officials receiving capacity building was exceeded. BaKTI provided implementation and capacity development support for the government officials to administer teacher performance evaluation scores, verify the reports, conduct payments of TKG based on either teacher presence or teacher service quality, and visit underperforming schools. The original target was 200 unique individuals, revised to 1,600 cumulative recipients during the December 2019 restructuring as it was difficult to track individuals. The project exceeded the revised target and 2,607 MoEC and district government officials received face to face and on-line capacity building support, comprised of 2,154 male (83 percent) and 453 female (17 percent). Number of school stakeholders receiving capacity building trainings substantially exceeded targets. The trainings covered the roles and responsibilities of Village Cadre, UC, and SC, facilitating socialization meeting, conducting diagnostic student learning assessment, collecting evidence for evaluating teacher performance, conducting evaluation of joint agreement indicators, and operating the e-KIAT Guru App & MIS. By end February 2020, all schools had established JSAs between teachers and parents to improve learning environments, for example through making learning materials or by agreeing homework regimes at school and at home. The target for this indicator was increased from 1,200 to 4,000 following good initial progress of the project. The project exceeded both the original and revised target with a total of 7,398 stakeholders at school level receiving training, which was comprised of 4,918 male (66 percent) and 2,480 female (34 percent). Percentage of TKG paid based on teacher presence exceeded. Payment for TKG recipient teachers in KIAT Guru schools was based on teacher presence, as recorded using e-KIAT Guru, and verified by the UC or SC. From September 2019 the target was met and exceeded with almost 100 percent of TKG being paid based on teacher presence by March 2020 (Table 3). The improved use of the camera application and recording system enhanced the recording of teacher unexplained absence and tardiness and assists in improved teacher management by Principal and Superintendents. Table 3. TKG payments based on teacher presence. Reporting Date June 2019 September 2019 November 2019 March 2020 % TKG paid based on teacher - 93% 91% 98.63% presence (Target 85%) Level of achievement 0% 109% 107% 116% Source: Project monitoring in June 2019, September 2019, November 2019, and March 2020 After March 2020, the project adjusted the activities to align with the MoEC priorities during the COVID-19 pandemic. The project facilitated online training for 78 primary school supervisors from 1,168 primary schools in five districts on: (i) procedures for schools to make preparation for new normal protocol; (ii) procedures to conduct diagnostic assessment at the beginning of the school year; and (iii) emergency curriculum for teachers to adjust learning materials. BaKTI also distributed project-developed materials, comprising of: (i) twelve packages of ready-to-use literacy and numeracy assessments that can be utilized by all primary schools in the five districts; (ii) revised joint agreement indicators to guide collaborations between teachers and parents during pandemic; and (iii) linked learning materials that can be used by teachers and parents alike. Page 17 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) The pivot necessitated reprioritizing of activities but resulted in increased support to schools during the pandemic. The testing of the KIAT Guru approach in 50 urban secondary schools was cancelled as it was not feasible to conduct field visits. In consultation with MoEC and the Bank’s team, BaKTI adjusted the project’s activities. Analysis from the project’s monthly online monitoring from July to October 2020 identified: (i) improvement in percentages of students that can continue learning with strict health protocols, from 59 to 74 percent; (ii) increased in small group learning activities and collaboration between teachers and parents, from 58 to 69 percent; and improved parental supports to home-based learning, from 46 to 55 percent.23 EFFICIENCY Assessment of Efficiency Rating: Substantial KGP2 operated efficiently, mobilized agreed counterpart resources, and generated expected benefits. The project costs were invested to establish, train and support the national, district and village stakeholders in establishing the JSAs, conducting student assessments, and developing and maintaining e-KIAT Guru application, including the purchase of an Android-based phone per school. The average annual cost for the government to sustain the KGP2 approach is expected to be USD 1,872 per school, or USD 14.18 per student. This investment is expected to be within the capacity of MoEC, District, Villages, and Schools to cover, with benefits accruing from the enhanced teacher-parent engagement in working on improved student outcomes. KGP2 operated efficiently given the short timeframe and context of the COVID-19 pandemic. The budget for KGP2 was developed based on KGP1, adjusted to standard GoI costs. It was estimated that MoEC would allocate a minimum of USD 2.5 million for TKG in 386 schools to be tied with teacher presence in 2019. The project successfully reached 410 schools. The five pilot districts committed USD 260,000 equivalent for implementation activities at the district level.24 The project utilized almost all available funds with a disbursement rate of 99.98 percent. The project reallocation disbursements aligned with the restructurings and reflected project activities and full implementation. The institutional mechanism was efficient for the grant implementation. BaKTI was appointed as a Grant Recipient to implement the project for the reason of flexibility and service delivery capacity. Awarding the grant to MoEC would have required longer and more complicated processing that would not have been possible for the required grant timeline. The selected NGO had the necessary technical and operational expertise to deliver the component activities within the required timeframe and budget. The project management of KGP2 amounted to a total of USD 38,670, or USD 94.32 per school, which was only 14 percent of KGP1 cost, with a total of USD 140,342 or USD 691.34 per school. This efficiency could be achieved as KGP2 piggybacked on KGP1 investments with the national and district governments, hired less qualified facilitators compared to KGP1, and supplemented the implementation with trained school supervisors co-financed by the government. Mobilization of counterpart funds occurred as projected. Based on the ICR mission, BaKTI’s team was successful in activating the national and district government stakeholders to allocate staff resources to support 23 The World Bank Group, December 2020 ISR. 24 Please refer to Annex 3 for the Efficiency Analysis. Page 18 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) the project in generating the required policies, mobilizing the required budgets, and engaging in supervisory processes. The village governments were to allocate USD 450,000 for implementation activities at the village level. The allocation of budgets per village was not tracked but based on information from the implementors, the budget allocation supported the costs of the training and monthly meetings. There were only two reported examples of villages that did not allocate the required resources. Each dollar of Bank investment for KGP2 generated USD 64 cents of GoI co-financing, which focused on training and other expenses for sustaining benefits beyond the end of the project. Data has been gathered to compare the one-time investment costs (training of trainers, salaries of trainers and operational costs) of the two approaches adopted in KGP2. Group A involved rollout through UC in 111 schools with a total cost of USD 259,980 which equates to USD 2,342 per school and USD 17.74 per student. Group B, which conducted rollout through SC in 96 schools, totaled USD 228,936, or equal to USD 2,385 per school and USD 18.07 per student. KGP2 costs were substantially lower than KGP1. Savings of 45 percent were achieved through reduced numbers of facilitated meetings and streamlined processes due to improved training and orientation. See Annex 3 for further detail on the costs. The district and village governments are expected to sustain the activities independently. In 2020, MoEC and district stakeholders confirmed that required budgets are in place for the next 12 months and there is likelihood, but not certainty, of continuation due to the political context where local leaders are changed annually. Three of five district governments committed district budgets to continue the project in 2021, which, along with MoEC’s commitment, is estimated to be USD 545,000. The enabling environment for continued student learning outcomes has been strengthened. The project helped issue all relevant government regulations in 2020. Furthermore, the effectiveness of the project’s pandemic response demonstrated continued results in line with the PDO. During the project’s closing workshop on December 17, 2020, the Director General for Teachers and Education Personnel announced that starting in 2021, MoEC will sustain KIAT Guru in 410 primary schools, expand the model to junior secondary schools, and integrate some of the project’s schools to be part of the national reform school models (Sekolah Penggerak) in remote primary schools. The process of generating required decrees for the government to sustain KIAT Guru independently is already in place. Reallocation of resources during the COVID-19 pandemic; yet activities continued. Teacher attendance and teacher service performance improved but were affected by the pandemic. During the period of COVID-19, the co-financing allocations were reported to have dropped due to competing priorities. Project data indicates that on average 18 percent of monthly meetings occurred between March to December 2020. Nevertheless, interviews with stakeholders indicated that even if formal meetings did not occur, in some schools, committee members communicated by WhatsApp or used local communication networks to continue the JSA process, so the actual performance is likely to be higher. Some committees also continued to meet using their own (school and private) resources. JUSTIFICATION OF OVERALL OUTCOME RATING Overall Outcome Rating: Satisfactory Given the project’s substantial achievement of PDO2 and in terms of efficacy and efficiency, the overall rating Page 19 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) is satisfactory (see Table 4 and paragraph 26). Table 4. Performance Assessment Assessment Criteria End of Project Relevance High Achievement of PDO1 Modest Achievement of PDO2 Substantial Efficacy Substantial Efficiency Substantial Overall rating Satisfactory Other Outcomes and Impacts Mainstreaming of tools and processes accelerated in response to COVID-19. MoEC is mainstreaming the project’s key principles to its priority programs, including: (i) utilizing technology to conduct student learning assessment; (ii) strengthening participations from parents and communities to collaborate with principals and teachers to create sustainable ecosystem; and (iii) improving teacher accountability.25 The tools developed under KIAT Guru are now being used at the national level to rollout remote education initiatives. The mainstreaming of the approach beyond the originally intended 386 rural primary schools to a broader context including potentially urban secondary schools demonstrates the potential sustainability of the tools. After the onset of the pandemic, the student assessment process was amended from only a selection of students to 100 percent assessment for all students to differentiate learning abilities to prepare suitable schoolwork that could be completed at home. Since April 2020, the use of the KIAT Guru approach has increased substantially due to the adaptability of the design in the COVID-19 context. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME KEY FACTORS DURING PREPARATION Political and Governance and Sector Strategies and Policies . During preparation, the project faced uncertainties in the political context given an impeding election in April 2019. The design required the national and district governments to issue regulations to tie TKG with teacher presence. The teacher pay for performance approach has political sensitivities as it would involve substantial negotiations with the unions when there are many other issues also to be addressed, so the project was prepared as a low-profile pilot to avoid potential negative political attention. To address this potential risk, the Bank engaged with the Executive Secretary of the Tim Nasional Percepatan Penanggulangan Kemiskinan (National Team for the Acceleration of Poverty Reduction, TNP2K26), under the Office of the Vice President, to facilitate support for KGP2. In December 2018, TNP2K sent a letter to the Coordinating Ministry for Human Development, and the Ministers of MoEC, Ministry of Finance (MoF), and the Ministry of National Development Planning (BAPPENAS) to inform them about the results of KGP1 and plans for KGP2. This created high-level awareness of and readiness to support KGP2 in the high-level government machinery that facilitated internal support as soon as the project 25 The World Bank Group, ISR December 2020. 26 The TNP2K was created through Presidential Regulation Number 15 of 2010 to promote coordination across ministries/agencies to improve the implementation of poverty reduction programmes, improve the living standards of the poor and vulnerable, as well as reduce inequality among income groups. Page 20 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) became operational. This approach enabled the BaKTI team, which was the implementing entity, to work with national stakeholders across ministries to facilitate the preparation and signing of the required policy guidelines even before the election and on a continuing basis throughout the project. Existing district and village ownership of implementation arrangements and budgets. The five district governments selected for KGP2 were the same as for KGP1. KGP2 benefitted from KGP1 where village governments funded the community empowerment activities, as guided by district regulations. The district governments already had strong ownership for the project, proven commitment through allocation of operational funding for KGP1 and in preparation for KGP2, and a willingness to engage and to continue to contribute resources on a gradually increasing basis to take over independent operation of KIAT Guru at the end of the project. Institutional capacity development and technical assistance for implementation and sustainability . During preparation, substantial variations in the quality of capacity amongst district government staff and school stakeholders was noted by the project team. The implementation of the project was dependent largely on the leadership and capacity of the school superintendent, headmaster and committee leaders. Capacity building trainings for all the KGP1 and KGP2 schools were planned; however, an important factor, supported by stakeholder feedback, was that the design incorporated opportunity for on-going technical support. The coordination mechanisms such as quarterly meetings were used as a mechanism to orient and problem-solve with district and village stakeholders. BaKTI assigned several full-time staff to be stationed at and hosted by the district government offices. These staff provided critical on-the-job mentoring and TA for the district government officials throughout the timeline of the project. The feedback from district stakeholders confirmed the importance of this design feature given the innovative approaches that were unfamiliar to them. KEY FACTORS DURING IMPLEMENTATION Institutional coordination at the national and district level are critical to establish the KIAT Guru approach. The level of stakeholder engagement at the national level was important to establish the policy guidance for the district level. A Steering Committee, also known as the National Coordination Team (NCT), was established under KGP1 and continued with KGP2. The NCT is chaired and co-chaired by director general level at MoEC and TNP2K respectively. The NCT members consist of the BAPPENAS, MoF, Ministry of Civil Servants and Bureaucracy Reform, Ministry of Home Affairs, and Ministry of Village, Disadvantaged Areas and Transmigration. Sub-National Coordination Teams (SCT) were established in each of the five pilot districts, under the leadership of the District Head. The roles of the NCT and SCTs were instrumental in: (i) the issuance of national and district government regulations that enabled community participation in improving education service delivery, (ii) appointing schools for expansion, (iii) making the performance-based payment of TKG, (iv) conducting project monitoring, and (v) developing policy recommendations at the end of the project. The engagement of the District Head for overall guidance, the legal section within the district office to generate the required decrees based on the national policy guidance was also important. The success of the institutional arrangements of KGP2 was aided by district circulars which made it clear that the release of village funds could also be utilized for the project. Flexibility in project management facilitated a multi-sectoral approach. The project worked with an NGO to ensure effective delivery at the village level in line with the financial disbursement mechanisms of the project. This approach drew on the expertise of the NGO and allowed flexibility in recruiting multi-sectoral technical Page 21 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) expertise (especially community empowerment, education, local governance) as well as field level facilitators. In addition, the project management had the capacity to engage with district level stakeholders to build coordination and capacity. When the project operations were faced with the realities of the COVID-19 pandemic, the BaKTI team were able to adjust to the requirements. This was instrumental in allowing the project to shift activities to address the urgent needs in the pandemic context. The project tested alternative pathways for implementation towards positive outcomes. The Bank provided technical assistance to help test two alternative approaches through village and school governance structures. Expansion through user committees supported by village government was facilitated by NGO facilitators and selected village cadre (local facilitators). These short-term facilitators assisted the user committees in establishment of the JSA, student assessment and e-KIAT Guru process. Expansion through school governance, on the other hand, was facilitated by school supervisors and directly implemented by school level stakeholders. The project found that both approaches were valid and similar outcomes were achieved. The expansion through the facilitated approach achieved better learning outcomes compared to those through school governance and even to KGP1 schools. However, it could not be determined whether this result is significant statistically compared to the control schools, as data could not be collected from these schools due to the pandemic. The district stakeholders preferred the facilitated approach because it provided more intensive support at the beginning of the project. The performance through the school governance approach was more sustainable, but also variable depending on the capability and workload of the school supervisors. The project experienced a range of challenges related to COVID-19 but pivoted quickly to support the government, as discussed earlier. The pandemic outbreak resulted in the closure of many schools and halted all the project’s field activities within these schools, including the recording of teacher presence. However, monthly teacher performance evaluation meetings continued in schools that were able to operate due to low COVID-19 cases. The MoEC and the five district governments had to reallocate some of their budgets for handling of the pandemic situation. The project changed the capacity building approaches from face to face to online. The training content was adapted to include a focus on supporting children to receive learning according to their abilities and paying further attention to the safety and security of all village stakeholders. Although the number of training hours had to be reduced, the number of beneficiaries reached expanded threefold, as district governments invited supervisors, principals, and teachers in non-project schools so they can conduct distant learning and safely reopen schools during the pandemic. As the grant’s implementing agency, BaKTI completed all project activities, and output targets were either met and, in many cases, exceeded. BaKTI invested significant efforts in establishing an effective team-based support network of technical specialists and facilitators across the five districts. With the strategic management and administrative support from BaKTI headquarter, the project was able to respond to the needs of local conditions whilst achieving clear focus on the expected development outcomes. The staff caliber was high with positive feedback during the ICR mission from government and community stakeholders on the good level of support provided. MoEC and district governments engaged actively throughout implementation. At the national level, MoEC issued the required policy guidelines in a timely manner to enable the KGP2 activities to proceed at the district level. District government stakeholders noted that they were impressed by the speed that policy guidelines were provided by MoEC and attributed some of this success to the facilitation support of BaKTI working with MoEC. At the district level, there was a high level of engagement in the district quarterly meetings as well as Page 22 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) in a series of activities necessary to achieve the project targets. The school superintendents played a key role and exerted commendable extra effort to support the pilot activities and engage with village stakeholders to establish and ensure functionality of the school-based community engagement mechanisms. At the village level, there was also positive engagement by local stakeholders to hold monthly meetings and report on the joint agreement activities. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME QUALITY OF MONITORING AND EVALUATION (M&E) Assessment of Monitoring and Evaluation Rating: Satisfactory M&E Design The design of the Results Framework was appropriate considering the objectives of the project and it was aligned with national priorities. The results framework benefited from the M&E experience during KGP1. There is sound logic between the intermediate and PDO level indicators, all of which contribute to the achievement of the PDO. The alignment of the results framework and M&E activities with national priorities further assists to demonstrate the relevance and good progress of the project in the national context. The project staff successfully collected sufficient data, including gender disaggregated data through the services of BaKTI monitoring and evaluation staff. At the school level, committees and operators were trained on how to generate and track their own progress and maintain records of their meeting activities and achievements. COVID-19 constrained data collection in the latter part of the project and field investigations to validate and interpretation of data was not possible, but monthly online monitoring was conducted between July and November 2020. Some challenges were experienced in aligning data collection with national systems and adapting to changes in context. The project did experience some challenges associated with monitoring the payment of TKG allowances because of internal MoEC systems. MoEC systems paid allowances on a quarterly basis. Project Quarterly Progress Reports (QPRs) covered similar time periods but were prepared prior to the release of the allowances by MoEC. As a result, allowance figures reported in the QPRs could not reflect current data and instead included previous amounts. Additionally, if the conditions had been different there may have been changes to the monitoring framework to reflect activities undertaken in response to COVID-19. However, given that these changes were undertaken in an emergency context and the team was adapting to circumstances, the adaptive approach of tracking activities was valid. M&E Implementation The regular and robust collection of data for most indicators facilitated a strong M&E performance. Data was collected and documented regularly by BaKTI for all indicators allowing for good tracking of project progress. School operators facilitated monthly monitoring data collection and provision to the district operator. The support provided to the district operator by the project’s data monitoring specialist helped to address data collection issues. The tracking of project progress was well recorded in QPRs except for teacher allowance figures as discussed above. Regular progress reports were presented to the national government stakeholders through the National Steering Committee and through targeted meetings with senior officials. Page 23 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) The ICR mission interviews with MoEC officials confirmed that the project provided regular, timely and complete information to national counterparts which facilitated the release of budgets and assisted in generation of the required policy guidelines to the districts to enable project implementation. Tracking of progress experienced some gaps and understanding the underlying reasons for results was constrained by the pandemic. While BaKTI could track the budget allocated, there were some difficulties in tracking actual expenditure due to the complexities and delays in government expenditure records making tracing of specific payments to KGP2 schools difficult to isolate. The project adapted to M&E challenges associated with COVID-19 and largely transitioned well to remote monitoring by using Google Form distributed through WhatsApp groups. The pandemic did not cause any delays in the collection of data and reporting requirements were maintained. However, the necessitated shift to remote monitoring did undermine the richness of some qualitative data. For example, the July 2020 ISR notes that “Project monitoring could not explain why there have been an increase in teacher tardiness and unexcused absences, as follow up spot check visits to the field could not be conducted.�27 M&E Utilization The tools and data generated through the project assisted broader evaluation activities to inform shifts in the project approach as well as national government activities in response to COVID-19. M&E activities at the project level and the knowledge of project staff informed scale-up of the rapid learning assessments to the national level to assist in the development of a ‘new normal’ protocol. Robust reporting of progress allowed the project to compare the effectiveness of the different approaches adopted over the project period. For example, the December 2020 ISR reports disaggregated results based on the level of project facilitation support to conclude that shorter term support to communities was more effective than longer term support.28 Similarly, monitoring of project progress and successful approaches by BaKTI informed the development of training material for use in remote learning during the COVID-19 pandemic such as linked teaching packages for teachers and parents. Justification for Overall Rating of Quality of M&E The overall rating for quality of M&E is Substantial. M&E design, implementation and utilization were all satisfactory despite some challenges and gaps. Data collection was strong over the project period and informed regular reporting of project progress. Gaps were present in tracking funding expenditure and understanding the factors contributing to level of achievement. Nonetheless, the data and outputs generated from project M&E activities have been widely used to inform both project level conclusions and national level adaptation to remote learning. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental and Social Safeguards compliance were Satisfactory. The project’s risk category was assessed as moderate (Category B – partial assessment under OP 4.01) and supervision was conducted through the Bank missions. This project provided TA to MoEC and five priority districts to scale up the scope of KIAT Guru interventions, which include development of training modules, capacity enhancement for a simplified CEM and introduction of e-KIAT Guru apps. No civil works were financed under the project.29 The project triggered OP 4.01 on Environmental Assessment and 4.10 on Indigenous Peoples. The latter was triggered due to the 27 The World Bank Group, ISR, July 2020. 28 The World Bank Group, ISR, December 2020. 29 The World Bank Group, ISR, July 2020. Page 24 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) project's geographic coverage, including rural areas where Indigenous Peoples as per the policy criteria are present. Given the project worked closely with each district and village to align the policy requirements to local conditions, Indigenous Peoples considerations were embedded in the facilitation processes. All Occupational Health and Safety (OHS) standards were applied and, during the COVID-19 pandemic, the project imposed no travel policy to minimize OHS. During project implementation BaKTI was responsible for implementing OHS procedures. During Bank supervision missions, no issues were identified, and safety of staff was considered satisfactory. Normal project activities requiring face to face stakeholder engagement had been converted to virtual communication during COVID-19 pandemic to minimize public health risks. Additional public health measures in line with the government regulations were prepared and introduced for stakeholder engagements.30 The TA activities, recommendations and advice provided through the project support were not expected to generate adverse environmental and social implications. The Project Operations Manual (POM) contained a negative list which established ineligible expenditure categories, which included involuntary land acquisition and physical construction fall under. Procedures for consultations with Indigenous Peoples and participating communities to ensure broad community support and social inclusion were governed in the POM and monitored during project implementation. Given the village-based approach, inclusion activities were supported during training on a case-by-case basis. Project facilitators and village cadres received appropriate training and mentorship support from community development specialists on community consultation and participatory engagement approaches.31 Fiduciary Management & Procurement Financial Management (FM) performance is rated Satisfactory. The FM system worked adequately during project implementation. Project accounting and reporting used BaKTI’s existing accounting system and had adequate internal control. BaKTI improved its financial system software in early 2020, from Quickbook to SAKTI (Sistem Aplikasi Keuangan BaKTI, BaKTI’s financing application system). The new system improves financial management performance and synchronizes data from all sub-system (projects) in an integrated manner. It has helped improve BaKTI’s overall financial management control as well. Internal Audit. BaKTI headquarters had an internal audit function and carried out reviews over subsidiary offices at least each semester during project implementation. The team conducted physical checks, field verification, direct confirmation with regional staff and external parties related to project implementation. BaKTI should keep the internal audit function and expand to all activities in the future. In order to improve independency and maintain the overhead cost for the internal auditor, BaKTI may hire or sub-contract the internal audit function to an external party (consultant). External Audit. BaKTI prepared an institutional financial report which included project expenditures. The Bank required BaKTI to submit the institutional audit report during project implementation and the audited financial statements were furnished to the Bank within six months after the end of the fiscal year. The auditor expressed unqualified opinions for all fiscal years of project implementation. BaKTI’s Financial Report for FY 2020 has been audited and should be submitted to the Bank prior to June 30, 2021. 30 The World Bank Group, ISR, July 2020. 31 Ibid. Page 25 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) BaKTI’s procurement performance was Satisfactory. The project procurement risk was rated as Moderate. Generally, procurement was carried out in accordance with the applicable World Bank’s Procurement Regulations for Investment Project Financing (IPF) Borrowers and the agreed Procurement Plan. All contracts were successfully processed and recorded in STEP32 and no major issues related to fiduciary aspects were found. The procurement capacity of BaKTI has improved under the project and they may potentially continue to apply the lessons learnt and best practices from this project in their future procurement. BANK PERFORMANCE Assessment of Bank Performance Rating: Satisfactory Quality at Entry The Bank’s performance at entry was Satisfactory. The project was responsive to GoI priorities and systems and designed KGP2 with a sound concept based on international experience and good practice. BaKTI was selected through a limited selection procedure. Eight NGOs were invited to attend an information session on the project, and to submit a proposal on providing information on managerial and operational experience and other administrative requirements (see also Key Factors During Preparation).33 Quality of Supervision The Bank team supervision was Satisfactory. The team carried out regular supervision and provided additional technical support both to BaKTI and MoEC and partners as required. Supervision was satisfactory and was supplemented with Bank technical and advocacy support. Risk-based supervision of project financial management was conducted. This involved desk supervision, including review of Interim Financial Reports and one supervision mission per year. Financial management supervision was to be conducted by Financial Management Specialist and a Bank consultant. Procurement supervision and enhanced hands-on support for implementation of the Bank’s STEP were provided by the Accredited Procurement Specialist with the support of a Procurement Program Assistant. Supervision missions were conducted as scheduled, and attention was provided to the project when changes were required in response to COVID-19. In addition, several meetings were conducted with senior government officials when required to facilitate project activities. The team worked on the required restructuring paper submissions. The Bank team also provided guidance on how the KGP2 could link with other initiatives to strengthen the social accountability and education service delivery ecosystem in Indonesia. RISK TO DEVELOPMENT OUTCOME Government feedback indicates a likelihood of sustainability of impacts over time with minimal risks. The broad engagement of all stakeholders involved in education and simultaneous capacity building activities 32 The World Bank Group - Systematic Tracking of Exchanges in Procurement. 33 The World Bank Group, (2019), Project Paper PP2940, Proposed Grant to the Bursa Pengetahuan Kawasan Timur Indonesia (BaKTI) for KIAT Guru Phase 2 – Improving Teacher Performance and Accountability. February 2019. Page 26 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) promotes a more coherent approach towards improving primary education service delivery in remote areas. Feedback from MoEC during the ICR mission emphasised the high value of KIAT Guru’s efforts to strengthen the ‘education ecosystem’ by increasing engagement of parents and community stakeholders in education. Ownership by stakeholders contributes to a higher likelihood of sustainability as project activities are likely to be continued. Furthermore, knowledge and capacity changes through training activities increase the propensity of stakeholders to continue project activities for ongoing results. The project actively invested in sustainability mechanisms . The engagement of district and village governments to provide operating budget for the activities demonstrated commitment during implementation. Budget allocations are set annually so continuity is difficult to predict. However, discussions with stakeholders indicate that most participating districts and villages have allocated funding to continue KIAT Guru for 2021 and 2022. As part of a planned exit process, in April 2020, the World Bank team held discussions with MoEC regarding maintaining project outcomes through continuing, mainstreaming, replicating or upscaling project activities through the Ministry, and particularly through policy reforms and regulations. The Ministry indicated a desire to upscale the KIAT Guru approach as a national policy for improving frontline education services in all remote schools. This was formalized through a request from MoEC to the Bank to request support for the scale up. The issuance of relevant regulations reduces the risks to development outcomes. The proactive engagement of national and district level stakeholders in the project approach worked to institutionalize the KIAT Guru approach. The guiding materials (manuals, draft regulations, etc.) helped to guide implementors across the five districts to operate independently. The materials also facilitated the transition to remote learning in response to COVID-19. The use of regulations provides a mechanism for institutionalization of results and minimizes risks to development outcomes. Nonetheless, there are risks that the procedural gains may be undermined by shifting and competing priorities of MoEC, for example, improving the level of educational achievement in secondary schools and negotiations with the teacher’s union regarding teacher working conditions. The KIAT Guru approach has been mainstreamed but gaps in capacity and documented M&E and scale-up processes, as discussed earlier, represent a risk to development outcomes. The internal capacity that has been built by the project supports the likelihood of sustainability. The transfer of the e-KIAT Guru monitoring system has occurred at national level and at district level but it is not yet fully operational with trained staff in place. The teacher score and student assessment systems are also immature. After the national coordination meeting in December 2020, where it was announced that MoEC will sustain the program, the Bank (through the accompanying KGP2 Advisory Services and Analytics [ASA]) has continued to provide TA to MoEC and district governments to take over the implementation and e-KIAT Guru management. In early 2021. MoEC conducted trainings for the operators at the school and district level to strengthen the capacities in managing the e-KIAT Guru app independently. MoEC has also set up a task force to coordinate directly with the district government officials. Training has achieved good results but if staff turnover occurs, this will lead to variability in capabilities, if refresher training is not available. Furthermore, some school level participants were computer literate and could easily operate the e-KIAT Guru app while others were less able. The project approach is innovative and carries inherent risk factors. The project paper stated that “the government may use the findings of the KGP1 in a new context without sufficient evidence of effectiveness,� Page 27 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) which was suggesting that the KIAT Guru approach was not yet sufficiently mature.34 An underlying premise of the KGP2 was to simplify and mainstream the facilitation processes and to test different approaches to scale-up. The innovation approaches in the pilot project are new and therefore unfamiliar to implementors. There are risks that the approach may be replicated within different contexts or without the similar level of support. The project paper also noted that this risk could be mitigated by providing “a clear description on the specific conditions to be applied to ensure that the proposed modalities will work at scale up.�35 The guidelines and supporting material development through KGP2, plus the strong participatory monitoring and evaluation of the project will assist in mitigating this risk. V. LESSONS LEARNED AND RECOMMENDATIONS Lessons Learned Both KGP1 and KGP2 showcased that it was possible for the government to pilot and make asymmetric policies that are applicable only to certain beneficiaries. Despite being a very heterogenous country, the business-as-usual approach for the GoI in issuing policies was to use a blanket approach applicable to all nation- or district-wide beneficiaries. In KGP1 and KGP2, MoEC and five pilot districts issued asymmetric policies that were applicable only to pilot schools. In 2020, during the pandemic, MoEC consulted the Bank’s task team to adopt this asymmetric policy approach to target additional school operational funds for schools serving disadvantaged students and regions. Interventions for educational outcomes require a multi-stakeholder approach and good coordination at national and district level to ensure coherence at a village level. KIAT Guru was effective in using the quarterly district stakeholder meetings as a mechanism to bring representatives from different sectors together. Although challenging to implement, it provided an active forum for discussion, to agree on solutions and for joint action. Similarly, the meetings were a mechanism for rapid feedback on any concerns so that these were addressed in a timely manner, regardless of the responsible entity. The combination of improved teacher supervision and activated village school committees achieved positive results in improved student learning outcomes. The two-pronged approach of KIAT Guru is mutually reinforcing, achieving results beyond only improved teacher attendance. The active dialogue between the parents and teachers stimulates the local education ecosystem that generates additional benefits such as improved communication, and other ad-hoc school improvement projects. The two different models of community engagement tested during the KGP2 were both effective. The KGP2 testing of two mechanisms for scaling up the approach found that both models trialled achieved similar results in terms of activation of committee engagement in education supervision. External facilitation of the regular meetings was important, regardless of the model, to provide technical support and guidance. Use of trained facilitators did assist in mobilizing the community and user committees, who were also supported by village government to empower their position vis-à-vis the school. In the more mainstreamed model, the role of the school supervisor was like the external facilitator but tended to take longer to mature. The one-time 34 The World Bank, Project Paper PP2940, Proposed Grant to the Bursa Pengetahuan Kawasan Timur Indonesia (BAKIT) for KIAT Guru Phase 2 – Improving Teacher Performance and Accountability. February 2019. Pg. 22. 35 Ibid. Page 28 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) investment costs for both models were comparable but their focused role in the project allows them to strengthen their facilitation capacity and conducted trainings for beneficiaries at the sub-district level. In contrast, the training for beneficiaries supported by the school supervisors need to be conducted at the district level, to take advantage of those with stronger facilitation capacity, and support from district-level project staff. Building KGP2 from the KGP1 pilot was an effective approach to have continuation of the learning . During design there was interest from stakeholders to shift from KGP1 to other schools for the purpose of equity and spreading benefits further into the district. The eventual design agreed was to build from KGP1 as well as expand into new schools. This provided double benefit because the implementation was able to track progress in the original cohort, assess implementation in the new cohort as well as identify any differences in the new innovations introduced in KGP2. This amplified the learning potential. The training of school supervisors and placing them as key implementation actors turned out to be an important mechanism in scaling up the KIAT Guru approach and other education reforms. Different from KGP1, the implementation of KGP2 was strongly embedded within the government system, focusing on improving the capacity of school supervisors and advocating for the strengthening of their role. The school supervisors traditionally are not the target recipients of capacity building by the government, as their role is considered administrative, and most of them are rather senior and nearing retirement age. KGP2 training content provided these supervisors with knowledge on substance and equipped them with tools to conduct school supervision. BaKTI district teams also advocated to the district governments to improve resources to enable school supervisors to meet regularly at the district level with the education department. In two districts, Sintang and Ketapang, district governments allocated additional funding for school supervisors to visit remote schools. Ketapang also hired more school supervisors to fulfill the national standard of one supervisor per ten schools. During the pandemic, the project conducted webinars with these trained supervisors, equipped them with MoEC and project materials to support schools to conduct distant learning, and supported them to mobilize school principals to disseminate the materials. Strong national, district and local engagement enabled the project to take advantage of opportunities for innovation and adaptation in a COVID-19 environment. The project faced a critical contextual change when the COVID-19 pandemic occurred. Potentially many of the gains achieved could have been lost in terms of teacher attendance and monthly meetings. However, the active relationship between the BaKTI, the Bank and MoEC enabled a rapid shift in activities in response to the situation. At the district and local level, the project staff and stakeholders remote dialogue resulted in a range of innovations to find solutions for home-based schooling and continued action by the teachers and community. Recommendations The detailed coordination mechanisms applied within KGP2 need to be orchestrated across the national, district and local level to generate the education ecosystem strengthening results of KGP2. One of the most important learning from KGP2 is that a multisectoral coordinated approach is critical to the success of the project. Any further initiatives drawing on the KIAT Guru experience need to incorporate mechanisms that facilitate strong and coherent partnerships. This approach of the project needs significant consideration in design of any future phases. This could be added into the TOC during the design of any future project phases. Page 29 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) The e-KIAT Guru app is still at an early stage and users need training and further ongoing support. There is a need to ensure the people in the district are comfortable using the e-KIAT Guru app and have adequate training to build capacity to use it effectively and confidently. Training processes need to be reviewed to ensure that there is sufficient support to the trained individuals at the school level, including the consideration of the risk of only one person per school being trained. As noted previously, the MoEC have announced that they will sustain the program. However, there is still also uncertainty about how the system would be maintained beyond the end of the project and concerns surrounding the absence of ownership of the system and analysis capability at the district level to use the e-KIAT Guru App to maximum effect. There is some risk to sustainability related to the use of the app without ongoing assistance. Guidelines need to be imbedded within the village funding mechanisms to include KIAT Guru activities in future village fund budgeting. The allocation of village funds is at the discretion of each village’s leadership. While there are guidelines from the national government on how to allocate village funds, there is variability across villages. With differing priorities across the villages, KIAT Guru sometimes seen as a lower priority than other uses for the funds. There is a need to secure ongoing funding for the program and to ensure financial sustainability. This requires further coordination by the national and district governments for sustainability of operations. Strategic action is required to capitalize on the learning from the pilot towards scale-up. The ICR mission demonstrated that there is strong interest at the district level in the program being extended to all remote schools across the district; however, the process for scale up is not yet clear. District leaders have signalled commitment of political will and funds to continue operations for 2021. However, the political process of the potential for annual changes in leadership and changing priorities means that longer term planning and budgeting is not yet possible. This relies on the need for continual re-orientation of district leadership to continue the program. Similar challenges occurred at the national level with MoEC. Adapting a similar approach to improve frontline service delivery. Although the project was implemented at a very small scale, both phases indicated the promise of social accountability and teacher performance-based pay in improving education service delivery and learning outcomes. A similar approach can be utilized for improving frontline services in other sectors. This can start with identification of a few service performance indicators that frontline service providers are accountable for, which can be independently and objectively verified by community members. Community members should have access to data and information on frontline service performance indicators in comparison to other regions. Service providers and community members should agree upon a simple and (ideally) locally-set joint service agreement, collaborate on the supply-side and demand-side efforts to achieve the indicators, and follow up on a frequent basis to discuss progress and address challenges. Finally, service providers’ allowances should be tied to a few objective indicators of service performance. KGP1 and KGP2 showcased that this allowance can be funded by either the district or national governments, that the amount of cut in total was less than 5 percent, but very effective in improving target service delivery outcomes. . Page 30 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: To improve teacher presence and teacher service performance in pilot schools. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Reduced percentage of teacher Percentage 6.15 2.00 2.00 8.32 tardiness 01-Mar-2019 31-Dec-2019 30-Nov-2020 31-Mar-2020 Comments (achievements against targets): Unmet PDO target for teacher tardiness. Results show that between June and September 2019 teacher tardiness increased, between September and October 2019 some progress was made, and teacher tardiness decreased marginally. Data from February shows a marked increase in teacher tardiness, reported to be a result of the COVID-19 pandemic as by February the pandemic had already begun, and the risk of spread was well known. However, data from March 2020 again shows a decrease in tardiness back towards the target. Final achievement was 8.32 percent which was higher than the baseline level. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Reduced percentage of Percentage 15.00 5.00 5.00 8.28 teachers with unexcused 01-Mar-2019 31-Dec-2019 30-Nov-2020 31-Mar-2020 Page 31 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) absence Comments (achievements against targets): Unmet PDO target for unexcused absences. Results show that progress from the baseline was evident, with the target being achieved in September 2019. However, unexcused absences surged in February and March 2020 corresponding to the onset of the pandemic. The overall teacher presence decreased from 95.13 to 90.80 percent in the same period demonstrating a link between attendance and unexplained absences. Due to the pandemic constraining access to the schools for monitoring visits, it was not possible to verify the reason for the increase, but it is likely that the reversal in performance was related to the external context of the pandemic rather than a decline in validity of the KGP2 approach. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Improved teacher performance Percentage 0.00 85.00 85.00 96.50 score 01-Mar-2019 31-Dec-2019 30-Nov-2020 31-Mar-2020 Comments (achievements against targets): Substantially exceeded PDO target on teacher performance score. Project stakeholders attributed the growth in score to the greater involvement and capacity of the district stakeholders including school supervisors in local education management, the improved relationship between teachers and parents through the Joint Agreement, and recognition of the efforts to maintain educational quality for students during the pandemic. A.2 Intermediate Results Indicators Component: The project has one project component. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Relevant government Yes/No N Y Y Y Page 32 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) regulations issued 01-Mar-2019 31-Dec-2019 30-Nov-2020 30-Nov-2020 Comments (achievements against targets): Achieved target for relevant government regulations to be issued in 2019 and 2020. This included the Minister of Education's regulation which identified the Project as a National Priority Program for delivery of Tunjangan Khusus Guru (Teacher Special Allowance or TKG). The project also facilitated and provided policy recommendation to MoEC for the issuance of the key regulations in 2020, which was not part of the original target. The project recorded 8 national level regulations and 61 district government regulations to support the implementation of program activities in five districts. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Monthly teacher evaluation Number 0.00 2800.00 3400.00 2816.00 meetings conducted 01-Mar-2019 31-Dec-2020 30-Nov-2020 30-Nov-2020 Comments (achievements against targets): Unmet revised target on monthly teacher evaluation meetings conducted. Progress towards this indicator was positive resulting in an increase in the target from 2,800 to 3,400 meetings in December 2019. However, in the COVID-19 context, social gatherings and meetings were constrained, resulting in an achievement level of 2,816 meetings. This equated to 81 percent of the revised target and exceeded the original target of 2,800. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of MoEC and district Number 0.00 200.00 1600.00 2607.00 government officials received capacity building 01-Mar-2019 31-Dec-2020 30-Nov-2020 30-Nov-2020 (disaggregated by gender) Page 33 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) Comments (achievements against targets): Substantially exceeded number of MoEC and district government officials receiving capacity building. BaKTI, the Implementing Agency, provided implementation and capacity development supports for the government officials to administer teacher performance evaluation scores, verify the reports, conduct payments of TKG based on teacher presence, and visit underperforming schools. The project recorded 2,607 MoEC and district government official received capacity building support, comprise of 2,154 male (83%) and 453 female (17%). Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of school stakeholders Number 0.00 1200.00 4000.00 7398.00 received capacity building trainings (disaggregated by 01-Mar-2019 31-Dec-2019 30-Nov-2020 30-Nov-2020 gender) Comments (achievements against targets): Substantially exceeded number of school stakeholders receiving capacity building trainings substantially. By end of February 2020, all schools had established joint agreements between teachers and parents to improve learning environments at school and at home. The target for this indicator was increased from 1,200 to 4,000 following good initial progress of the Project. The project exceeded both the original and revised target with a total of 7,398 stakeholders at school level receiving training which was comprised of 4,918 male (66 percent) and 2,480 female (34 percent). Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Percentage of TKG paid based Percentage 0.00 85.00 85.00 98.63 on teacher presence 01-Mar-2019 31-Dec-2019 30-Nov-2020 31-Mar-2020 Comments (achievements against targets): Substantially exceeded percentage of TKG paid based on teacher presence. From September 2019 the target was met and exceeded with almost 100 Page 34 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) percent of TKG being paid based on teacher presence by March 2020. Page 35 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1a: Improve teacher presence 1. a. (i) Reduced percentage of teacher tardiness and Outcome Indicators 1. a. (ii) Reduced percentage of teachers with unexcused absence 1. Relevant government regulations issued. 2. Number of MoEC and district government officials received Intermediate Results Indicators capacity building. 3. Percentage of TKG paid based on teacher presence PDO 1.a. (i) 8.32% of teacher tardiness compared with a baseline of 6.15% and a target of 2% PDO 1. a. (ii) 8.28% of teachers with unexcused absences compared with a baseline of 15% and a target of 5% Key Outputs by Component Intermediate Results indicators (Linked to the achievement of the Objective/Outcome 1) 1. 8 national and 61 district level regulations have been issued (Accomplishment 100%) 2. 2,607 MoEC and district government officials received capacity building of a target of 1,600 (Accomplishment 163%) 3. 98.63% of TKG paid based on teacher presence compared with a target of 85% (Accomplishment 116%) Objective/Outcome 1b: Improve teacher service performance Outcome Indicators 1b. Improved teacher performance score Page 36 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) 4. Monthly teacher evaluation meetings conducted. Intermediate Results Indicators 5. Number of school stakeholders received capacity building trainings PDO 1b. 96.5% teacher performance scores Intermediate Results indicators Key Outputs by Component 4. 2,816 monthly teacher evaluation meetings have been conducted (Linked to the achievement of the Objective/Outcome 2) compared with a revised target of 3,400 (Accomplishment 83%) 5. 7,398 school stakeholders received capacity building trainings compared with a revised target of 4,000 (Accomplishment 185%) Page 37 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Actual at Percentage of Category Description Approval Project Closing Approval (USD) (USD) Goods, non-consulting services, consulting services, training & workshops, 1,348,000 1,347,762 99.98 and incremental operational costs36 Total 1,348,000 1,347,762 36Incremental operational costs include local contractual support staff salaries (implementation support personnel); travel and other travel-related expenditures; equipment rental and maintenance; vehicle operation, maintenance and repair; office rental and maintenance; materials and supplies; utilities and communications expenses; consumables; bank charges; and advertising expenses. Page 38 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) ANNEX 3. ECONOMIC ANALYSIS KGP2 operated in a manner that was efficient in terms of satisfactory use of available resources, mobilization of counterpart funds and in relation to cost benefit for generation of outcomes. The procurement of an experienced and skilled team to implement the project, continuing from KGP1 ensured continuity of resources that avoided a new orientation phase and capitalized on the relationships and resources already established. The project utilized the learning and capacity built during KGP1 as a basis for design and to enhance achievements. Applying the KGP2 interventions into the KGP1 cohort of 270 schools allowed the project to build upon the results achieved under the first phase while at the same time establishing the same process in 140 new schools. This longitudinal approach contributed to a higher likelihood of sustainability due to reinforced district capacity. Furthermore, adding the pilot activities of phase two (facilitation versus internal school governance) allowed for an additional layer of experimentation to measure differences in outcomes and to identify the most effective and efficient mix of activities. GoI paid for 39 percent of the total project implementation costs and matched 64 cents for every dollar that the Bank invested. At design, it was noted that the cost estimates for implementing KGP2 were based on the costs associated with implementing KGP1 but with adjustment related to the changes in methodology, expansion of scope and with consideration of exchange rate fluctuation. The project was funded by USAID’s Local Solutions to Development Single Donor Trust Fund (SDTF), with a total of USD 1,347,97337 from April 2019 until December 2020. Over the same period, MoEC, district and village governments allocated a total of USD 866,742 (Table 3. 1), bringing the total project cost to USD 2,214,742. Table 3. 1. Government of Indonesia’s Funding Contributions Year MoEC District Village School Total 2019 USD 325,942 USD 241,308 USD 124,971 USD 41,733 USD 733,954 2020 USD 9,337 USD 81,770 USD 32,105 USD 9,575 USD 132,787 Total USD 335,279 USD 323,078 USD 157,076 USD 51,309 USD 866,742 Source: analysis of project expenditure tracking. GoI co-financing focused on training and workshop and other expenses which could be sustained beyond the project’s timeline. Based on the category of expenses (Table 3.2), MoEC financed training and workshops at the national level, while those conducted at the district level were resourced by district governments. In addition, some district governments also allocated transportation funding for the school supervisors and hired additional staff to implement the project. Village governments allocated funding to support the monthly meetings as well as operational expenses of the user and school committees. On average, village governments provided USD 191 per school per year, equivalent to 0.3 percent of annual village funds. Hence, even after the project ended, in theory activities at the village level can be continued by village governments utilizing the village funds. The district of Ketapang also circulated a letter to all the participating schools to allocate their school operational funds to purchase Android-based phone utilized for e-KIAT Guru. 37 Some variation in these numbers is due to fluctuations in exchange rates. Page 39 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) Table 3.2. Expenditures from SDTF and GoI by Category of Expenses Category SDTF GoI MoEC District Village School 1. TA/ Consulting services USD 474,967 USD - USD 17,534 USD - USD - 2. Training & workshop USD 347,026 USD 293,774 USD 269,229 USD 157,076 USD 32,939 3. Goods USD 47,329 USD - USD - USD - USD 18,369 4. Non-consulting services USD 200,789 USD 41,505 USD 36,315 USD - USD - 5. Incremental Cost USD 239,191 USD - USD - USD - USD - 6. Indirect cost USD 38,670 USD - USD - USD - USD - Total USD 1,347,973 USD 335,279 USD 323,078 USD 157,076 USD 51,309 Source: analysis of project expenditure tracking. The SDTF funding, on the other hand, focused on supporting the GoI on one-time investment needed to introduce the approach. The SDTF allowed much more flexibility compared to GoI co- financing and enabled BaKTI to identify and fill the funding gaps. Most of the SDTF was utilized for technical assistance/ consulting services, training and workshop, transportation costs for project staff to visit the schools, and purchasing of Android-based phones. Findings from cost analysis of the expansion to new schools piloting the two approaches provides insights on the cost of scaling up. The 410 schools covered in KGP2 consisted of: (a) 203 schools from KGP1, which continued to be supported by school supervisors, and (b) 207 expansion schools, 111 of which were facilitated by the project’s community facilitators (as part of Group A approach: village governance), and 96 were supported by school supervisors (as part of Group B approach: school governance). The average number of students per school is maintained at 132, to make it comparable with KGP1 cost analysis. The one-time investment cost to roll out KIAT Guru through village governance is USD 2,342 per school, or USD 17.74 per student. The expansion model through village governance (Group A) necessitates the hiring of NGO facilitators to set up the user committee, develop joint agreements, train local village cadres and support up to three meetings at the village level. One hundred and eleven schools implemented this Group A approach, with a total cost of USD 259,980 (Table 3.3). MoEC and BaKTI funded the training of the facilitators, with a total of USD 43,425, or USD 391 per school, and USD 2.96 per student. The facilitators then conducted three trainings to project beneficiaries on the following topics: (a) developing joint agreements; (b) conducting evaluation of joint agreements, which included implementation of citizen-led learning assessment; and (c) using e- KIAT Guru. The cost of these trainings was USD 106,995, or USD 964 per school, and USD 7.30 per student. The project facilitators were hired for eight months with funding largely originating from the SDTF with a few facilitators hired directly by the district governments. The cost of transportations for the facilitators to visit between 5 to 6 schools under their purviews were covered by BaKTI. Table 3.3. One-time Investment Cost for Expansion through Village Governance (Group A) One-time Investment One-time Investment One-time Investment per School per Student Training of Trainers USD 43,425 USD 391 USD 2.96 Training of Project Beneficiaries USD 106,995 USD 964 USD 7.30 Salary USD 60,117 USD 542 USD 4.10 Operational Cost (transport, copy materials, etc) USD 31,472 USD 284 USD 2.15 Initial Meetings USD 17,970 USD 162 USD 1.23 Total USD 259,980 USD 2,342 USD 17.74 Source: analysis of project expenditure tracking. Page 40 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) The one-time investment cost to roll out KIAT Guru through school governance averaged USD 2,420 per school, or USD 18.33 per student. The expansion model through school governance (Group B) leveraged the existing school supervisors to revitalize the school committee and develop joint agreements. A total of 96 schools implemented the Group B approach, with a total one-time investment of USD 228,936 (Table 3.44). MoEC fully funded the training of the supervisors, with a total of USD 27,885, or USD 290 per school, and USD 2.20 per student. The cost of training in Table 3.4 is an estimate given that the trained school supervisors covered all 410 schools. The supervisors, supported by project facilitators, conducted three trainings for the project beneficiaries covering the same topics as Group A. These trainings were more expensive compared with Group A, because they were centralized at the district level, to account for more limited number of supervisors who could conduct trainings independently. The salary of the supervisors came directly from the district governments, estimated to be USD 651 per month for eight months of facilitation to schools. The operational cost that could be tracked only cover additional expenditure for the project, with most supervisors utilizing transport money they received as part of their business-as-usual visit to schools. Table 3.4. One-time Investment Cost for Expansion through School Governance (Group B) One-time Investment One-time Investment One-time Investment per School per Student Training of Trainers USD 27,885 USD 290 USD 2.20 Training of Project Beneficiaries USD 124,446 USD 1,296 USD 9.82 Salary USD 58,499 USD 609 USD 4.62 Operational Cost (transport, copy materials, etc) USD 3,519 USD 37 USD 0.28 Initial Meetings USD 17,953 USD 187 USD 1.42 Total USD 232,302 USD 2,420 USD 18.33 Source: analysis of project expenditure tracking. The one-time investment cost of both KGP2 approaches were 46.3 percent and 47.2 percent of per school costs of KGP1 for group A and B respectively. The Group A approach, at USD 17.74 per student, was 44 percent KGP1 cost, and slightly cheaper than Group B. The Group B approach, at USD 18.07 per student, was 45 percent KGP1 cost. The cost savings came from lessons learned from KGP1 (Table 3.5). The CEM process was reduced from a total of seven meetings in KGP1 to four meetings in KGP2. KGP1 facilitators were employed over 15 months, compared to 8 months in KGP2. Each facilitator visited a school with an average of 11 visits, compared to 3 visits. As the facilitation process has been simplified, less experienced facilitators were employed in KGP2. Table 3.5. Comparison of KGP1 and Group A and B of KGP2 KGP2 Group A KGP2 Group B Cost KGP1 USD % of KGP1 costs USD % of KGP1 costs One-time investment N/A N/A 1,026759 259,980 232,302 total different scope different scope One-time investment 5,058 2,342 46.3% 2,420 47,8% per school One-time investment 40 17.74 44.4% 18,33 45,8% per student Source: analysis of project expenditure tracking. Page 41 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) Table 3.6. One-time Investment Cost for KGP1 One-time Investment One-time Investment One-time Investment (203 Schools) per School per Student Training 143,889 709 6 Salary 501,483 2,470 19 Transport 252,888 1,246 10 Initial Meetings 128,499 6,333 5 Total 1,026,759 5,058 40 Source: analysis of project expenditure tracking. The average annual cost to sustain KGP2 approach was USD 1,872 per school, or USD 14.18 per student, equal to 85.3 percent of the annual cost to sustain KGP1, at USD 2,182 per school or USD 17 per student. This cost estimate was generated from the 203 KGP1 schools which were sustained as part of the government support and oversight (Table 3.5). It included the cost for developing and maintaining e-KIAT Guru application, including the purchase of an Android-based phone every two years. The project beneficiaries in these schools received refresher trainings to introduce the simplified version of the joint agreement and the e-KIAT Guru application. On a monthly basis, the stakeholders at the village level met to discuss the joint agreement progress and conducted evaluation meetings at the end of the semester which cost USD 57,823, or USD 285 per school, and USD 2.16 per student based on pre-pandemic data. At the district level, the supervisors met on a quarterly basis to verify the data collected at the village level, and the District Coordination Team conducted another meeting. Some school supervisors then conducted spot check on underperforming schools, with associated operational costs. The supervisors’ salaries were not accounted for, as the activities fall under their roles and responsibilities. See Table 3.7 for cost per component of annual sustaining costs. Table 3.7. Annual cost to sustain KGP2 Annual Cost per Annual Cost Annual Cost per School Student e-KIAT Guru development, phone purchase, and USD 101,699 USD 501 USD 3.80 maintenance Training USD 182,914 USD 901 USD 6.83 Monthly & evaluation meetings at village USD 57,823 USD 285 USD 2.16 Verification & coordination meeting at district USD 8,004 USD 39 USD 0.30 Operational Cost (transport, copy materials, etc) USD 29,477 USD 145 USD 1.10 Total USD 379,917 USD 1,872 USD 14.18 Source: analysis of project expenditure tracking. Page 42 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) ANNEX 4. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Summary of Stakeholder Consultation The ICR consultation took place over two distinct periods in January 2020 and November 2020. This was due to extension of the project due to the occurrence of the COVID-19 pandemic and resultant closing date extensions. There was no significant difference in the feedback between the two periods apart from the updates regarding adaptation of the KGP2 processes because of the pandemic. Summary of National/Ministry Level Stakeholders The feedback from senior MoEC officials during ICR mission meetings was that the objectives and approach of KIAT Guru is very strongly aligned with Government priorities related to education and that the focus of the project had been guided by MoEC in collaboration with other Ministries, particularly the BAPPENAS and the Ministry of Villages. In particular, there is a shared emphasis on the importance of improving teacher performance. The stakeholders reported that they had been adequately informed of the progress of the KGP2 activities. There had been changes in personnel in the senior roles in MoEC during the project, but since the new incumbents had been drawn from within the Ministry, there was still good knowledge regarding the project. BaKTI, with the Bank staff, had been instrumental in ensuring good orientation of new leadership. KIAT Guru activities contributed to the development of policies and were mainstreamed and institutionalized for sustainability. Based on KIAT Guru results, and the Bank’s policy inputs, MoEC issued a national regulation in 2020 on the data-driven appointment of schools for the distribution of additional school operational funds and special allowance. KIAT Guru’s school and user committees membership criteria have been institutionalized in line with the 2016 MoEC Regulation number 75 for school committees. The Student Learning Assessments developed under KIAT Guru have been adapted nationwide to conduct minimum competency assessments. The alignment as well as positive relationships forged with district education department and local governments has contributed to decisions by four districts to continue the program through resource sharing schemes. The government is interested in how the KIAT Guru approach can be used more broadly, especially in remote areas. Another shared priority relates to ‘building the education ecosystem’ – particularly in engaging more stakeholders, especially parents and communities in education. The government has expressed strong appreciation of KIAT Guru as a mechanism to test different approaches. Government representatives noted some barriers to ongoing upscaling which would need to be overcome if KIAT Guru tools are to be used more widely. The main barrier is school digitization which is an ongoing government priority and would assist in nationwide use of the application. The necessity of technological connectedness has been highlighted in the COVID-19 context and has demonstrated the importance of the education ecosystem. In response to COVID-19 KIAT Guru assisted to organize webinars for teachers and school supervisors about safe learning in the COVID Page 43 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) context, to be able to continue the learning process and follow healthy protocols. Overall, national stakeholders were very positive regarding KIAT Guru. The alignment with national objectives was appreciated and the tools developed have helped to fill a broader gap that emerged because of the COVID-19 pandemic. Furthermore, national stakeholders expressed satisfaction with the technical assistance of the World Bank. In particular, the willingness of the Bank to overcome challenges associated with the pandemic to continue to assist was appreciated. Summary of District Level Stakeholders The stakeholders in the five district governments met during the ICR mission (face to face and virtually) expressed very strong support for KGP2. They had all been included in KGP1 so already had active knowledge of the process. The feedback from the district stakeholders was that a unique feature of the project was the intensive coordination. They appreciated the efforts of the local BaKTI team in orientating each stakeholder separately as well as together through the quarterly meetings so that the specific requirements of each stakeholder were recognized and addressed. An important factor raised by all district stakeholders was the existence of BaKTI’s district teams. The onsite presence of technical staff was a key enabling factor in enabling project implementation effectively within the available timeframe. At the beginning of the project, there was uncertainty regarding the likelihood that the required policy guidelines and directives would be generated from the national level and that the Bupati’s legal office would be able to issue the local regulations in time. District stakeholders were impressed that this had been achieved and attributed it to the facilitation of BaKTI’s technical specialists at the district level. The experimental nature of KGP2 was of interest to the district stakeholders but was not the prime interest. They were aware of the findings of the KGP1 pilot and supported the direction but were not fully aware of the details of the experimental nature of the pilot. Instead, the main concern was to understand the student and teacher performance data and to identify mechanisms whereby the process could be transferred to all rural schools in the districts. Of particular emphasis by the district stakeholders was how KGP2 worked with the district superintendents to improve their capacity to supervise teacher attendance and performance and to reactivate the school committees. They noted that there was increasing relationship between the school superintendents and the village leaders, this had other benefits in addressing other educational matters across the district. The main benefit of KIAT Guru for the district stakeholders was the potential to improve the educational standards of the district (a perceived level of competition between districts on development performance parameters). The commitment of the district stakeholders was evidenced by their willingness to continue the activities commenced through KGP2 and allocate sufficient budget to support ongoing activities. Village government leaders The village government leaders had a varied response to KIAT Guru. While some leaders were Page 44 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) fully supportive of the project, others had competing priorities and were not fully aware of the process and the benefits. However, most villages made sure there were resources available for the committee meetings in order to develop the joint agreements. In most villages a representative of the village attended the consultation which demonstrated their level of engagement with the project. At a village level they appreciated the project and were able to allocate some funds but there was no guarantee of continuing funding, although most believed that it would be possible. However, in consultation after COVID-19, some local governments were unable to continue with financial support through the school committees because of the need to use the budget for COVID related activities. Nonetheless, some of the school committees were continuing to meet anyway even without available funds. School Principals. School principals were supportive and interested in the project. They agreed that attendance was increasing. However, given that rural schools are difficult environments, there is still nonattendance at times because teachers are unable to reach the school. The application of the student assessment has created discussion at the school level, with some parents in support and some parents concerned. At the beginning of the project, only a small sample of students were selected, and the principals reported concern from parents about who was selected and why. This issue was addressed after COVID when 100 percent assessment was applied. The school principals had to organize the joint agreements between the teachers and parents and attend the committee meetings. Several headmasters reported that the project had helped to reactivate the school committees, which had been passive in the past. Other headmasters noted that the school committees were very active even before KIAT Guru. In this respect the role of the school supervisors was important to assist the principals to engage more fully with the school committees to implement and monitor the program. Teachers. The teachers found the KIAT Guru process of recording attendance difficult to manage at first because they were unaccustomed to using the technology and not used to having to record time so precisely using a camera. For some teachers they were previously punctual and had no problem with the system. Others found the increased pressure of timekeeping difficult initially, but most did respond positively to the requirements as time went on. Nonetheless, most teachers brought up the challenges of working in remote schools where public transport is limited and unreliable or where teachers had to travel long distances over difficult terrain to attend class. Parents.38 The parents met during the ICR mission expressed interest and appreciation for the great attention paid to their children’s education and the renewed efforts to improve educational outcomes. They were committed to spending time for training and for the monthly meetings. Approximately 30 percent of those met indicated that in addition to the engagement with the School Committee, they became involved with implementing small projects to improve the school environment or teaching experience, such as fund raising for small equipment or joining classes to support practical skill activities. 38Note: The parents met during the ICR mission were representative parents for the school board/committee. These were appointed in different ways in different villages, sometimes through election, others nominated as class or area-based representatives. For this reason, the opinions should not be considered as representative of all parents. Page 45 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) The concept of the JSA was easily understood although the scoring process for teachers was initially cumbersome and required much debate. After several meetings, the village and school stakeholders became accustomed to the process. They could see the benefits of the e-KIAT Guru system and believed that there was an improvement in teacher attendance. There was mixed support for the sample student assessment process, with some appreciating the opportunity to benchmark school progress in relation to national and district averages. But other parents felt that the process was not sufficiently fair because the sample of students was small. Several also raised ethical concerns about selecting a few children and that it could place undue pressure on those participating students. The teachers and headmaster stated that the participating students were not placed under pressure but were supported to complete the assessment without judgement. Nonetheless, the concerns were raised and after some discussion, were deemed valid. This concern was mitigated after the onset of the pandemic because the process was amended to 100 percent assessment for all students to differentiate learning abilities to prepare suitable schoolwork that could be completed at home. Page 46 of 47 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2 (P167216) REFERENCES BPS (Indonesian Center of Statistics Agency). (2018). Profil Kemiskinan di Indonesia, Maret 2018. (Poverty Profile in Indonesia, Maret 2018). Jakarta: BPS. De Ree, J., Muralidharan, K., Pradhan, M., and Rogers, H., (2018). Double for Nothing? Experimental Evidence on an Unconditional Teacher Salary Increase in Indonesia. Quarterly Journal of Economics 133 (2): 993–1039. Rosser, A (2018). Beyond Access: making Indonesia’s Education System Work. https://www.lowyinstitute.org/publications/beyond-access-making-indonesia-s-education-system- work SMERU. (2010). Teacher Absenteeism and Remote Area Allowance: Baseline Survey. Jakarta, Indonesia: SMERU Research Institute. Susanti, D, Priebe, J & Bah, A (2020) The hard truth: Challenges of primary education in rural and remote Indonesia. https://blogs.worldbank.org/eastasiapacific/hard-truth-challenges-primary-education- rural-and-remote-indonesia The Republic of Indonesia, (2003). Act of the Republic of Indonesia, number 20, year 2003 The World Bank Group, (2015), Country Partnership Framework for The Republic of Indonesia for the period FY16-FY20. pg. 64-65, http://documents1.worldbank.org/curated/en/306831620760881407/pdf/Indonesia-Country- Partnership-Framework-for-the-Period-FY21-FY25.pdf The World Bank Group, (2016). Indonesia's Rising Divide. Washington. D.C.: World Bank Group. http://documents.worldbank.org/curated/en/267671467991932516/Indonesias-rising-divide The World Bank Group, (2018), Databank: Education: Indonesia. https://data.worldbank.org/topic/education?locations=ID The World Bank Group, (2018), Education Statistics: Country at a Glance: Indonesia, https://datatopics.worldbank.org/education/country/indonesia The World Bank Group, (2018). Learning to realize education’s promise. World Development Report. Washington, D.C: World Bank Group. The World Bank Group, (2019), Project Paper PP2940, Proposed Grant to the Bursa Pengetahuan Kawasan Timur Indonesia (BAKIT) for Kiat Guru Phase 2 – Improving Teacher Performance and Accountability. February 2019. The World Bank Group, (2019). Implementation Completion Report for Indonesia: Improving Teacher Performance and Accountability. The World Bank Group, (2020) Implementation Status & Results Report. December 2020. The World Bank Group, (2020) Indonesia Country Partnership Framework 2021-2025. The World Bank Group, (2020), Implementation Status and Results Report. July 2020. The World Bank Group, (2020). The Promise of Education in Indonesia. The World Bank Group, (June 2019). Restructuring Paper on a Proposed Project Restructuring of Improving Teacher Performance and Accountability Phase 2. Approved on March 5, 2019. Page 47 of 47