ICRR 14533 Report Number : ICRR14533 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/03/2014 Country : Indonesia Project ID : P097104 Appraisal Actual Project Name : Bermutu-better US$M ): Project Costs (US$M): 195.06 173.34 Education Through Reformed Management And Universal Teacher Upgrading L/C Number : C4349; L7476 Loan /Credit (US$M): Loan/ US$M ): 24.5/61.5 18.1/59.4 Sector Board : Education US$M): Cofinancing (US$M ): 52.0 50.9 Cofinanciers : Board Approval Date : 07/05/2007 Closing Date : 12/31/2013 12/31/2013 Sector (s): Tertiary education (78%); Central government administration (19%); Sub-national government administration (3%) Theme (s): Education for all (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Judith Hahn Gaubatz Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Financing Agreement (page 4) and the Project Appraisal Document (PAD, page 3), the objective of the project was as follows : To contribute to the improvement of the overall quality and performance of teachers of the Republic of Indonesia through enhancing such teachers' knowledge of subject matter and pedagogical skills in the classroom . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: 1. Reforming University-Based Teacher Education (Appraisal: US$ 20.5 million; Actual: US$ 15.1 million): This component aimed to develop an accreditation process for teacher training institutions, to ensure that curricula are revised in line with international best practice . Activities were to include: development of standards and procedures for accreditation, provision of accreditation incentive grants to develop capacity of institutions to meet the new standards; provision of distance learning development grants to develop instructional materials for online training; and provision of overseas scholarships for PhD candidates and non -degree training opportunities. 2. Strengthening Structures for Teacher Improvement at the Local Level (Appraisal: US$ 134.2 million; Actual: US$ 127.8 million): This component aimed to support mechanisms for in -service teacher training and teacher classroom assessment at the district level (75 out of 515 total districts) . Activities were to include: development of procedures to recognize prior learning of incumbent teachers; establishment of professional working groups for teachers, principals, and supervisors; development of in -service training modules to be delivered in the professional working groups; and development of a teacher induction program for new teachers . 3. Reforming Teacher Accountability and Incentive Systems for Performance Appraisal and Career Advancement (Appraisal: US$ 5.3 million; Actual: US$ 8.4 million): This component aimed to sustain and enhance the quality and accountability of teachers once they are certified . Activities were to include: development of policies and procedures for teacher promotion, performance -based appraisals, and under-performance. 4. Improving Program Coordination, Monitoring and Evaluation (Appraisal: US$ 33.8 million; Actual: US$ 22.0 million): This component aimed to support monitoring systems and evaluative activities to inform implementation experience. Activities were to include: support for the collection of teacher and school census data; integration of the teacher and school data into one monitoring system; rapid assessments of the teacher certification and in-service teacher training activities; video study to assess teacher performance; and survey -based teacher absenteeism studies. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost : The actual project cost was US$ 173.34 million, compared to the appraised cost of US$ 195.06 million. The difference was due to favorable exchange rate fluctuations and lower actual outputs for some project activities (see below). The actual cost for Component 1 was US$ 15.1 million, compared to the appraised cost of US$ 20.5 million. The difference was due to the lower number of scholarships awarded (19, compared to the anticipated 30), lower amounts for distance learning grants awarded, and gains from exchange rate fluctuations . The actual cost for Component 3 was US$ 8.4 million, compared to the appraised cost of US$ 5.3 million. The difference was due to the implementation period for this component lasting six years, instead of the expected two years. The actual cost for Component 4 was US$ 22.0 million, compared to the appraised cost of US$ 33.8 million. The difference was due to the smaller sample of schools assessed (360, compared to the anticipated 600), and gains from exchange rate fluctuations . Financing The project was to be financed by an IBRD Loan of US$ 24.5 million and an IDA Credit of US$ 61.5 million, for combined Bank financing of US$ 86.0 million. The actual IBRD amount was US$ 18.1 million and the IDA Credit was US$ 59.4 million, for a combined actual amount of US$ 77.5 million. The Government of the Netherlands also provided co -financing of US$ 50.9 million, out of an appraised amount of US$ 52.0 million. Borrower contribution The appraised Borrower contribution was US$ 57.06 million, and the actual contribution was US$ 44.9 million. Dates The Loan Agreement was amended on two occasions : In November 2011, an amendment was made to reallocate funds to accommodate the larger number of teachers participating in the professional working group. In October 2012, an amendment was made in response to the government's official exchange rate change, which affected disbursement plans and fund availability . The project closed as planned on December 31, 2013. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High Although Indonesia has achieved nearly universal access to Relevance of the project objective is rated High. primary education, performance on international educational assessments has been relatively poor, especially when compared to countries with similar socio -economic characteristics. This has raised concerns about the quality of education in Indonesia, namely the quality of the classroom experience . Therefore, the relevance of the project objective to improve the quality and performance of teachers is High . It is consistent with the government's recent strategic thrust to improve the quality of teaching, as reflected in the 2005 Teacher Law. The Bank's Country Partnership Strategy for FY 2004-08 prioritizes improving the quality of education, and while the Strategy for FY2013-25 also recognizes this as a critical priority for the country, it notes that the government has expressed a preference to no longer seek external financing for education . b. Relevance of Design: Relevance of the project design is rated Substantial . Project interventions were likely to achieve the intended outcome of improved teacher quality . The project directly supported implementation of a new Teacher Law, which aimed to improve teacher professionalism, performance, accountability, and welfare . The economic analysis in the PAD (Annex 10) highlights the positive relationship between higher teacher qualifications (specifically, having a four-year degree vs. one-two year diploma) and Grade 8 students' math scores. While the government was funding the core initiatives of the Law (salary increases, teacher training and performance incentives), the project provided technical assistance to these new programs, including upgrading current skill levels and maintaining those skill levels over time . The project's pre-service training interventions were national in scope, while the in-service training interventions were piloted in a sub -set of districts (initially 80 of 441 total districts in the country). These districts were targeted based on several criteria, including education development indicators, the number of teachers with less than the required four -year degree for certification, and poverty rates. 4. Achievement of Objectives (Efficacy): To contribute to the improvement of the overall quality and performance of teachers of the Republic of Indonesia through enhancing such teachers' knowledge of subject matter and pedagogical skills in the classroom Achievement of this objective is rated Substantial . Almost all project activities were completed as planned, and there were substantial achievements in improving the quality of teachers (i.e. increased number of teachers meeting qualifications, increased number of teachers receiving in -service training) as well as some evidence of achievements in improving the performance of teachers (i.e. improved pedagogical skills, reduced rate of teacher absenteeism). The project's nationwide activities (pre-service training programs to meet academic qualifications ) reached the 1.7 million teachers (out of approximately 2.7 million teachers in Indonesia) who were considered under-qualified at the time of project appraisal. The project's district-level activities (professional working groups) covered 75 out of a total 515 districts. Outputs Nationwide Development of accreditation tools and procedures for pre -service teacher training programs . Provision of 26 accreditation incentive grants to pre -service training programs. The target was to award 26 grants. Upgrading of distance-learning training programs to meet mandated standards . Development of policies and procedures for continuing education and career development for teachers . Development of regulations regarding incentives and rewards for teacher performance, including a performance appraisal instrument. The teacher performance management program was implemented only in 2013, thus its effectiveness is not yet known . Establishment of teacher database to track progress in teacher placement, academic qualifications, and certification. This database was to be integrated with the schools database; however, this integration was not completed until just before project closing . The ICR (page 34) reports that "although it is too early to ascertain the effectiveness and degree of utilization of this integrated teacher database, it represents an important tool... for decision-making." Conducting of impact evaluations on teacher certification and pre -service teacher training. Support for 19 PhD candidates, out of a targeted 30, in overseas degree programs . The shortfall is due to difficulties in obtaining visas on time to begin overseas studies . Project districts Establishment of 6,107 professional working groups (for teachers, principals, and supervisors ). These working groups provided training on subject matter, pedagogical skills, and school management . This surpassed the target of 4,500. Development of regulations and procedures to recognize prior learning of incumbent teachers, in order to accelerate certification for these teachers . Development of regulations for school induction services for new teachers . Conducting of an impact evaluation of the professional working groups, a survey of teacher absenteeism, and trend analysis on student and teacher performance . Conducting of video assessment of teacher performance . The ICR (page 9) reports that the assessment "provided valuable insights into teaching practices " but no specific information is provided . Outcomes Intermediate outcomes The number of pre-service training programs achieving accreditation was 81. The evaluation of the programs found that the accredited programs "tend to attract a better quality intake of students than non-accredited institutions," and that the intake of the former is larger (ICR, page 34). Another finding was that the quality of accredited institutions in terms of number of certified lecturers, number of lecturers conducting research, and the number of graduates becoming certified basic education teachers was also higher (ICR, page 35). The number of incumbent teachers gaining recognition for prior learning from accredited programs was 879,832, surpassing the target of 700,000. The number of new teachers being provided with school induction services in project districts was 1,429. This fell short of the target of 3,000. Overall outcomes The number of new graduates from accredited programs meeting the mandated competency standards was 13,179. This fell short of the target of 15,000, although the ICR (page ix) notes that as the first cohort graduated only in 2013, the outcome data is still to be updated . The number of teachers nationwide meeting academic qualifications as mandated by the Teacher Law increased from 900,000 in 2007 to over 1.7 million in 2013, surpassing the target of 1.4 million. However, the impact evaluation of the teacher certification process found that although "students under certified teachers showed better learning outcomes than similar students under non -certified teachers, teacher professional scores were very low, indicating limited teacher professional competencies' gains " (ICR, page 35). The number of teachers in project districts using subject - and age-appropriate pedagogy increased from 17,000 in 2007 to 214,380 in 2013, surpassing the target of 190,000. The ICR (page 14) reports findings from the impact evaluation of professional working groups (which were intended to support teachers in improving instructional practice ), including that "teachers' participation in these working groups improved their pedagogical skills as indicated by their abilities in teaching preparation, delivery and formative assessment of their students, as compared to equivalent teachers that did not participate in similar groups ." However, specific data are not provided . The teacher absenteeism rate in project districts decreased from 19.0% in 2007 to 6.6% (primary schools) and 6.3% (junior secondary schools ) in 2013, surpassing the target of 15.0%. 5. Efficiency: Efficiency is rated Substantial . The PAD economic analysis (Annex 10) did not calculate an economic rate of return. However, it presents a cost-effectiveness analysis of two possible interventions to improve learning outcomes: upgrading teacher qualifications and reducing class size . While the latter did contribute to improved test scores, the cost of this approach was estimated to be higher than the approach of upgrading teacher qualifications. The ICR economic analysis (Annex 3) does not provide an updated analysis; however, almost all project activities related to teacher upgrading were implemented as planned and achieved substantial outcomes, at 88.9% of appraised project cost. The project closed as planned with no closing date extensions . In addition, project management and evaluation activities (Component 4) were effectively carried out at an actual cost of US$ 22.0 million, compared to the appraised cost of US$ 33.8 million. Other specific project activities also indicate efficiency in project implementation, including the 879,832 incumbent teachers who benefited from the recognition of prior learning procedures established by the project, which surpassed the target of 700,000. Of those, 792,331 were able to meet remaining standards through distance learning courses supported by the project, which as noted in the ICR (page 30), enabled those teachers to remain on their jobs while pursuing their qualifications . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of the project objective is rated High, and relevance of the project design is rated Substantial . Achievement of the objective to improve the quality and performance of teaching is rated Substantial, due to substantial achievements in improving the quality of teachers (i.e. increased number of teachers meeting qualifications, increased number of teachers receiving in -service training) as well as some evidence of achievements in improving the performance of teachers (i.e. improved pedagogical skills, reduced rate of teacher absenteeism). Efficiency is rated Substantial . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The teacher reform agenda, as reflected in the 2005 Teacher Law, has strong political support . The policies, regulation and processes to improve teacher management and quality in the country have been well -established and are likely to be sustained . The government, including through the support of other donors, is expected to provide the budget allocations to the Ministry of Education and Culture and to districts to continue the programs . However, as noted in the ICR (page 19) the teacher promotion program has not yet been firmly established and therefore its related outcomes remain at modest risk . a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: a. Quality at entry: The project design drew upon significant analytic work that had been conducted by the Bank leading up to the Teacher Law in 2005. This included the paper, "Managing the Transition to Decentralization " and others on teacher employment, deployment, and certification, and a survey of ongoing teacher support groups . This analytic work provided a strong foundation for effective policy dialogue with the government during project preparation. Numerous actions were also taken to ensure quality at entry, including participation in three quality reviews and preparation of documents to ensure project readiness ("readiness filter agreement," draft project operations manual, co -financing agreement, draft procurement plan, request for consultancy proposals, and manuals for the grants programs, distance education modules, and professional working groups). The project design drew upon lessons from previous experience in Indonesia and internationally, including the importance of classroom assessment (for both students and teachers ) for quality improvement; and the importance of continuing education and non -financial incentives as tools for ongoing professional development . The risk assessment was extensive, including preparation of a comprehensive Anti-Corruption Action Plan to mitigate potential fiduciary risks . However, financial management procedures were not fully in place prior to project effectiveness and led to initial delays (see Section 11). The M&E framework was comprehensive, including relevant and measurable indicators . at -Entry Rating : Quality -at- Satisfactory b. Quality of supervision: The project team, led by a single task team leader throughout the implementation period, provided continuous support primarily from the field -based office. Almost all project activities were completed as planned. The project team ensured satisfactory resolution of initial financial management problems, as well as other implementation issues and helped to put the project firmly on track by the mid -term. The ICR (page 21) also notes that there was ongoing policy dialogue with the government, particularly when findings emerged from the rapid impact assessments . Safeguards compliance and M&E implementation were similarly satisfactory (see Sections 10 and 11). Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: From the outset, the government demonstrated strong commitment to the project objective given its high relevance to the ongoing teacher reform agenda . During the initial project period, there were delays in the government's budget allocations to the decentralized units, including counterpart contributions for the project, which affected the pace of implementation . Extensive consultations between the Bank and the government helped to facilitate the budget allocation process and ensure adequate funding by 2010 (ICR, page 7). There was an extensive restructuring of the Ministry of National Education in 2011 (renamed the Ministry of Education and Culture) which led to further implementation delays; however, the ICR (page 8) notes that the restructuring was viewed by the Bank as a positive move for the project to further strengthen the integration between teacher management and basic education management, and that the implementation delays were resolved by the 2011 mid-term. Government Performance Rating Satisfactory b. Implementing Agency Performance: Three Directorates of the Ministry of National Education were responsible for implementation, with an overall project coordination unit established to coordinate activities and monitor implementation progress . The three Directorates were: Quality Improvement of Teacher and Education Personnel (Components 2 and 3); Higher Education (Component 1); and Research and Development (Component 4). Although there were initial implementation challenges (caused primarily by delays in government budget allocations, Ministry restructuring and financial management weaknesses ), most of these issues were resolved by mid -term and project activities were completed as planned . There were no problems reported in safeguards compliance (see Section 11). Although there were no qualifications in audits, there remained some shortcomings in fully addressing all audit findings prior to project closing . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E design included both routine monitoring to track implementation progress as well as a series of impact evaluation studies. The monitoring arrangements included the use of existing monitoring systems (the schools database) and the development of new systems (teacher database). The evaluative studies included rapid assessments (to provide immediate feedback on key project activities ) and longer-term impact evaluations. The M&E indicators were appropriate, relevant and measurable, including baseline and target figures . b. M&E Implementation: Data from the schools and teacher databases were reported regularly, although the integration of the two systems was not completed until almost project closing . Impact evaluations on teacher training, teacher certification, and the professional working groups were conducted, as well as an absenteeism survey and video assessment of teacher performance . c. M&E Utilization: The ICR (page 9) reports that the Ministry of Education and Culture "regularly provided the Bank with data and information that allowed the teams to make judgments on progress toward achieving the PDO and the project's overall implementation" as reflected in numerous Bank Aide -Memoires and Implementation Status and Results reports. M&E Quality Rating : Substantial 11. Other Issues a. Safeguards: The project was classified as an Environmental Category "C" project and therefore an environmental assessment was not required . The safeguard policy on Indigenous Peoples (OP 4.10) was triggered due to the presence of isolated and vulnerable populations within some of the project districts . An Isolated Vulnerable Peoples' Framework (IVPF) was developed, aimed at ensuring that both teachers and isolated groups within those districts would have access to the project. Although there was limited attention given to the Framework during the initial project period, a social safeguards consultant was subsequently hired to ensure compliance . Although a planned grievance mechanism was never implemented, the ICR (page 12) reports that "compliance with the IVPF moved forward with no grievances reported during project implementation ." b. Fiduciary Compliance: Procurement: Procurement was limited to the selection of consultants and there was no purchase of goods or civil works. Control of procurement processes was enhanced by putting in place a detailed procurement plan, an operational manual, and a reporting mechanism for the Anti -Corruption Action Plan. Although there were some initial delays in hiring procurement consultants, procurement performance was considered in the satisfactory range throughout the project period . Overall, there were no procurement problems reported and the ICR (page 13) indicates that procurement was carried out in accordance with Bank guidelines . Financial management: There were initial challenges in financial management, primarily due to the fact that financial management procedures and guidelines were not established until one year into the project period . The ICR (page 10) reports the following problems in the first part of the project period : concerns regarding the verification process for block grants; differences between financial reports from grant recipients and actual costs; delays in following up on Bank observations on travel expenditures; delays in submitting the Inspector -General's intra-audit report; and delays in following up on the external audit findings . Over the course of the project period, agreed action plans were put into place and issues arising from audits were addressed, although some remained unresolved by project closing . The ICR (page 10) reports that all audit reports were unqualified . c. Unintended Impacts (positive or negative): None reported. d. Other: 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lessons drawn by IEG: Rigorous project preparation, including prior analytic work and project readiness activities, can help ensure successful project implementation . In the case of this project, prior analytic work ensured the technical soundness of the project design and numerous documents were prepared prior to effectiveness to support project readiness . Effective policy dialogue, based on rigorous analytic work and evidence from ongoing assessments, can be a key support to project achievements . In the case of this project, policy dialogue helped to strengthen the working relationship with the government and evidence from the real -time assessments of specific project activities was shared with the government and helped to inform decision -making processes. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is concise and has a clear focus on outcomes . The quality of the evidence is satisfactory, as well as the analysis of the evidence to substantiate project achievements . The ICR draws on the robust M&E conducted for the project, including monitoring data and findings from project assessments . a.Quality of ICR Rating : Satisfactory