The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) Angola Learning for All Project (P122700) AFRICA EAST | Angola | Education Global Practice | IBRD/IDA | Investment Project Financing | FY 2014 | Seq No: 14 | ARCHIVED on 07-Jun-2021 | ISR45073 | Implementing Agencies: Ministry of Education, Republic of Angola Key Dates Key Project Dates Bank Approval Date: 26-Sep-2013 Effectiveness Date: 16-Jun-2014 Planned Mid Term Review Date: 28-Mar-2016 Actual Mid-Term Review Date: 28-Mar-2016 Original Closing Date: 30-Apr-2018 Revised Closing Date: 28-Feb-2022 pdoTable Project Development Objectives Project Development Objective (from Project Appraisal Document) The proposed Project Development Objectives are to improve teachers’ skills and knowledge as well as school management in project designated areas; and to develop a system for systematic student assessment. Has the Project Development Objective been changed since Board Approval of the Project Objective? No Components Table Name Component 1: Improving Teachers’ Skills and Knowledge and School Management in P:(Cost $51500000.00 M) Component 2: Establishing a System for Student Assessment:(Cost $7000000.00 M) Component 3. Project Management:(Cost $9500000.00 M) Contingencies Overall Ratings Name Previous Rating Current Rating Progress towards achievement of PDO Moderately Satisfactory Moderately Satisfactory Overall Implementation Progress (IP) Moderately Satisfactory Moderately Satisfactory Overall Risk Rating Substantial Substantial Implementation Status and Key Decisions Over the past seven years, significant progress has been made under all three components of the project. Component 1: Improving Teachers’ Skills and Knowledge and School Management in Practices. The entire in-service teacher training has been successfully implemented and has also been adjusted to tackle the Covid-19 crisis. Moreover, all ZIPs have been set up and almost all school councils are active; school improvement plans and school grants for at least 725 schools have been approved; and all targeted schools are now using school report cards. Also, the component was adapted to support the mitigation measures of the Covid-19, such as the printing and distribution of accelerated learning materials for 2 million students, in-service training with structured lessons plans for 60 thousand teachers, purchase of equipment for home-office work of the MED team, purchase of TVs, radios, and flash drives for the distance learning, materials for the deep-cleaning of schools, and support hygiene interventions for the reopening of schools. 6/7/2021 Page 1 of 10 The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) Component 2: Establishing a System for Student Assessment. Of note is the following: (i) the EGRA will be done in September 2021, (ii) the development of the national student assessment program in grades 4 and 6 and the pilot of the national exams in grade 6 will be conducted by September 2021. Component 3: Project Management. Over the past 6 years, significant attention has been given to increasing the capacity of both the PIU staff and within the MED for implementation of World Bank financed-projects. Considering that the upcoming education operation under preparation will require an increase in the capacity of the PIU staff, the project has strengthened the PIU team. The government continues to strive to implement activities to improve the availability of basic education statistics and strengthen the Education Management Information System (EMIS) and is committed to disseminate the National Statistical reports for the years of 2017, 2018, and 2019 by June 2021. Risks Systematic Operations Risk-rating Tool Risk Category Rating at Approval Previous Rating Current Rating Political and Governance -- High Substantial Macroeconomic -- High Substantial Sector Strategies and Policies -- Moderate Moderate Technical Design of Project or Program -- Substantial Substantial Institutional Capacity for Implementation and -- Substantial Substantial Sustainability Fiduciary -- Moderate Moderate Environment and Social -- Low Low Stakeholders -- Moderate Moderate Other -- -- -- Overall -- Substantial Substantial Results PDO Indicators by Objectives / Outcomes To improve teachers’ skills and knowledge as well as school management in project designated areas IN00939639 ►Demonstrated improvement in trained teachers’ skills and knowledge as a result of the training (Percentage, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 76.10 76.10 80.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 29-Oct-2021 The in-service teacher training program started in 2017 - all 6 modules were developed and implemented Comments: over a 3 year period (2 modules per year delivered during teacher breaks each May and August). Sample 6/7/2021 Page 2 of 10 The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) based tests during the first two years from Modules 1-4 showed significant progress with more than 80 percent of teachers demonstrating greater capacity in both Mathematics and Portuguese. However, in 2019 once a strengthened system to administer the tests was in place, the pre- and post-tests for Modules 5-6 were administered to all participating teachers (17,096). The results from the May 2019 session show that only 66 percent of teachers were able to demonstrate improvements, but the August 2019 session showed that the result improved to 78 percent. The May and August tests results are now being disaggregated by Province and ZIP to determine which teachers should be targeted for the new round of training with structured lesson plans planned for September 2021. The government will conduct in September and October 2021 the census-based pre and post tests to the same sample of teachers (17,906) that were evaluated in May and August 2019. This will be critical in order to achieve the target of 80 percent of participating teachers that demonstrate improved content knowledge. IN00939641 ►Percentage of active school councils (schools targeted by the project only) (Percentage, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 100.00 100.00 100.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 167 ZIP School Councils are now active. (NOTE: The definition of being active was determined initially as being able to show meeting minutes from at least 2 meetings during that year. However, ZIP Councils noted that delays in procuring Comments: the computers and solar panels for the ZIPs made it difficult for them to type and send electronic copies of meeting minutes. Hence, it was agreed that the ZIP Facilitators would count school councils as being active based on evidence that meetings took place without necessarily requiring typed minutes.) To develop a system for systematic student assessment IN00939640 ►Number of sample-based assessment exercises carried out (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 1.00 2.00 2.00 3.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 30-Jul-2021 To date, the 2 sample-based exercises carried out included EGRA and SDI in 2016. There were some reliability issu it was ultimately decided that it would need to be repeated in order to have results that were credible. An internationa experience implementing EGRA in other developing countries has now been recruited and the new round of EGRA w 2020. A Brazilian institution (CAED) has also now been recruited to provide technical assistance to develop the natio Comments: 4 and 6. Since primary schools reopened in January 2020, the project should be able to successfully complete at lea assessments (surpassing the target of 3) in September 30, 2021. One of the key achievements beyond the assessm unit responsible for overseeing learning assessments has now been officially established within the Ministry of Educa for National Assessment and Accreditation - DNAA) with full-time staff, budgetary resources, etc. Intermediate Results Indicators by Components Component 1: Improving Teachers’ Skills and Knowledge and School Management IN00939642 ►ZIP facilitators recruited and trained (Number, Custom) 6/7/2021 Page 3 of 10 The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 53.00 53.00 30.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 To begin the work to establish the ZIPs, ZIP Councils, and develop the School Improvement Plans, it was necessary with how to effectively work with community. Initially, there was just one facilitator hired for each Province. Given the geographic size and number of municipalities for each, it was agreed that additional support would be required to rea the first phase and at the time of the Mid-Term Review, it was agreed to also start working with the Inspectorate unit Comments: tasks to ensure greater sustainability and alignment within the Ministry. In total, 53 people (combination of contracted inspectors) have been working to support the activities to improve school management and reach the targets for ZIPs Councils, approved PEEs, school report cards, etc. There are just 4 contracted facilitators remaining while the rest o these activities are inspectors. IN00939645 ►ZIPs set up and active (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 167.00 167.00 167.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 All ZIPs to be supported by the Project were established during the first two years of implementation. All Comments: 167 ZIPs (one in each Municipality) have been active since 2016. IN00939648 ►ZIP center schools equipped (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 167.00 167.00 167.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 All planned resources for the ZIPs targeted by the Project have now been delivered. The ZIPs are serving as resource centers and places to conduct in-service teacher training activities. ZIP schools received: (i) equipment - including a computer, printer, and solar panels; (ii) furniture - table, chairs, shelves, etc.; and (iii) pedagogical materials - including books, games to teach literacy and numeracy skills, maps, support materials/equipment for special education. Due to the state of Angola's economy, purchasing and delivering these materials to the schools took much longer than anticipated - 2016-2019. While computers and printers were delivered way back in 2016/17, the installation of the solar panels in all schools was only just finalized in September 2019. There was initially significant concern that security Comments: would be an issue and much of these investments would go missing. However, due to mitigation measures taken initially, this hasn't been the case and the materials are still present. For the pedagogical materials, the focus needs now be on how teachers can effectively use these materials in the classroom and positively impact student learning. The majority of schools in Angola do not have libraries so the schools do not have in place a system for managing these types of materials. Creating a manual and videos to establish a basic system for management of materials and effective use of them is the focus during this final phase of implementation. IN00939651 ►Teacher training schools equipped (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 18.00 18.00 18.00 6/7/2021 Page 4 of 10 The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) Date 01-Nov-2012 28-Feb-2021 28-Feb-2021 28-Feb-2021 The Project has supported 18 Teacher Training Institutions - one in each Province of the country. These institutions have been directly supporting the implementation of the in-service teacher training program. They benefitted in terms of equipment and pedagogical support by receiving copiers and printers, laptops for the 43 teacher trainers helping to train trainers at the Municipal level and implement the in-service Comments: teacher training program, internet to access the moodle platform for the distance learning component for the training of trainers, and they also received the same pedagogical materials as the ZIPs - games to teach early literacy and numeracy skills, maps, etc. IN00939653 ►Number of school kits produced and distributed (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 842.00 842.00 842.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 All of the 842 school kits have been delivered to the schools. These kits include: a small library of 40- 50 books for early grades; games to teach literacy and numeracy skills; other pedagogical support materials; and for the schools without adequate space/conditions to store these materials (more than 75%), the project also supported the purchase and delivery of shelves, a desk and a chair to help manage Comments: the storage and use of these kits by teachers at the schools. The set of early literature books were delivered in early 2017, the other support materials in 2017/18 and then the furniture in 2019. This activity is now complete. IN00939655 ►Number of education projects approved (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 725.00 725.00 520.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 All 842 schools supported by the Project developed School Improvement Plans (PEEs). These PEEs are for the next 3 years and include a one-time school grant that finances related pedagogical materials identified together by the School Councils/community members. Based on the most recent information Comments: provided in the Project Report, out of the 842 PEEs prepared, 725 were officially endorsed and approved. IN00939656 ►Number of grants disbursed (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 725.00 725.00 520.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 Comments: At the time of the restructuring, a change was made to the flow of funds for the Project due to a policy enacted by the Ministry of Finance which prohibited the project from being able to transfer funds to the Municipal and school level. The school grant program was redesigned to take this into account. The schools developed overall school improvement plans (PEEs) together with the community, parents association, and school leaders. Then, together with the PEE, an expansive list of eligible materials that could be financed through the school grant was provided and they selected which items they wanted. The size of the grant for each school was determined by their student population. All 842 schools prepared PEEs together with grant request. At the time of the mission, 725 of these had been deemed to be of sufficient quality and had been approved by the education authorities. Based on the approved school 6/7/2021 Page 5 of 10 The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) grants, funds were then transferred each of the 18 Provincial Education Department in order to purchase the materials requested and it was completed the purchase the materials and delivered to the schools. IN00939657 ►Set of school report cards produced and disseminated (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 842.00 842.00 842.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 The template and module for a new school report card that focuses on education quality and information from an assessment conducted as part of the development of School Improvement Plans (PEEs) was finalized in 2018. Given the lack of electricity and printing/copying capabilities, especially in rural areas, it was decided to create laminated posters for the school report cards at the central level and Comments: then send these to the municipalities for their use. These laminated posters are being displayed on the wall of the school for parents and members of the community. The first set of report cards were produced and disseminated in 2019. IN00939658 ►In-service teacher training modules aligned with learning goals developed (six modules) (Text, Custom) Baseline Actual (Previous) Actual (Current) End Target Value No 6 out of 6 6 out of 6 Yes Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 All six modules of the in-service teacher training program now have been developed, piloted, and finalized. The modules cover the following areas (with an emphasis on Portuguese and Mathematics throughout): (i) Portuguese; (ii) Mathematics; (iii) Classroom Assessment; (iv) Multi-grade classrooms; (v) Teaching children of different ages and accelerated learning; and (vi) Special Education. The modules Comments: were developed with the support of the University of Setubal in Portugal in partnership with the Foundation Calouste Gulbenkian. The manuals and supporting materials have been in very high demand - a number of Provincial Governors have written formally to request that the materials be made available to all teachers. IN00939659 ►Number of teacher kits produced and distributed (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 17,922.00 17,922.00 15,000.00 Date 01-Dec-2016 05-Jun-2020 05-Jun-2020 28-Feb-2021 This includes all teachers, directors, and sub-directors trained in addition to the teacher trainer/supervisors. NOTE: All of the training beneficiaries are from the 842 schools supported by the Project. Since some of the teachers at these schools have changed (new ones on some years, transfers, Comments: etc.), the actual number is now greater than the target of 15,000. The kits were defined as the training manuals and other materials that are part of the training program. As of August 2019, these teachers have now received all manuals and supporting materials for all 6 modules. IN00939660 ►Number of additional qualified primary teachers resulting from project interventions. (Number, Corporate) Baseline Actual (Previous) Actual (Current) End Target 6/7/2021 Page 6 of 10 The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) Value 0.00 20,929.00 20,929.00 15,000.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 Based on the figures provided after the last in-service teacher training program in August 2019, the total number of primary school teachers that have participated in the in-service teacher training program has now reached 20,929. The majority of these teachers have participated in the entire 3-year program which Comments: began in 2017. This figure represents almost 20 perent of the total of primary school teachers in the country. Component 2: Establishing a System for Student Assessment IN00939643 ►System for learning assessment at the primary level (Yes/No, Custom) Baseline Actual (Previous) Actual (Current) End Target Value Yes Yes Yes Yes Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 30-Jul-2021 This indicator is meant to determine if the basic elements of a learning assessment system exist at the primary level, that assessment is able to provide useful information for education policy and practice. While the basic elements we project, the institutional arrangements had not been formalized and assessment activities were being conducted on a support of the project, a new unit within the MED specifically to oversee assessment activities was created in early 20 approximately 10 full-time staff that have been benefitting from a significant amount of capacity building activities and implement the upcoming EGRA III and National Assessment in grades 4 and 6, but the unit is facing capacity constra Comments: specialized staff. Where there is room for improvement is the second part of the indicator which is "the degree to wh to provide useful information for education policy and practice". While EGRA and SDI were conducted in 2016/17, is EGRA data prevented it from being able to be published and used for decision-making. Given capacity constraints th system, the level of outside technical assistance required to produce data that is credible and reliable is significant. H EGRA and the National Students Assessment will be conducted by July 2021 to ensure quality controls at every leve allow even greater monitoring during data collection. IN00939647 Utility of the learning assessment system (Number, Custom Supplement) Baseline Actual (Previous) Actual (Current) End Target Value 1.00 2.00 2.00 3.00 IN00939649 ►Assessment framework reviewed (Yes/No, Custom) Baseline Actual (Previous) Actual (Current) End Target Value No No Yes Yes Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 The activity was completed in February 2021 after a qualified expertise willing to support the work to develop the national assessment program (i.e., review of the assessment framework, creating the item Comments: bank, developing testing instruments, etc.) in Angola was hired. A contract with the Center of Educational Public Policy and Evaluation (CAED) in Brazil was sign to complete this activity. IN00939652 ►Minimum standards defined as well as bank of items and standardized tests constructed (Yes/No, Custom) 6/7/2021 Page 7 of 10 The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) Baseline Actual (Previous) Actual (Current) End Target Value No No Yes Yes Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 Learning standards for primary grades have been developed with the technical assistance (CAED) in Comments: February 2021. IN00939654 ►Impact evaluation phases completed (Text, Custom) Baseline Actual (Previous) Actual (Current) End Target Follow-Up Data Value Design Design of Baseline Design of Baseline Analysis, and IE report Date 01-Nov-2013 05-Jun-2020 05-Jun-2020 31-Jan-2022 The project was initially designed to include a complex impact evaluation of various pilot activities (i.e., use of ZIPs, in-service teacher training program, school grants, etc.) However, given the limited capacity within the MED to oversee this activity, it was ultimately decided to just have one control and one treatment group - treatment schools that received all of the project interventions and those that did not. Two issues that emerged during implementation were the following: 1) issues with the quality and Comments: reliability of the baseline data collected - EGRA II and SDI; and 2) contamination of the control and treatment groups. The Ministry of Education enacted a policy to adopt ZIPs nationwide and the teacher training program has been highly sought after and in some Provinces scaled-up beyond the target schools. Considering that EGRA II and SDI data were too problematic to be publishable, it was agreed with MED to use EGRA III and the National Assessment to conduct a quasi-experimental evaluation using the propensity score matching methodology by January 2022. Component 3. Project Management IN00939644 ►Number of provincial, municipal and teacher training schools officers trained in management around the project (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target Value 0.00 3,627.00 3,627.00 254.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 Given the scale of the project and multiple changes in education leaders at different levels (Central, Provincial, Municipal, and School level leaders) after the change in governments, the level of effort required and number of people to be trained to order to effectively implement the project has been much greater than initially envisioned. Some of the education leaders have benefited from multiple capacity building activities while others were introduced to the project more recently. According to the records of participation of different leaders, the total number trained has now reached 3,627 individuals. These Comments: include individuals from the Central Ministry, all 18 Provinces, all 167 Municipalities, and all 842 schools targeted by the Project. This is the World Bank's first operation fully dedicated to education in Angola. The project has been able to create a large base of education leaders that are now familiar with various rules and procedures in implementing World Bank projects. This foundation will be critical as the World Bank works to broaden its engagement in the sector. IN00939646 ►Direct project beneficiaries (Number, Custom) Baseline Actual (Previous) Actual (Current) End Target 6/7/2021 Page 8 of 10 The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) Value 0.00 26,002.00 26,002.00 19,576.00 Date 01-Nov-2012 05-Jun-2020 05-Jun-2020 28-Feb-2021 According to the last progress report, the number of direct project beneficiaries reached is now 26,002. This number benefitting from in-service teacher training program; teacher trainers and others from the Provincial Education Depar Education Departments, Teacher Training Institutions; Central Ministry staff (INIDE, INFQE, GEPE, DNAA); accounta Education Departments; and school leaders from the 842 schools targeted by the project. When the project was des indicator did not include the students at the schools. Based on the information provided, there are more than 500,00 Comments: are also benefitting from the greater attention being paid to education quality. In terms of the percentage of the benef are female, this decreased from 56 to 48 percent between 2018 and 2019. This is due to the fact that a large share o from the 842 schools targeted by the Project are male. When deciding which teachers will benefit from the additiona planned for 2020, gender will be taken into consideration to ensure the project reaches the target of 50 percent of wh IN00939650 Female beneficiaries (Percentage, Custom Supplement) Baseline Actual (Previous) Actual (Current) End Target Value 50.00 48.00 48.00 50.00 Performance-Based Conditions Data on Financial Performance Disbursements (by loan) Project Loan/Credit/TF Status Currency Original Revised Cancelled Disbursed Undisbursed % Disbursed P122700 IDA-52540 Effective USD 75.00 75.00 0.00 63.26 5.21 92% Key Dates (by loan) Project Loan/Credit/TF Status Approval Date Signing Date Effectiveness Date Orig. Closing Date Rev. Closing Date P122700 IDA-52540 Effective 26-Sep-2013 16-Jan-2014 16-Jun-2014 30-Apr-2018 28-Feb-2022 Cumulative Disbursements 6/7/2021 Page 9 of 10 The World Bank Implementation Status & Results Report Angola Learning for All Project (P122700) PBC Disbursement Achievement Disbursed amount in Disbursement % PBC ID PBC Type Description Coc PBC Amount Status Coc for PBC Restructuring History Level 2 Approved on 29-Jan-2017 ,Level 2 Approved on 27-Feb-2020 ,Level 2 Approved on 26-Feb-2021 Related Project(s) There are no related projects. 6/7/2021 Page 10 of 10