Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) Report Number : ICRR0020318 1. Project Data Project ID Project Name P105897 Reading Education Project Country Practice Area(Lead) Papua New Guinea Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-98729 30-Jun-2014 19,200,000.00 Bank Approval Date Closing Date (Actual) 02-Mar-2011 31-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 19,200,000.00 19,200,000.00 Revised Commitment 19,200,000.00 19,200,000.00 Actual 19,200,000.00 19,200,000.00 Sector(s) Primary Education(79%):Public Administration - Education(21%) Theme(s) Education for all(100%) Prepared by Reviewed by ICR Review Coordinator Group Kimberly Kamlesh Parekh Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives According to the original Grant Agreement and the Project Appraisal Document (PAD), the original project development objective was “to improve the reading skills of elementary and primary education students” and “to build the policy, monitoring, and evaluation capacity of the Curriculum Development Division (CDD) (formerly the Curriculum Development and Assessment Division (CDAD)) to select, procure, and distribute books and reading support materials and to undertake assessment of early grade reading together with provincial officials” (Grant Agreement 2011, PAD 2011). In a March 25, 2014 Level One Restructuring, the project development objective was revised “to promote better teaching and learning of reading skills of elementary and primary education,” and the second objective was explicitly removed, with procurement and assessment activities folded into the first objective (Grant Agreement Amendment 2014). Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 21-Mar-2014 c. Components According to the Project Appraisal Document, the project had four components: (1) Increasing the Availability of Books and Learning Materials by Establishing Classroom Libraries in Elementary and Primary Schools (appraisal cost US$14.3 million; actual cost US$13.76 million). This component was to support the central selection and procurement of children’s early grade reading materials, including supplementary reading books and simple bookshelves, for classroom libraries. (2) Promoting Reading through Professional Development, Professional Teacher Networks, and Public Reading Campaigns (appraisal cost US$1.7 million; actual cost US$2.79 million). This component was to support the professional development of teachers on use of classroom libraries to encourage children to read and to improve their reading habits, and on the creation of localized reading materials. (3) Strengthening Early Grade Reading Assessment (EGRA) (appraisal cost US$2.6 million; actual cost US$1.49 million). This component was to support EGRA to: (i) produce a baseline of early grade reading performance by province during the project period (2011-2013); (ii) build local capacity for EGRA at the central and provincial levels and to monitor and report progress against the established baseline; (iii) analyze the implications of EGRA findings for instructional and curriculum improvements, policy directions, and future investment opportunities; and (iv) use EGRA findings as an input into a reading information campaign aimed at mobilizing parental support for reading. (4) Project Management and Monitoring (appraisal cost US$0.6 million; actual cost US$1.17 million). This component was to support the management and coordination of the implementation of the project, including technical assistance, financial management, procurement, monitoring and evaluation. During the March 2014 Level One Restructuring, the following component changes were made: Component 3 Strengthening Early Grade Reading Assessment was modified to: (i) reduce the number of provinces from 15 to four (one per region of the country) where the EGRA diagnostic surveys would be carried out, and (ii) introduce a post-EGRA intervention to improve teaching practice in a small number of selected provinces that would be used to monitor improvements in learning outcomes against the new PDO. During a May 2015 Level Two Restructuring, the following component changes were made: Component 1 Increasing the Availability of Books and Learning Materials by Establishing Classroom Libraries was modified such that schools and villages, instead of the project, would finance bookshelves from their own budgets. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The actual cost of the project was US$19.2 million, or 100% of the original estimate of US$19.2 million. Spending on Components 2 and 4 was higher than expected (164.12% and 195% respectively), while spending on Component 3 was lower than expected (64.78%). Funds were reallocated because the project design initially underestimated costs. PNG's isolated and remote nature and security issues contributed to increased costs for Component 2, and poor capacity in financial management and procurement contributed to increased costs for Component 4 (TTL Interview, 2016). Financing: This project was financed by the Education for All – Fast Track Initiative. Project funds were targeted and disbursed at US$19.2 million. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) Borrower Contribution: There was no planned Borrower contribution. Dates: The project was approved on March 3, 2011 and became effective on November 31, 2011. The project was expected to close on June 30, 2014 but actually closed on December 31, 2015. A Level 1 Restructuring revised the project's objectives and extended the closing date to June 30, 2015 due to project start-up delays, particularly with procurement and additional time needed to demonstrate learning outcomes (March 2014 Restructuring Paper, p. 9). A Level 2 Restructuring further extended the closing date to December 31, 2015 due to additional time needed to complete delivery of elementary learning and vernacular reading materials and ensure proper time needed for monitoring and evaluation (May 2015 Restructuring Paper, p.8). The project required an additional closing date extension because the Government was ambitious in estimating the time that it would take to complete the project. Long approval processes marked by significant turnover in senior management and leadership led to delays. 3. Relevance of Objectives & Design a. Relevance of Objectives Although the Government and World Bank’s strategic plans reflect significant quality needs at the start and close of the project, as reflected in the objectives, those strategic plans also contain an equal or greater emphasis on access that is missing from the project. For that reason, the relevance of both the original and revised objectives is rated substantial rather than high. The Government of Papua New Guinea’s Universal Basic Education (UBE) Plan (2010-2019) primarily focused on access and retention but also quality to a lesser extent. UBE identified low gross enrollment rates of 78% and net enrollment rates of 45% for grades 1-6 with significant regional variation (PAD, p. 2). It also noted significant barriers to access such as primary school fees, overage enrollment, remote and poor schooling facilities, etc. (PAD, p. 2). However, UBE also included the need for increased quality measures. The Government has simultaneously supported various initiatives such as an outcomes-based curriculum, school learning improvement plan, centrally-funded provision of textbooks and other relevant learning materials, etc. The World Bank’s Country Partnership Strategy in Papua New Guinea (2013-2016) second pillar on "gender equitable improvements in lives and livelihoods" includes support for education quality in order to bolster access to education. It specifically suggests a World Bank focus on its niche areas of support, including better service delivery for reading support in early grades, improved reading results for both boys and girls, and strengthened evidence on education sector expenditures (World Bank Country Partnership Strategy, p. 17). The World Bank Group's decision to support quality over access was due to increased attention towards access issues by the government (as demonstrated by tripling their budget for access in the past ten years) and other development partners such as UK Department for International Development, European Union, Japan International Cooperation Agency, and New Zealand. Quality, as demonstrated by learning outcomes, was considered very poor. For example, according to EGRA in 2010, only three out of ten students could read some sentences after three years of schooling. Thus, the World Bank decided to focus on quality (TTL Interview, 2016). In general, the project objectives are substantially demonstrative of both government and World Bank education strategies. The project’s first objective was to improve reading by increasing the availability of books and learning materials through the establishment of classroom libraries and promoting reading through professional development networks and public reading campaign components. The project’s second objective was to build capacity of CDD to select and procure reading materials and undertake reading assessments by strengthening early grade reading assessment components. The Level One Restructuring technically removed the secondary objective, but it is important to note that the secondary objective related to capacity building of officials, particularly in the procurement of materials and implementation of early grade reading, was subsequently included in the first objective after the restructuring. Rating Revised Rating Substantial Substantial b. Relevance of Design The relevance of design rating is substantial at the original project design and high after the 2014 restructuring. At the time of the original project design, the project’s objectives were substantially linked with the project’s components and activities. The project’s first objective of improved reading was appropriately linked with increasing the availability of books and learning materials by establishing classroom libraries and promoting reading through professional development networks and public reading campaigns. The Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) project’s second objective of building capacity of CDD to select and procure reading materials and undertake reading assessments was linked to strengthening early grading reading assessment. The project components, however, were not explicitly focused on building the capacity of the CDD, and this part of the objective was subsequently removed in the 2014 restructuring. However, the procurement and early grade assessment activities remained and were merged into the first objective; therefore, the project after restructuring implicitly built the capacity of CDD and other government officials. In general, the project activities supported the project components and objectives. The 2014 restructuring added additional activities such as piloting of reading booster programs. Some elements of the project's original scope were ambitious. For example, the project initially targeted 15 provinces for early grade reading assessment, which is ambitious considering the remoteness of Papua New Guinea. The March 2014 restructuring reduced the number of provinces from 15 to 4, which was more doable. Rating Revised Rating Substantial High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To improve the reading skills of elementary and primary education students. Rationale PHORIGINALNARRATIVE Rating Substantial Revised Objective To promote better teaching and learning of reading skills of elementary and primary education. Revised Rationale The original and revised objectives are essentially the same in content. As the objective was not materially revised, the above discussion applies. Revised Rating Substantial PHEFFICACYTBL Objective 2 Objective To build the policy, monitoring, and evaluation capacity of the Curriculum Development Division (CDD) (formerly the Curriculum Development and Assessment Division (CDAD)) to select, procure, and distribute books and reading support materials and to undertake assessment of early grade reading together with provincial officials. Rationale Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) Outputs In addition to the information presented in the discussion of the first objective, where provision of materials/libraries significantly exceeded targets: • 3 national reading events were conducted, exceeding 1 targeted national reading event. • 4 early grade reading assessment baselines and endlines were conducted, meeting restructured targets. • 4 early grade reading assessment results were disseminated to National Department of Education and other stakeholders, meeting targets. Outcomes • 2 provinces piloted Booster Reading programs, meeting targets. Rating High Revised Objective N/A Revised Rationale The 2014 Restructuring removed Objective 2 and inserted the EGRA baseline into Objective 1. Revised Rating Not Rated/Not Applicable 5. Efficiency The project rating for efficiency is modest. The Project Appraisal Document (Annex 9) discuses social returns to education in Papua New Guinea but does not provide an economic analysis specifically for the project. All project activities were implemented within the restructured closing date of December 31, 2015 (18 months extension from the original closing date). Project benefits related to quality in elementary and primary education are considered likely to be achieved. Project staff consider unit costs for project interventions like the provision of materials or training of teachers to have been reasonable due to the remote nature of Papua New Guinea, but comparison costs for similar projects were not provided. According to the ICR, the costs for major project interventions were as follows: • Classroom Libraries - $21/student • Elementary Literacy Materials - $7/student • Elementary Vernacular Language Books - $8/student • Professional Development by Teacher - $9/student • Professional Development by Classroom Libraries - $30/student • Reading Booster Pilot Program - $186/student Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) There were moderate implementation shortcomings. Financial management challenges and procurement delays were persistent before the first restructuring. Financial management was challenging because interim financial reports were not submitted; monthly reports for the designated account were not submitted; and reporting and accounting systems, processes, and procedures were not in place. Procurement was challenging because procurement staff could not provide updates on the status of preliminary procurement actions. There was also a lack of capacity among key personnel and changes in the results framework before the restructuring (ICR, pp. 10, 46, 47). Post-restructuring, there continued to be some financial management and procurement challenges. For example, procurement delays continued as some materials needed clarification of copyright-related issues before disbursement (ICR, p.11). Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Outcome under the original objectives – Moderately Satisfactory Relevance of the project's objectives is rated substantial, as government strategic plans focused primarily on access, in contrast to the project's focus on quality. The relevance of project design is rated substantial, as there was coherence between the project's planned activities and its intended outcomes. Achievement of the objective to improve reading skills is rated substantial, with evidence of measurable learning gains that are reasonably attributable to project interventions. Achievement of the objective to build capacity of the CDD to procure/distribute materials and undertake assessments is rated high, as the project significantly exceeded targets for procuring and distributing reading materials and libraries, and because early grade reading assessments were successfully implemented. Efficiency is rated modest due to financial management challenges and procurement delays. Outcome under the revised objectives – Moderately Satisfactory The ratings are the same as above, with activities under the first objective folded into the second. Only the first objective (improve teaching and learning of reading skills) remained after the restructuring, with substantial achievement. Taken together, these ratings are indicative of moderate shortcomings in the project's preparation and implementation, and therefore an overall Outcome rating of Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating While key project activities were completed and targets were met, the sustainability of outcomes is uncertain. The Government of Papua New Guinea has not allocated funding for READ PNG activities to continue in 2016. However, there is a strong commitment by the National Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) Department of Education (NDoE) to sustain successful components of READ PNG such as the classroom library program. As such, NDoE has submitted paperwork to the Department of Planning for a follow-on READ PNG project (TTL Interview, 2016). The CDD has also allocated funds for project activities (ICR, Annex 7). For example, the government has budgeted for maintaining and expanding classroom libraries (TTL Interview, 2016). The project also built the capacity of NDoE and provincial education staff in areas like distribution of materials and training of teachers. In general, institutional risk is lessened due to increased familiarity with World Bank policies and procedures. While there has been improved transparency and accountability, procurement and financial management is still considered to be a substantial risk (ICR, p.21). Finally, the project promoted communities to take over some aspects of financing and managing the project. The 2015 Level Two Restructuring modified Component 1 Increasing the Availability of Books and Learning Materials by Establishing Classroom Libraries such that schools and villages, instead of the project, would finance bookshelves from their own budgets. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry The project drew upon lessons learned from a previous education project (Education Development Project) which had a small libraries component. The libraries component of the previous project monitored education inputs, instead of outputs and outcomes, particularly around teaching and learning. READ-PNG therefore included teaching and learning output and outcome indicators. The results framework was relatively coherent with defined objectives, activities, and indicators. The output and outcome indicators sufficiently measured achievement of the project's objectives. However, there were moderate shortcomings, as some of the targets were seen as overly ambitious, and costing for EGRA activities was underestimated (ICR, pp. 21-22). Institutional and implementation frameworks with the National Department of Education and Curriculum Development Division (CDD) were appropriate, relevant, and accurate, as CDD is the relevant government agency with responsibility for educational materials and assessment. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project actively engaged the Government at both national and provincial levels. The project helped build the capacity of National Department of Education teams in problem areas such as financial management and procurement, and progress was systematically documented. The Bank team conducted 11 implementation support missions, including a mid-term review. It was able to identify that the project would need restructuring, though the first restructuring took place three years after the start of the project in 2014, and the second restructuring took place four years after the start of the project in 2015. The project notes that the first restructuring delay was due to: (1) problems associated with letters of credit; (2) an October 2013 financial management review documenting ineligible expenditures and unaccounted funds that had to be addressed before restructuring was allowed; and (3) changes in the organizational structure of GPE that delayed GPE secretariat approval of restructuring. The Level One Restructuring further clarified and removed the secondary objective and one outcome indicator. However, it is important to note that activities related to the secondary objective, for capacity building of officials on procurement of materials and implementation of early grade reading, were subsequently folded into the first objective after the restructuring. The restructuring also set more realistic expectations by extending the deadline twice because of procurement delays, and limiting EGRA surveys to four provinces (instead of 15 provinces) due to the remote and isolated nature of Papua New Guinea. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The Government of Papua New Guinea demonstrated full support for the project through the EFA/FTI Catalytic Fund. It committed national and provincial staff members in implementing the project. However, there were significant challenges (ICR, p. 22). The government's centralized and hierarchical system required approval at senior levels. The National Executive Committee had to approve all projects, and high turnover in the secretary's office led to continuous delays. This turnover was set in a context of insecurity, instability, and volatility. The project in particular had long delays related to procurement and cumbersome tax processes for materials (TTL Interview, 2016). Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The Government of Papua New Guinea’s CDD was the implementing agency. The project served to build CDD's capacity on activities related to early grade reading assessments, including administration of baselines, surveys, and analysis of data. It also helped CDD to use data to inform curriculum development. There was poor capacity among CDD officials related to financial management and procurement, which significantly delayed implementation (see Section 11b, and ICR, p. 23). These challenges with financial management and procurement reduced project efficiency to modest. However, capacity building efforts of the CDD in the areas of procurement and early grade reading assessment later in the project period were significant, justifying a moderately satisfactory rating for implementing agency performance. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The original design of the project intended to use: (1) EGRA as the primary evaluation tool to determine impact, (2) CDD to monitor and report data, and (3) beneficiary classrooms to report directly to provincial education offices when they received books and other materials. The M&E framework’s output and outcome indicators adequately measured the project objectives, though some targets were initially seen as ambitious. The M&E framework was slightly modified in the March 2014 Level One restructuring, including: (1) clarification of project objectives; (2) restating and modifying project objectives and outcome/intermediate indicators; (3) reduction of the number of EGRA provinces; and (4) project extension (ICR, p. 10). b. M&E Implementation Early challenges with reporting were due to: (1) limited availability of data due to slow implementation of inputs; (2) limited capacity of national and provincial staff in understanding indicators; and (3) limited capacity to collect data. There were a few changes to the indicators during the 2014 restructuring. The "Classroom Teaching Practices Reflect Enhanced Knowledge in Reading Instruction and Use of Learning Materials" indicator and "Students in Elementary and Primary School Will Demonstrate Improvements in Their Literacy Skills as Defined by EGRA" indicators were dropped. These are appropriate changes, as the former indicator would have been difficult to measure, and the latter indicator was already demonstrated through other existing indicators. The early grade reading assessment with baseline and endline results for both Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) treatment and control groups proved to be of great value in assessing project outcomes. c. M&E Utilization The Pilot Program impact evaluation assessed achievement of results. It was also used to further develop a new standards-based curriculum and relevant teacher training activities. M&E Quality Rating Substantial 11. Other Issues a. Safeguards The project was identified as a Category C operation. The OP 4.10 Indigenous People’s Plan was considered but not required because all project beneficiaries were regarded as indigenous. Compliance with project safeguards was consistently rated satisfactory throughout implementation. The ICR's Annex 7 notes attention towards gender through the disaggregation of data, indigenous people by conducting EGRA surveys in vernacular language and providing vernacular language elementary materials, and environment through inclusion of environment-specific criteria when considering classroom library books. b. Fiduciary Compliance Procurement: Under the project, the original procurement process required the Government (CDD) to implement large-scale international competitive bidding procurement processes for the first time. The initial procurement design included an inadequate number of dedicated staff and mission support. In 2013, the Bank modified processes to directly purchase materials from suppliers. It also increased technical support to CDD national and provincial staff for procurement. These changes significantly reduced procurement delays. The ICR notes moderately satisfactory ratings throughout implementation. Financial Management: Challenges in financial management were attributed to a lack of capacity of CDD financial management staff and limited knowledge of Bank requirements. This led to challenges with timeliness of withdrawal applications and submission of interim financial reports, and reporting and accounting systems, policies, and procedures. The Bank created several training programs with the Government on financial management and other issues. The project conducted and complied with annual, external, unqualified, legally-binded audits throughout the project and at the end of the project. The ICR notes moderately satisfactory ratings throughout implementation. Challenges with financial management and procurement and resulting delays reduced project efficiency to modest. However, capacity building efforts of the CDD in areas of procurement and early grade reading assessment were significant, resulting in improvements later in the project period. c. Unintended impacts (Positive or Negative) None reported. d. Other --- Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Moderately Satisfactory Moderately Satisfactory --- Risk to Development Outcome Substantial Substantial --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR identifies the following lessons (ICR, p. 23): 1 . EGRA education projects with strong links between objectives and activities can measure learning outcomes with strong evaluation processes. Strong results frameworks, including EGRA surveys, can be used to demonstrate learning outcomes. 2 . EGRA activities can have a substantial impact on informing education policy. The EGRA pilot program results were used to inform the new standards- based curriculum and create relevant teacher training activities. 3 . EGRA surveys are useful in measuring impact. The EGRA survey in this case demonstrated specific reading outcomes for students. Specifically, the baseline and endline surveys revealed that girls and boys had significant improvement in phonemic awareness, letter sound identification, and familiar word reading in two pilot provinces as a result of the project. 4 . Education procurement processes need established and proven capacity in order to ensure on-time delivery of materials. Procurement staff must be familiar with World Bank Group policies and procedures, and the Bank must allocate staff to build capacity on basic procedures like international competitive bidding in order ensure timely delivery of materials. 14. Assessment Recommended? Yes Please explain Yes. This project marks GPE's shift from access to quality and the WBG's priority in early grade reading assessments. It had significant output and outcome achievements, including student-level improvements in reading and policy-level reform in curriculum development, that could be usefully studied and verified. 15. Comments on Quality of ICR The ICR is clear, candid, and concise. It articulates the results framework effectively, highlighting the linkages between project outputs and outcomes. It also clearly explains the changes that took place during project restructurings. Further detail on fiduciary management of the Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reading Education Project (P105897) project, including procurement and financial management, could have been included. a. Quality of ICR Rating Substantial