Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) Report Number : ICRR0020614 1. Project Data Project ID Project Name P116521 PG-Flexible and Open Distance Education Country Practice Area(Lead) Papua New Guinea Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-48530 30-Jun-2016 6,000,000.00 Bank Approval Date Closing Date (Actual) 11-Jan-2011 30-Jun-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 5,000,000.00 0.00 Revised Commitment 5,000,000.00 0.00 Actual 4,668,628.82 0.00 Prepared by Reviewed by ICR Review Coordinator Group Kimberly Kamlesh Judyth L. Twigg Joy Behrens IEGHC (Unit 2) Parekh 2. Project Objectives and Components a. Objectives The original project development objective was: (1) to increase the number of out of school youth who complete recognized secondary diploma and certificate equivalency programs (Loan Agreement, p. 5). The revised project development objectives after a 2015 Level 1 Restructuring were: (1) to align the distance education curriculum with the national curriculum; and (2) to strengthen Flexible and Open Distance Education's (FODE) information management system (2015 Restructuring Paper, p. 14). Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 06-Mar-2015 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components The original project components were: (1) Component 1 Updating of Curriculum and Instructional Materials (Estimated Cost US$1.5 million) - This component aimed to: (1) improve the quality of the distance learning curriculum by supporting the development of new and/or upgrading of existing curricula for grades 7 - 12 and the quality and availability of instructional materials; and (2) improve the quality and availability of instructional materials by increasing the accessibility of instructional materials, improving the quality of documents produced, and lowering production and distribution costs. (2) Component 2 Improving the Quality of Service Delivery (Estimated Cost US$0.7 million) - This component aimed to improve the quality of services provided by Flexible and Open Distance Education (FODE) for both the central FODE campus and 20 provincial centers through staff training, revised administrative processes, establishing and implementing information technology (IT) systems, and recruiting and training tutors in provincial FODE centers. FODE is recognized primary and secondary equivalency education for out of school children who typically cannot access formal primary and secondary schools. (3) Component 3 Grade 11 - 12 Matriculation Program Expansion (Estimated Cost US$1.8 million) - This component aimed to expand FODE's Grade 11 - 12 equivalency program to at least one secondary school in each of PNG's 20 provinces. It was intended to build on recent pilots and expand the Grades 11 - 12 equivalency program by reimbursing schools for use of their facilities, providing classrooms with equipment and instructional materials, and financing costs for teachers for after-hours teaching in each school. (4) Component 4 Improving Information Technology, Monitoring and Evaluation (Estimated Cost US$1.0 million) - This component aimed to provide FODE with IT technology and train FODE staff on the use of such technology. It also included funding for small scale pilots on low-cost IT to improve service delivery. The components were revised at the 2015 restructuring: Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) (1) Component 1 Curriculum Development (formerly Component 1; Estimated Cost US$1.80 million; Actual Cost US $1.64 million) - This component was revised to prioritize developing curriculum for all Grade 7 - 12 subjects. This also included aligning this curriculum with the national curriculum; creating learning assessments for the revised curriculum; piloting the printing of Grades 7 - 10 curriculum; and creating an implementation strategy. (This revised component did not include the originally-planned development of instructional materials.) (2) Component 2 Information Management Infrastructure (formerly parts of Components 2, 3, & 4; Estimated Cost US $2.52 million; Actual Cost US $1.96 million) - This component was revised to establish connectivity between FODE headquarters and its 22 provincial centers; align FODE information and communication technology (ICT) with Department of Education (DOE) ICT; and upgrade FODE Student Information Systems. Support was to be provided for procurement and installation of IT hardware and software and consulting services for integration and use of IT and Student Information Systems. (This revised component did not include parts of the original Component 2 on training staff, creating administrative processes other than IT, and financing tutors. It also removed Component 3 on Grade 11 - 12 Matriculation Program Expansion because PNG higher education institutions no longer recognized the Grade 11 - 12 Matriculation Certificate from the older curriculum.) (3) Component 3 Project Management, Monitoring and Evaluation (newly created Component; Estimated Cost US $0.68 million; Actual Cost US $0.97 million) - This component was created to support the management of the project, including planning and implementation, budgeting and finance, procurement, and monitoring, reporting, and evaluation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The actual cost of IDA financing was US$4.57 million, or 97.6% of the original estimate of US$5.00 million. Spending on revised Component 1 was slightly less than expected (US $1.64 million or 91.1% of the revised US$1.80 million estimated); spending on revised Component 2 was less than expected (US $1.96 million or 77.8% of the revised US $2.52 million estimated); and spending on revised Component 3 was slightly more than expected (US $0.97 million or 157.0% of the revised US$0.68 million estimated). Funds were reallocated because of the 2015 Level 1 restructuring, including changes in the project objectives and project components (ICR, p. 27). Financing: The total estimated cost of the project was US$6 million, including US$5 million from an International Development Association (IDA) Specific Investment Credit and US$1 million from the Government of Papua New Guinea. The actual cost was US$5.931 million, including US$4.57 million from IDA and US$1.361 million from the Government. The ICR (p. 27) notes that the Bank financing of US$4.57 million falls short of the intended US$5 million due to exchange rate losses. Borrower Contribution: The Borrower's estimated contribution was US$1 million, but the actual contribution was US$1.361 million. Dates: The project was approved on January 11, 2011 and became effective on November 3, 2011. The Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) project closed on June 30, 2016 as expected. The project experienced significant delays in implementing the original project objectives and components. However, the March 2015 Level 1 Restructuring revised the objectives and components, and the project was able to close on schedule. 3. Relevance of Objectives & Design a. Relevance of Objectives The relevance of objectives is rated substantial both before and after the restructuring. The objectives are substantially relevant to Government strategy and to country conditions in the education sector. The Government of PNG's National Education Plan (NEP, 2005-2014) prioritized access, retention, and quality in basic education (PAD, p. 1), and the current Education Strategy 2020 also prioritizes FODE. At project preparation, fewer than 10% of students had access to formal secondary schools, and so FODE was considered a legitimate path (TTL Interview). In particular, the Government recognized distance education as a strategy to engage out-of-school youth, including those who had never enrolled and those who had dropped out from formal school (ICR, p. 1). The FODE Center was the public institution that provided upper primary and secondary education through distance learning, supported by ad hoc on-site teachers and tutors (PAD, p. 1). At the time of project design, formal secondary schools supported 65,000 students, while FODE supported 46,000 students. However, the success rates of FODE were limited, with only 335 completion certificates issued in 2008 and 684 in 2009. The objectives are also substantially relevant to Bank strategy at project appraisal and at closing. A Bank assessment at appraisal indicated that FODE lacked funding to ensure quality curriculum, instructional materials, and teacher/tutor skills (PAD, p. 2). The PNG Country Assistance Strategy (CAS) 2008-2011 supported both policy and institutional development for distance education at the national level and service delivery at the local level. The World Bank Education Sector Strategy 2020 prioritizes service development for out-of-school youth in the Pacific region (PAD, p. 2). The Bank's Country Partnership Strategy at closing (2013-2016) places education support in a "niche" role under its second pillar to achieve gender-equitable improvements in lives and livelihoods, recognizing the need to harness the Bank's comparative advantage to support evidence-based policy dialogue and appropriate choices in program design. This project's support for FODE is explicitly cited as a key existing intervention supporting the base for this pillar (County Partnership Strategy, p. 17). After restructuring, the project's objectives were narrowed to take into account the level of effort required for curriculum development. Considering that this was the first time FODE had revised the curriculum, the revision is considered appropriate. Rating Revised Rating Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) Substantial Substantial b. Relevance of Design Relevance of design is rated modest before the restructuring and substantial after the restructuring. The pre-restructuring rating is modest because the original design did not take into account the limited capacity of the borrower and implementing agency, and did not plan for adequate levels of activity to compensate. The post-restructuring rating is substantial because capacity limitations were taken more directly into consideration, leading to a narrower and more achievable project design. As originally envisaged, the project's design was intended to cover a very broad set of components and related activities. At that time, FODE had not updated its curriculum in over 30 years. The original project activities set out to update this curriculum, including all subjects for Grades 7-10; create instructional materials; hire and recruit teachers; streamline administrative and IT processes for FODE headquarters and provincial centers; align the Grades 11 - 12 one-year curriculum with the curriculum in formal secondary schools to allow for entry into higher education; and significantly update IT and monitoring and evaluation (M&E) systems. This wide range of activities was to be implemented under very challenging circumstances involving remote locations. There was insufficient planning for security concerns that ultimately limited the ability to provide internet connectivity to FODE centers around the country. Overall, the planned activities are considered to have been significantly too ambitious given time, context, and financial constraints, without the requisite necessary capacity development. At the time of restructuring, FODE's lack of capacity to implement the original design was recognized. In the restructuring, the project decided to focus on critical aspects for the core development of FODE; consolidate components and activities; and exclude components and activities no longer relevant to PNG education. For example, original Component 1 was given priority and curriculum development came to the forefront; original Components 2, 3, 4 were consolidated into Component 2, and original Component 3 was eliminated, when PNG higher education institutions would no longer recognize Grade 11 - 12 certificates. Instead, FODE was to directly create its own Grades 11 - 12 curriculum as part of its curriculum design and development. Rating Revised Rating Modest Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Increase the number of out of school youth who complete recognized secondary diploma and certificate equivalency programs. Rationale Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) Outputs • The majority of grades 7 - 12 curriculum was written. • Staff capacity was built through curriculum writing workshops led by distance education consultants. • An analysis of curriculum development was completed, including mapping of resources, planning for consultations, and timelines. • Commonwealth of Learning (an intergovernmental organization promoting the development and use of open learning and distance education) provided technical assistance on curriculum development and distance education service delivery. Workshops and exchanges, including study tours, took place. • Tutoring manuals for Grades 11 - 12 subjects were developed. • A FODE information technology (IT) needs assessments was conducted. Hardware such as laptops and printers were distributed to all provincial FODE centers, and basic training was provided to FODE administrators on IT systems. • Technical assistance led to a stakeholder review of Student Information Systems. • No data for Grade 8, Grade 10, and Grade 12 enrollment rates, completion rates, or exam takers was provided. These indicators were dropped at the restructuring because available data was not credible. Outcomes •No data on Grade 8, Grade 10, and Grade 12 graduates with certificates was provided. This indicator was dropped after the restructuring because available data was not credible. Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective The first revised project development objective after the 2015 Level 1 Restructuring was to align the distance education curriculum with the national curriculum. Revised Rationale Outputs • 26 subject matter learning materials for Grades 7 - 12 were piloted and made ready for printing. This met revised Level 1 Restructuring targets, though 28 was originally specified due to a miscalculation. A new process of curriculum development and quality assurance included assessment of FODE students at entry, a standards- and competency-based curriculum to ensure equivalency with national curriculum, and quality control mechanisms including piloting and sampling. • A roll-out strategy for the aligned distance education curriculum was developed and approved. This met Level 1 Restructuring targets. Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) Outcomes • 26 subject matter curricula for Grades 7 - 12 were produced and approved by the Governing Council. The project met its target, as it was intended to produce curricula for all 26 subject areas for Grades 7 - 12. Revised Rating Substantial PHINNERREVISEDTBL Objective 1 Revision 2 Revised Objective The second revised project development objectives after the 2015 Level 1 Restructuring was to strengthen Flexible and Open Distance Education's (FODE) information management system. Revised Rationale Outputs • Network connectivity was established in FODE HQ and 21 provincial centers. This exceeded the Level 1 Restructuring target of at least 10 provincial centers. • The FODE management information system was upgraded and implemented. This met the Level 1 Restructuring target. • FODE annual project plans were provided. This met the Level 1 restructuring target. • Student Information Systems were upgraded. Outcomes • A new internet data system for FODE was created and implemented in all provincial centers. The project partially met its target, as it was able to create and implement the new internet system, but the system was not created in time for use in planning processes. Revised Rating Substantial PHREVISEDTBL 5. Efficiency The project rating for efficiency is modest. According to the PAD, the project did not attempt to calculate economic and social rates of return for this investment since the last household survey was carried out more than 15 years before the start of the project (PAD, p. 10). However, the ICR notes that FODE in and of itself is efficient and yields internal rates of return between 5% and 6%, depending on which grade is completed. There is a positive net economic benefit for Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) students attending and completing FODE grades (ICR, pp. 19-20). There were moderate implementation shortcomings. Before the restructuring, financial management was considered successful, with timely submission of interim financial reports and financial audits in 2012 and 2013 resulting in unqualified opinions (Level One Restructuring Paper, p. 7). However, the ICR states that FODE project staff lacked capacity to produce financial statements (ICR, p. 9) and that there were challenges in timely submission of interim financial reports and financial audits (ICR, p.14). These statements are probably more accurate given the lack of Bank support to PNG in the education sector and limited knowledge of Bank procedures. Procurement also faced repeated delays due to weak staff capacity, though there was improvement with the hiring of a procurement advisor. Nevertheless, the project faced delays because of lengthy internal government procurement practices, lack of understanding of FODE staff of those processes; and lack of follow-up (Level One Restructuring, p.7). The ICR adds that FODE project staff lacked understanding of Bank procurement processes (ICR, p. 9). After the restructuring, Component 3 focused specifically on project management, including financial management and procurement. However, financial management during this period continued to struggle. The project continued to have challenges with timely submissions of withdrawals, unaudited interim financial reports, and audited financial statements. During this period, the project had an overdue audited financial statement. Procurement improved due to the impact of the hired procurement advisor pre-restructuring, though weak capacity of FODE staff continued to remain an issue. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 100.00 ICR Estimate  6.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome At the time of restructuring, US$2.52 million out of a total of US4.57 million, or 55% of Bank financing, had been disbursed. Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) Outcome under the original objective - Moderately Unsatisfactory Relevance of the project's objectives is rated substantial, as the project was in line with Government of Papua New Guinea education strategy and World Bank CAS strategy on youth. The relevance of project design is rated modest, as project's planned activities were overly ambitious, and there was insufficient capacity development planned through the components. Achievement of the objective to increase the number of out-of- school youth who complete recognized secondary diploma and certificate equivalency programs is rated modest, with considerable curriculum and IT infrastructure development achieved but lack of measurable evidence from FODE on outcomes. Efficiency is rated modest due to financial management and procurement challenges. Outcome under the revised objectives - Moderately Satisfactory Relevance of the project's objectives is rated substantial, as the project was in line with Government of Papua New Guinea education strategy and World Bank CAS strategy on youth. The relevance of project design is rated substantial, as the restructured project activities were more in line with capacity and likely to achieve the reformulated objectives. Achievement of the objective to align the distance education curriculum with the national curriculum is rated substantial, and achievement of the objective to strengthen Flexible and Open Distance Education's (FODE) information management system is also rated substantial, with evidence of project-financed outputs leading to intended outcomes. Specifically, the project was able to meet revised outcome targets: 26 subject matter curricula for Grades 7 - 12 were produced and approved by the Governing Council, and a new internet data system for FODE was created and implemented in all provincial centers. Efficiency is rated modest due to continuing financial management and procurement challenges. According to OPCS/IEG harmonized guidelines, when objectives and/or key outcome targets are revised, the overall outcome is derived by weighting outcome under the original and revised objectives according to the percentage of Bank financing disbursed under each set of objectives/targets. In this case, the calculations are (.55)(3 = MU) + (.45)(4 = MS) = 1.65 + 1.8 = 3.45, which rounds to 3 (MU). Because the restructuring took place after the majority of Bank financing had been disbursed, the overall Outcome rating is Moderately Unsatisfactory, indicative of significant shortcomings in the project's preparation and implementation. a. Outcome Rating Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating While key project activities were completed and revised targets were met, the sustainability of outcomes is uncertain. Having said that, the project is credited with building FODE staff capacity in curriculum development and information systems. Specifically with curriculum development, project staff worked to ensure that the newly created FODE curriculum was aligned with the national formal school curriculum through a standards-based approach. This is considered to be relevant and innovative in basic education. However, project documents provide evidence for continued risk related to weak staff capacity and continued challenges in financial management and procurement. Finally, there has been a decline in public expenditure in education that may Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) adversely affect sustainability of achieved outcomes (ICR, p. 23). The Department of Education has submitted a proposal to the Department of Finance to fund a follow-on project to FODE, but the Government is at a standstill in making decisions because of upcoming elections (TTL Interview). a. Risk to Development Outcome Rating High 8. Assessment of Bank Performance a. Quality-at-Entry The project's design was in line with the Government of PNG's National Education Plan (2005-2014) and World Bank Country Assistance Strategy (CAS) (2008-2011). However, the FODE curriculum had not been updated in over 30 years, and the project team underestimated the time and effort required to update the curriculum (ICR, p. 23). In addition, the project's monitoring and evaluation arrangements were inadequate to measure achievement under the original objectives and targets (see Section 10). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank conducted 21 review processes, including 8 technical support missions, 13 implementation support missions, and a mid-term review (ICR, p. 14 and 23). The ICR documents detailed missions that led to the recruitment of key national and international technical staff, built capacity of staff, organized technical workshops, improved project and financial management processes, and facilitated completion of curricula (ICR, p. 14). The Bank team's active engagement with the project led to the restructuring that improved activities and outcomes. This restructuring came late during implementation, but the ICR notes that the delay was influenced by factors beyond the Bank's control, and FODE and the Bank teams were already monitoring based on the revised design in late 2013. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) a. Government Performance During the project design phase, the Government of PNG committed to aligning the FODE equivalency curriculum with the formal school curriculum, and this was achieved. However, it is unclear why the FODE Grades 11 – 12 curricula were no longer accepted by PNG higher education institutions during the course of the project. There could have been greater synergy between the Department of Education’s FODE and the Department of Higher Education, Research, and Science and Technology during project prepatation. In terms of financing, the Government agreed to contribute $US 1 million towards the project. However, during project implementation, there were ongoing challenges related to expenditures on construction of the national FODE center and taxes (GST) that were not originally included in IDA financing. The Government overspent its original budget by $US 0.361 million because of overspending on construction of the national FODE center (TTL interview). The Government was also responsible for delays in the project's financial management and, critically, in its restructuring (ICR, p. 24). Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance This project was the Government of PNG’s first World Bank-financed project in education in over ten years. As a result, there was limited capacity within FODE staff to implement the project, resulting in ongoing project management challenges (financial management and procurement) and technical challenges (curriculum development) (ICR, pp. 24-25). Having said that, the project did learn from these challenges, and the staff is credited with having grown in capacity throughout the project period (ICR, p. 11). In particular, FODE built staff capacity to create a standards-based subject matter curriculum for Grades 7-12, ensuring rigorous quality assurance during the curriculum development process. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The original project design, including objectives, components, activities, targets, and indicators, was very ambitious. On the reporting side and in addition to project indicators, FODE was expected to report on students enrolled in FODE, sitting for FODE exams, and completing FODE. On the evaluation side, FODE was expected Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) to produce a research study on the demographics of FODE students, a tracer study of FODE students (both completers and drop outs), and a youth study. This schedule of reporting was too ambitious, and FODE did not have systems in place to document student progress. The expected evaluation was also too ambitious, and plans for it were ultimately eliminated after the restructuring (ICR, p. 15). b. M&E Implementation The pre-restructuring M&E design was challenging and data was not collected as originally intended. However, the post restructuring M&E design led to the development of a FODE database system that allowed for collection and tracking of student data (ICR, p. 15). The project's key performance indicator was changed from number of Grade 10 certificate completers (because of lack of data) to number of subject curricula approved and data available for planning and budgeting (Level One Restructuring, p. 15). The new indicators were considered appropriate given the revision of objectives, adequately measuring the PDO, and data on these indictors were more readily collectible by project staff (ICR, p. 9). c. M&E Utilization While the project was able to create a new database system for FODE, FODE was not able to use the data for planning and budgeting purposes (ICR, pp. 18-19). M&E Quality Rating Modest 11. Other Issues a. Safeguards This project was an environmental Category ‘C’ operation. It triggered the environmental safeguard OP/BP 4.01 because the IDA credit and GoPNG financing was intended to renovate the FODE headquarters and provincial centers. As this activity involved only minor renovation work in existing headquarters and provincial center, an environmental assessment was not required. The restructuring removed all renovations from the project. Project safeguard compliance is considered to be satisfactory (ICR, pp. 15- 16). b. Fiduciary Compliance The project consistently faced challenges with financial management, including timely submissions of Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) withdrawal applications, unaudited interim financial reports, and audited financial statements (ICR, p. 16). Taxes (GST) were paid through the IDA credit, instead of with Government financing, which was not allowed or previously agreed upon. There were consistent financial management supervisory visits to address concerns, and a final financial management visit in 2017 is expected to address remaining issues (ICR, p. 16). Procurement also experienced a number of challenges, primarily because of weak FODE capacity to carry out procurement activities. The project addressed concerns by hiring an International Procurement advisor halfway through the project, improving planning and implementation (ICR, p. 16). c. Unintended impacts (Positive or Negative) None reported. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Unsatisfactory Unsatisfactory Risk to Development High High --- Outcome Moderately Moderately Bank Performance --- Unsatisfactory Unsatisfactory Moderately Moderately Borrower Performance --- Unsatisfactory Unsatisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR (pp. 25-26) highlights several useful lessons, including: 1. Extensive needs assessments when re-engaging within a sector can help the Bank to better understand Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PG-Flexible and Open Distance Education (P116521) the sector, Government and implementing partner capacity, and overall constraints in the sector. In this project, project staff poorly understand the time and effort required to design and roll out curriculum and the capacity of staff to implement the project. 2 . Understanding the limitations of existing platforms for data collection is essential when creating project indicators. In this project, project staff assumed that FODE staff had the ability to collect student-level information about course enrollment, test enrollment, and completion rates. The data system was limited, however, and a new system was produced only by the end of the project. Because of these limitations, the project had to revise its objectives and indicators at the restructuring. 3 . Curriculum design and development takes time, and the project design phase is the appropriate point to conduct relevant and appropriate curriculum needs assessments. After 40 years of not updating the FODE curriculum in PNG, it was considered necessary to update all subject matter modules for Grades 7 – 12 curricula (ICR, p. 25). 14. Assessment Recommended? Yes Please explain The World Bank Group has invested in education projects that are intended to provide out-of-school children and youth a second chance to pursue education. The lessons of this project's challenges and achievements in achieving those goals could be usefully examined, ideally in tandem with those of the PNG Read project. 15. Comments on Quality of ICR The ICR is candid and concise, clearly explaining the project's results framework, the plausible linkages between its objectives, components, activities, and indicators, and the revisions that were implemented at restructuring. Its lessons are detailed and appropriate for the project. However, there were some elements that could have received more emphasis, in particular the process and impact of aligning FODE curriculum with the national curriculum. The discussions on safeguards and impacts on gender and other vulnerable groups are relatively thin. There is also a lack of discussion on the overall sustainability of the project's achieved outcomes. a. Quality of ICR Rating Substantial Page 14 of 14