ICRR 13457 Report Number : ICRR13457 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/22/2011 PROJ ID : P045174 Appraisal Actual Project Name : Education US$M ): Project Costs (US$M): 70.9 43.2 Development Project Country : Lebanon Loan /Credit (US$M): Loan/ US$M ): 56.6 42.6 Sector Board : ED Cofinancing (US$M ): US$M): 2.1 0 Sector (s): General education sector (100%) Theme (s): Access to urban services and housing (50% - P) Administrative and civil service reform (50% - P) L/C Number : L7010 Board Approval Date : 03/30/2000 Partners involved : French Bilateral Closing Date : 12/31/2005 12/31/2009 Cooperation Evaluator : Panel Reviewer : Group Manager : Group : Judith Hahn Gaubatz Robert Mark Lacey IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: According to the Project Appraisal Document (PAD, page 2), the original project objectives were: (i) to enhance the capacity of the Ministry of National Education, Youth and Sport (MNEYS) to function as an effective manager of the education sector; and (ii) to restore the credibility of the public education system through improvements in quality and efficiency, and increased access . According to the Loan Agreement, the objectives were "to assist the Borrower in improving the management and institutional capacity of the public education system, promoting equity of educational opportunities and enhancing the quality and efficiency of the public education system ." Although the objectives in the PAD and Loan Agreement were essentially the same, the PAD objectives are used for the purposes of this review as they were more specific in identifying the MNEYS as the focus of capacity building activities . The project objectives were formally revised in 2004 (at which point US$3.22 million, or 7.0%, of the loan had disbursed), to reflect a shift in focus from actual implementation of reforms to readiness for the reform process, as well as an expansion of focus to include vocational education. The revised objectives were : (i) to assist the Borrower in building the knowledge base, analytical tools and the institutional capacity required to initiate reform of the general and vocational education systems; and (ii) ii) to enhance access through a modest program of school construction in priority areas of need . Note: The MNEYS and the Ministry of Technical and Vocational Education were merged into a single Ministry of Education and Higher Education (MEHE) in 2000. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 05/17/2004 c. Components (or Key Conditions in the case of DPLs, as appropriate): The original project components were as follows : (1) Management and Institutional Development (Appraisal : US$4 US$ 4.4 million ): This component aimed to improve financial processes for more efficient use of resources, business processes for more systematic decision -making, and human resources for improved management . Activities included the establishment of information systems, organizational restructuring of the MNEYS, rationalization of teaching staff, piloting of a local school empowerment initiative, and a sector financing study . US$ 49. (2) Quality Enhancement (Appraisal : US$49 49 .0 million ): This component aimed to facilitate equitable access to public schools. Activities included upgrading of school facilities, teacher training, development of tools for educational assessment, and provision of computer resources . US$ 2.6 million ): This component supported the operations of the project (3) Project Implementation (Appraisal : US$2 management unit (PMU), housed in the Education Center for Research and Development (ECRD), a semi-autonomous institution under the direct control of the Minister of Education . The revised project components were as follows : US$ 9.8 million ): This component supported sector (1) Policy, Planning and Information Management (Actual : US$9 development in both the general and vocational education systems . Activities included the development of a national education strategy, establishment of an education management information system (EMIS), a financing study, and an organizational restructuring of the MEHE . 2) Strengthening School Leadership, Pedagogical Capacity and the Assessment of Academic Achievement (2 US$ 5.3 million ): This component supported professional development activities, including implementation of (Actual : US$5 a school leadership program for principals, training for general and vocational education teachers, and development of academic assessment tools . 3) Upgrading Education Facilities (Actual : US$23 (3 23 .6 million ): This component supported the construction and US$ 23. equipping of new school facilities, and the provision of information technology infrastructure in some secondary schools and all vocational education schools . US$ 4.5 million ): This component supported the operations of the PMU, which was (4) Project Management (Actual : US$4 moved to the MEHE soon after the restructuring . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Following recommendations from a 2005 Country Portfolio Performance Review (CPPR), the Government and Bank agreed to cancel a number of vocational and technical education (VTE) activities (mainly equipment procurement) due to slow progress in implementation . Accordingly, the planned loan amount decreased from US$56.6 million to US$44.6 million. Neither the Borrower or the co-financing partner, French Bilateral Cooperation, provided the planned amount of funding - the reason for the lack of contribution is not provided . At the time of restructuring, the closing date was extended from December 2005 to December 2007. It was extended twice more - to December 2008 and then to December 2009 - to allow for the completion of activities . 3. Relevance of Objectives & Design: The relevance of the original objectives is rated Substantial . Improved education outcomes through better management of the education sector remains a priority issue for the Bank and the Borrower, as articulated in the Bank's FY08-09 Interim Strategy Note. The objectives of the project aimed to address weaknesses in the education sector that were well understood by the Bank and the Borrower . However, the aims were overly ambitious within the time frame of the project, given the low capacity of the Borrower . The relevance of the original design is rated Modest . The project design appropriately focused on major sector issues such as information management, sector financing, and teacher rationalization . However, given the fragile operating environment - political instability and weak capacity - the time and effort needed to reorganize the sector and build adequate institutional capacity stretched beyond the time frame of the project . Moreover, there was an absence of policy dialogue that would have greatly aided the reform efforts . The relevance of the revised objectives is rated Substantial . The focus on preparing the sector for major reforms through improving the knowledge base and providing analytical tools was substantially relevant, given the uncertain political and institutional context . Pursuing an integrated approach to general and vocational education reform, from a policy standpoint, was also relevant . The relevance of the revised design is rated Substantial . The scope of project activities was more narrow and realistic, explicitly concentrating on improving institutional capacity in sector policy and management, school administration, and teachers. However, the inclusion of VTE activities, aside from the policy -related work incorporated with the general education analytic outputs, appears questionable given the known institutional weaknesses in the VTE sub -sector. 4. Achievement of Objectives (Efficacy): Original objectives Achievement is rated Negligible due to lack of evidence on outcomes . There is also no information reported in the ICR on outputs from the original project components . Revised Objective 1: To assist the Borrower in building the knowledge base, analytical tools and institutional capacity to initiate reforms of the general and vocational education systems . Achievement of this objective is rated Modest . While there were contributions in building the knowledge base and providing analytical tools, there was limited evidence of strengthened institutional capacity . Outputs : Preparation of a National Education Strategy, which included "detailed strategies and objectives " for sector development in both general and vocational education . The ICR reports that the final product was based on an extensive process of situational analysis and consultation with numerous stakeholder groups . Preparation of a comprehensive education finance study, which identified all financial resource flows in and out of the sector and created an econometric model to assess the financial implications of various policy options . The study covered both the general and vocational education systems . Preparation of an organizational structure study, including recommendations for a new structure, proposed functions for each unit, development of work flow diagrams, and staff training in both general and vocational education . Participation of 450 school principals in a Leadership Development Program . The program was designed to improve leadership and management skills in implementing strategic sector reform plans . Development of an EMIS and training of MEHE staff . Establishment of a teacher development program, including training centers and teacher evaluation tools . From 2005-2008, 25,000 teachers participated in the training . Development of databanks (Question Bank System, Examination Generation System, and Examination Management System) to enhance efficiency and consistency of academic testing and assessment . Provision of equipment for vocational education facilities was not completed. An Information Management Unit (IMU) within MEHE was not established as planned, due to high staff turnover and recruitment difficulties. The ICR reports that the MEHE incorporated the functions of the IMU into the Educational Secretariat in 2007, as an "interim measure". Outcomes : The Cabinet formally approved and adopted the National Education Strategy in April 2010. A series of technical papers on education finance were prepared using data from the financing study and model . A Program-Based Budget approach has been adopted by the MEHE for the 2009-11 period. The MEHE has endorsed the new organizational chart . Draft laws and regulations needed to implement the changes are being finalized. A formal relationship has been established with the Lebanese University to support future delivery of the School Leadership Program, and a Ministerial Decree has been issued establishing a 15 percent salary incentive for personnel completing the Program. The EMIS was not fully operational at the time of project closing; however, project transition arrangements include plans to streamline use of the EMIS in the day -to-day administration of the MEHE. Revised Objective 2: To enhance access through a modest program of school construction in priority areas . Achievement of this objective is rated Modest, due to lack of clear evidence of increased access . Outputs : Construction of eleven school facilities in priority areas, as planned . The ICR cites findings of the Government's School Construction Technical Audit in October 2009, which included reports that architectural aspects were good, quality of construction was good or acceptable, and all procured heavy equipment was installed . The Audit also reports shortcomings including higher than planned costs, mistakes that undermined the rational use of some spaces, roof leaks, and heavy equipment that was not operational due to lack of budget . Outcomes : Although construction of eleven schools took place as planned, the Government's School Construction Technical Audit reported "low student enrollment" in 2009, indicating that the provision of these facilities had not enhanced de facto access. The ICR reports that enrollments were higher in the following year, though no data are provided . 5. Efficiency (not applicable to DPLs): Efficiency of the original project is rated Negligible , as most project outputs were not implemented . Efficiency of the revised project is rated Modest . No economic or financial rate of return analyses, nor any alternative measures of efficiency, such as cost effectiveness, were prepared at the close of the project . The ICR reports that (i) the average construction cost per square meter increased by 40 percent compared to original estimates, citing higher material and energy prices; and (ii) equipment procured for schools such as water treatment equipment, water pumps, and electricity generators were often too large . There is no information provided on the efficient use of project resources regarding the knowledge and capacity building activities . ERR)/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The outcome of the original project is rated Unsatisfactory . Although the project objectives were substantially relevant to the country and Bank priorities, the design was overly ambitious given the fragile operating environment . Achievement of the project objectives to improve management and quality of the education sector was Negligible . The outcome of the revised project is rated Moderately Unsatisfactory . The project objectives and design were more consistent with the capacity level of the Borrower . There were contributions in analytical work and strategic studies; however, there was limited evidence of strengthened institutional capacity, and there was modest achievement of the objective to increase access . Efficiency of the revised project was modest . According to ICR guidelines, the combined outcome rating is determined by weighting the percentage of the loan that was disbursed before and after restructuring . As US$3.22 million was disbursed before restructuring, or 7.6%, the final outcome rating is Moderately Unsatisfactory . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Risk to development outcome is rated Significant . The Government has demonstrated commitment to moving forward with the reforms supported by the project . Key project outputs - such as the National Education Strategy and organization restructuring plan - have moved forward to the next steps of implementation . The project PMU was institutionalized as the Educational Secretariat for Development and Support (ESDS) within the MEHE. Moreover, the Minister of Education requested that the Bank to identify resources for transitional support while initiating a second phase of the sector reform program . A second Bank education project - Education Reform for Quality Schools - was scheduled for appraisal in early FY 2011. However, the political and security situation remains highly fragile and poses significant risk to the outcomes . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Quality at entry is rated Unsatisfactory . The ICR reports that previous Bank efforts to rebuild the civil service had been unsuccessful and that "implementation experience revealed that the country was not politically ready for, or institutionally capable of, comprehensive reform efforts ." Even with this knowledge, the Bank team prepared an ambitious project and faced numerous obstacles inherent in such an operating environment . Measures to mitigate risks were inadequate . Supervision is rated Moderately Satisfactory . A Quality of Supervision Assessment (QSA) reported that the supervision efforts by the Bank were satisfactory ("intensive, opportunistic, focused on problem solving, and entailed a substantial amount of technical assistance and capacity building ") although it also noted the need for more realistic project ratings. The project benefited from task team continuity, with two task team leaders both located in the Country Office. The Bank team, recognizing that the original project was beyond the capacity of the Borrower, moved forward with a much needed restructuring to simplify the project and continue implementation despite the unstable and fragile operating environment . However, additional VTE activities were included in the revised project design despite the fact that Bank management had decided not to extend the VTE project beyond its closing date of December 2003, based on a review of the portfolio and "assessment of the fragility of the institutional environment ". The ICR reports that the Bank agreed to include the VTE activities because the incoming Minister had stressed the adverse impact of a contrary decision and because of “demonstrated improvements� in project implementation. at -Entry :Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government performance is rated Moderately Unsatisfactory . The Government demonstrated initial strong commitment to improving performance of the education sector, as reflected by the convening of the First National Conference on Education in 1996 and a National Committee created by the Minister of Education in 1999 to prepare the foundation for a comprehensive education sector strategy . However, following the appointment of a new Minister just after project effectiveness, government commitment waned . The ICR reports that government institutions were not functioning in the last three years of the project, in no small part due to the turmoil in the country (assassination of the Prime Minister in 2005, military conflict with Israel in 2006, Presidential and Parliamentary elections in 2009). Implementing Agency performance is rated Moderately Unsatisfactory . Original project activities were to be implemented by multiple agencies, including MNEYS, ECRD, and the Council for Development and Reconstruction (CDR). The ICR reports that the agencies could not agree on their respective roles and that MOUs to ensure cooperation and coordination were not signed despite repeated Bank follow -up. The project gained momentum after restructuring and a number of activities were implemented as planned, including those still within the responsibility of the ECRD . Under the auspices of the MEHE, the PMU evolved to become the ESDS and was institutionalized within MEHE by the close of the project . However, there were notable shortcomings in implementation . Procurement of VTE equipment did not take place, and the ICR reports on the "lack of readiness of the former VTE Project PMU Director to provide constructive support and share information." Also, IT activities were not completed as they required specific skills that were not easily developed by the PMU. Another implementing agency, CDR, was not able to address delays in school construction in a timely manner and did not give adequate priority to the project following the 2006 hostilities which required other major emergency reconstruction efforts . a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : The M&E framework for the restructured project was primarily focused on outputs, and did not adequately identify or track capacity -building outcomes. M&E Implementation : Initial M&E efforts were focused on monitoring implementation progress and reporting outputs . The ICR reports that this was deemed necessary at the time in order to build implementation momentum and facilitate coordination of sub -components. An international expert eventually joined the task team and produced a report to help evolve existing M&E systems towards capturing outcomes . However, PMU staff turnover and recruitment delays prevented progress on this front . M&E Utilization : The ICR reports that the M&E system was utilized to monitor implementation progress; however, as noted above, there was insufficient tracking of information to assess development outcomes . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : Although the ICR reports that safeguard compliance was satisfactory during implementation and that there were no significant environmental or social safeguard issues reported, it provides no information to support this assertion. The PAD identified the project as an environmental category "B" project due to the school construction activities (and relevant issues of sanitation and waste disposal ). Fiduciary : The ICR reports that there were no significant procurement or financial management problems other than those noted below. Both procurement and financial management procedures were simplified during restructuring, with two procurement post-reviews reporting effective assistance from the Bank . Two internal control issues were noted, including the inability of CDR to address some construction and goods procurement issues in a timely manner and the inability of the PMU to introduce a fixed -asset management system that could record equipment and furniture procured under the project in an institutional and sustainable manner . The ICR does not report on project audits – including whether they were presented in a timely manner or whether there were qualifications . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development Moderate Significant The political and security situation Outcome : remains highly fragile and poses significant risk to the outcomes . Bank Performance : Moderately Moderately Quality at entry is rated Unsatisfactory Satisfactory Unsatisfactory while Supervision is rated Moderately Satisfactory. Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Sufficient institutional capacity is a prerequisite for embarking on a major reform initiative . In addition to technical input, a political consensus -building process is critical to support a significant reform agenda. In a post-conflict setting, targeted reforms should be implemented on a step -by-step basis while building capacity for more sector-wide reforms. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: ICR Quality is rated Satisfactory . The ICR provides a thorough discussion of context to understand the project's performance. However, there was limited assessment of outcomes with regard to capacity building, aside from completion of outputs. Efficiency analysis is inadequate, and insufficient information is provided on compliance with environmental safeguards and fiduciary issues (there is no information on project audits ). a.Quality of ICR Rating : Satisfactory