ICRR 14571 Report Number : ICRR14571 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 01/29/2015 Country : Mongolia Project ID : P108776 Appraisal Actual Project Name : Mongolia: Education US$M ): Project Costs (US$M): 29.4 29.4 For All Fast Track Initiative (efa-fti) Catalytic Trust Fund L/C Number : Loan /Credit (US$M): Loan/ US$M ): 29.4 29.4 Sector Board : Education US$M): Cofinancing (US$M ): Cofinanciers : Board Approval Date : 03/16/2007 Closing Date : 12/31/2008 12/31/2013 Sector (s): Primary education (100%) Theme (s): Education for all (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Judyth L. Twigg Pia Schneider Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the 2009 Catalytic Fund Program Document (p. 4), the project’s objectives were “to: (i) improve access to and completion of basic education (primary and secondary education ), with particular attention to rural areas; (ii) improve the quality of basic education; (iii) expand access to pre-school education in rural areas; and (iv) improve project management.� b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project contained four components (as stated in the 2009 Program Document, p. 4): 1. Extending access (appraisal, US$ 6.23 million; actual, US$ 6.21 million) was to rehabilitate and improve service facilities in 30 rural schools and dormitories, provide furniture and equipment to schools and dormitories, and build a new general secondary school complex . 2. Improving teaching and learning environments (appraisal, US$ 9.39 million; actual, US$ 9.39 million) was to provide continuous in-service training, including teacher training (primary school teachers in teaching methodologies for math, language, and information /communication technologies (ICT), first grade teachers in methods to cope with intake of new six -year-olds, and dormitory teachers to provide better care to boarding students), supply of mobile ICT stations and electronic books for teachers, and rehabilitation and equipping of laboratories in teachers’ colleges in Ulaanbaatar and selected provinces . 3. Extending access to pre -school education in rural areas (appraisal, US$ 13.15 million; actual, US$ 13.16 million) was to provide mobile ger pre-schools and rehabilitate the kindergarten teachers ’ college in Ulaanbaatar. 4. Project management (appraisal, US$ 0.63 million; actual, US$ 0.64 million) was to support operational costs for the Project Coordination Unit (PCU) and technical audit of the Grant . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost: The project cost US$ 29.4 million, as planned. Actual costs by component were in line with that planned at appraisal. Financing: The project was funded by a US$ 29.4 million Grant from an EFA-FTI free-standing single-purpose Trust Fund. The grant added about 4% to the government’s annual expenditure for pre -school, primary and secondary education. According to the Program Document (p. 4), this was critical because the government budget mainly covered salaries with very little left for investments in infrastructure or quality improvements . Borrower Contribution: No Recipient contribution was planned or made . Dates: In December 2009, the project’s closing date was extended by two years, from December 31, 2008 to December 31, 2010, due to a delay in disbursement of the first tranche in 2009 and a delay in procurement of civil works (especially the construction of a new school complex in Baganuur ). In December 2010, the closing date was extended by another two years, from December 31, 2010 to December 31, 2012, to complete construction of the Baganuur school complex . In December 2012, the closing date was extended by another one year, from December 31, 2012 to December 31, 2013, to modify some output indicators. In particular, the number of new secondary schools to be constructed under the first component was increased to three . 3. Relevance of Objectives & Design: a. Relevance of Objectives: High . Despite strong economic growth, at appraisal there were significant Relevance of Objectives is rated High. urban-rural disparities in Mongolian education access and quality, revealed by a 2006 household survey showing a 7.3 percentage point gap in primary education completion rates between urban and rural students, and an even larger gap at the secondary level . Similar urban-rural gaps on national assessments revealed disparities in education quality . Most pre-primary children who were enrolled in early childhood education programs lived in urban areas, placing six -year-olds in rural areas at a disadvantage in readiness for primary schooling. Mongolia’s Education Sector Master Plan (ESMP, 2006-2015) stresses improvements in access and quality specifically in rural areas and for poor /vulnerable groups; the project was prepared to help deliver results explicitly linked to this plan. The Bank’s Country Assistance Strategy at the time of appraisal (2005-2008) stressed establishment of an accessible and high-quality education system that would nourish the intellectual development of Mongolian citizens while fostering economic growth . The Bank’s current Country Partnership Strategy (CPS, 2013-2017) contains a pillar for addressing vulnerabilities through improved access to services and better service delivery, safety net provision, and improved disaster risk management, with better delivery of education services (specifically early childhood education ) as explicitly desired outcomes (pp. 24-25). Another CPS desired outcome, falling under a pillar for economic growth and employment, specifically targets the creation of opportunity in rural areas (pp. 22-23). The project’s objectives are therefore highly relevant to country context and to Bank and Government strategy . b. Relevance of Design: Relevance of Design is r ated Modest . The project’s planned activities were explicitly linked to its objectives, with interventions under each component directly and plausibly connected to those objectives . Gender-specific activities and targeting were appropriately not included, as Mongolia is unique in having a “reverse gender gap,� with enrollment rates higher for girls than for boys . However, because of the EFA-FTI guidelines that required objectives and indicators to be expressed in terms of overall sector goals, the project design did not allow for a results chain that could demonstrate the attribution of progress specifically to project interventions . This was a common challenge for EFT-FTI projects of this generation : how to evaluate project accomplishments under global-level objectives and indicators while still giving projects credit for strong performance on their individual short-term goals (ICR, p. 10). This project had an opportunity to address the problem in 2009, but this was not done. 4. Achievement of Objectives (Efficacy): Improve access to and completion of basic education (primary and secondary education ), with particular attention to rural areas, is rated Modest . Attribution of observed results to project interventions is challenging, as the indicators are at the national level and do not differentiate between rural and urban results . Outputs : Three new schools were built, including the Baganuur Secondary School Complex covering grades 1-11 and two smaller schools. Baganuur is an urban area, but the two smaller schools are in rural regions . This did not meet the target of three new rural schools . 55 rural school buildings and 23 rural dormitories were rehabilitated, essentially meeting the target of 57 and 23, respectively. 30 school toilets and hand washing places were remodeled, meeting the target of 30. Teaching kits and learning materials, furniture, and dormitory and kitchen facilities were provided, meeting quantitative targets for all items. Furniture and equipment was supplied to six schools for children with special needs, meeting the target of six . Outcomes : The gross enrollment ratio in primary education increased from 96.4% in 2005 to 102.7% in 2014, exceeding the target of 96.2%. The gross enrollment ratio in secondary education increased from 94.2% in 2005 to 98.0% in 2014, exceeding the target of 95.5%. No information is provided specifically on rural areas . The completion rate in primary education increased from 98% in 2005 to 100% in 2014, meeting the target of 99%. The completion rate in secondary education increased from 97% in 2005 to 99% in 2014, meeting the target of 99%. No information is provided specifically for rural areas . Improve the quality of basic education is rated Modest . Outputs : 2,200 teachers (out of a target of 3,000) were trained in teaching methodologies in math, language, and ICT in 2007, and 1,962 (meeting the target of 1,962) in 2009. Training meeting targets for each year took place for first-grade teachers on the intake of new six -year-olds: 1,449 (target 1,432) in 2008; 1,602 (target 1,602) in 2009; and 1,778 (target 1,778) in 2010. 590 dormitory teachers were trained in provision of care to boarding students, surpassing the target of 500. 580 social workers were trained to provide services to six -year-olds, meeting the target of 580. In-service training in teaching methods in five subjects was provided to 6,826 teachers in secondary schools, first grade teachers, and dormitory teachers, meeting the target . As a result of project interventions, 4,162 additional primary teachers were qualified, meeting the target . 550 mobile ICT stations were provided for teachers, meeting the target of 580. 375 new kits of furniture and equipment were supplied, meeting the target . Electronic books were provided for 6,000 teachers, doubling the target of 3,000. Seven laboratories were rehabilitated and equipped in the Ulaanbaatar teachers ’ college, meeting the target of 7. 27 laboratories were similarly rehabilitated and equipped in three other regions, meeting the target of 27. Outcomes : As the ICR points out (p. 12), evidence of project impact is not clear, as there are no data on student learning outcomes or teacher performance . ial . The lack of an urban-rural Expand access to pre -school education in rural areas is rated Substant ial. breakdown for enrollment data to substantiate actual increases in rural pre -school enrollment is a shortcoming, but given that the outputs were exclusively in rural areas, it can be plausibly argued that at least some of the enrollment increases were in rural areas and can be attributed to the project's interventions . Outputs : 171 mobile ger pre-schools (specifically targeted at children of rural herders ) were deployed, meeting the target. One kindergarten teachers’ college was rehabilitated, meeting the target . All newly deployed ger kindergartens were supplied with necessary furniture, equipment, and learning materials, meeting the target . Outcomes : The gross enrollment ratio in pre-school education increased from 53.8% in 2005 to 77.6% in 2013, exceeding the target of 72.7%. Improve project management is rated Substantial . The training the Bank provided in procurement and financial management resulted in substantial improvement in these areas (see Section 11). In addition, targets were met for the establishment of the PIU and of an Educational Donors ’ Consultative Mechanism (EDCM) to facilitate donor coordination. According to the ICR (p. 14), the EDCM acted as the project's Steering Committee and was active in reviewing and endorsing the project's annual technical and work plans; "the mechanism played an important role in the project's success ." 5. Efficiency: Efficiency is rated Substantial . Neither the 2009 Program Document nor the ICR conducted a traditional economic analysis, instead providing a sector public expenditure analysis . However, there is evidence of substantial efficiency in terms of the investment of project resources . The project deliberately financed activities to cover parts of the Education Sector Master Plan that were not funded by the Government or other donors (ICR, p. 5). Donor collaboration and division of labor enhanced project efficiency (ICR, p. 14); for example, the project focused on building and renovating rural classrooms while the Japan International Cooperation Agency did the same in urban areas. Administrative costs were only 2% of total project costs and did not exceed appraised costs despite several project extensions . Unit cost analysis shows that project costs for grade 1 teacher training, facilities costs per child, desk /chair costs, and annual energy consumption per square meter in new schools were markedly lower than in comparable projects (ICR, pp. 22-23). These cost savings in many cases resulted from deliberate decisions made during project implementation, including conducting training “in- house� using MES trainers rather than relying on contractors, and to procurements that took advantage of economies of scale. Delays in contracting for the Baganuur school complex, totaling four years and requiring two project extensions, reduced project efficiency; while most of these delays were due to project activities and Bank procedures being new to the Ministry and the PCU, some were outside the control of the project (spiraling construction costs and heavy rains) (ICR, p 12). ERR)/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of Objectives is rated High, as the project ’s objectives were deliberately and precisely aligned with long-term sector strategy. Relevance of Design is rated Modest, as the lack of rural -urban breakdown made it difficult to assess and measure achievement of two of the project's objectives . Achievement of the objective to improve access and completion of primary education particularly in rural areas is rated Modest in the absence of outcome data specifically for rural areas and therefore difficulty attributing observed outcomes to project interventions. No evidence is presented related to quality of basic education, leaving that objective also Modestly achieved. Expansion of access to pre -school education in rural areas is rated Substantial, as the project's pre-school interventions were entirely targeted at children of rural herders . Improvements in project management were also Substantial, as the Bank's training in procurement and financial management resulted in significant increases in effectiveness and efficiency in these areas over the project's lifetime . Efficiency is rated Substantial, with important evidence supporting the argument that project resources were invested in a cost-effective manner. Taken together, these ratings are indicative of moderate shortcomings in the project ’s preparation and implementation, and therefore an Outcome rating of Moderately Satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: At project closing, Mongolia had graduated from EFA -FTI, as it had been reclassified as a middle -income country. As a transition gesture, however, a “graduation grant� of US$ 10 million was awarded in 2012 to continue work in early childhood education; this was seen to be of particular importance because of an abrupt upturn in the pre-school population. This new grant is a direct extension of the project, covering 100 additional mobile ger-kindergartens and 17 fixed pre-school buildings in urban and peri -urban areas. A new Bank-funded project to improve the quality of basic education (US$ 30 million, 2014-2019) has just launched and will expand on project interventions related to in -service teacher education . Innovations begun under the project, such as a Canadian, wood-frame construction design for kindergarten buildings that enhances energy efficiency and cost-effectiveness, are now being used country -wide. However, site visits have indicated that some project inputs, such as dormitory equipment /upgrades and first-grade classroom furniture and toys, are showing signs of deterioration; it is not clear that the Government plans to make funds available for maintenance and replacement of these inputs where required . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project was able to move forward after a very short preparation period (less than three months) because of an extensive prior process that aligned project goals and activities with those of the Government ’s Education Sector Master Plan. However, it was a challenge to fulfill the terms that existed for EFA -FTI grants at that time, especially the need to plan for and receive financing for just one year at a time even though the identified gaps would take significantly longer to fill, and the need to use global outcome indicators even though project interventions were more limited in scope . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The Bank team constructively engaged with the Government and other donors . In the early years of the project when procurement and financial management were below Bank standards, the team provided appropriate capacity building and mentoring (ICR, p. 16). Once it was decided to generate an overall program document in 2009, the Bank effectively expanded documentation to allow for more integrated planning of project activities and improvements in procurement and financial management . However, the Bank team did not supplement the results framework and global indicators preferred by EFA -FTI with indicators more specific to project interventions . Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government, in particular the Ministry of Education and Science (MES), was a strong partner, having paved the way for EFA-FTI support through the preparation of a strong poverty reduction strategy and education sector strategic plan . Government commitment was further evidenced through its constructive relationship with the donor community and its mandate to the various branches of the MES that they cooperate fully with the Project Implementation Unit (PIU). Ministry ownership was evidenced through, among other things, incorporation of grant funds into its annual budget . Government Performance Rating Satisfactory b. Implementing Agency Performance: A project coordination unit was established within a pre -existing Asian Development Bank PIU, headed by a Deputy Project Implementation Coordinator . This experienced platform facilitated rapid movement forward . Collaboration with relevant departments in the MES was smooth . There were numerous shortcomings with financial management and procurement in the project ’s early years (see Section 11), but most of these were overcome by 2009-2010. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The initial three annual grant agreements (2007, 2008, and 2009) specified key performance indicators for those years, but overall project performance indicators were specified in the 2009 Program Document. At first, the project followed EFA-FTI guidelines, using higher-level sector goals as outcome indicators : national-level primary and secondary enrollment and completion rates, net intake into grade one, and transition rates from primary to secondary education . By the middle of the project’s second year, however, it became clear that it would be difficult to assess project attribution to these highly aggregated indicators; the scale of the interventions was not likely to have made much of an impact on national -level trends. When the retrofitted project design was formalized in 2009, key performance indicators were broken down by gender but not by urban-rural residence, making it impossible still to determine whether urban -rural gaps were being addressed, and still not resolving the issue of attribution of observed outcomes to project interventions and of assessing achievement of the objective to increase access and completion rates with particular attention to rural areas . In addition, the target for one indicator (gross enrollment rate in primary education ) had a target lower than the baseline. b. M&E Implementation: Output data were collected and tracked, and global data were collected and reported on outcome indicators . Project attribution, however, is difficult, a common challenge with EFA -FTI indicators of this time period. c. M&E Utilization: According to the ICR (p. 8), output data were used to inform action on a variety of issues : output quality and quantity, budgetary issues, logistics, etc . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project triggered Environmental Assessment (OP/BP 4.01). According to the ICR, due to the fact that EFA-FTI program was approved by the Catalytic Fund Strategy Committee and not the Bank ’s Board of Directors, an Environmental Assessment was not required and not conducted at the time of preparation . The ICR (p. 8) states that the task team was attentive to Mongolian law governing school safety, standards, and environmental management, and all civil works complied with these laws . The 2009 Program Document included classified the project as Category C for its modest environmental impact . An Environmental Assessment was undertaken prior to the construction of the Baganuur School complex and did not raise concerns. The 2009 Program Document also included a social assessment, noting the existence of ethnic minority groups, particularly Kazakhs in rural areas in the far Western district . The project included this district as one of its intervention sites, providing teacher training and rehabilitation to schools serving Kazakh children . b. Fiduciary Compliance: Early in the project period there were problems with financial management, including comparison of actual and budgeted expenses, establishment of a proper chart of accounts, financial management reporting, documenting the receipt of goods, and tagging assets . By late 2009 most of these problems had been overcome, and the financial management rating was Satisfactory from 2010-2013. All financial audits were on time and unqualified (ICR, p. 8). The project’s ambitious agenda required the awarding of many contracts within a short period of time as the project launched. Because of the PIU’s inexperience with Bank procurement procedures, there were early difficulties. The Bank responded during the project ’s first year with procurement training and additional support, such that considerable improvement was observed by the third year . However, there were delays and challenges with selection of a contractor for the Baganuur school complex that ultimately required a second tendering and two project extensions; the school ’s innovative and eco-friendly design added a level of complexity to the activity that contributed to the delays . The PIU completed and fully furnished the school by project closing. c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (pp. 17-18) offers the following useful lessons : Low-income countries operating under severe financial constraints can benefit greatly from “bridge funding� that funds capital expenditure and other development expenses until they can begin to cover such expenditures on their own. This lesson may ring particularly true for countries, like Mongolia, undergoing economic transition. Even when financed through an instrument whose guidelines require use of broad, global objectives and indicators, projects should be careful to supplement those requirements with project -specific indicators so that attribution of results can be plausibly established . In particular, the indicators should fit the scale of the intervention. A wide variety of constraints and variables, including institutional capacity, weather, and construction costs, can impact construction efforts . Projects need to anticipate such variables and supervise closely, retaining a civil works engineer as part of supervision teams where appropriate . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is solid in its explanations of several complex issues, particularly the evolution of the project ’s objectives and indicators in its early years and the challenges of attribution stemming from the requirement to adhere to EFA-FTI guidelines. However, it is almost too concise, putting several discussions in annexes that could have considerably enhanced the narrative of the main text (this applies in particular to evidence supporting the Efficacy and Efficiency ratings ). a.Quality of ICR Rating : Satisfactory