Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) Report Number : ICRR0020670 1. Project Data Project ID Project Name P129381 GH-Global Partnership for Ed. (FY12) Country Practice Area(Lead) Ghana Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-13140 31-Oct-2015 75,500,000.00 Bank Approval Date Closing Date (Actual) 10-Oct-2012 31-Aug-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 75,500,000.00 75,500,000.00 Revised Commitment 75,500,000.00 75,500,000.00 Actual 75,500,000.00 75,500,000.00 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Estelle Rosine Joy Behrens IEGHC (Unit 2) Raimondo 2. Project Objectives and Components a. Objectives According to the Grant Agreement (page 5), and the Project Appraisal Document (PAD, page 8), the project objective was as follows: •To improve the planning, monitoring, and delivery of basic education services in deprived districts of the Recipient’s territory. Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components 1. Sub-Grants to Deprived Districts to Support Key Education Objectives (Appraisal: US$ 42.36 million; Actual: US$ 48.41 million): This component provided support for non-salary expenditures to 75 deprived districts, according to each district's Annual Program of Work. Support was in the form of grants, averaging US$5 per student/year, and were to be used towards in-service teacher training, upgrading of teacher qualifications through the Unqualified Teacher Diploma in Basic Education (UTDBE) program; and instructional materials. Project support also included capacity building for education staff and administrative staff; as well as evaluative work. Grant funds were to be channeled through decentralized mechanisms, to enable local education agencies to take direct responsibility for allocating and monitoring use of funds. 2. School Sub-Grants (Appraisal: US$ 22.06 million; Actual: US$ 20.38 million): This component provided support for non-salary expenditures to all basic education schools in deprived districts, according to each school's School Performance Improvement Plans (SPIP). Support was in the form of grants, averaging US$ 4 per student/year, and were to be used towards improving teaching and learning such as through instructional materials, furnishings and toilets, information technology, and conducting of School Performance Appraisal Meetings. Project support also included capacity building for school administrators and School Management Committees to develop and monitor SPIPs. Funds were to be channeled to schools, to generate demand for improved services and to improve social accountability in the use of resources. 3. Project Management and Institutional Strengthening (Appraisal: US$ 4.58 million; Actual: US$ 6.71 million): This component supported project management and M&E activities at central, regional and district levels. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost • The appraised and actual project cost was US$ 75.5 million. • There were significant cost savings due to favorable exchange rate gains; the additional funds were allocated to Component 1 for district grants. Financing • The project was financed entirely by a US$ 75.5 million grant from the Global Program on Education. Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) Borrower contribution • There was no planned Borrower contribution. Dates • May 2013: The Grant Agreement was amended to add "works" as an eligible category for District grants. • August 2015: Project funds were reallocated among categories, including from "unallocated" to Component 1 for District grants. The project closing date was extended from October 2015 to August 2016, in order to allow for completion of activities (namely, teacher trainees in completing UTDBE diploma program and the impact evaluation). 3. Relevance of Objectives & Design a. Relevance of Objectives The country of Ghana is approaching universal access to the first two portions of basic education, kindergarten and primary school (99% and 96% gross enrollment rate, respectively (2012)), owing to an increased supply of schooling and the elimination of school fees by the government. Enrollment rate for the third portion, junior high school, is lower at 81%, and therefore attaining universal access to all three portions of basic education (kindergarten, primary, and lower secondary) is a continued priority. Efforts include addressing the wide regional disparities between the north and south on education outcomes, which are associated with higher poverty rates in the north and linked to inequality of public funding for education (60% of primary school students are in the north but per child expenditure falls within the bottom third of the country). Improving the quality of education is also a priority, given low performance on proficiency exams and assessments, as well as addressing inequities in access and outcomes. The project objective is highly consistent with both country and Bank priorities. Ghana's Education Sector Plan for 2010-2020 highlighted the goals of providing equitable access to quality basic education, improving the quality of teaching and learning, and improving the management of education service delivery. The Bank's Country Partnership Strategy for FY 2013-2016 prioritizes targeting improvements in services to deprived areas. Rating High b. Relevance of Design The project design was strongly aligned with the country's decentralization program, in which local District Education Departments oversee the operation of basic schools. Project funding was provided through Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) decentralized mechanisms (based on districts' and schools' individual work plans), while accountability was increasingly localized through school management committees and school report cards. These various decentralized mechanisms, already tested and in place at the time of project appraisal, were to be further strengthened through project support. In addition, the project districts were categorized by the Ministry of Education as "deprived" districts (poorer education outcomes and lower share of education spending) and weighted according to poverty levels, thus ensuring effective targeting mechanisms. The process of preparing and supervising district and school annual work plans was likely to improve planning and monitoring capacity, by establishing processes to deliver specific outputs and to assess school performance. The menu of activities funded by the district and school grants was substantially likely to lead to improved delivery of basic education. Given the limited public funding available for non-salary spending needs (97% of the public education budget is allocated to salaries), the eligible activities focused on non-salary expenditures such as teacher training, learning materials, and minor rehabilitation for school facilities. These are critical educational inputs that would likely improve the teaching and learning environment. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To improve planning of basic education services in deprived districts Rationale The project focused on “basic” education (kindergarten, primary, and junior high school) in "deprived" districts (comprising 75 out of a total 216 districts in the country). Around 7,000 schools were expected to receive project support. Outputs • Preparation of Annual Work Programs by 100% of participating districts. • Preparation of School Performance Improvement Plans (SPIP) in 100% of participating schools (target: 75%). Impact evaluation findings confirmed that 80% of Head Teachers were able to produce a completed SPIP in the official template upon request. • Training to Head Teachers on preparation of School Performance Improvement Plans. • Training of 2,288 regional and district education staff on financial management and data collection (target: 500). However, the ICR (page 11) notes high staff turnover resulting in schools and districts losing institutional knowledge on learned procedures and guidelines. Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) Outcomes • 100% of participating districts disbursed 75% or more of their District grants, as planned in their Annual Programs of Work (APW). This surpassed the target of 80% of participating districts. APW reports from all districts are collated and reported in annual financial analysis reports. • The execution rate for school grants steadily increased over the project period, from 72% in 2012-13, to 88% in 2014-2015, and to 92% at project closing. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To improve monitoring of basic education services in deprived districts Rationale Outputs • Training of 5,993 Head Teachers and 1,094 Circuit Supervisors on use of the School Report Cards, for a total of 7,087 staff trained (target: 6,930 total staff). • Conducting of school visits twice per year at 100% of project schools by Circuit Supervisors. In comparison to all schools in the country, Circuit Supervisors visited those schools more than once per year in only 77% of schools. • Preparation of annual M&E reports, incorporating data from all districts, by regional education directorates. Data was also incorporated into the annual Education Sector Performance Reports presented at annual sector meetings. Outcomes • 98% of participating schools had updated School Report Cards (with 93% of schools displaying their report cards on notice boards). This surpassed the target of 75% of participating schools. Feedback from school report cards were to inform school improvement plans and the use of school grants. The impact evaluation reported that 97% of district education directorates agreed or strongly agreed that data from school report cards “had helped them in improving girls’ school attendance by an increased focus on gender in their APWs." The ICR (page 14) notes that improvements observed in teacher/student Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) attendance rates or improvements in access or retention rates were tracked through School Report Cards; however, this specific data are not provided. • A fiduciary assessment conducted in a sample of 105 schools found that school sub-projects were inspected by the district office and that expenditure of grants was also tracked by the district office. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective To improve delivery of basic education services in deprived districts Rationale Outputs • Training of 6,480 teachers through the Unqualified Teachers Diploma in Basic Education program (target: 5,000). However, the ICR (page 11) reports significant attrition, as the initial enrollment was 7,027 participants. • In-service training of 42,541 teachers (target: 30,000) in the core areas of numeracy, literacy and science. 97.8% of all project schools participated in the in-service training. The ICR notes that data tracked yearly participants in the program without tracking whether the same teacher was trained more than once in different areas. As a subset, 2015 data was reported as follows: 28,056 teachers participated in the math course, 14,485 in kindergarten course. The % of trained teachers in project districts increased 10-18% from 2012/13 to 2015/16, compared to a 3-5% increase in non-project districts. • See also outputs reported above which contributed to improved planning, monitoring and financing of non-salary expenditures, all of which are important inputs to the delivery of basic education. Outcomes • The % of teachers who obtained a satisfactory rating or higher on Lesson Observation Sheets during training increased from 36.0% in 2010 to 75.7% in 2016. This surpassed the target of 50.0%. According to the project team, the observations were conducted during actual school teaching sessions (teachers enrolled in these training programs remained in their teaching positions during the school year but worked towards upgraded qualifications through distance education courses) at the end of enrollees' first year, at the start of their third year, and during their fourth year. • While not clearly attributable to the project interventions due to the short project period, the proportion of Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) P3 students that achieved proficiency in English on National Education Assessment tests increased from 12.4% in 2011 to 16% in 2013. This surpassed the target of 15.0%. Scores in math increased from 9% to 13% over the same period, surpassing the target of 11%. Results from later years were not available as the most recently administered test (2016) was administered to P4 students. The ICR also reports early grade reading/math assessments (EGRA/EGMA) that also reflected statistically significant improved scores between 2013 and 2015. • The ICR (page 14) notes that indicators on student/teacher attendance rates and retention rates were tracked through School Report Cards and that 'improvements were observed throughout the project period"; however, no specific data are reported in the ICR. The beneficiary survey findings reported in the ICR (Annex 5) include that the project "reduced the financial burden on parents of having to pay additional fees for miscellaneous expenditure to schools"; again, no specific data are provided. However, • Observations made during field visits included the following: some cases of poor record keeping by Head Teachers, school management committees not operating as effectively as they should, overcrowding due to increased attendance, and difficulties monitoring remote schools. The ICR page 13) reports that mitigation measures were undertaken to address the above issues. • Also, there were ongoing parallel efforts in teacher training managed by the Colleges of Education, which may have also contributed to the improved test scores, as well as other interventions such as school feeding programs which may also have increased access rates. Rating Substantial PHREVDELTBL PHREVISEDTBL 5. Efficiency The appraisal analysis (PAD, Annex 7) highlighted project design elements that likely led to efficient use of project resources. These included targeting of deprived districts, use of existing mechanisms and institutional arrangements, and the use of in-service rather than pre-service teacher training. The appraisal also included a fiscal analysis that showed that the school/district grant program would have only limited fiscal impact on the government. There was no calculation of economic rate of return. The ICR analysis (Annex 3) confirmed the effective implementation of these particular project design elements. Additionally, there was a high execution rate for grants at both school and district levels. Cost savings due to currency depreciation and non-use of contingency funds were used to increase allocations for district grants and in-service teacher training. Project implementation was completed in three and a half years, compared to original project period of three years. Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) The ICR analysis also included data on comparative cost effectiveness of two project interventions: i) construction of toilets and urinals and ii) in-service training. For construction of toilets and urinals, the unit cost of a four-toilet block under the project was GH cedi 32,801, which compared to a unit cost of GH cedi 40,100 for government construction and GH cedi 36,083 for UNICEF construction (albeit the latter has higher standards). The unit cost of a two-urinal block under the project was GH cedi 5,665, which compares to a unit cost of GH cedi 5,950 for government construction and GH cedi 6,368 for UNICEF construction. For in- service teacher training, delivery of training modules in project districts cost GH cedi 39.48 per beneficiary in mathematics, compared to GH cedi 45.8 per beneficiary in non-project districts. In science and literacy, the training modules in project districts cost GH cedi 34.41 and 37.57 per beneficiary, compared to GH cedi 48.8 and 42.7 per beneficiary in non-project districts. There was no economic rate of return analysis in the ICR. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the project objective is High and relevance of the project design is Substantial. Achievement of the objective to improve planning, monitoring and delivery of basic education in deprived districts is rated Substantial for the sub-objective of improved planning, Substantial for the sub-objective of improved monitoring, and Substantial for the sub-objective of improved delivery of basic education, due to evidence of outcomes that met targets. Efficiency in the use of project resources is rated Substantial. Therefore, the overall outcome is rated Satisfactory. a. Outcome Rating Satisfactory Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) 7. Rationale for Risk to Development Outcome Rating Capacity improvements are likely to be sustained, as capacity development activities were implemented largely within existing government systems. Teacher training, utilization and analyses of Annual Programs of Work and School Report Cards will continue to be implemented by the government. School Management Committees are likely to remain in operation, with overall improved functionality. However, due to overall fiscal constraints, it is not clear whether the government will be able to continue funding district and school grants. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry Project objectives were strongly consistent with government sector priorities and existing institutional arrangements in the education sector. The key project interventions were based on pilots that had already been tested in the country, thereby contributing to likelihood of achievement. The project design took into account other experience from prior projects, including that monitoring systems needed to be strengthened at the decentralized levels and that transparent and objective criteria for targeting deprived areas needed to be developed (in part to minimize political interference). Project interventions also took into account ongoing donor interventions (DfID scholarship program for lower secondary girls and USAID early grade literacy and numeracy program). The risk assessment was comprehensive, rating overall risk as Substantial, due to potential staff turnover and weak capacity at the decentralized level; however, effective mitigation measures, such as extensive stakeholder consultations and a detailed project implementation manual, were included in the project design. The M&E design was overall strong, with detailed M&E data collection arrangements but some minor shortcomings in selection of indicators. Quality-at-Entry Rating Satisfactory b. Quality of supervision The Bank team conducted regular supervision missions with joint field visits with other donors, which contributed to effective donor coordination. Implementation challenges were quickly identified during supervision missions, aided by the consistency of a single task team leader throughout the implementation period and an in-country core team. The Bank team launched two in-depth fiduciary reviews and also identified effective mitigation measures such as continual training, simplified school level templates, and increased support from Circuit Supervisors. As noted in the ICR (page 32), the Bank team was commended in the Borrower's project completion report for securing all approvals (i.e. for no-objections, work plans, budgets) with minimal delays and for prompting the Government to submit relevant documentation at the appropriate Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) time. One shortcoming noted was with regards to M&E, where "additional attention to [results framework] calculations and reporting during supervision missions may have prevented some challenges in reporting." Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. Assessment of Borrower Performance a. Government Performance Government commitment and performance was strong, aided by the consistency of a single Minister of Education during the implementation period. The project's use of existing institutional arrangements contributed to increased ownership by government agencies. However, there were delays in disbursement of the government's capitation grants throughout the project period, which may have affected the impact of project school grants as supplemental funding. Government Performance Rating Satisfactory b. Implementing Agency Performance The Ghana Education Service (GES), under the oversight of the Ministry of Education, was responsible for overall project implementation. Almost all project interventions were carried out as planned and on time, with the exception of the impact evaluation which was significantly delayed. Extensive monitoring and data collection was carried out and used to inform project decisions. The ICR (page 34) notes that the GES "is commended for quarterly meetings with district officers to check and discuss project activities and status report of the districts, as well as the high quality project status reporting by the Ministry prior to implementation support missions." Fiduciary performance was overall satisfactory, with the exception of the delayed procurement of the contractor for the impact evaluation, and compliance with legal covenants and safeguards was also satisfactory. Implementing Agency Performance Rating Satisfactory Overall Borrower Performance Rating Satisfactory 10. M&E Design, Implementation, & Utilization Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) a. M&E Design The M&E framework was generally sound, with measurable indicators and realistic targets. M&E arrangements are notable for including multiple evaluative measures, such as impact evaluations, regular analyses of district Annual Programs of Work and School Report Cards, as well as for including capacity building measures to ensure ability to monitor activities. One key project indicator - P3 students achieving proficiency in math and English - was ambitious given the short time frame of the project as would not have been directly attributable to project interventions. As suggested in the ICR (page 14), other indicators of improved quality such as teacher/student attendance rate or improved retention rate may have been more appropriate. b. M&E Implementation Extensive M&E activities were largely carried out as planned. In addition to the planned activities referenced above, project implementation also included regular supervision visits to schools, internal and external audits, and two fiduciary reviews. M&E templates to be used at the school level were simplified to ensure utilization. There were some shortcomings in data collection, for example: collection and validation of decentralized data on teacher training from over 7,000 schools was difficult and could have been improved with computerization of reporting; the number of teachers trained through in-service program was reported as the cumulative number of trainings each year rather than the actual number of teachers participating; and the endline value for P3 students proficiency tests could not be calculated in the final year of the project as the 2016 exam was administered to P4 students instead. The two planned impact evaluations on the Unqualified Teachers Diploma in Basic Education program and the district/school grants program were carried out. However, the latter was significant delayed due to difficulties in selecting the contractor and also difficulties in redesigning the evaluation as implementation had already started - consequently, the final evaluation was limited in scope (lacked control group and baseline data) and quality of evidence. c. M&E Utilization The ICR (page 15) reports that the regular analysis of district Annual Programs of Work was "the first time the central Ministry had a clear sense of what activities (and their cost) were being implemented by schools using their capitation grants" and also enabled a "feedback loop between the central Ministry, districts, and schools." The ICR also reports that the impact evaluations provided robust data on management at the district and school levels, parent and community involvement, teacher behavior in the classroom, UTDBE cost-effectiveness, student learning outcomes, and access and completion rates at schools. M&E Quality Rating Substantial 11. Other Issues a. Safeguards Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) The project was classified as a Category "B" project due to the potential minor rehabilitation of school buildings. The safeguard policy on Environmental Assessment (OP/BP 4.01) was triggered. An Environmental and Social Management Framework was developed, and the project implementation manual included a list of negative items that could not be funded through the district/school grants to prevent triggering additional safeguards. An Environmental Management Plan was not initially developed, although a checklist was subsequently developed during the project period to aid in supervision. Safeguard reviews were conducted by the Bank team in May 2014 and March 2016. Although there were initial delays in undertaking environmental and social screening of sub-projects, the ICR (page 16) reports that all minor works were eventually screened and well-documented and did not trigger additional safeguards. b. Fiduciary Compliance Financial management: The ICR (page 17) confirms that the client has complied with all legal covenants related to financial management arrangements. Audits were submitted in a timely manner and there were no qualifications. Interim financial reports were also submitted regularly. Two fiduciary reviews were conducted (December 2014 and August 2016) and verified overall good performance based on a sample of 35 districts and at least three schools per district. The reviews noted mainstreaming of financial management arrangements with existing government processes, complemented by strict procedures and processes laid out in the project implementation manual. Several minor shortcomings were noted as follows: changes in accounting personnel at the district level, delays in making payment for district claims, and poor filing systems in some districts and schools. Procurement: Procurement performance was overall satisfactory, with the exception of delays in collating district level procurement plans and the aforementioned delay in procuring the contractor for the impact evaluation. An in-depth procurement audit was used to overcome some shortcomings in reporting at the decentralized levels. c. Unintended impacts (Positive or Negative) An unintended positive impact: The district/school grant program generated some interest among NGO and civil society organizations, who independently tracked grant activities and produced reviews (2013, 2015) that verified grant allocations, utilization, and community satisfaction. d. Other --- 12. Ratings Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk to Development Modest Modest --- Outcome Bank Performance Satisfactory Satisfactory --- Borrower Performance Satisfactory Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Lessons drawn from the ICR: • Project design built on tested pilot interventions, along with capacity building support, contributed significantly to smooth and timely uptake of project interventions. In the case of this project, the decentralized institutional arrangements, grants program, and unqualified teacher training program had all been recently piloted and lessons learned were incorporated in the project design. Continued capacity support to local government and school staff was also included. • Transparent and objective targeting criteria ensured appropriate political support and effective implementation. In the case of this project, clearly defined criteria based on education and poverty indicators ensured that that funds were targeted to the districts most in need, and limited political capture of benefits. • The unqualified teacher training program appears to be a cost-effective alternative to the more formal diploma program. In the case of this project, data demonstrated significant lower costs. including in underserved areas, alongside similar performance ratings compared to the more conventionally trained teachers. However, while the participants in the unqualified diploma program appear more likely to remain in the underserved areas (as they are more likely to speak the local language and are from the local community), there is also a higher attrition rate in completing the training. 14. Assessment Recommended? Yes Please explain With numerous processes put in place by the project (decentralized funding mechanisms, school report cards, Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Global Partnership for Ed. (FY12) (P129381) district work programs), it would be informative to learn whether educational outcomes have improved as a result; for example, teacher attendance rates, school enrollment rates, school completion rates. 15. Comments on Quality of ICR Despite having a more process-oriented PDO, the ICR was results-oriented and provided a satisfactory level of evidence and data to verify impact. For example, the ICR reported the execution rate of budgets (rather than only the number of budgets prepared or submitted); a comparison of the frequency of supervision visits between project schools and non-project schools (rather than only the number of supervision visits conducted in project schools); evidence of data informing decision-making by the government (rather than only the publication of data in government reports); and the proportion of School Report Cards actually displayed on bulletin boards (rather than only the number of schools that completed School Report Cards). The ICR also provides extensive details on safeguards, M&E and financial management to verify performance. a. Quality of ICR Rating Substantial Page 14 of 14