Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) Report Number : ICRR0020315 1. Project Data Project ID Project Name P100657 SOCIAL INCLUSION PROJECT Country Practice Area(Lead) Bulgaria Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-76120 31-Oct-2013 66,130,316.00 Bank Approval Date Closing Date (Actual) 04-Nov-2008 31-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 59,000,000.00 0.00 Revised Commitment 46,302,674.90 0.00 Actual 30,675,220.59 0.00 Prepared by Reviewed by ICR Review Coordinator Group Gisela M. Garcia Soniya Carvalho Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives The PDO of the Bulgaria social inclusion project (SIP) is to promote social inclusion through increasing the school readiness of children below the age of 7, targeting low-income and marginalized families, including children with a disability and other special needs (Financing Agreement p.4). There is no difference in the wording between the PAD and the Financing Agreement. The PAD indicates that the SIP would finance the pilot phase of a school readiness program that integrates municipal social, education and health services projects in a select group of pilot municipalities for two years. The European Social Fund (ESF) would finance the roll-out across the rest of the country after an evaluation Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) of the impact of its components (PAD p.4-6). b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Component I: Integrated social and childcare services (Appraisal cost EURO 37.39m, Actual cost EURO 22.87m). This component funded grants to eligible municipalities to cover a menu of services and infrastructure for the provision of integrated social and childcare services for marginalized groups with young children (aged 0-6) or with children with a disability. The menu of services included integrated parenting programs, individualized counseling, and mobile community outreach that covered pre- and post- natal parenting skills and orientation on available health and social services. The menu also included different measures to promote access to child care for low-income children or children with special needs (fee reduction, creation of family services, transport, access to health care). Child care services included formal kindergarten and preschool services as well as alternative child care services. This component also funded infrastructure rehabilitation and construction for municipal kindergarten or childcare service buildings (not to exceed 50 percent of the overall loan volume) as well as educational materials and training for service providers. Component II: Capacity-building (Appraisal costs EURO 2.51m, Actual costs EURO 0.67m). This component funded capacity-building activities at central and municipal level in inter-agency/cross-sectoral cooperation, sub-contracting of services to and cooperation with NGOs and accessing European Funds. The component also funded project auditing and construction works inspection for municipal sub-projects. This component was also originally intended to contribute to fund the cost of the impact evaluation and the promotion of capacity building around it, but the impact evaluation was cancelled. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Actual total project costs were EURO 23.59m (59% of the EURO 40m estimated at appraisal); borrower contribution (EURO 2.47m) was only 2.55% of the estimated amount at appraisal (EURO 96.73m). Borrower contribution was severely limited by the ceiling expenditure imposed by the government in response to the financial crisis. The total project cost shown above is higher than actuals. The project was restructured five times (all level 2). The first restructuring (May 2010) reflected changes in the implementing agency within the same ministry. The second restructuring (May 2011) reduced the time commitment of municipalities to preserve services after project completion (from 10 to 5 years in the case of infrastructure subprojects and from 5 to 3 years in the case of subprojects with services only) and Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) allowed advance payments to municipalities in specific cases. The third restructuring (Dec 2012) revised the project’s results framework changing key performance indicators (see Section 10) and reallocated some consultancy and training funds to grants. The fourth restructuring (Sept 2013) scaled down and cancelled project activities and its corresponding allocations responding to government budget restrictions. The fifth and last restructuring (Sept 2015) extended the project closing date for three months. 3. Relevance of Objectives & Design a. Relevance of Objectives The Project was relevant to the government's social inclusion agenda, which aimed to address important inequalities in early education (lower educational attendance in early childhood among children from poor backgrounds with resulting lower educational attainment overall and high dropout rates). The PDO was consistent with several strategy documents: (i) the 2005 Joint Inclusion Memorandum with the European Commission; (ii) Strategies for Social Protection and Social Inclusion 2006-2008; (iii) the National Program for the Development of School Education and Preschool Education and Preparation 2006-2015, (iv) Decade of Roma Inclusion Initiative and Action Plans, and (v) the 2007-2013 Bulgaria programming document for the European Social Fund (ESF). (PAD p.24). The PDO remained relevant during implementation. Although two years of preschool education in Bulgaria is compulsory for all children aged 5 or older (implemented by municipalities since school year 2012/2013), as per the ICR most vulnerable groups are still largely excluded. The 2012 results in Program for International Student Assessment Test (PISA) for Bulgaria show promising results for children attending at least a 2-year preprimary education program with greater effects for children of low socioeconomic status and for students like Roma who speak a different language at home (ICR p. 17). Furthermore, Bulgaria’s National Strategy for Promotion of Literacy 2014-2020 and the National Strategy for Educational Integration of Children and Students from Ethnic Minorities 2015-2020 continue to emphasize the importance of promoting Early Childhood Development for vulnerable groups. The project was also very relevant to the Investing in People pillar of the 2016 Bulgaria Country Partnership Strategy, especially to objective 4 (enhanced school outcomes for better employability) and objective 5 (improved access to Early Childhood Development, among other social services, for the bottom 40 percent and marginalized groups). The CPS states the goals of 'improving quality and inclusiveness of education in Bulgaria to enhance cognitive, socio-emotional and technical skills of Bulgarian youth, in particular by focusing on equipping disadvantaged students, including Roma, with the right opportunities from early childhood through formal education.' (CPS p.36-37) Project objectives are also consistent with one of the policy areas identified in the Systematic Country Diagnostic (SCD): boosting skills and employability of all Bulgarians. The SCD notes the need to support policies that will "expand early childhood development programs to improve schooling outcomes, which is especially critical for the poor." (ICR p. 16) Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) Rating Substantial b. Relevance of Design Mandatory stakeholder consultation at the municipal level during project implementation was meant to ensure relevance with local needs and complementarity with existing services. Municipal-level integration of preschool education, child health services, and social services for vulnerable children and their parents is expected to be more efficient than providing them separately. Integration of services is also expected to achieve higher impacts by better serving the needs of low-income and marginalized families. Components addressing both the demand and the supply side of early childhood services were appropriate for accomplishing the stated objectives in the Bulgarian context. A kindergarten quota of 30% vulnerable children per kindergarten group was established to ensure the inclusion of marginalized children. This quota was based on the lessons learnt from previous experiences and decided in a consultative process involving concerned ministries, agencies and local governments (ICR p.12). The project's focus on children aged 0-6 and their parents (as opposed to focus only on school-aged children) recognized the importance of investing as early as possible in child development, especially for disadvantaged families. This is consistent with the broader literature in ECD. Including in the design the development of a rigorous impact evaluation mechanism was essential to informing future design and report on its effectiveness in light of the expected roll out. Yet, there were some shortcomings. Divergent views within the government on what the project should be supporting (and more specifically the balance between infrastructure investments and other services) led to significant delays. Targeting mechanisms were not all sharply defined at entry. Self-targeting among families who were recipients of Guaranteed Minimum Income (GMI) or Child Allowance (CA) benefits was the criterion included in project design documents to identify low income and marginalized potential beneficiaries. This criterion proved insufficient early during implementation due to the high CA threshold. The criterion had to be adjusted to capture different types of vulnerabilities (low income, long term unemployment, ethnic minorities, different forms of disabilities, etc.) (ICR Annex 5). The high risk of relying only on impact evaluation data was not considered at entry. Cancellation of the impact evaluation not only limited the ability to learn from the pilot project before the program would be rolled out, it also negatively affected project monitoring as many of the indicators included in the M&E framework were to be collected through the impact evaluation. Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) Objective 1 Objective To promote social inclusion through increasing the school readiness of children below the age of 7, targeting low-income and marginalized families, including children with a disability and other special needs Rationale Social inclusion was promoted by targeting selected early childhood development services offered at the municipal level to low-income and marginalized families and families with children with disabilities. Services varied by municipality and include community mediation, summer preschools, Roma teacher assistants in kindergarten, kindergarten fee reductions, health consultations and screening, parenting programs and family counselling, early intervention of disabilities, and individual education support to children with disabilities. Participating municipalities were chosen based on the percentage of families receiving Guaranteed Minimum Income and a self-reported needs assessment including socio-economic data on families. The needs assessment aimed to capture different types of vulnerabilities (low income, long term unemployment, belonging to underprivileged ethnic minorities, different forms of disabilities, no health insurance, not attending kindergarten or other child care facilities, etc.). The ICR does not report on how this information was captured or verified by municipalities. The ICR also does not state whether the most needy municipalities were, in fact, the ones selected to participate. Outcomes: Increasing the school readiness of children was the only outcome indicator that survived the changes in the results framework and was measured for vulnerable children attending formal kindergarten supported by the project. A comprehensive School Readiness Test (SRT) was performed at baseline in 98 municipalities across Bulgaria (some SIP-supported and others not). The school-readiness passing rate among vulnerable children in project municipalities was used as a baseline. At end line, all 6-year-olds who attended SIP- supported formal kindergarten for two years (897 children) were tested using the same test performed at baseline. While the baseline and the end line are not strictly comparable (as the former is much larger than the latter and included a control group), there was a noticeable improvement in performance: 80% of the vulnerable children who attended two years in formal kindergarten supported by the SIP passed the SRT in 2015, compared with 47% before the project interventions in 2012. A target was not set at appraisal. Statistically significant improvements were reported in all indicators of kindergarten performance, with larger municipalities performing better than smaller ones. The largest improvement reported was in the ability to speak Bulgarian (99.8% of vulnerable children could speak Bulgarian at project closure compared to 86% in 2012). Despite some large improvements in school performance {or say more specifically what we mean by "impressive results"}, in the absence of a control group, it is difficult to fully attribute the improvements observed to this project. The baseline report indicates that there were several other concurrent projects that included activities aiming at social inclusion and integrating children with disabilities. The baseline also reports increasing numbers of newly-created kindergarten places and increasing numbers of children from Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) marginalized families and with disabilities attending kindergarten. Results of the 2012 PISA study found that attending at least two years of preschool education raises low achievers’ scores by up to 10 points and the scores of those who speak a different language at home by up to 19 points (ICR p.33). This information at baseline suggests that improvements might already have been underway. The ICR also does not report any trend in overall performance of marginalized children in kindergarten across Bulgaria, which might have provided comparison information. Outputs A quota of 30% of vulnerable children per kindergarten and across kindergartens was set based on previous experience in attempting to integrate marginalized children into kindergarten in Bulgaria. The information available in the ICR and the follow up conversation with the project team confirms that compliance with this quota was monitored by the Bank and by the municipalities themselves. School readiness outcomes were supported by the opening of 2,357 new places in kindergartens and preschool education groups (surpassing the target of 1,600); 4,420 children aged 3-7 enrolled in kindergartens and preschool groups (target of 3,000); 471 children with disabilities and other special needs enrolled in mainstream kindergartens and preschool groups (target of 150); and 4,311 children benefitting from the “Early Intervention of Disabilities” service (target of 1,500). The project supported the construction and rehabilitation of kindergartens, and the enrollment of vulnerable children in them through community mediators, and also offered remedial education programs in summer preschools. Education services for children ages 3 to 6 were offered for two years. Social and childcare services supported by the SIP were offered only between 8-6 months. Integrated services focused mostly on the education needs of children aged 3 to 6. Complementary services for children aged 0 to 6 and their parents were only implemented for a few months. Although the PAD describes this project as a pilot (which could be expected to have relatively higher attention and resources devoted to M&E), deficiencies in M&E were noted. Although some improvements were noted as described above, lack of information or evidence to support attribution of those improvements to interventions in this project lead to an assessment of barely substantial efficacy. Rating Substantial PHREVDELTBL PHREVISEDTBL 5. Efficiency At entry, a cost-benefit analysis (CBA) was conducted using the baseline (no intervention) as the counterfactual. The internal rate of return varied from 6-14% based on different scenarios. The economic Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) analysis was based on the assumption that the program would be nationwide. There was a good discussion of the potential impacts based on the analysis of a wide scope of benefits and relevant literature (from Bulgaria and international). Expected benefits were reasonable based on the scope of the project and included: (i) returns to education; (ii) cost-savings in annual per capita student expenditures; (iii) returns to IQ; (iv) cost-savings per arrest and per year of incarceration; and (v) cost-savings in annual social assistance and child allowance for poor families. The CBA was less comprehensive in the discussion of costs; the specific costs estimated are not clearly specified, and there was no consideration of the fiscal impact of the project on government expenditures. At closing, the CBA was not replicated, as the project targeting criteria and its scale changed considerably compared to the original objectives. A cost benefit ratio was calculated for kindergarten services, comparing the actual cost of kindergarten for marginalized children to the expected benefits over 20 years. Expected benefits were estimated using long- term returns on investment to education based on the literature and evidence on students’ performance for Bulgaria, based on the 2012 PISA. Expected outcomes included in the analysis were reasonable and reflected the narrower scope of the project. The expected outcomes included: (i) better school outcomes in later stages, (ii) reduced crime, and (iii) increased productivity in the workforce. Costs included in the cost-benefit ratio calculations were the actual cost of construction (comparable figures in neighboring countries were also present). The analysis resulted in a benefit-to-cost ratio for the project of 2.15. Administrative Efficiency The ICR reports strengthened institutional capacity to provide integrated services in project municipalities (ICR p.23). Stronger institutional capacity could result in higher efficiency gains in the long term. However, the capacity building provided by the project was very limited in both time and scope compared to the original estimate, and thus the capacity improvements are difficult to attribute to the project. Funding constraints associated with the financial crisis negatively impacted the project throughout implementation, causing important delays and efficiency losses. Effectiveness was delayed due to the discrepancies between the Bank and the borrower on the type of activities to be funded under the project. Disbursement was slow in the first years of implementation due to changes in the implementing agency and limited fiscal space on the part of the government. Implementation of the full package of services was severely restricted by the inability of the government to first secure funding and then securing the service specialists needed to provide those services. Taken together, these delays and issues reduced the time available for municipalities to implement the full package of services from the 24 months originally planned to 6-8 months (ICR p.22). Efficiency Rating Modest Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  10.00 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The substantial achievement of the indicators related to school readiness is noteworthy and consistent with other evidence. Yet, there are moderate shortcomings associated with the attribution of those results to the project as the ICR does not discuss other non-project factors leading to these outcomes. Relevance of project objectives was rated substantial due to the alignment of the project objectives with the government strategies. Relevance of design was rated modest due to shortcomings in targeting and the reliance of data coming from the impact evaluation for project M&E. Efficiency was rated modest due to the shortcomings in administrative efficiency associated with the budget ceiling constraints and consequent delays during implementation. Taking all these together, the overall outcome rating for the project is Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating Investing in early childhood education and its importance for future skills and employability is highlighted in many official documents and strategies (see section above on Relevance of Objectives). EU funding was secured through December 2018, and municipalities committed to preserving services (for 5 or 3 years depending on whether subprojects included infrastructure investments or services only) (ICR p.16). All these suggest sustained and continuous investments in early childhood education in the future. However, sustainability may encounter difficulties as activities are scaled up, especially considering the greater need for human resources and the fact that municipalities are currently subsidizing kindergarten fees for vulnerable students. a. Risk to Development Outcome Rating Modest Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) 8. Assessment of Bank Performance a. Quality-at-Entry As discussed in the relevance of project objectives, project design responded to Government’s priorities both in early childhood development and social inclusion. It incorporated lessons of earlier, related projects (PAD p.10, ICR p.9) and resulted in an innovative, one-stop-shop of early childhood interventions for vulnerable children and their parents. Its different arms were supposed to be evaluated to inform the program rollout. The preparation process was highly participatory, including several rounds of consultations among potential beneficiaries, community-based organizations and childcare service providers, municipal officials, and NGO representatives. Consultations confirmed the relevance of the activities proposed by the project and provided greater understanding of the reasons behind the lower utilization of kindergarten and child care services among low income parents (ICR p.9). However, several shortcomings in quality at entry are noted: (i) the project took 2.5 years from Project Concept Note to Approval (the ICR does not discuss the reasons for this delay); (ii) the targeting mechanism was not well defined; (iii) the project’s original results framework was too reliant on data to be collected for the impact evaluation (baseline and target indicators were to be collected after project effectiveness, and the risk of this happening too late - or not happening - was not considered); (iv) the risk of funding constraints was not properly identified at entry, although, as the ICR acknowledges, there were early signs that this risk could have been better anticipated (ICR p.10). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision As per the ICR, Bank staff provided continuous support during implementation to help the government address implementation delays due to restrictive budget ceilings. The five project restructurings suggest that staff reacted to implementation issues. At the time of the third restructuring, the Bank showed proactivity in conditioning the extension of the closing date on budget allocation. The supervision budget was used to finance the end-line survey needed to assess project results, showing commitment to the pilot nature of the operation. Yet, shortcomings in the Bank's performance are noted: (i) the project essentially supported activities for kindergarten-age children as opposed to a varied supply of services for children aged 0 to 6 as envisioned at entry, and this shift was not accurately reflected at restructuring; (ii) during the third restructuring, indicators of relative measures of inclusion and welfare, and indicators of behavior change were dropped, even though they were considered important to assess progress towards expected results (see M&E implementation); and, (iii) the revised results framework included mostly output indicators, with unambitious targets or no targets. Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance Despite the divergent views on what the project should finance during preparation and the constrained financial environment, the government showed commitment to continue with program activities. Project design included a Monitoring Committee integrated by representatives of ministries, agencies and organizations, directly concerned with early childhood development, inclusion and education but supervision documents suggest this committee was not used as planned. As highlighted in the ICR, the main shortcoming in government performance had to do with the budget ceiling that negatively impacted the project throughout implementation (ICR p.25), although, as noted above, there were early signs that this could have been better anticipated. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The implementing agency changed a year after the project became effective as the original unit closed. Both implementing agencies were within the Ministry of Labor and Social Protection (MLSP). Two years of implementation delays are associated with this change. The project implementing unit and the 66 municipalities that implemented subprojects worked well in a very resource-constrained environment. Several delays associated with the budget ceiling constraints negatively affected project implementation from the beginning. Both the implementing agency and the involved municipalities used creative ways to 'do more with less.' Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) Strong monitoring and evaluation provisions were included in the PAD due to the pilot nature of the project. Funds were allocated for an impact evaluation and for associated capacity building efforts for monitoring and evaluation. Many indicators included in the project’s original Results Framework relied on data to be collected for the impact evaluation. M&E was intended to be used as "a managing-by-results tool as roll-out progresses by testing features of the project to modify design and improve effectiveness over time" (PAD p.13). Baseline data was supposed to be collected soon after project effectiveness and follow-up surveys were to be conducted during years two and four of implementation. Qualitative analysis was supposed to complement the quantitative data collection for the impact evaluation, through focus group discussions at different stages of project implementation (PAD p.14). To assess the PDO, the following age-specific pre-primary and early primary-school outcome indicators were identified at appraisal in the results framework: • Cognitive development and school readiness scores, including (i) memory, verbal and visual-motor skills development; copying scores (age 3-5); (ii) fluency in Bulgarian (age 5-6); (iii) achievement test results, including reading tests (age 6-8); • Child nutrition in target population (as proxy by anthropometric measures). The service integration and greater availability of subsidized full-day child care was expected to contribute to an increase in employment among low-income parents but this outcome was not included in the M&E. b. M&E Implementation Due to delays in project implementation, baseline data was completed three years after project effectiveness (May 2012). The information was added to the revised Results Framework, after the MTR, during the third restructuring (December 2012). Before that, the Project’s Implementation Status and Results Reports did not include routinely updated monitoring data on the project’s indicators. (ICR p.14) The impact evaluation was cancelled due to lack of government funding. The follow up conversation with the project team also suggests issues regarding the random selection of participant municipalities which, in turn, raises questions about the usefulness of the impact evaluation. The Project’s Results Framework was revised twice, with changes intended to limit the reliance on data from impact evaluation and to adjust the targets to a shorter implementation period (from the original 18-24 months to 12 months). Revised indicators, however, were not better than the original ones. The revised indicators mostly measured outputs; all indicators originally included that aimed to capture change in behavior (like immunization, nutrition or parenting skills) were dropped due to concerns regarding project attribution or complicated measurement. Revised indicators do not give indication of progress towards inclusion, and they refer to absolute numbers rather than rates or relative measures, for example replacing in many cases 'rate of enrollment' by 'number'. Beneficiary surveys or stakeholder workshops were not carried out as planned due to lack of funding. Similarly, the the qualitative and quantitative studies planned for the impact evaluation were not carried out. Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) c. M&E Utilization Government budget constraints resulted in the cancellation of the impact evaluation. There is no evidence that M&E data were used as "a managing-by-results tool," as was originally intended. Project design included a Monitoring Committee integrated by representatives of ministries, agencies and organizations, directly concerned with early childhood development, inclusion, and education, but supervision documents suggest this committee was not used as planned. To assess project results, school readiness tests were administered among kindergarteners both for the baseline report (May 2012) and again in June-July 2015. The ICR does not report on any utilization of the baseline report. At completion, the Bank validated the testing process through random observations in 10% of the sites and helped with processing and analyzing the data. 757 tests were used to compare data "before" and "after" project implementation. (ICR p.14) Per the ICR, these data were used by the government to justify continuing the provision of services for early childhood development. (ICR p.14) M&E Quality Rating Modest 11. Other Issues a. Safeguards The Project was categorized as "Financial Intermediary" with subprojects of potential Category C and B. Per the ICR, local environmental requirements were considered adequate, guidelines for subprojects' environmental prescreening and analysis were included in the project's Operational Manual. The project’s compliance was rated satisfactory throughout implementation (ICR p.15). At the time of the MTR, independent consultants checked the quality of works and compliance with local construction and safety standards in a sample of subprojects. Minor issues with environmental prescreening were identified but were promptly addressed and resolved (ICR p.15). b. Fiduciary Compliance Per the ICR (p. 15), financial Management arrangements were satisfactory and adequate controls and procedures were in place throughout project implementation. All interim financial reports and annual project audits were received on time, with clean opinions and no internal control issues. This was confirmed at MTR and in the ICR. Adequate procurement processes were followed and efficiency sought through centralized procurement for civil works and equipment (ICR p.16). Procurement-related delays were due to the budget ceiling imposed at the Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) government level. c. Unintended impacts (Positive or Negative) d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory Sustainability may encounter difficulties as activities are scaled up, especially considering the greater need Risk to Development Negligible Modest for human resources and the Outcome fact that municipalities are currently subsidizing kindergarten fees for vulnerable students. Moderately Moderately Bank Performance --- Satisfactory Satisfactory Moderately Moderately Borrower Performance --- Satisfactory Satisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons A realistic risk assessment conducted not only at entry but also continued during implementation can help to anticipated issues that arise during implementation (such as, in this project, the large amount of specialists Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SOCIAL INCLUSION PROJECT (P100657) needed to provide social services at scale), and, consequently, can help identify timely mitigation measures. Changes during implementation should be adequately reflected at the time of restructuring and also in the results framework. Delays in getting the necessary funding from the government changed the design of the project giving insufficient time of exposure to the social services (from the expected 18 to 24 months at entry to 12 months). This was not properly reflected in any of the restructurings nor it was reflected in the results framework. Cancelling the impact evaluation limited the expected learning from this pilot project. While piloting was not an explicit objective in the PDO/Financing Agreement, the PAD indicated this was a pilot project. Due to the government ceiling spending and issues affecting the selection of municipalities, the impact evaluation was cancelled. The PDO was assessed by a comprehensive school readiness test applied at baseline and end line to children benefitting from SIP supported formal kindergarten without a proper control group. Funding for this assessment came from the project supervision budget, but other assessments (quantitative or qualitative) were not conducted. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR follows the Bank ICR guidelines and it is internally consistent. The analysis is candid but incomplete (there is no information on the baseline study and limited information is provided on the subprojects funded despite amounting to 82% of the total loan commitment). The evidence on efficacy could be strengthened with a discussion of attribution. a. Quality of ICR Rating Substantial Page 14 of 14