IEG Report Number: ICRR14790 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 08/24/2015 Country: Bangladesh Project ID: P118701 Appraisal Actual Project Name: Bangladesh - Project Costs (US$M): 497.41 475.84 Employment Generation Program For The Poorest L/C Number: Loan/Credit (US$M): 150.00 148.09 Sector Board: Social Protection Cofinancing (US$M): Cofinanciers: Board Approval Date : 11/30/2010 Closing Date: 06/30/2014 06/30/2014 Sector(s): Other social services (96%); Public administration- Other social services (4%) Theme(s): Social safety nets (70%); Improving labor markets (30%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Elena Bardasi Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: The project development objective was "to provide short-term employment on community sub-projects to enable households to better cope with vulnerability, while strengthening Program implementation" (Financial Agreement, 2010, Schedule 1, p. 5). The same formulation is used in the Project Appraisal Document, Annex 4, p. 40. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project had two components (the US$ figures by component provided in parenthesis are Bank costs only): Component #1 - Main Component (total estimated cost US $144.75m; actual US$144.75m): This component aimed to support the expansion of the Government of Bangladesh social protection program "Employment Generation Program for the Poorest (EGPP)" and the introduction of innovations. This component had three focus areas: (i) Targeting. The project aimed to improve targeting at the geographical level through the use of the poverty map to ensure channelling of resources to the poorest upazilas; at the household level through setting wages below the prevailing market wage to encourage self-selection of the poorest into the program and through the use of proxy indicators for household poverty to screen into the program households headed by a manual laborer and with less than 1/2 acre of land; and along gender lines, to ensure that women represented at least 33 percent of project beneficiaries during the course of the project period; (ii) Governance. This focus area aimed to introduce innovations into the program to increase clarity and transparency of rules and procedures; improve information and public disclosure, by mandating community notice boards and making selection criteria and beneficiary lists publicly available; make provision for non-wage costs, allowing up to 20 percent of the cost of each EGPP subproject to be earmarked to purchase materials and skilled labor; improve the wage payment systems to reduce fiduciary risk and empower the poorest; enhance the capacity at the field level for better supervision by introducing field level supervisors provided with transport and access to training; and develop an enhanced grievance redress system to accept and address complaints; (iii) Monitoring and evaluation . This focus area aimed to develop a set of standardized summary indicators and monitoring reports (including external independent validation of the program's implementation, impact evaluations, and qualitative assessment); provide financing for critical staffing at the central and local levels; and introduce a digitized MIS (in the form of hardware, software, and data entry clerks) at the upazila or district level, including connectivity to transmit data from the local to the central level. Component #2 - Operational Support Component (total estimated cost US $5.25m; actual US$5.21m): This component aimed to provide direct support to the Minister of Food and Disaster Management (MOFDM) for implementation and capacity strengthening. The activities under this component were meant to develop capacity at the central and upazila level for implementation of the Main Program Component of EGPP, but also of other programs of the MOFDM. Moreover, they were also meant to strengthen cross-sectoral collaboration among the ministries and with other stakeholders. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project costs: The total project cost was US$475.84 million, US$21.57 million less than the US$492.16 million originally planned. Financing: The project was financed with an International Development Association (IDA) loan in the amount of US$150.00 million, which was entirely disbursed. The project provided finance to the Government of Bangladesh EGPP Program in the form of financing 70% of the wage costs, up to a maximum of US$144.75 million (component #1), over the three-year period 2011-13. A results-based approach was adopted, that is, disbursements were linked to targets included in the disbursement-linked indicator (DLI) framework. Annual disbursement from the IDA credits were conditional on meeting the targets for the relevant year. Essentially, the main component of the project reimbursed a portion of the costs sustained by the Government conditional on achieving the (annual) targets articulated in the DLI framework, and component #2 provided capacity building to create the conditions to achieve the targeted results. Bank-executed Trust Funds were used to support the monitoring and evaluation activities (pay for an independent evaluation of activities undertaken under component #2). Borrower contribution : The government contributed US$325.84 million, or US$21.57 million less than the US$ 347.41 originally planned. Dates: The original credit was approved in November 2010 and became effective in January 2011. The project closed on June 30, 2014, which was the original closing date. A level-two restructuring was approved in November 2013 to revise the DLI 2013 target for geographic targeting. This change was a technical amendment and did not change the originally agreed principle of the DLI; it only increased the percentage of resources allocated to higher poverty rates. A second level-two restructuring was approved in May 2014 to (i) adjust the target for the "number of person-days of short-term employment created" indicator from 60 to 48 million (due to the increase in the wage rate that had to be absorbed by a lower total number of days) and (ii) introduce the PDO-level indicator 2014 targets (phase 1 - until the end of the project) for all indicators, which inadvertently had not been originally defined. As the "person-days of employment" indicator is considered an output indicator, the revision of its target does not trigger a split rating. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of Objectives is rated High. The project was fully coherent with the country conditions. At appraisal, 31.1 percent of the population (or 47 million people) was poor and 17.4 percent (or 26 million) extremely poor. Rural farmers engaged in casual labor were particularly vulnerable, due to the strong seasonal variations in food production as well as natural disasters (ICR, p. 13). Hence the relevance of a workfare program aimed to provide short-term employment when the conditions for these rural households were most difficult (i.e. during the lean seasons). The IDA support to EGPP is aligned with the priorities of the Government of Bangladesh, as reflected in its Sixth Five-Year Plan for FY11-15. This document recognizes that Social Safety Net Programs are an integral part of the anti-poverty strategy of the Bangladeshi government and play a crucial role to address risk and vulnerability. The Employment Generation Program for the Poorest is highlighted as one of the major social protection programs, which is part of an integrated, comprehensive, and sustainable social safety net system. The goal of the project is also fully aligned with the Country Assistance Strategy (CAS) of the Bank, 2011-2014, specifically with the strategic objective for more inclusive growth "improving social service delivery," The aim of this strategic objective is to improve targeting of social protection programs, strengthen program governance, strengthen institutions, promote micro-insurance for the extreme poor, and develop capacity to scale up safety nets effectively during shocks including natural disasters. Moreover, this objective explicitly includes the expansion of targeted social assistance through "support(ing) Government’s Employment Generation for the Poorest Program to address seasonal poverty in rural areas, with a special focus on vulnerable women" (CAS, p. 28). The CAS indicates that the World Bank was to support EGPP in the first year of the CAS. b. Relevance of Design: Relevance of design is rated High. The project's design was sound and aimed at supporting, expanding, and strengthening the largest government safety net program focused on employment generation. To achieve its objective, the project correctly emphasized the need of improving targeting, governance, and M&E. Specific elements of design were introduced to increase the likelihood of achieving the desired objectives: (i) the adoption of a results-based approach, which conditioned annual disbursements from the IDA credit to meeting the milestones agreed for the relevant year; (ii) the refinement of the targeting methods (the use of the poverty map to improve the geographical targeting and the adoption of two criteria - appropriate selection of the wage rate to ensure self-targeting and use of proxies for household poverty - to target the poorest households within the selected areas; (iii) the specific effort to target women, not only through the adoption of a quota, but also through the provision of light work close to the local community; (iv) the increased clarity and transparency of guidelines and implementation; (v) the provision for non-wage costs to allow for subprojects others than simple earth removal and reducing the risk that wage funds would be diverted to non-wage spending; (vi) the improvement of wage payments through the use of formal financial channels to increase efficiency and ensure separation between resource management and payment processing with the goal of reducing misappropriation, payment mismanagement, and abuse; and (vii) the substantial strengthening of the implementation capacity, through hiring of additional supervisors at the local level, provision of training, and the development of an enhanced grievance redress system. These elements of design, some of which were innovative, considerably strengthened the results chain of the project. 4. Achievement of Objectives (Efficacy): 1. Provide short-term employment on community sub-projects to enable households to better cope with vulnerability is rated Substantial. Outputs: - The number of individuals who participated in each of the seven phases of public works was over 700,000 on average per phase, surpassing the target of 600,000 per phase. In total, 3.02 million individuals participated in the project; - The % of women participating in the project was 36%, meeting the target; - The project generated 195 million person-days of employment (for the 3.02 million participants), surpassing the revised annual target of 48 million person-days of employment. The project performed slightly below the original target of 60 million person-days per year, except in FY13 when 59.1 million person-days of employment were created, very close to the target. Outcomes: - Household vulnerability was reduced (EGPP households had a larger level of non-food expenditure than non-EGPP households), based on the results of the impact evaluation. The ICR reports preliminary findings based on cross tabulations from the baseline and the follow up surveys: it was found that consumption levels were higher for EGPP participants than non-EGPP participants. Moreover, male-headed EGPP households had a larger level of non-food expenditure than non-EGPP households based on the midline survey, while female-headed EGPP households had the same results but only when data from the endline survey was used. (ICR, p. 26 footnote 13). The results of the impact evaluation emerging after the ICR was completed indicated that participation in EGPP led to an increase in total monthly consumption of about Taka 1480, or about US$20, against a maximum monthly average benefit of US$17, as reported by the project team. - The % of beneficiaries from households classified as 'extremely poor' was over 90%, surpassing the target of 80%; ['extremely poor households' were defined as those who owned less than 0.5 acres of land and whose head was a manual worker]. 2. Strengthen program implementation is rated Substantial. Outputs (Targeting): - The % of wage costs allocated to very poor upazilas (with poverty rate equal or higher than 35%) was 55% in FY11, 60% in FY12 and FY13, and 45% in phase 1 of FY14, surpassing the revised target of 45%; but lower than the original target of 60%; - The % of women participating in the project was 36%, surpassing the target of 33%. Outputs (Governance): - The % of sub-projects with signboards at work sites and publicly available beneficiary lists was 72% at the end of the project, slightly short of the target value of 75%; - All EGPP participants were receiving payments through formal financial channels, surpassing the 95% target; - From 2011 all field supervisor positions had been filled, including in upazilas with poverty rates of 21% or higher; - Over the final two phases of the project, 78% of sub-projects on average used 10-20% of project resources for non-wage costs, slightly surpassing the target of 75%; - At the end of the project, 100% of upazilas had a functioning grievance redress system surpassing the target of 75%; Outputs (Monitoring and Evaluation): - At the end of the project 100% of community sub-projects were using standardized monitoring formats, surpassing the target of 90%; - 83% of upazilas had, at the end of the project, an electronic beneficiary database, short of the 100% target. The difference reflected delays in MIS procurement; - By the end of the project three household surveys had been carried out, one each year from 2011 to 2013, according to the target; - In all upazilas independent process documentation activities had taken place by the end of the project, according to the target. The result was derived from a random sample of 20 upazilas in six phases of the program. Outcomes: - The % of beneficiaries from households classified as 'extremely poor' was over 90%, surpassing the target of 80%. 5. Efficiency: Efficiency is rated Substantial. The ICR replicates the cost effectiveness analysis originally presented in the PAD, and concludes that the project - based on the results achieved - is a more cost-effective mechanism for reaching the poor than the counterfactual program where each household in the population receives a uniform transfer. The calculation follows a commonly used decomposition proposed by Ravallion (1999, The World Bank Research Observer, 14/1, p. 31-48). The results presented in the ICR still need to rely on assumptions regarding the net wage gains from participating in the program (with respect to non-participating) and indirect benefits of the assets produced accruing to the poor. Using alternative assumptions (as or more conservative than those used in the PAD) for these two components, the ICR finds that cost effectiveness is higher than originally calculated in the PAD. The cost-effectiveness increase with respect to the PAD is the result of improvements introduced by the program, specifically the provision of non-wage costs and the improved targeting. The provision for non-wage costs decreases the labor intensity and increases the social benefit-to-cost ratio of the program, in both cases positively impacting the cost effectiveness. Improved household and geographic targeting also positively impact the cost effectiveness by improving the outreach to the poor as well as the share of benefits from assets that goes to the poor respectively. As a result, the cost effectiveness of EGPP is estimated to be, in absence of indirect benefits generated by the assets created, 0.38 or 0.45 (depending on the assumption of net wage gains) and 0.76 or 0.83 when indirect benefits are accounted for. In any case, as the poverty rate was 31.5 percent at project closing, these figures outperform the counterfactual of a uniform allocation of the budget to the population. (Even with a poverty rate of 44 percent, when the project started, the project is cost effective under the assumption of at least some indirect benefits for the poor.) The figures presented in the PAD were 0.33 or 0.38 and 0.63 or 0.76 respectively. There were delays in procurement - due to unexpected turnover of staff with procurement expertise - which affected the development of the new MIS software. This is turn delayed a DLI (efficient record-keeping mechanism) and therefore disbursement of funds. While it is hard to quantify the impact, it is expected that these delays had some negative effect on project efficiency. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project's objectives were highly relevant to the country conditions, the country strategy, and the CAS at time of appraisal and closure. Its design was also highly relevant, as it was based on a sound result chain and included some innovative elements such as a result-based approach to disbursement. The project performed substantially in terms of both targeting the extremely poor as well as women and reducing household vulnerability, as documented by the impact evaluation. The project also succeeded in strengthening capacity at the central and upazila level, despite delays and shortcomings in setting up an electronic beneficiary database. Efficiency (cost effectiveness) was also substantial. a. Outcome Rating: Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development outcomes is rated moderate. The initially limited capacity of the implementing agency was strengthened during project implementation through training and hiring of more staff, especially field supervisors. The commitment of the Government remained very high throughout implementation and at closure; the budget resources allocated to the project increased by about 40 percent over the project period (ICR, p. 31). In 2013, the program absorbed 12 percent of the government budget for social protection (which in turn represented 14 percent of the overall government budget), but was not expected to generate additional fiscal commitments (except for increases in the wage rate, set in line with increases in inflation) (ICR, p. 28). Some risks remained associated with the integration of the program into a harmonized social protection system. After the project closed, the program implementation was moved out of an ad-hoc office and mainstreamed under the Department of Disaster Management (DDM) under the Ministry of Disaster Management and Relief. This risks putting too much pressure on the DDM, which is responsible for four other safety net programs. However, one year after implementation this risk had not materialized (ICR, p. 31). a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project's design was based on and informed by the 100 Day Employment Generation Program (100 DEGP) that preceded the EGPP. The 100 DEGP was introduced in 2008 as the first attempt in Bangladesh to alleviate poverty through guaranteed employment and income in the short run. Independent assessments of the 100 DEGP were conducted by the World Bank (2009), the Centre for Policy Dialogue (2009) and the NFPCSP-FAO-BRAC (2008). (The Bank team was also involved in assessing the first phase of the EGPP.) Lessons from these assessments - as well as from the Bank's worldwide experience with implementation of public works - were incorporated in the EGPP design, specifically with respect to the following aspects: (i) the use of poverty maps for better geographic targeting; (ii) the introduction of a quota to ensure a minimum participation of women; (iii) the provision for a minimum amount of non-wage costs, to support the creation of larger sub-projects; (iv) the adoption of an improved payment system; and (v) increased transparency through the use of information boards at project sites. The ICR highlights some minor shortcomings. The local implementation capacity was overestimated and measures to further strengthen it had to be put in place during implementation; the results framework did not include some important outcome indicators; target values for FY14 were inadvertently omitted in the original results framework; and the methodology for the household surveys was not fully defined. Quality-at-Entry Rating: Satisfactory b. Quality of supervision: The Bank's team conducted regular supervision missions. IDA disbursements occurred on schedule, and the project closed on time with no need of extension. The supervision missions identified and promptly supported the government to address all challenges that arose during implementation, including the need to adjust geographical targeting to better reach the poorest upazilas, improve the payment systems in remote areas, strengthen field-level monitoring, introduce signboards at worksites to increase transparency, and improve recording of attendance. A small procedural irregularity was generated by incorrect advice from the Bank's legal team, which did not recommend restructuring when the government (with the agreement of the Bank team) proceeded in January 2012 to change the allocation rule to provide more resources to upazilas with higher poverty rates and lower resources to less poor areas (a change that was reflected in a revision of the EGPP program guidelines). Later on, the Bank stressed that this change - despite being a minor one - should have been treated as a restructuring, as any change in legal agreements requires restructuring. When delays in the development of the MIS became apparent, the Bank team used Bank-executed trust fund resources to conduct a qualitative evaluation and regular spot checks in each phase of the project to provide feedback on the three focus areas of expected reforms (ICR, p. 20). The ICR (p. 32) rightly comments that the restructuring could have been undertaken earlier, but as this was not about any aspect that could have impacted the implementation of the project, the delay was of no concrete consequence. The one aspect of supervision performance that had a meaningful consequence was the lack of prompt consideration of the methodology of the household surveys, which were supposed to collect information on the impacts of the project on development outcomes. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The program was already being implemented when the Bank's loan started. This means that the program was already fully "owned" and embraced by the Government of Bangladesh, which remained fully committed throughout implementation, despite three government changes. The ICR (p. 33) notes that the high level of policy engagement and the dialogue with the Bank ensured progressive and continuous improvement in operations and strengthening of the program's poverty focus. The ICR (p. 33) also stresses that the Government did not interpret the engagement with the Bank simply as a means to get budgetary support for the EGPP, but also as an opportunity to receive technical assistance to undertake systemic social protection reforms. Government Performance Rating Highly Satisfactory b. Implementing Agency Performance: The project management team was housed in the Ministry of Food and Disaster Management, which subsequently became the Ministry of Disaster Management and Relief (MODMR). The coordination between the project implementation team and the Bank was good, which ensured regular monitoring of implementation and timely intervention when challenges arose. A shortcoming was represented by the difficulties in implementing the MIS, due to procurement challenges generated by rapid turnover in procurement expertise. The ICR indicates that this problem was addressed and resolved thanks to the support of the project team, although this involved a meaningful delay in MIS implementation and the inability to achieve fully the MIS target (ICR, p. 34). There were incidents of mis-procurement: in FY12 on three contracts totaling US$17,450; and on another two contracts (one in FY12 and one in FY13) totaling US$10,527. As the project had already closed when the incidents were found and the contracts already issued, the funds were cancelled and were in the process of being returned to the Bank when the ICR was completed (ICR, p. 23). Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The project's M&E was to be based on the Minister of Disaster Management and Relief's regular program monitoring system, appropriately strengthened and streamlined. The new MIS was to include a set of standardized indicators and monitoring reports redesigned to facilitate data collection on "inputs, outputs, intermediate outcomes and compliance with program guidelines" (PAD, p. 31). The project was going to introduce a digitized monitoring unit at the upazila level, as well as connectivity that would allow information to be transferred electronically from local levels to the center. Some important indicators were not included in the results framework. The results framework did not include indicators of households' ability to cope with vulnerability. Instead, the PAD indicated that an impact evaluation was to collect baseline and follow-up data to measure "the impact of the program on household net incomes and coping strategies in response to shocks" (PAD, p. 32), in addition to assessing the effectiveness of targeting. Considering the complexity of measuring vulnerability and the difficulties in deciding attribution, this was an appropriate choice. Another indicator missing from the results framework was a measure of leakages in the program. However, perceptions of the existence of leakages and corruption were to be collected through qualitative assessments complementing the M&E framework (see below). The coverage of the program was measured in absolute numbers (million of beneficiaries reached), but there is no indicator measuring the percentage of total poor households reached by the program. The ICR does not discuss this point, and does not provide information on how equally eligible beneficiaries were selected (if the program could not serve all poor households). As the program aimed at reducing vulnerability by providing employment during lean seasons, another missing indicator was the timing of employment provision, as delays in providing employment could cause undesirable overlapping between the program and the harvest period. Interestingly, the ICR (p. 30) notices that there were indeed lags between the approval and start of works, which would push the start of employment into the harvesting season, but classifies those as "unintended outcomes and impacts," while in fact these were perfectly foreseeable outcomes. Otherwise, the M&E framework included an overall meaningful set of indicators to measure the targeting and take-up of the program, increased transparency, and improvement in the capacity for monitoring and evaluation. Several monitoring and evaluation indicators were to be used as DLI (disbursement-linked indicators). Among those were some of the most meaningful indicators (namely those measuring improvements in targeting), a de facto commitment of the project to promptly set up and collect its monitoring system. To complement the set of indicators included in the results framework, the monitoring framework provided for independent assessments, in particular: (i) an external validation, to be carried out by a local think-tank, to check a random sample of about 3% of union parishads during each phase of the program implementation; (ii) an impact evaluation, based on a baseline and follow-up survey of program participants and non-participants in a representative sample of upazilas, to collect information on household demographics, assets, income generating activities, risk coping strategies, and participation in safety net programs; and (iii) qualitative assessments to assess perceptions of beneficiaries and non-beneficiaries regarding the targeting, the working of the program, the existence of leakages and corruption, the value and quality of the infrastructure sub-projects, and the impact on migration and social empowerment. b. M&E Implementation: The new digitized MIS was put in place only toward the end of the project, because of delays in the procurement process. For this reason, the regular MODMR M&E, integrating the additional data provided by the new forms and reports, was used during the duration of the project. The third-party validations originally planned took place, including qualitative assessments and beneficiary focus group discussions, spot checks, and process evaluations. The impact evaluation was also carried out; it was based on three surveys (baseline, midline and endline) supported by two trust funds. c. M&E Utilization: The M&E data were used both to adjust implementation and to trigger disbursements. Some changes and adjustments in design were made during implementation in response to findings based on data collected. For example, it was found that project beneficiaries only utilized the EGPP bank account to receive their payments and not for other bank operations. It was then decided to require that a small portion of wages be retained in the checking accounts as savings, which further strengthened the beneficiaries' interest in having a EGPP bank account. Similarly, changes in supervision at the work sites were made based on result assessments. The impact evaluation findings were used in conjunction with the Bank's midterm and endline assessments, which allowed the Bank's team to triangulate findings and better validate results. The Government used the findings from the 2011 midline impact evaluation survey to justify an increase of its allocation to the program in FY13. M&E Quality Rating: Substantial 11. Other Issues a. Safeguards: The following Safeguards were triggered: the Bank’s Operational Policy on Environmental Assessment (OP 4.01) (the project was identified as category B, both in the PAD p. 93 and the ICR p. 23); the Bank’s Operational Policy on Involuntary Resettlement (OP 4.12); and the Bank’s Operational Policy for Indigenous Peoples (OP 4.10). Both the PAD and the ICR recognize that issues were likely to be minor, as the sub-projects were small-scale earthworks and no private land acquisition or large-scale displacement was expected. The ICR indicates that, as far as the Environmental Safeguard was concerned, an Environmental Management Framework was already in place, which included screening tools and procedures to mitigate potentially adverse impacts, a monitoring mechanism, and procedures for disclosure. Environmental audits were included in the independent third-party assessment, but the ICR does not report the results of those audits. As far as the Involuntary Resettlement and Indigenous People safeguards were concerned, a Social Management Framework was developed, which included a Resettlement Policy Framework and an Indigenous Peoples Planning Framework. Details of this framework are not provided. The ICR indicates that no issues arose during implementation and there were "no deviations or waivers from the bank's [social] safeguard policies" (ICR, p. 23). b. Fiduciary Compliance: There were delays in the timeliness of disbursement under component #2 and reporting on spending on both components #1 and #2. The Comptroller and Auditor General of Bangladesh audited the program's financial statements annually. Audit reports signaled that the financial statements were free from material error. However, because no audit of expenditure under component 1 was carried out in FY11 and FY12 (as this was outside of the auditor's remit), the audited financial statements for these two years remained beyond audit assurance. At the time the ICR was being prepared, these issues were in the process of being resolved (ICR, p. 23). The project suffered from rapid turnover of procurement specialists, which caused delays in contracting of goods and services. In particular, this delayed the contracting of the IT firm that was to develop the MIS software. As a result, the MIS was put in place only toward the end of the project. Moreover, there were also delays in setting up an efficient record-keeping mechanism for the DLI (although the ICR does not report delays in disbursement). There were instances of misprocurement (3 contracts in FY12 for a total of US$17,450; 2 contracts in FY12 and FY13 for a total of US$10,527). At the time the ICR was prepared, the funds had been cancelled and were in the process of being returned to the Bank. c. Unintended Impacts (positive or negative): None reported. d. Other: A minor technical modification to the geographic allocation of program resources was agreed upon and communicated through a management letter from the Bank to the Government. It was incorporated in the EGPP Program Guidelines but was not processed as restructuring as per advice from the Bank’s legal team. This advice later turned out to be incorrect (OPCS and LEG policy clarified that any change in legal agreements requires restructuring, including minor changes) (ICR, p. 17). 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome: Bank Performance: Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Three lessons -- all presented in the ICR (pp. 34-35) -- are highlighted here. They are all well grounded in the evidence presented: 1. The adoption of a Disbursement -Linked Indicator framework can be very effective in keeping the focus on output, outcomes, and results and ensuring continuity in the dialogue between the Government and the Bank . The DLI framework included all the relevant indicators (with the exception of the vulnerability measure to be collected through the impact evaluation). It forced a clear articulation of a complete results chain and the definition of a rich M&E framework. It also kept the dialogue going and focused on results. 2. A diverse and complete set of evaluation mechanisms ensures proper documentation of results and can help overcome unexpected shortcomings . In addition to the M&E framework, the project had planned for independent evaluations, including an impact evaluation, spot checks, qualitative assessments, and external validations. When it became clear that the new M&E framework could not be put in place on time, spot checks allowed for positive feedback loops to identify areas in need of course correction. Moreover, different indicators could be collected with the most appropriate means (for example, the vulnerability measure through impact evaluation). 3. Strong project design and proper stocktaking of existing experience allows for introduction of innovations and can ensure better inclusion of specific groups . This project's design focused on marginal, but significant improvement in an already promising government program under implementation. It introduced an innovative payment system. It introduced not only a quota, but also greater attention to working conditions favored by women to increase female participation. It also took notice of existing evaluations of the previous version of the program to introduce improvement. 14. Assessment Recommended? Yes No Why? To better understand how the DLI framework impacted the achievement of the project. It is also a large and successful project that can offer opportunities for learning. 15. Comments on Quality of ICR: The ICR is concise, clear, well written and organized. The document is centered on the results achieved and the elements that facilitated achievement of those results. The only issue to note is that the ICR does not clearly describe the results of the impact evaluation, which are quite crucial for understanding the impact of the project on the main PDO indicator, the vulnerability measure. The ICR cites issues with the household surveys, but does not describe what the problems were, and this is quite central to understanding the achievements of the project. a.Quality of ICR Rating : Satisfactory