ICRR 14457 Report Number : ICRR14457 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/17/2014 Country : Africa Project ID : P075776 Appraisal Actual Project Name : Africa Stockpiles US$M ): Project Costs (US$M): Programme - Project 1 L/C Number : Loan /Credit (US$M): Loan/ US$M): Sector Board : Environment Cofinancing (US$M): US$M ): Cofinanciers : Board Approval Date : Closing Date : 06/30/2012 Sector (s): Central government administration (40%); Solid waste management (33%); Petrochemicals and fertilizers (20%); Other industry (7%) Theme (s): Pollution management and environmental health (50% - P); Other environment and natural resources management (25% - S); Environmental policies and institutions (25% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : David M. Colbert Robert Mark Lacey Christopher David IEGPS1 Nelson 2. Project Objectives and Components: a. Objectives: According to the Global Environment Facility (GEF) Trust Fund Grant Agreements with the participating countries (TFGA, Schedule 2, p. 14), the objectives of the Africa Stockpiles Program (ASP) are: “to assist the Recipient[s] in: (a) eliminating inventoried Publicly Held Obsolete Pesticide stocks and Associated Waste; and (b) implementing measures to reduce and prevent future related risks .� The statement of objectives in the Project Appraisal Document (PAD, p. 6) is identical. Neither the TFGAs nor the PAD defined the ASP ’s Global Environmental Objective (GEO), but the PAD indicated that it would be a “direct implementation of the Stockholm Convention on POPs [Persistent Organic Pollutants] and the associated GEF Operational Program aiming to reduce the impact of POPs on the global food chain, pollution of transboundary waters, land and biodiversity � (p.4). b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The ASP's four components were as follows : Component 1: Country Operations (GEF: US$ 21.74 million; Total: US$ 51.37 million; actual N/A): This component encompassed the full range of disposal, cleanup and prevention activities to achieve the PDO in the participating countries. It included four sub-components: (i) cleanup and disposal; (ii) prevention of pesticide accumulation; (iii) capacity building; and (iv) project management and monitoring. Country-specific details on these activities were outlined in the respective country annexes . Component 2: Technical Support (GEF: US$ 3.26 million; Total: US$ 4.31 million; actual N/A): This component provided the technical and specialized expertise required for implementation, supervision and monitoring of the country-level activities outlined under the first component . This included technical advice requested by the countries for the design and delivery of training; production of technical guidelines for cleanup and prevention operations; assistance in managing technical aspects of procurement and supervision of specialized contractors; advice on linking countries with other specialized agencies and organizations; enhancement of health and safety; assessment of laboratory capacities; and oversight of monitoring and evaluation (M&E). Component 3: Crosscutting Activities (Total: US$ 2.44 million; actual N/A): This component supported knowledge management, awareness -raising, strategic studies and outreach services through a coordinated multi-country, multi-partner approach. Its key sub-components included outreach to promote ratification of conventions relevant to the PDO, capacity building for nongovernmental organizations (NGOs), knowledge management, communications, and strategic studies, such as an evaluation of alternative technologies for the disposal of obsolete pesticides, better understanding of environmental health risks associated with the specific pesticides and assistance with prioritization and decision -making within each country. Component 4: Project Coordination (Total US$ 2.44 million; actual N/A): This component coordinated the implementation and monitoring of the ASP across the various partners . Key activities include developing and ensuring implementation of a fund-raising strategy; acting as a Secretariat to the ASP Steering Committee; maintaining awareness at the policy level about the project; reporting to donors; preparing work plans, progress reports and financial reports; coordinating independent project monitoring; and coordinating the design of follow-on projects. These ASP components were never formally revised through a World Bank -approved restructuring. However, in June 2011, following and based on the mid-term review, the projects of four of the six ASP participating countries (i.e. Ethiopia, South Africa, Tanzania and Tunisia ) underwent design adjustments, including clarification of project outcomes and revision of project components, in order to facilitate the implementation of ASP activities and the achievement of desired outcomes . The projects of the remaining two countries (i.e. Mali, Morocco) did not undergo adjustments, either because external conditions or lack of commitment impeded further project implementation. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost: The PAD and ICR both report an estimated project cost of US$ 60.0 million. The ICR, however, does not report the actual (or latest estimate of) project costs, indicating that these were not available as of December 2013 (p. 24). The actual project costs remain difficult to estimate because of (i) the limited data provided by the multiple sources of project financing and (ii) the difficulty in determining the actual contributions of the participating countries . Individual country project costs are presented in the technical annexes : Ethiopia (US$ 2.2 million), Mali (US$ 2.4 million), Morocco (US$ 0.3 million), South Africa (US$ 0.7 million), Tanzania US$ 6.8 million), and Tunisia (US$ 3.8 million). Financing: The ASP was financed by a US$ 25.0 million grant from the Global Environment Facility (GEF) and by contributions from ASP participant countries ($2.7 million). Co- and parallel financing was mobilized through the World Bank Development Grant Facility ($2.7 million), the Multi-Donor Trust Fund ($4.2 million), the African Development Bank ($10.0 million estimated at appraisal, actual N /A), Canada ($2.3 million); bilateral financing from Belgium ($4.1 million estimated at appraisal, actual N /A), Finland ($1.0 million estimated at appraisal, actual N/A), France ($1.7 million), Japan ($1.1 million estimated at appraisal, actual N /A), Netherlands ($1.0 million), FAO ($0.1 million estimated at appraisal, actual N /A) and Crop Life International ($2.1 million) (ICR, pp. 24-25). Borrower Contribution: At appraisal the ASP participant countries agreed to contribute US$ 3.9 million in kind; whereas the actual contribution the ICR reports at US$ 2.7 million or 71% of the appraisal estimate. Dates: The ASP was approved on September 8, 2005, while the six individual ASP country projects were approved on various dates from 2005 to 2007. The country projects underwent mid -term reviews in 2008 and 2009; as noted above, four of the six country projects were subsequently restructured in 2011. Because of significant implementation delays both at the ASP and at the country project levels, the original ASP closing date (June 30, 2011) was extended by two years to May 31, 2013, as were the various closing dates for the four restructured country projects (ICR, p. iv). 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial . The ASP’s objectives remain relevant to the national obsolete pesticide stockpile elimination strategies of the six participating countries (Ethiopia, Mali, Morocco, South Africa, Tanzania, and Tunisia ), all of which had ratified the Stockholm Convention on POPs and the Basel Convention on Trans -boundary Shipment of Hazardous Waste. The objectives are also relevant to the World Bank Group ’s Country partnership Strategies for these countries, and to their higher -level objective of supporting implementation of international agreements on pollution control. During the second half of the 20th century, the African continent accumulated large quantities of obsolete pesticides as a result of inadequate stock management, failure to distribute them to farmers, bans on several pesticides, and/or uncoordinated or inappropriate supply from donor agencies . Often stored close to human settlements, these obsolete stockpiles present a serious risk to human and livestock health, ground and surface water, land use, and the broader environment . Moreover, some of these obsolete pesticides include persistent organic pollutants (POPs), posing serious long-term danger to human health and the environment because of their mobility, toxicity, bioaccumulation potential, and persistence . The objectives of the ASP addressed the risks presented by these obsolete pesticides in Africa by developing and implementing cost efficient disposal and prevention activities in the six countries . Each of the countries had developed a National Implementation Plans for elimination of POPs and to contribute to their development strategies in the areas of public health, poverty alleviation, environmental protection and agriculture. The ASP was intended to assist them in the implementation of these plans . b. Relevance of Design: Modest . The ASP was designed to address the risks of obsolete pesticides by supporting country-specific disposal and prevention activities based on establishment of a partnership between the six African countries and their collaborating partners, including the World Bank and the United Nations Food and Agriculture Organization (FAO). The ASP's design, however, included a number of shortcomings that hampered effective implementation of the project from the outset: (i) the PADs at the country level were deficient in critical country-specific details necessary to ensure smooth implementation, most notably country-specific results frameworks; (ii) the uniform approach embodied in the design did not take individual country contexts sufficiently into account (templates configured at the program level were applied to all countries irrespective of context); (iii) the design of the PDO and of the components was over-ambitious; (iv) the design paid insufficient attention to specifying and defining partnership arrangements – with the exception of a set of “partnership principles,� no formal documents defined the partners’ respective roles nor the modus operandi of their collaboration; and (v) the design was built on insufficient knowledge of the specific conditions obtaining in each country – comprehensive inventories to determine the size and nature of stocks and associated disposal requirements were to be carried out once implementation had started. Furthermore, the ASP's design as an Adaptable Program Lending operation is questionable - any synergies or economies of scale anticipated by this design failed to materialize during project implementation. In all likelihood, the results achieved in three of the countries would have been more efficiently obtained by individual obsolete pesticide remediation projects in each of those countries . 4. Achievement of Objectives (Efficacy): The project development objective is to assist the Recipient (s) in: (a) eliminating inventoried publicly-held obsolete pesticide stockpiles and associated waste and (b) implementing measures to reduce and prevent future related risks. (a) Eliminating pesticide stockpiles and associated waste Modest . Outputs Ethiopia (after restructuring): Detailed inventory of publicly held obsolete pesticides, including database and risk-based prioritization, completed; Country Environmental and Social Assessment (CESA) identifying mitigation measures prepared and implemented . Assessment of treatment and disposal options completed, preparations for new storage warehouse underway, container management strategy endorsed . Mali: Mali Inventory of obsolete pesticides, including database and risk -based prioritization, completed; CESA identifying mitigation measures prepared and implemented; soil contamination surveys completed . Morocco : Inventory of obsolete pesticides, including database, completed; but neither risk -based prioritization nor CESA prepared. No collection centers established . South Africa (after restructuring): Booking line for identifying public and private -sector stockpiles established, but only one of three provincial Environmental Management Plans (EMPs) prepared, obsolete pesticides collected and transported in only one of three provinces . Study of disposal technology completed. Tanzania (after restructuring): Inventory of obsolete pesticides, including database and risk -based prioritization, completed; CESA prepared and implemented . Assessment of treatment and disposal options completed, six collection centers rehabilitated and pesticide stocks received . Tunisia (after restructuring): Detailed inventory of publicly held obsolete pesticides, including database and risk-based prioritization, completed; CESA identifying mitigation measures prepared and implemented . Outcomes The ASP disbursed 75% of its GEF resources but had disposed of only 37% of the inventoried publicly-held obsolete pesticides and associated waste by closure . The ICR presents evidence that the level of risk emanating from publicly held obsolete pesticides and associated wastes to nearby communities and critical natural resources was determined and quantified through the completion, during implementation, of national inventories in five of the six participating countries . Although this information eventually provided the Governments with the baseline data and knowledge necessary to inform eventual disposal of stockpiles, the program’s mitigation efforts resulted in disposal of inventoried publicly held obsolete pesticides in only three countries (i.e. Ethiopia, Tanzania, and Tunisia ). The remaining countries faced various project implementation challenges that prevented achievement of disposal activities (i.e. Mali, Morocco and South Africa). By closure, a total of only 3,164 of the inventoried 8,949 tonnes of publicly held obsolete pesticides and associated waste were eliminated overseas in an environmentally and technically sound manner . Following the restructuring in four countries, the obsolete pesticides were given priority for overseas disposal; the remaining associated wastes were environmentally safeguarded in country . Tanzania was the only country in which all obsolete pesticides were properly disposed of; the percentages in Tunisia and Ethiopia were 85% and 35% respectively. In South Africa, hazardous stocks were disposed of in only one out of three target provinces . In Mali and Morocco, no disposal took place . Implementing measures to reduce /prevent future risks . Modest . Outputs Ethiopia : Awareness-raising activities completed, legal /regulatory framework for pesticide management strengthened, national pesticide management strategy developed . Capacity for environmental assessment strengthened, training on improved methodologies provided . Technical and operational support provided . Mali: Mali Training provided, legal/regulatory framework for pesticide management updated, empty container strategy developed and implemented, communication plan implemented, national prevention plan developed. Technical and operational support provided . Morocco : Various awareness raising activities completed, aspects of the legal /regulatory framework improved, but national Integrated Pest Management (IPM) strategy not prepared. Limited technical assistance and training provided . Limited technical and operational support provided . South Africa : Various awareness-raising activities completed; environmental assessment capacity strengthened by training. Equipment, operational support and training provided . Tanzania : Awareness-raising activities completed; legislation to update the legal framework formulated, strategy covering pesticide management and IPM prepared . Capacity for environmental assessment strengthened; Study tour on pesticide collection and disposal provided; training in project management, pesticide inventory preparation and database management provided . Technical and operational support provided. Tunisia : Awareness-raising and communication activities completed, two legal regulations strengthening the legal/regulatory framework adopted, IPM strategy prepared and implemented, training plan for improving pesticide management practices prepared and implemented, national action plan on reducing re-accumulation of obsolete pesticides adopted . Project management capacity strengthened by training . Limited technical and operational support provided . Outcomes The ICR indicates that the original target of not accumulating new obsolete pesticide stockpiles and associated waste was too difficult to measure accurately . At any given point the amount of accumulated obsolete pesticides was difficult to determine. For example, at appraisal it was estimated that there were approximately 4,750 tonnes of obsolete pesticides to be dealt with; inventories conducted during implementation, however, revealed approximately 800 additional tonnes of obsolete pesticides in the participating countries . When the associated wastes were included the total amount became 8,949 tonnes. According to the ICR, the ASP was more successful in raising public awareness on the health and environmental hazards of pesticides and in encouraging safe pesticide handling through various outreach activities across the participating countries . The World Wildlife Fund (WWF) developed an overarching ASP communication strategy and related toolkit for the participating countries, which assisted all the countries in developing national communication strategies . However, this was actually implemented in only four of the participating countries . 5. Efficiency: The PAD did not include economic or financial analyses, arguing that the ASP was performing a public good in reducing risk to human health and the environment and that these benefits are non -measurable (p. 19). The ICR (pp.17-18) rated efficiency as "low", since (i) no accurate quantitative data were available on the financial and human resources invested in the ASP ’s preparation by any of the partners; (ii) because the Project Coordination Unit (PCU) never became operational, there was no M&E of the resources spent by ASP ’s partners; and (iii) the Bank’s operational focus on country -specific activities makes it difficult to capture adequately allocated funds and associated expenditures of a regional program with national sub -projects, ultimately resulting in high transaction costs and inaccurate reporting . The ICR also notes that the efficiency of capacity building and prevention activities is difficult to quantify since the benefits do not readily translate into quantifiable results . With respect to the efficiency of eliminating the pesticide stocks and associated wastes, the ICR determined that even though the unit cost for disposal (US$ 3,103/tonne disposed) was slightly lower than originally anticipated, the operation was inefficient because the ASP had disbursed 75% of its GEF resources but disposed of only 37% of the inventoried publicly-held obsolete pesticides and associated waste at program closing . Efficiency is rated negligible . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: While the ASP’s objectives and design were rated substantially and modestly relevant, respectively; efficacy of both development objectives was modest and efficiency was negligible . From the outset, the project had trouble delivering its outputs and at mid -term review had to be restructured in an attempt to achieve outcomes in only three of the six participating countries . For these reasons, the overall outcome is rated unsatisfactory . In the case of Ethiopia, Tanzania and Tunisia, it should be noted that risks to the environment and public health were significantly reduced through complete disposal of inventoried obsolete pesticides and effective safeguarding of associated wastes. a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR notes that the risk to the development outcome varies markedly across the six participating countries and thus has been assessed at the country -level. Tunisia : risk is rated low based on the Government ’s demonstrated commitment and capacity, as well as improved legislation for pesticide management . Ethiopia and Tanzania : risk is rated moderate and confined to the re -accumulation of obsolete pesticides, and potential weakening of the Governments ’ commitment to sustaining and building on the ASP ’s outcomes. South Africa : risk is rated moderate given the uncertainty about Government and private industry willingness to approve and implement the newly developed Integrated Waste Management Plan to collect and dispose of both private and public stocks . Mali: Mali risk is rated high due to Mali's post -conflict and transitional situation, its strong focus on agricultural intensification: and the relatively low priority obsolete stock elimination is receiving in the country's recovery plans. Morocco : risk is also rated high due to the government ’s lack of capacity and resources to safeguard and dispose of its stocks. a. Risk to Development Outcome Rating : Non-evaluable 8. Assessment of Bank Performance: a. Quality at entry: The ICR notes that the Bank and its project partners conducted numerous preparatory missions and generated considerable background information, technical analyses and plans during preparation . More than four years elapsed from the time of GEF eligibility in July 2001, to the program’s approval by the Board in September 2005. The long preparation time resulted from a lengthy partnership -building process at the program-level caused by a complex envisaged implementation and partnership arrangement. Despite this effort, however, the PADs at the country level were deficient in critical country-specific details necessary to ensure smooth implementation of the project, most notably country-specific results frameworks. At the program level as well there were challenges, particularly in finding common agreement among the partners on the ASP ’s design and implementation arrangements . Moreover, the Bank’s position during these negotiations was undermined by the high turnover of its ASP Task Team Leaders (TTLs). This difficult partnership-building process ultimately resulted in a program-level PAD that gave insufficient attention to defining the roles and responsibilities of each project partner and to specifying the nature of project collaboration, including oversight . Furthermore, the design in the PAD for the overall M&E program was weak and in the end never implemented, which resulted in shortcomings in that area (see Section 10a below). at -Entry Rating : Quality -at- Unsatisfactory b. Quality of supervision: The fact that a functioning PCU was not established, together with what the ICR describes as confusion over the partnership arrangements at the program level, impeded effective ASP implementation during the first three years and significantly affected project implementation at the country level as well . For example, lengthy discussions between the Bank and FAO on standard formats and guidelines for the participating countries delayed implementation progress . When the Bank and FAO failed to agree on establishing an interim PCU as a secretariat, the Bank team itself undertook a number of the PCU tasks in an attempt to ensure satisfactory implementation . Moreover, the Bank’s own internal implementation arrangements presented challenges because the ASP spanned two Bank administrative regions in Africa and included a program-level TTL and multiple co-TTLs, which resulted in lengthy review clearance processes of project-related documents, high transaction costs and inaccurate reporting . Efforts undertaken to solve these system-related challenges were ultimately unsuccessful . Following mid-term reviews in 2008 and 2009, the Bank team augmented its efforts to address effectively the project’s implementation delays and increase the likelihood of program success in the participating countries . Corrective actions included increased and more targeted operational and technical support through consultants, extension of closing dates, and reallocation of existing GEF funds to maximize outcomes . Project design weaknesses were partially corrected through the restructuring of the programs of four of the participating countries, by both revising targets and better aligning project activities with country -specific environments. These measures allowed Ethiopia, Tanzania and Tunisia to make substantial progress in achieving project outcomes through overseas disposal of obsolete pesticide stocks and effective safeguarding of associated wastes . Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The ICR characterizes ASP implementation as a complex partnership at the program level, hindered by substantial delays for a host of reasons, and highly varying performance at the country level, hindered by country-specific constraints. It indicates that program-level activities were terminated in 2010 because several partners had exhausted their resources but that the project continued to be implemented at the country level into 2012-13. The evidence presented in the individual country technical annexes suggests that the performance of the Governments in the six participating countries ranged from satisfactory to moderately unsatisfactory . All six Governments stated their support for the ASP and its objectives at the outset; some even had limited experience with pesticide management . Each of the Governments, however, met country -specific challenges in implementation. These included lack of high -level oversight, poor project management, inadequate capacity and insufficient resources for project implementation; and disruptive changes in institutional responsibilities. The evidence presented supports the ICR's ratings for the performance of the Governments of Mali, South Africa, Tanzania and Tunisia (all of which made progress in achieving project outcomes ) as satisfactory; Ethiopia (which had poor project management but partially achieved project outcomes ) as moderately satisfactory; and Morocco (which underwent institutional changes and ultimately withdrew from the project) as moderately unsatisfactory . Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The evidence presented in the country technical annexes suggests that the performance of the implementing agencies in the project countries varied markedly, ranging from satisfactory to unsatisfactory . Various institutional challenges impeded implementing agency performance in the six countries; including delays resulting from weak institutional capacity, lack of and /or high turnover of qualified staff, difficulties with procurement capacity, etc. The evidence presented in the annexes confirms the ICR's performance rating for the implementing agencies of Tanzania and Tunisia (both of which substantially achieved project outcomes ) as satisfactory; Mali (which made progress in achieving project outcomes until the internal conflict impeded project activities) as moderately satisfactory; Ethiopia and South Africa (both of which had project management problems and difficulty in achieving project outcomes ) as moderately unsatisfactory; and Morocco (which suffered institutional changes and failed to make progress in achieving project outcomes ) as unsatisfactory. As noted in Section 8b above, Implementation Agency performance at the Program level was unsatisfactory. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The PAD describes a complex system for program -level M&E of outcomes/results (pp. 14-15), including key indicators (pp. 6-7) and a project M&E framework with monitoring arrangements (pp. 34-38), all of which was included in the ASP’s Operational Manual. As the ICR indicates, however, there were discrepancies between the development objectives and the results framework /monitoring arrangements and a number of the indicators were vaguely formulated (e.g. "no further accumulation of new obsolete pesticide stocks ") and did not cover all expected outcomes, making it difficult to assess results effectively . Moreover, the ICR notes that country -specific M&E systems were not developed during project preparation . Thus, there were no country -level results frameworks at the time of project launch, leaving the countries with only output indicators linked to the project components. b. M&E Implementation: The ICR indicates that the ASP never established an overall M&E system at the program level throughout its implementation, largely because the PCU, whose task was to coordinate the independent M&E of the ASP, never became operational. As a fallback, the ICR notes, the project attempted to monitor implementation of activities during its regular meetings . The Bank produced comprehensive annual consolidated technical and financial reports for dissemination to its partners and donors . However, these Bank reports did not include systematic monitoring of implementation progress against the Outcome and Intermediate Outcome Indicators . Thus, it was difficult to assess overall progress in achieving project outcomes /results. Similarly, because the project had not anticipated establishing M&E systems at the country level, the participating countries operated without an effective M&E system in place until the mid -term review in 2008-2009. At that point, the Technical Support Unit (the technical assistance arm at the program level ) provided the countries with a tool for performing M&E. Some countries used the tool; others found it too complex to utilize effectively . In any case it was introduced too late in the process to be effective . For practical purposes, country -level M&E activity only began in 2011, after an M&E framework had been introduced as part of the project restructuring in four of the participating countries. c. M&E Utilization: The fact that no systematic M&E program was implemented at either the program or country levels precluded the utilization of M&E results for either meaningful measurement of implementation progress or modification of project activities for enhancing achievement of project outcomes . M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: The ASP triggered two safeguard policies : Environmental Assessment (OP 4.01) and Pest Management (OP 4.09). Environmental Assessment: In compliance with the Bank’s environmental and social safeguard policies and because of potentially significant adverse environmental impacts raised by management /disposal of obsolete pesticides, the Bank classified the ASP Category A, requiring a full environmental assessment . The PAD reports that a detailed Framework Environmental Assessment was undertaken to assess the potential environmental and social impacts and an Environmental and Social Management Framework was prepared to identify appropriate mitigation measures. Based on these documents, the project prepared an Environmental and Social Assessment Synthesis Report, discussed it in public consultations in the participating countries, and disclosed it in the Bank’s InfoShop in March 2004. The document provided guidance to the participating countries on how to prepare country-specific safeguard instruments . To this end, the ICR reports that each participating country (although elsewhere it is stated that Morocco was an exception ) then prepared a Country Environmental and Social Assessment (CESA), which included requirements for spite -specific environmental management plans (EMPs) to address potential adverse environmental and social impacts at project sites . The technical annexes state that the participating countries complied with the requirements of OP 4.01 in preparing CESAs and EMPs that ensured implementation of appropriate mitigation measures at project sites . World Bank supervision missions also reported that the CESAs and EMPs were implemented effectively . Pest Management: According to the ICR, the project did not finance the procurement of any pesticides, nor did it lead to their increased use. The project intended to reduce reliance on pesticide use by promoting integrated pest management (IPM), improving pesticide procurement and management practices and building capacity to prevent future accumulation of pesticides . To this end, the ASP provided training on promoting IPM, complying with the International Code of Conduct on pesticide distribution and use and using the Pesticide Stock Management System (PSMS) to manage pesticide stockpiles . The ICR notes that all the participating countries have access to the PSMS and that "many" (number unspecified) have continued to use it after closure of the ASP. The ICR reports that the activities proposed by the project and executed in all participating countries met the requirements of OP 4.09. b. Fiduciary Compliance: Procurement: Based on the evidence presented, the ICR rated the project's procurement performance unsatisfactory. The vast majority of project procurement was carried out at the country -level where developing and maintaining adequate procurement capacity posed problems throughout the life of the project (e.g. resulting in delays in procuring consultants to perform the national inventories of obsolete pesticide stocks, to secure overseas disposal of pesticide stocks, etc .). Details can be found in Section 2.4 in the technical annexes. The ASP’s external partners, in contrast, each used their own procurement procedures to procure necessary services, consultancies and pay for operating costs without any reported problems . There were no reported cases of misprocurement. Financial Management: The ICR rated the project's financial management moderately satisfactory throughout the life of the project. This is largely because the Bank, as the signatory of all project GEF trust fund grant agreements, as well as the CIDA and MDTF administration agreements, supervised the use of these funds through regular missions, reviews of progress and financial reports, periodic external audits, follow -up meetings, and regular contact as needed . The Bank produced monthly financial reports, which could be accessed via the Bank’s donor portal or upon request . There is no overall reporting on external project audits and whether the auditors’ opinions were qualified. In the case of some countries, it is stated that the audits were timely and “satisfactory.� In other cases, there is no information provided . c. Unintended Impacts (positive or negative): None d. Other: None 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Unsatisfactory Limited achievement of outcomes in Unsatisfactory only three of six countries . Risk to Development Moderate Non-evaluable High variability in risk levels makes Outcome : overall evaluation difficult. Bank Performance : Moderately Unsatisfactory There were major shortcomings in Unsatisfactory Quality at Entry (see Section 8a above). Borrower Performance : Moderately Moderately Significant performance shortcomings Satisfactory Unsatisfactory across the board impeded project implementation (see Section 9b above). Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The following lesson is drawn by IEG : Multi- Multi -regional, multi -country Adaptable Program Lending operations present considerable implementation challenges which are unlikely to be successfully addressed unless the anticipated synergies, economies of scale or other advantages of the project design are clearly shown to be concrete and readily achievable . Complex regional, multi-country operations like the present one necessarily carry higher risks /barriers to effective implementation than individual country -based projects. The cumulative weight of these risks can severely impede project implementation at both the regional and national levels . For this reason, a thorough evaluation of the anticipated benefits of a regional project design and the likelihood of their achievement should be undertaken before project design decisions are made . In this case, individual country operations would likely have achieved, or even exceeded, the results actually obtained, and at a lower cost in resources . The following lessons are drawn from the ICR with some adaptation of language : The Bank is not presently well equipped to handle multi -regional and multi -country programs . The Bank's internal systems are not geared for administering an umbrella program with multiple sub -projects in more than one region. Bank support for similar approaches in the future should weigh transaction costs and efficiency constraints in designing regional programs . Furthermore, the Bank's internal processing arrangements for regional/multi-national programs should be made sufficiently flexible to accommodate their characteristics and needs rather than applying coding and budget systems that hinder successful implementation . For multi -donor projects, clearly defined roles and responsibilities for all partners and their collaboration need to be agreed and operationalized prior to project effectiveness . The ASP was a complex program, which depended on close collaboration and cooperation among several partners, drawing upon their comparative expertise in pesticide management, project design, implementation, communications, and monitoring and evaluation. To ensure clarity for all partners, each entity ’s roles and responsibilities, together with guidance on how the partnership should function, should be defined and agreed on during project preparation. Concise time-bound and budget-related terms of reference for all partners, as well as clearly defined oversight and coordination responsibilities, would greatly improve project design and help to ensure smooth implementation. The dimensions of the task need to be carefully determined and sufficiently well -prepared before implementation . Project activities, such as a national obsolete pesticides inventory and a CESA, are lengthy processes but crucial for determining the dimensions of the disposal task ahead . Ideally, these activities should not be conducted during implementation as was done in this case; they should either be conducted during preparation or during a first implementation phase, followed by a second that comprises adequately timed and budgeted disposal activities . This could include an approach under which the inventory and the CESA are prepared simultaneously. 14. Assessment Recommended? Yes No Why? There may be important lessons to be learned from the ASP's experience . 15. Comments on Quality of ICR: The ICR is comprehensive and candid in its presentation and analysis of the evidence generated by this complex multi-regional, multi-national environmental remediation project . However, although its focus is results-oriented, its presentation and analysis of the project ’s various outputs and outcomes are not always clear, particularly for the country -level projects. There is greater clarity in its analysis of the performances of the Bank and the Borrower and in its assessment of project M&E . The presentation and analysis of project details are consistent throughout, if complicated by the multi -national nature of the project . Although exhaustive in number, the lessons identified are based on a careful analysis of the challenges and insights presented by the complex project. A clear summary discussion of external project audits at both program and individual country level would have been helpful. a.Quality of ICR Rating : Satisfactory