FILE COFY Report No. 4104 Annual Report on Operations Evaluation September 3, 1982 Operations Evaluation Department FOR OFFICIAL USE ONLY U Document of the World Bank This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. FOR OFFICIAL USE ONLY THE WORLD BANK Washington, DC 20433 U S A Office of Director-General Operations Evaiuation September 3, 1982 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Annual Report on Operations Evaluation Attached, for information, is this year's "Annual Report on Operations Evaluation." The report reviews the current status of the process by which the Bank, in its continuing search for ways to improve the effectiveness of its development assistance, evaluates the results of its operational activities and communicates them to its members and its staff. The report also contains a summary of the current status of the recommendations in OED major policy reviews and evaluation studies. Mervyn L. Weiner by Shiv S. Kapur Attachment This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. FOR OFFICIAL USE ONLY ANNUAL REPORT ON OPERATIONS EVALUATION Table of Contents Page No. I. Introduction 1 II. The Project Performance Audit Reporting System 1 Borrower's Role 4 III. Monitoring and Evaluation in Projects 4 IV. Post-evaluation in Member Governments 7 V. The Operations Evaluation Department 8 VI. Other Evaluation Activities 10 IFC 10 Research 11 Economic Development Institute 11 VII. Follow-up 12 Annex A: Modified Project Performance Audit 13 Reporting System Annex B: Regions' Status Report on Borrower 16 Governments' Involvement in the Post- Evaluation of Bank/IDA-Financed Projects (All countries with five or more PCRs scheduled through CY83) Exhibit 1: Operations Evaluation in Mexico 23 Annex C: Regions' Status Report on Borrower 26 Governments' Involvement in the Post- Evaluation of Bank/IDA-Financed Projects (Selected countries with less than five PCRs scheduled through CY83) Annex D: The Current Status of Recommendations of 30 OED Policy Reviews and Evaluation Studies Annex E: OED Reports Issued During FY82 55 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. ANNUAL REPORT ON OPERATIONS EVALUATION Abbreviations used JAC Joint Audit Committee OED Operations Evaluation Department PPAR Project Performance Audit Report PCR Project Completion Report CPS Central Projects Staff OPS Operations Policy Staff IFC International Finance Corporation EDI Economic Development Institute M&E Monitoring and evaluation IRMD Information Resource Management Department ANNUAL REPORT ON OPERATIONS EVALUATION I. Introduction 1. The purpose of this report is to review recent develop- ments in the Bank's operations evaluation activities and the status of still outstanding recommendations of past reports of the Opera- tions Evaluation Department (OED). The report assumes that readers are familiar with "Operations Evaluation: World Bank Standards and Procedures," Second edition, August 1979. Details of the earlier evolution of operations evaluation in the Bank are available in past Annual Reports: R76-245 cf October 12, 1976 to R81-246 of September 14, 1981. II. The Project Performance Audit Reporting System 2. The operations evaluation process has now evolved to the point where a selective procedure for project performance audits has become appropriate. Details of this new procedure were set forth in the OED Work Program and Staff Budget for FY83 approved by the Joint Audit Committee (JAC) on April 14, 1982 and by the Executive Directors on May 4, 1982 (R82-109). The new system requires that all PCRs will continue to be submitted to and read by OED staff and be issued to Executive Directors and the President. Although only a selected number of PCRs will henceforth be subjected to perfor- mance audit by OED, all PCRs, or the draft project performance audit report, as the case may be, will continue to be submitted to borrow- ers and their comments will be reflected in the final document released to the Executive Directors and the President by the Director-General. The new system will facilitate the audit of groups of projects, thus enabling OED more effectively to examine topics such as experience with the sector policy and institution building objectives of Bank operations. The fuller description of the modified PPAR system contained in the OED budget document is annexed (Annex A) for ease of reference. 3. In FY82, 159 PCRs were received and PPARs for 122 projects were issued, almost equalling the 124 PPARs issued in FY78, the highest number yet. Although original estimates of PCRs to be received and PPARs to be issued in a given year have often been substantially different from past actuals, these estimates have been improving and the deviations from actuals have been declining. For FY82, the estimate of PCRs contained in the Bank's Administra- - 2 - tive Budget happened to almost coincide exactly with the number received by OED. (Some deviation between any single year's esti- mated and actual number of PCRs received and PPARs issued may be considered a normal consequence of unexpected changes in the sched- uling of other activities in the Regions and in OED.) The table below shows the status of PCRs received from FY79 through FY82 and the estimate for FY83. It shows, as expected, an upward trend of PCRs received and brought forward as well as of PPARs issued. This upward trend may be expected to continue for the next few years. Status of PCRs Received Actual and Estimated FY79 to FY83 a/ A c t u a 1 Estimated FY79 FY80 FY81 FY82 FY83 PCRs Brought Forward 67 106 87 136 173 c/ PCRs Received 130 90 143 159 185 Total 197 196 230 295 358 d/ PPARs Issued 91 109 94 122 185 e/ 106 87 136 173 173 a/ The FY83 estimate is based on the numbers of loans and credits with closing dates on or before the end of Calendar Year 1982 and which were more than 85% disbursed, adjusted as a result of discussions between OED and Projects staff. b/ At various stages in the audit process. c/ OED estimate, adjusted in conformity with Regional projections. d/ Based on the proposed modified audit system, which envisages for FY83, 105 PPARs and 80 PCRs issued without audit. e/ The carry-over would be substantially greater in the absence of modified audit system. -3- 4. Guidelines to OED staff on the procedures to be adopted in selecting PCRs for audit have been prepared and are now being applied. They will shortly be reviewed by the JAC. Experience with selectivity will also be reviewed periodically with the JAC. In the past, note has been made in Annual Reports of the varying quality of PCRs. Recent instructions to Regional staff by the Senior Vice President, Operations, with regard to the procedures to be followed in the preparation of PCRs under the modified PPAR system make it clear that the responsibility for PCR quality rests with Regional management. 5. Since 1977, Sub-Committees of the Joint Audit Committee have reviewed a number of PPARs each year to assess the adequacy of the PPAR system. In respect of FY82, the Sub-Committee reviewed six PPARs and also examined the follow-up actions taken by management and OED on the recommendations made by former Sub-Committees (JAC/82- 9, June 16, 1982). One of the general recommendations made by the Sub-Committee and endorsed by JAC was that the findings of PPARs should receive wider dissemination in the Bank and that the proce- dure adopted for this purpose by some operational policy departments should be adopted by others. 6. While the established formal mechanisms for the review and dissemination of the major findings contained in OED reports provide for effective dialogue between OED and senior operational staff, they do not yet ensure uniform dissemination throughout the Bank. This is a subject which OED has under review and plans to comment on more fully in next year's Annual Report. 7. The cost to the Bank of the project performance audit process in FY82 continued to approximate 14 staffweeks per project (10 and 3.8 staffweeks for the PCR and PPAM respectively). In FY83 the cost of preparing each PCR is expected to remain at 10 staff- weeks and, for preparing audits in respect of those projects select- ed for audit, about 6 staffweeks. The time to be spent by OED staff on PCRs not audited is estimated at half a staffweek for each PCR, resulting in an overall FY83 average of 3.6 OED staffweeks per PCR. The overall cost to the Bank of the audit process for FY83 is estimated at about 52 staffyears, comprising 35 for PCR preparation and the balance for the OED input. This compares with the program of 403 staffyears for other supervision work. 8. The Office of Environmental Affairs of the OPS Project Advisory Staff (OEA) continues to examine the environmental aspects of projects where relevant. Projects due for audit are reviewed by OEA to identify those likely to have environmental implications, and -4- periodically OEA sends a list of such projects to OED. Eight PPARs distributed to the Board in FY82 had been reviewed by OEA, of which five were agricultural. projects, two public utilities and one DFC. Borrower's Role 9. While few PCRs are as yet completed by borrowers, there has been an increase in FY82 as against the previous year. Thus, 22 PPARs issued in FY82 included PCRs 1/ prepared by borrowers compared with 10 in FY81, 8 in FY80 and 2 in FY79. As noted in paragraphs 18 and 19 below, an increasing number of borrowers are now contributing substantially to their preparation and more borrower PCRs may thus be expected in the coming years. Comments on project experience continue to be made to OED by most borrowers, either to OED missions or by correspondence (78% of the PPARs issued in FY82 reflected such comments). OED country visits were made to 43% of the projects for which PPARs were issued in FY82; this percentage is higher than the one-third indicated in the OED Work Program and Budget and reflects missions to single countries covering several projects and missions which occurred in FY81 which resulted in PPARs for FY82. Borrowers' written comments, when received, continue to be reproduced in full in PPARs. 10. An undertaking was given at the meeting of the Executive Directors which considered the Report on Operations Evaluation for FY81 to present annually henceforth a report on the status of borrower involvement in the completion reporting process in respect of each borrower with a substantial number of completed Bank/IDA- financed projects coming up for review. The first such status report was included in last year's Report and the second is present- ed in Annexes B and C of this Report. III. Monitoring and Evaluation in Projects 11. Monitoring and evaluation (M&E) elements continue to be included in Bank projects in certain sectors. Thus, in the agricultural and rural development sector, as noted last year, while M&E was a component in only 20 (60%) of the 33 projects in the sector 1/ In certain cases with Bank staff revisions. - 5 - approved in FY74, it was an element in 68 (80%) of the 86 such projects approved in FY81. In FY82, 61 (88%) of the 71 projects in this sector contained an M&E component and 22 made explicit finan- cial provision for these activities. Thus 45% of FY82 projects included a costed M&E component; this compares with 35% of projects in FY81. OED has so far prepared three reports on experience with the M&E components of projects: the first report dealt with six projects in the agriculture sector, the second with experience in six education projects and the third with eight irrigation projects. 1/ The three reviews already completed pointed to rather disappoint- ing experience with this component, while recognizing that some useful work had been done and that M&E components had potential for contributing substantially to improving future projects. As indi- cated below, steps have since been taken and more are under way to make M&E a more effective tool of project management. 12. Design Guidelines and a Handbook for Monitoring and Evaluation of Agriculture and Rural Development Projects were issued by Central Projects Staff in November 1981. The Handbook has been supplied to all projects in this sector supported by Bank operations that include provision for M&E. It will shortly be published by Johns Hopkins Press and thus be available to a wider audience. 13. A detailed Case Study of the Philippines referred to in last year's Annual Report entitled "Monitoring Systems and Irriga- tion Management" has been completed by the Agriculture and Rural Development Department of Central Projects Staff (now Operations Policy Staff) and is shortly to be published. That same Department is also preparing a series of shorter Case Studies covering four other examples of M&E systems in Thailand (Tree Crops), Honduras (Rural Credit), Liberia (Area Development) and Zaire (Livestock). 14. The growth of interest in M&E by the borrower executing agencies is also evidenced by the growing number of requests for advice and operational support on the design and implementation of these systems, which are handled by a group of Operations Policy specialist staff. 15. In the urban sector, reporting on the joint effort under- taken by the Bank and the International Development Research Centre (IDRC) in Canada was nearly completed during the year. A report comprising the program's principal findings will soon be published as a Staff Working Paper. During the course of the year, final evaluation reports have also been produced on the El Salvador and Zambia projects, and both will shortly appear as Staff Working 1/ SecM77-745, SecM79-788 and SecM81-91 respectively. -6- Papers. A draft final report on the First Philippine Urban Project has been completed. In addition, IDRC plans to publish an annotated bibliography of all research reports produced by the evaluation teams in El Salvador, Zambia, Senegal and the Philippines and by staff of Development Economics Department, Urban and Regional Economics Division (DEDRB). 16. As a result of a Bank reorganization, responsibility for the coordination of monitoring and evaluation of urban development projects has passed from DEDRB to the Urban Development Department, and the process of completing the work is continuing there. A draft set of manuals on the methodology for the evaluation of urban development projects has been produced and is currently being reviewed. A book-length manuscript by a Bank staff member on the experience to date is also being reviewed. In addition, work continues on the design and general implementation guidelines for Bank-financed urban shelter programs. 17. The education sector is continuing its systematic efforts to build M&E activities into projects at the design stage and to draw lessons from evaluation activities conducted in past projects. In FY82, 16 of 19 projects included M&E components (about the same percentage as in FY81 when 15 of 18 projects included such a compo- nent) and 11 of these projects included funding for the component. The Education Department has continued its program of training by holding workshops for education sector staff and by participating in EDI courses for project managers. Evaluation specialists in the department consult with sector staff on the design of project M&E activities early in the project cycle, in Washington and when feasible in the field. Work has begun on preparing guidelines for incorporating evaluation activities in projects. A second group of activities centers on drawing lessons for the future from the results of earlier evaluation work. The first such effort will be a review of the more than 1.25 tracer studies that have been included in past education projects. This review is expected to yield not only substantive knowledge, but also valuable procedural information about how to improve the design and implementation of such studies. Finally, the Education Department has under way a research project, the DiSCus Study, which will assess the impact and effectiveness of diversification of secondary education of Bank-financed projects in Colombia and Tanzania. IV. Post-evaluation in Member Governments 18. Evaluation efforts by member countries, as might be expected, vary considerably both as regards individual countries and as to sectors within eac' country. But, as will be seen from Annexes B and C, there has been progress in some areas. Thus, for example, it is expected Lhat PCRs for certain sectors will be prepared by the countries themselves (as many as ten each in Indone- sia and the Philippines). In Mexico (details of whose evaluation arrangements are contained in Exhibit I to Annex B) heavy involve- ment in the PCR process is expected in tourism and irrigation projects. Provision in Bank projects to facilitate a country's own evaluations is expected to contribute substantially to local efforts in agricultural projects in Liberia and Nigeria, and for development projects generally in Thailand. 19. Annexes B and C show that agencies of borrowers are not generally obliged to account for the performance of projects under their jurisdiction to their own governments, although a number of borrowers are now seeking to correct this deficiency in their development management arrangements. Although this matter is not yet a common part of the Bank's operational dialogue with borrowers, there is good reason for staff to discuss with more borrowers how the Bank might help them address more systematically the key invest- ment performance questions, which are no different for borrowers than for the Bank: (i) in what ways did the completed project differ from the original proposal and why?; (ii) was the project as completed still a "worthwhile" investment for the borrower bearing in mind the actual cost and re-estimated benefits?; and (iii) what are the key lessons of the project experience for both the borrower and the Bank? Project implementation reviews are a start, but only a start, in this direction. The perception of some borrowers and their agencies (and also some Bank staff) that the Bank expects borrowers to emulate its own project evaluation arrangements and reporting formats is unwarranted. It is also unfortunate if it has the effect of inhibiting borrowers from formulating their own answers to these questions in their own way for inclusion in PPARs. 20. Post-evaluation continues to be a normal component of EDI's sector and national economic management courses for member government officials. V. The Operations Evaluation Department 21. Last year's Annual Report on Operations Evaluation (R81-246) was issued on September 14, 1981, and the Seventh Annual Review of Project Performance Audit Results (R81-265) on October 14, 1981. The Annual Review, with only the names of countries and projects removed, was issued for the general public in December 1981 following the precedent set in respect of the Reviews issued in 1977 and subsequent years. This year's Annual Review will be issued shortly. 22. The findings of the 780 project performance audits on which reports were issued by OED through June 30, 1982 have been classified and entered into OED's computerized memory and retrieval system, which has now been in place for more than three years. The first computerized Concordance to Project Performance Audit Reports was issued on August 31, 1978 (SecM78-699). Updated versions of the Concordance covering project audits and other reports of OED have been issued periodically, most recently on September 30, 1981 (SecM81-830). A further revised and updated version of the Concor- dance is expected to be published shortly. A study has been com- pleted in OED, with the cooperation of the Information Resource Management Department (IRMD) and in consultation with some user departments, to expand and improve upon the system with a view to making these findings of the Bank's extensive operational experience more readily available for use by staff. The Bank's capital and systems development budget for FY83 contains provision for funds to enable work to begin on the expanded system during the current fiscal year. Some delay in implementing the expanded system is anticipated, partly as a result of the recent reorganization of IRMD whose cooperation in formulating and putting in place an appropriate system is of critical importance. 23. During FY82, OED issued seven special studies, among which were four Impact Evaluation Reports in the "second look" series: one on the first livestock project in Kenya (SecM81-806) and three on irrigation projects--the Muda and Kemubu projects in Malaysia (SecM81-713), the Lake Alaotra project in Madagascar (SecM81-736), and the San Lorenzo Irrigation and Land Settlement project in Peru (SecM82-485). A fifth Impact Evaluation Report--on the Atlantico Irrigation project in Colombia (SecM82-662)--was issued on July 15, 1982. In addition, the following studies were published: an Interim Report on Procurement Issues issued on July 15, 1981 (SecM81-636); Sector Operations Review: Agricultural and Rural Development Program in the Philippines (SecM82-118); and a Review of Training in Bank-Financed Projects (SecM82-163). All the foregoing studies have been reviewed by the JAC. A report on Compliance with Loan Covenants has been completed and was distri- buted to the Executive Directors on September 2, 1982 (SecM82-782). -9- 24. Reports on Strengthening Agricultural Research and Extension: The World Bank Experience, and the Water Supply and Waste Disposal Program in Tunisia are both at an advanced stage of preparation and are expected to be distributed shortly. In addi- tion, work is under way on the first of a proposed series of Reviews of Experience in Institution Development, on a review of Smallholder Livestock Development, and on the third of a series of Country Sector Operations reviews in agriculture (Malawi/Upper Volta). It is also planned to complete six further Impact Evaluation Studies in the current fiscal year. 25. The OED work program and budget for FY83 contains (in Annex B) an indicative compilation of topics on which choices about themes and subjects for future studies are to be based. As approved by the JAC, a three-year rolling program of special studies is being drawn up in consultation with other departments in the Bank, including the Country Policy Department (CPD), with a view to integrating better OED's work planning with the Bank's operational policy and research planning. The proposed program of studies will be subject to JAC review and approval at the time of considering OED's work program and budget for each fiscal year. 26. The OED Budget for FY83 was formulated on the basis of zero staff growth apart from the addition of one secretarial posi- tion. A modest provision has been made in the capital budget for the improvement of the computerized memory and retrieval system referred to in paragraph 22 above. Accordingly, OED's authorized staff consists of 28 professionals, 5 research assistants and 20 support staff. Previous Annual Reports have noted the difficulties in the rotation in and out of OED. In the past year, the situation has improved somewhat. As of mid-August 1982, three vacancies existed in the Department. and a fourth will occur in late September; all these arise from the recent transfer of staff out of the Depart- ment into the operational mainstream of the Bank. As noted last year, there persists in the minds of many staff a feeling that a transfer to OED may pose special problems for them later on reassign- ment. Efforts continue to be made, with the help of PMD, to correct this perception, which has no basis in fact, and to stress the value of a period of service in OED as an important contributing factor to career development. Meanwhile, OED has recently recruited three staff from outside the Bank. 27. Staff of member governments continue to visit OED. In FY82, seven officials from five countries visited OED for periods generally ranging from one to two weeks, although in certain cases for a considerably longer time, on programs which exposed them to the project cycle as experienced in the Regions, OPS and OED. Two officials, one from France and the other from Germany, worked with OED for six-month terms during the calendar year 1982, one in FY82 - 10 - and the other in FY83; further such assignments are anticipated, as are joint evaluations with co-financiers of projects such as with Caisse Centrale of France for three Oil Palm projects in the Ivory Coast and the German Federal Ministry for Economic Cooperation for the Phosphate Fertilizer project in Morocco. In addition, liaison with other development agencies continues as regards the evaluation process, particularly with the Regional Banks. VI. Other Evaluation Activities 28. Although OED does not now play an active part in the other evaluation activities of the Bank, previous Annual Reports have briefly noted these activities and their current status. The following paragraphs contain an updating of these activities in various specific areas. In addition, the Bank's operations policy staff continue to review experience in certain sectors and policy matters in accord with their broad mandate which includes "evalua- tion of the effectiveness of Bank operational work, and proposal of measures to improve it." (Organizational Manual Statement No. 1.20, para. l[g], August 1982). Strategy sector papers on such topics as education, transport and water supply, as well as reports on poverty redressal and urban development operations, have either been com- pleted or are under way and on completion are distributed widely to Bank staff. IFC 29. IFC continues to prepare and send to OED all completion reports it prepares on its projects. In FY81, IFC prepared fifteen such reports, covering various industrial projects including mining, engineering, steel, cement, paper and fruit juices in twelve coun- tries; in FY82 ten completion reports were prepared on various industrial projects in eight countries. These reports represent an advance on earlier work, particularly in respect of the analysis of lessons drawn from experience, as applicable to the Corporation's future work. 30. IFC is currently examining its experience in the cement industry for the purposes of its second sector evaluation report. For the first time, IFC will base its evaluation on completion reports prepared by it--nine reports covering ten cement projects in eight countries. The terms of reference of this study was discussed with OED staff. Among other aspects, the study will analyze the role of transportation costs and the impact of state intervention on the development of the industry. The report is in the final stages of completion. - 11 - Research 31. The most recent report on the World Bank research program (R82-85 and IDA/R82-36 dated March 24, 1982) was discussed at a meeting of the Executive Directors on April 13, 1982. The report came at a time when the research work of the Bank was being reorga- nized and greater emphasis given to applied research as well as a major shift in the locus of the management of research. Applied research is now the shared responsibility of the operations complex and the Economics Research Staff (ERS) and will account for an increased proportion of all Bank research work. As noted in earlier Annual Reports, the need to relate Bank research more effectively to Bank operations was a major finding of the external panels set up to evaluate Bank research. 1/ The changes referred to above respond to this finding. 32. The report summarizes the evaluations of eleven research projects undertaken in FY81, bringing to 120 the total number of research projects evaluated to date. OED continues to be repre- sented on the internal evaluation panels responsible for evaluating completed research projects. Economic Development Institute 33. EDI has continued its practice of evaluating each of its courses at its conclusion at which time participants are asked their opinions about the course content and its relevance to their work, the quality of the teaching and the teaching materials distributed, and suggestions on how future courses could be improved in the light of the experience gained from the present course. EDI has not yet undertaken systematic tracer studies on its fellows although a few ad hoc studies have been undertaken. No full-scale tracer studies are as yet planned. 34. A fifth evaluation study on EDI's training activities overseas was completed during the past year. This study was carried out to evaluate the effectiveness of a training program in which EDI is collaborating in the Philippines. A report is presently being prepared by the consultant who carried out the study and will be summarized in EDI's next Annual Report (which is expected to be circulated shortly) together with the evaluation of EDI's program in Pakistan which was completed in FY81. 1/ Report of the General Research Advisory Panel (R79-221, August 1979). - 1) - 35. A task force has been established by the Managing Commit- tee under the chairmanship of Mr. Shahid Husain, Vice President of East Asia and Pacific Region, to review the objectives and operating procedures of EDI and to suggest the lines of further development in the eighties. The task force is addressing such matters as the objectives of EDI and how they relate to those of the Bank, the quality and relevance of its courses, and the adequacy of the institute's institutional building efforts. The task force is expected to complete its work by October 1982. VII. Follow-Up 36. The PPARs (which include PCRs) for each calendar year are reviewed on a sectoral basis at meetings arranged by OPS and attend- ed by representatives of the Regions, OPS and OED staff. Under the modified PPAR system, this procedure will continue to be followed and, in addition, OED's Annual Review of Project Performance Audit Results will review all PPARs and PCRs issued in the preceding calendar year. 37. Since the 1978 Annual Report on Operations Evaluation, the main recommendations of OED special studies and management's reactions to them have been set forth in an Annex to each Annual Report. Issues stemming from the recommendations to which manage- ment has not fully responded have been repeated in subsequent Annual Reports. Annex D of this Report provides information on the status of the recommendations contained in the OED study on The Supervision of Bank Projects (SecM8O-129, February 22, 1980) where action or comment by the Bank remains to be indicated. In addition, the Annex contains the principal recommendations of the three major OED studies distributed in FY82, together with management's reaction to them, as well as an abbreviated listing of the more generalized issues emerging from recent Annual Reviews of Project Performance Audit Results. -13 - ANNEX A Page 1 MODIFIED PROJECT PERFORMANCE AUDIT REPORTING SYSTEM If the increases in the number of completed projects expected in the next few years are to be accommodated without some combination of still further increases in the allocation of OED staff for their review and a further reduction in the resources for special studies, OED's role in the PPAR system must become even more selective than it is now. There are two ways in which this could be done. One would be by a more aggressive OED effort to increase the proportion of abbreviated audits beyond the 30% level at which it is now. The other would be by sending certain completion reports, especially those now selected for abbreviated audit, to Executive Directors without any OED review. For the reasons outlined below, the second alternative is recommended. The immediate budget benefit of eliminating abbreviated audits would be minor--less than one staff year (of 185 projects for audit in FY83 and some 80 projects [43%] assumed as abbreviated--see para. 18--the savings @ 2 days saved per project would be 32 staffweeks). However, the potential benefits in strengthening DED's substantive evaluation work should be considerable once the principle of selective audit becomes clearly established and is found to function well. A disengagement from OED's present practice of auditing the performance of all completed projects need not jeopardize nor be perceived to jeopardize the integrity of the project evaluation system. There is ample evidence that project completion reporting by operational staff is now firmly established in the Bank and is, for the most part, comprehensive and frank; that gross inadequacies in completion reporting or in identifying issues of general interest can usually be caught by mature and experienced evaluation officers just reading a PCR (although such a reading without reference to operational files would be unlikely to identify issues that may have been overlooked); and that the policy of 100% OED review, which was important during the years in which comprehensive reporting was becoming established, is not needed any longer just to confirm that PCR authors respect the Bank's professional reporting standards. PCR authors will know that all their reports will continue to be read in OED and by the borrower, will be sent to Executive Direc- tors, and may be selected by OED for audit. 1/ Extracted from the FY83 OED Work Program and Staff Budget (R82-109). - 14 - ANNEX A Page 2 Recent JAC Subcommittees to review PPARs have also raised questions about the value of abbreviated performance audits; and OED staff engaged in such audits have been concerned for some time that PPAR readers do not always distinguish abbreviated from more tho- rough OED reviews and therefore sometimes consider abbreviated PPARs to be less thorough than they should be, even though the distinc- tions are explicit in each preface. Some outsiders have also misunderstood OED's policy of auditing every completed project to mean that each one is subjected to an in-depth performance audit, which has not been the case for some while. For all these reasons, it seems timely to reconsider the policy of 100% OED performance audit. A modified approach is accordingly proposed below. It should lead to a more focussed and effective performance auditing system and thereby enhance the evidence which the Bank now provides to member governments and the public about the effectiveness of its development assistance. The Bank's management has no objection to the proposal. The modified PPAR system hereby proposed would provide for the following: - a PCR with Preface, Basic Data and Highlights would be prepared by the Region and/or the borrower on every completed project; - each PCR would be read in OED to determine whether or not the project should be selected for audit. Projects selected for OED audit would be handled as they are now with the abbrEviated classification eliminated and the preparation of in-depth reviews increased. For all other projects, however, OED would advise the Region to seek forthwith the borrower's comments on the PCR and then reflect these comments in their PCR text. The final PCR, with a copy of the borrower's comments annexed, would be sent by the Director-General to the Executive Directors and the President. Each Preface would indicate whether or not the project had been subjected to OED audit; - OED would continue to prepare an annual review of all PPARs and PCRs issued during the preceding calendar year, and thereby preserve intact the benefits of the present comprehensive project evaluation system. -15 - ANNEX A Page 3 OED's proposed FY83 budget (Table 2) has been prepared on the assumption that the modified PPAR system will be endorsed, and that roughly 43% of the FY83 PCRs would be issued without OED audit, compared with the current 30% proportion of abbreviated audits. As a start, the criteria for selecting PCRs for issue without review would be the same as for abbreviated audit now, i.e., no significant time or cost overrun or shortfall in re-estimated economic return and no significant issues of general interest, or an acceptable analysis of the project experience in the PCR. Of these criteria, however, special weight will be given to the quality of the PCR to determine if the project should be reviewed by OED. Experience with this new arrangement would be subject to review by Executive Direc- tors at least three times each year on the basis of (i) OED's Annual Report on Operations Evaluation, (ii) OED's draft work program and budget and (iii) the report of the JAC Sub-Committee on PPARs. - 16 - ANNEX B Page 1 REGIONS' STATUS REPORT ON BORROWER GOVERNMENTS' INVOLVEMENT IN THE POST-EVALUATION OF BANK/IDA-FINANCED PROJECTS (All countries with five or more PCRs scheduled through CY83) A. Eastern Africa (1) Tanzania - Seven PCRs in three sectors (agriculture, IDF and urban) are scheduled in CY83. Significant participa- tion by the project entities is expected in the two pro- jects in industrial development finance and urban. Bor- rower participation in monitoring and evaluation for agricultural projects has been quite limited and the extremely difficul; economic situation in the country make prospects for their increased participation in the future bleak. B. Western Africa (1) Liberia - Five PCRs in four sectors (agriculture, educa- tion, highways and power) are planned through FY83. The Planning and Programming Department of the Ministry of Public Works (MPW), with the technical assistance provided under the Third and Fourth Highway projects, has prepared a preliminary PCR fo:. the Third Highway Project which concen- trates on the physical implementation of the project and the ex-post economic evaluation. MPW has been requested to prepare additional information on performance of consul- tants and contractors. MPW will also prepare a PCR for the Fourth Highway Project. The monitoring and evaluation unit set up under the Lofa County ADP I has provided limited assistance with the preparation of the PCR for this project. However, under the recently approved Lofa County ADP II, a central Govern- ment monitoring and evaluation is to be established. With the experience gained under Lofa I and the establishment of the central unit, future PCRs for agricultural projects should be prepared by Liberian staff with the assistance of Bank supervision missions. The Bank expects that the project implementation unit for the education projects will prepare basic data and some background papers on key project elements; limited Govern- ment assistance is also envisaged in the preparation of the PCR for the Power IV project. - 17 - ANNEX B Page 2 C. East Asia and Pacific (1) Indonesia - Preparation of thirteen PCRs in six sectors (agriculture, education, energy, transport, urban and OPS-population) is scheduled through FY83. Project agen- cies are expected to prepare ten of these PCRs. The degree of involvement by the borrower in the preparation of PCRs has varied considerably among projects depending primarily on the capacity of the implementing agency. In general, the burden of preparing PCRs in draft (rather than only assembling data) has shifted increasingly to the borrower as the capacity of these agencies has improved. For example, while the contribution will continue to vary from agency to agency, the initial expe- rience with some of them in the agriculture sector has been encouraging. Such involvement has been common in the power sector for some time. There has not been, however, any substantial involvement by the borrower in the review of PCRs. There is no established procedure for review by a central agency such as the National Planning Agency, and the feasibility or desirability of establishing such a procedure has not yet been discussed with the Government. (2) Thailand - There are five PCRs in four sectors (agricul- ture, education, IDF and OPS-energy) scheduled for prepara- tion in FY83. Four of these PCRs are expected to be prepared by project agencies. The Government of Thailand continues to play an increasing role in the preparation of PCRs and the implemen- ting agencies now provide much of the input for those prepared by Headquarters staff. Provision was made under SAL I for the Government to establish improved mechanisms for monitoring and evaluation of all development projects and the Bank and the Government are now preparing a joint project processing system to ensure better coordination during all phases of the project work. (3) Philippines - Preparation of eleven PCRs in five sectors (agriculture, education, energy, IDF and transport) is expected in FY83. Project agencies are expected to prepare ten of these PCRs. - 18 - ANNEX B Page 3 The Philippine Government has directed that all PCRs prepared by the Executing Agencies should be sent to the National Economic and Development Authority (NEDA) which reports to a Board that is a sub-committee of the Cabinet. NEDA will review the PCRs and draw therefrom lessons that are of wider relevance. Review procedures are being developed and the Government intends to seek OED's assistance in strengthening the post project performance review system and in establishing appropriate standards for such reviews. We expect that each forthcoming PCR would include NEDA's comments on the report. D. South Asia (1) Bangladesh - Sixteen PCRs in various sectors (agriculture, education, transport, industry including one technical assistance credit,, two imports program credits and two fertilizer imports credits) are presently scheduled. The Planning Commission and the External Resources Division will be responsible for the coordination of PCRs, all of which are prepared by the Planning Cells in the various ministries and agencies concerned, with varying amounts of Bank assistance. Each forthcoming PCR will describe Bangladesh's involvement in its preparation. Our expe- rience during FY82 indicates a general need for improvement in the quality of the PCRs done by the respective agencies. (2) India - Fourteen PCRs in six sectors (agriculture, irriga- tion, education, power, urban and water supply) are sched- uled for completion in 1983. Three of these PCRs are expected to be prepared by project agencies concerned with Bank staff inputs. The others would be prepared by the Bank staff based on material provided by project authority. The Department of Economic Affairs of the Ministry of Finance is the focal point for evaluation and feedback of project experience. - 19 - ANNEX B Page 4 (3) Pakistan - Nine PCRs in four sectors (transportation, industry, power and irrigation) are scheduled through 1983. Initial analysis and data collection for all of these will be done by the project agencies concerned. Assistance from the Bank staff will be needed. Our experience in FY82 was that the industry and power sectors do quite well in preparing draft PCRs and only minor assistance is needed from the Bank staff. In the other sectors, somewhat greater amount of input from the Bank staff may be required to finalize PCRs. The Planning and Development Division of the Ministry of Finance and Economic Affairs has responsibility for post-evaluation work, with the concerned sections of the Planning and Development Division as well as the technical ministries/ agencies involved taking responsibility for reviewing the PCRs and giving their comments. Follow-up action is undertaken as considered appropriate. E. Europe, Middle East and North Africa (1) Algeria - Five PCRs are scheduled for completion in five sectors (education, highways, railways, industry and DFC). The Ministry of Public Works is expected to partici- pate in a limited manner in PCR preparation for the Third Highways Project (Loan 1407-AL). SONELGAZ, the borrower in the power sector, has willingly participated in PCR prepa- ration but their involvement did not cover institutional and financial matters. (2) Egypt - Seven FCRs are scheduled for completion in five sectors (railways, agriculture, education, power and industry). Having aided the Bank mission in the collection of information for one PCR, the Drainage Authority has now 1/ A Handbook of Instructions for Performance Audit was published by the Auditor-General of Pakistan in 1981 to assist in develop- ing a process of systematic performance audit of development projects and programs. The Handbook discusses in some detail the different stages of the project's life including, for example, identification, preparation, appraisal and implementa- tion and noting relevant checks to be applied while undertaking the exercise. - 20 - ANNEX B Page 5 taken on the responsibility of preparing the second for the series of projects in the subsector. In the case of railways, Egyptian Railways has aided in the compiling of information on the two projects (Credit 284-UAR and Loan 1098-EGT) they were involved with and the Rural Electrifi- cation Authority has begun preparing a PCR on the Rural Electrification Project (Loan 1453-EGT). (3) Morocco - Six PCRs are scheduled for completion in three sectors (irrigation, power and DFC). The Office National de l'Electricite assisted in the preparation of the Power I PCR (Loan 936-MOR) but their report did not cover the broad objectives of the project. The Bank expects greater and more analytical involvement in the Sidi Cheho Hydropower Project report (Loan 1299-MOR). (4) Syria - Five PC:Rs are scheduled for completion in four sectors (education, railways, power and water). The only borrower involvement at this time is in the power sector, although cooperation can be expected from the Water Supply Establishment in Aleppo and Damascus. Although not required, the Public Electricity Establishment has had its consultants for the Power Station under the First Mehardeh Thermal Power Project do a report; however, as is usual in such cases, considerable Bank staff assistance to the borrower may be required to complete the PCR covering the other components of the project. (5) Yugoslavia - Five PCRs are scheduled for completion in two sectors (agriculture and highways). In the agriculture sector, even though not obliged to do so by the Loan Agreements, borrowers have and are preparing PCRs of acceptable quality in accordance with Bank guidelines and with little Bank staff input. The Republic and Provincial Highways Departments and the Council of Road Organizations are continuously increasing their participation by provid- ing substantial amounts of data. In the railways and ports subsectors, it is expected that borrower involvement in future PCR work will be significant. - 21 - ANNEX B Page 6 F. Latin America and the Caribbean (1) Brazil - Five PCRs are forecast for 1982-83 in addition to the two already issued. While all PCRs are expected to be prepared by the Bank, the participation of the agencies is expected to be extensive in most cases (e.g., ports, railways, fertilizer, agricultural research and agro-indus- tries) and will take the form of comprehensive reports. A start was made in obtaining greater central govern- ment participation in the preparation and review of PCRs in 1978, when a special evaluation unit was established in the Federal Government's Planning Institute. However, this unit never acquired sufficient staff to permit it to be active in project evaluation. In 1980 the unit was trans- ferred to the Secretariat for Control of State Enterprises, where its activities were redirected toward reviewing overall programs rather than individual projects. Regional staff will continue to discuss with the Planning Institute the desirability of establishing a project evaluation function within the Institute. (2) Colombia - Five PCRs are forecast for 1982-83, in addition to one already issued. In all cases, these would be prepared by the Bank, with input from the executing agen- cies. As in the past, a comprehensive report is antici- pated from the power sector; agriculture and small-scale enterprises are also expected to play significant roles, with rather modest contributions coming from water, educa- tion and railways. The National Planning Department (DNP) regularly comments on PPARs, as do several executing agencies, and the Central Bank in some cases. Currently, DNP is considering adding a project evaluation capability through designing a methodology for selecting projects to be evaluated as well as evaluation criteria, and then con- tracting out the studies. At DNP's request, OED is provid- ing advisory assistance in this undertaking. (3) Ecuador - Six PCRs are forecast for 1982-83 in addition to one already issued. Except for Vocational Training where significant participation by the executing agency in the form of a relatively comprehensive report is expected, limited assistance by the agencies (primarily taking the form of data collection) is likely. - 22 - ANNEX B Page 7 (4) Mexico - Six PCRs are forecast for 1982-83 in addition to two already issued. The executing agencies' role in helping in the preparation of the PCRs by Bank staff is expected to range from heavy involvement in the case of tourism and irrigation projects (where relatively compre- hensive completicn reports are to be prepared by the agencies) to agricultural and livestock credit and highways where the final reports will furnish a partial basis for the Bank's work and the mere provision of data in the case of aviation. Evaluation units have been organized in the major external borrowing intermediary institutions (such as Nacional Financiera, Banco Nacional de Obras Publicas and Banco Nacional Pesquero Y Portuario [Banpescal ) and several of the ministries (Programming and Budgeting [SPP] and Public Finance [SHCP] ).l/ These units assist the executing agencies in ex post evaluation of projects and are strengthening their in-house capacity to prepare PCRs. An attempt is also being made to strengthen the national public sector evaluation and monitoring system to disseminate the results of evaluation and apply the lessons for improving project cycles and investment programs generally involving public funds. 1/ See Exhibit 1 for further background. - 22 - ANNEX B Page 7 (4) Mexico - Six PCRs are forecast for 1982-83 in addition to two already issued. The executing agencies' role in helping in the preparation of the PCRs by Bank staff is expected to range from heavy involvement in the case of tourism and irrigation projects (where relatively compre- hensive completion reports are to be prepared by the agencies) to agricultural and livestock credit and highways where the final reports will furnish a partial basis for the Bank's work and the mere provision of data in the case of aviation. Evaluation units have been organized in the major external borrowing intermediary institutions (such as Nacional Financiera, Banco Nacional de Obras Publicas and Banco Nacional Pesquero Y Portuario [Banpesca] ) and several of the ministries (Programming and Budgeting [SPPI and Public Finance [SHCP] ).l/ These units assist the executing agencies in ex post evaluation of projects and are strengthening their in-house capacity to prepare PCRs. An attempt is also being made to strengthen the national public sector evaluation and monitoring system to disseminate the results of evaluation and apply the lessons for improving project cycles and investment programs generally involving public funds. 1/ See Exhibit 1 for further background. - 23 - EXHIBIT 1 Page 1 OPERATIONS EVALUATION IN MEXICO 1. In recent years the Mexican Government has made consider- able efforts to strengthen the institutional infrastructure required for the indispensable task of evaluating to what extent and how effectively the objectives of economic and social development programs have been realized. It must be mentioned here that we have received assistance in this task from international finance organi- zations and particularly from the World Bank, primarily in the form of ex-post evaluation of projects partially financed by them. It should be noted that the lessons learned from the evaluation have resulted in the findings being taken into account in the formulation of new strategies for the future progress of the Nation and that a proper system exists to ensure that this is done. 2. At the Federal Government level, the operations evaluation process comprises the Sistema Nacional de Evaluacion - SNE (National Evaluation System), which is attached to the Office of the President of the Republic, the Secretaria de Programacion y Presupuesto - SPP (Secretariat of Programming and Budgeting), the Secretaria de Hacienda y Credito Publico - SHCP (Secretariat of Finance and Public Credit), the financial agencies of the Federal Government, the executing agencies, and, at the legislative level, the Contaduria Mayor de Hacienda - CMH (Official Auditing Department). The SNE was formed in 1977 in response to the realization that systematic evaluation is an essential tool for accomplishing sectoral goals, providing the means for continuously adjusting Government strategy, and that comprehensive and ongoing evaluation is the way to achieve the conscious and active participation of each and every segment of the population and of the various factors of the economy, thereby ensuring that they take an equal share in the obligations, rights, responsibilities and benefits involved. The system advises, sup- ports and coordinates the various departments and agencies, and is made up of representatives from these and a general coordinator (assisted by support staff). The basic function of the SNE is to issue guidelines for self--evaluation procedures and to put before the departments and agenc.es, for their consideration, any amend- ments additions and deletions that may prove necessary to ensure their efficiency, effectiveness and consistency with the progress and results of their plans and strategies. 3. SPP conducts a continuous evaluation of public expenditure with reference both to development plans and programs and to budge- tary performance and public works. More specifically, the Direc- torates General of BudgEting (Agricultural, Industrial, Social Development and Administration and Finance) and of Budgetary Policy analyze and study the impact and effects of expenditure on the national economy and their relationship with development plans. - 24 - EXHIBIT 1 Page 2 In the Subsecretariat of Evaluation, general guidelines are estab- lished to monitor and evaluate public works, self-evaluations are obtained from the departments, and Federal Government agencies are audited. It is important to point out here that the programs and budgets of the various departments are assigned on the basis of the socio-economic impact generated by the budgets in previous years and any increases requested therefore have to be justified on the strength of past performance. 4. In accordance with the provisions of the General Law on Public Debt, the Secretariat of Finance and Public Credit, through the Directorate General of Public Credit, is responsible for seeing that the proceeds of financing representing public debt, including international loans, are used for the implementation of projects, activities and undertakings that support the economic and social development plans. With regard to international financial organiza- tions, the Secretariat of Finance and Public Credit has, through the Directorate General of Public Credit, in conjunction with SPP, formed several working groups composed of financial agents of the Federal Government, i.e., Nacional Financiera, S.A. (NAFINSA), Banco Nacional de Obras y Servicios Publicos, S.A. (BANOBRAS), and Banco Nacional Pesquero y Portuario (BANPESCA), and the executing agencies that have entered into loans with those organizations. It should be mentioned that the working groups are backed by the operations evaluation units that have been set up in each and every financial and executing agency. The main task of these groups has been to comply with contractual conditions such as preparing project comple- tion reports (PCRs), devising methodologies for special impact or evaluation reports, and discussing with representatives of the international organizations the methodological aspects of the evaluation process. It should be pointed out that IBRD has advised these working groups on a continuous basis. 5. Finally, the Contaduria Mayor de Hacienca (CMH), which is a specialized office of the Chamber of Deputies (under the jurisdic- tion of the Comision de Vigilancia (Oversight Commission) conducts, through the Directorate General of Economic Research and Analysis, a series of evaluations of national accounts and of performance as regards the accomplishment of the priority economic policy goals set by the Federal Government. CMH checks, inter alia, that the opera- tions have been carried out in accordance with the Revenue Law (Ley de Ingresos), that legislation on public debt has been complied with, that the budget is being implemented correctly, that those represented are receiving the benefits indicated in the public - 25 - EXHIBIT 1 Page 3 investment programs, etc. This process takes place in two stages: (1) validation, where the figures submitted by the various depart- ments are checked, and (2) consolidation, where selective audits are made in order to evaluate and review the financial operations of government agencies and to consolidate the national accounts, this being a prerequisite for the performance report of the Chamber of Deputies. Source: The above text is a translation of information provided by the Government in June 1982. - 26 - ANNEX C Page 1 REGIONS' STATUS REPORT ON BORROWER GOVERNMENTS' INVOLVEMENT IN THE POST-EVALUATION OF BANK/IDA-FINANCED PROJECTS (Selected countries with less than five PCRs scheduled through CY83) A. Eastern Africa (1) Sudan - Four PCRs are scheduled in the agriculture, ports and railways sectors in CY83. The borrower is expected to prepare a PCR on the Port project in accordance with the legal agreement. Borrower participation in the other sectors is not expected. The Second Technical Assistance Project (Credit 1153-SU) approved in May 1981 provided for consultancy services to examine the possibility of estab- lishing a central monitoring and evaluation system in the Ministry of National Planning; however, this work has not yet begun. The Bank is encouraging prompt action in this area. B. Western Africa (1) Ivory Coast - Four PCRs in three sectors (IDF, highways and urban) are scheduled through FY83. Two IDF projects will be prepared by Bank staff on the basis of materials sent to the Bank on a regular basis by project authori- ties. One urban project will be based on data and informa- tion provided by the project entity. Although there is no clearly defined process of reviewing the experience of major Bank projects within the Government, sectoral minis- tries as well as ministries of planning and finance normal- ly review and forward comments on Bank-prepared PCRs. Regarding the highway project, the Bank does not expect significant borrower involvement in the preparation of the PCR. (2) Cameroon - Four PCRs in three sectors (agriculture, education and IDF) are planned through FY83 in which project agencies are expected to contribute basic data for the reports and in some cases prepare preliminary reports. - 27- ANNEX C Page 2 (3) Nigeria - Four ?CRs in two sectors (three in agriculture and one in power) are scheduled through FY83. As in the past, project agencies are expected to provide basic data and, in some instances, to prepare preliminary completion reports which will form the basis of Bank PCRs. The monitoring and evaluation units set up under the Agricul- tural Technical Assistance Project (Loan 2029-UNI) are expected to actively participate in the preparation of PCRs for future agricultural projects. Meanwhile, a general technical assistance project currently under preparation should strengthen the capacity of Ministries of Planning and Finance to exercise a more active role in the review of PCRs. The PCR for the Power IV Project will be prepared by Government consultants; funds for consulting services for this purpose have been included in the Power VI Project. (4) Senegal - Four PCRs in three sectors (two in agriculture and one each in highways and urban) are scheduled for FY83. None of the agriculture project reports are expected to be prepared by either the project agencies or Government monitoring and evaluation units. Project reporting has been generally good and project entities are preparing the basic data which will be used to prepare the PCRs. A technical assistance project will assist in establishing monitoring capabilities within the Ministry of Planning, which would at a later stage be entrusted with the prepara- tion and review of PCRs. The executing agency for sites and services project prepared a completion report which has been translated into English and will be appended to the Bank's PCR. The Government-prepared PCR covers the identification and design phase of the sites and services project. The remaining chapters of the report are under preparation by the Government and will be discussed with the Bank when a Bank PCR is available for review and discussion with Government. Regarding the PCR for the Highway III Project, the Maintenance Division of the Ministry of Public Works have asked their consultants to prepare a report on the physical implementation of the Project. C. East Asia and Pacific (1) Korea-- Only one PCR in the transport sector is scheduled for FY83. A project agency will be responsible for preparation of the PCR. - 28 - ANNEX C Page 3 The Government of Korea is developing a system for post-evaluation of major projects. During a Government reorganization in the fall of 1981, this function was assigned to a newly created Bureau of Post Evaluation and Coordination in the Economic Planning Board. The new Bureau has developed an initial work program which focuses on the evaluation of ongoing projects. At the same time, it has prepared post-evaluation guidelines which have been distributed to the Ministries and Agencies of Government who implement projects, is arranging for training of staff in post-evaluation theory and techniques (including visits to the Bank's Operations Evaluation Department), and is planning to complete a manual of post-evaluation proce- dures by the end of 1982. (2) Malaysia - Three PCRs in three sectors (energy, transport and population) are expected to be prepared during FY83. Two of these PCRs will be prepared by project agencies. From the discussions held in June 1981 with the Economic Planning Unit (EPU), it became apparent that no government agency in Malaysia has the mandate to undertake an evaluation of the PCRs prepared by the implementing agencies. Also, in view of the pervasive staff shortages besetting agencies such as EPU, the prospects for initia- ting such a process in Malaysia are not encouraging. Consequently, the Region is concentrating its efforts in strengthening the capabilities of the implementing agencies to come up with PCRs of high quality. D. Europe, Middle East and North Africa (1) Greece - Four PCRs in two sectors (agriculture and educa- tion) are expected to be completed during FY83. The borrower has appointed a team to prepare the PCR for the Groundwater Development Project (Loan 754-GR). The Bank expects similar participation as other loans close in the Greek portfolio. (2) Romania - Three PCRs in the agriculture sector are sched- uled for FY83. Both borrowers in Romania, the Bank for Agriculture and Food Industry (BAFI) and the Investment Bank (IB), are directly involved in the completion process - 29 - ANNEX C Page 4 and are preparing PCRs in accordance with Bank guidelines. Four have been submitted to date, three in the industry sector and one for agriculture. Bank staff involvement has been moderate and following revisions suggested by Bank staff, the PCR quality has been acceptable. The experience gained in preparing these PCRs should enable Bank staff involvement to be minimal in the future. (3) Tunisia - Four PCRs in three sectors (agriculture, high- ways and water supply) are expected to be prepared during FY83. Borrower participation has been or is expected to be limited in the highways, tourism and urban sectors (al- though the borrower hired consultants to prepare a report on the public transport component of the Urban Project). In the power sector, the borrower, Societe Tunisienne de l'Electricite et du Gaz (STEG) prepared a report of good quality on the Power III Project (Loan 1355-TUN). The Ministry of Education, it is hoped, will contribute substan- tially to an upcoming PCR in that sector; however, details have yet to be agreed. (4) Turkey - Four PCRs in two sectors (DFCs and railways) are expected to be prepared during FY83. Borrowers in the agriculture, railways, water supply, urban and power sectors have willingly supplied data as input to Bank- prepared PCRs. For future completion work in the power sector, it is hoped that the Bank's major borrower, Turkiye Electrik Kumkura (TEK), will take an increasingly larger role. - 30 - ANNEX D THE CURRENT STATUS OF RECOMMENDATIONS OF OED POLICY REVIEWS AND EVALUATION STUDIES Introduction Annex D-I attached hereto provides information on the status of the recommendations contained in the Operational Policy Review: The Supervision of Bank Projects dated February 22, 1980, which updates Annex C of the 1981 Report on Operations Evaluation Annexes D-II to D-IV contain the main observations of an Interim Report on Procurement Issues in Bank-Financed Projects dated July 15, 1981 and Reviewing of the Agricultural and Rural Development Program in the Philippines dated February 10, 1982 and of Training in Bank-Financed Projects dated March 1, 1982. Annex D-V outlines certain points which have emerged as of recurring concern in the Annual Reviews of Project Performance Audit Results. The sequence of the recommendations set forth in the attachments is not necessarily as in the Reports themselves. -31- ANNEX D INDEX Part Report Page No. I. The Supervision of Bank Projects 1 II. An Interim Report on Procurement Issues in Bank-Financed Projects 3 III. Sector Operations Review: Agricultural and Rural Development Program in the Philippines 11 IV. Review of Training in Bank-Financed Projects 14 V. Annual Reviews of Project Performance Audit Results 21 ANNEX D Page 1 I. OPERATIONAL POLICY REVIEW: THE SUPERVISION OF BANK PROJECTS (SecM80-129, February 22, 1980) The above report and the management response to its recommendations (R80-142, May 29, 1980) were discussed at a meeting of the Executive Directors on July 15, 1980 and Annex C to the last Annual Report set forth a summary of the principal recommendations contained in the report together with management response. The following notes those recommendations where action remains to be taken by management. Summary of Principal Recommendations I. Increased Borrower Involvement (i) Borrower Reports and Their Use Further steps need to be taken to make borrower reports and their use more effective. There is evidence that reporting intervals are sometimes too short. A logical corollary of reporting requirements being discussed during preparation and appraisal would be to have them set out in the appraisal report. The Bank proposes to take appropriate adminis- trative action on the monitoring of reporting by borrowers. II. The Conduct of Supervision Missions (ii) Mission Composition More conscious selection of supervision staff seems indicated for the needs of particular projects given borrowers' views that Bank missions are not always appropriately staffed. In addition, supervi- sion training and guidance for new staff seems indicated. The Bank is reviewing the guidance being given to new staff, and the need for more intensive staff training focussed explicitly on the supervision function. ANNEX D 33 - Page 2 (iii) Resident and Regional Missions and Resident Projects There is a need to review the present role in supervi- sion of resident and regional missions and resident projects staff, including their possible role in follow-up in between missions and in reviewing documents. The role of' resident missions was reviewed in a study by OPD on the Role of World Bank Field Offices (SecM80-792) October 20, 1980. No further review is presently planned. III. Supervision Reporting and Follow-Up (iv) Reporting to the Bank Improvements in the coverage and continuity of issues in supervision mission reporting could be achieved (a) by requiring managers to sign off/on reports before their distribution and (b) if less discretion on topics reported upon were allowed and more structured reporting required so that certain core issues would receive systematic and continuous attention. Furthermore, greater use could be made of devices such as critical path analysis to monitor implementation. On a related matter, it was noted that some project files are in a poor state and borrowers' progress reports frequently could not be traced. The Bank intends to take appropriate administrative action regarding the apparent deficiencies in project files. IV. Other Assistance and Approaches to Supervision Work (v) Increased Role for Programs Staff There should be increased participation in supervision missions by Programs staff. (vi) Manual on Procedures More specific written guidance on procurement and disbursement procedures should be available to both borrow- ers and Bank staff and handbooks thereon would be valuable. The Bank hopes to make a start in the near future on the systematic collection of information on local procure- ment practices and procedures with the view to cataloguing and analysing them and making them readily available to Bank staff. - 34 - ANNEX D Page 3 II. AN INTERIM REPORT ON PROCUREMENT ISSUES IN BANK-FINANCED PROJECTS (SecM81-646, July 15, 1981) The OED Interim Report on Procurement Issues in Bank- Financed Projects (Report #3557) dated July 15, 1981 was circulated on July 21, 1981 (SecM81-636) and was considered by the Joint Audit Committee on September 9, 1981 (JAC/M81-9). Subsequently, the OED Report, together with the Staff Comments thereon (SecM81-1037 dated December 31, 1981) has been the subject of two seminars for Execu- tive Directors, on January 22 and January 29, 1982. The princi- pal recommendations are summarized hereunder. The management's comments are summarized under the recommendations. Summary of Principal Recommendations A. Advanced Planning and Local Procedures Many procurement problems can be traced to insuffi- cient preparation in the early stages of the project cycle and therefore: (i) a procurement checklist should be provided to assist Bank staff in appraisal; (ii) special country reviews should be made of local procurement procedures jointly with Government and Bank staff; and (iii) increased training should be given to borrower staff. The Bank agrees with all the above and specifically considers a checklist a useful tool, particularly for inexperienced project officers. Some divisions have prepared checklists and the Central Procurement Unit in Projects Advisory Staff (PAS) is preparing a checklist which it expects to issue in a guideline. ANNEX D - 35 - Page 4 In addition, the Bank agrees that special country reviews should be made. With Government's support, the potential benefits from country reviews are high and Bank staff are prepared to assist in such cases. Training seminars help to reduce problems and disputes and to expedite project procurement and implemen- tation. About a dozen country seminars have been under- taken so far. Present staff resources permit three or four additional borrower seminars a year. Consideration is also being given to holding regional seminars, possibly in collaboration with Regional Development Banks, where smaller countries could come together in a group. The principal difficulties in training for borrowers are Bank staff availability and the high turnover of borrowers' staff. A significant impact would require periodic seminars with some extension of content. B. Threshold for Prior Approval (i) Higher threshold levels should be used for prior review of documents by the Bank; (ii) Thresholds should be explicitly discussed in the staff appraisal report taking into account the thresholds applied to earlier projects in the same sector and country; and (iii) Recourse to post rather than prior review should be considered in respect of stronger borrowers with established relationships with the Bank. ANNEX D - 36 - Page 5 The purpose of prior review is to protect the borrower from the penalties of misprocurement after the event, and to assure the international business community of the fairness and objectivity of the procurement process. There is also an element of technical assistance to borrowers in the review work of Bank staff. Prior review helps to ensure that major contracts are awarded in accordance with the procurement arrangements agreed at negotiations, to avoid inadvertent misprocurement, and to assist project implementation. However, it has some- times been done without a considered assessment of the resulting workload for borrower and Bank staff. Guide- lines have been issued to staff in this respect. They require explicit mention of the thresholds for prior review in Staff Appraisal Reports. They fix 80% of the total procurement value as a desirable goal (which may be modified as the number of contracts and workload require- ments may dictate); typically, this can be accomplished by reviewing about: 20% of contracts. Determining the appro- priate threshold for prior review in this manner is more efficient in the use of Bank and borrowers staff time. Another factor in fixing the thresholds is the experience of the borrowers; occasionally quite high thresholds have been established or they have been dispensed with alto- gether. There are also some borrowers who request low -37 - ANNEX D Page 6 thresholds to minimize the chance of misprocurement. Consequently, differing thresholds may be selected for Bank-financed projects among countries or even in the same country. Regardless of the threshold selected for prior review or the contract size, borrowers are free to consult with the Bank on any procurement issue. C. Organization and Training (i) The Bank should continue its efforts to train projects officers in procurement; (ii) The Bank should provide more guidance on procurement to the project staff such as by appointing more specialists in procurement; and (iii) The location of additional procurement spe- cialists requires consideration. While placing them in a central procurement unit working across regional boundaries either in an advisory or line function has the advantage of economy, an alternative would be to have, in addition to a central procurement unit, one or more procurement specialists assigned to the office of each Regional Projects Director to be more readily available and bring to the work the knowledge of local conditions and procedures of the countries within the Region. The Bank agrees that training is an important aspect of the work program of the Central Procurement Unit in PAS. Bank procurement seminars comprising one week of half-day sessions were started in 1975 and 40 seminars have been conducted for some 20 to 25 participants per seminar. An advanced seminar has been begun recently ANNEX D - 38 - Page 7 as well. These efforts will be continued at the rate of about four basic and two advanced seminars annually. One day or shorter seminars on special topics are also being considered for staff with direct responsibilities for procurement. The Bank also agrees that procurement specialists should be available in Projects Departments, and that PAS should complement their work. At present, the Central Procurement Unit has three procurement advisers in addi- tion to the Chief Procurement Adviser. Several project divisions have a procurement specialist who handles or coordinates all procurement work in the division, and this arrangement appears to serve well. Additional staff could be productively used in handling procurement matters. Increased recruitment of such staff must, of course, be judged in the context of overall budget priorities. D. Forms and Procedures (i) The Bank should develop a model or standard set of tender documents to provide guidance to borrowers and Bank staff. The Bank agrees, but notes that the differences in legal frameworks that exist among borrowers and the variety of goods and works to be procured make it very difficult to provide a single standard set of tender documents. However, the absence of a specimen or sample document to guide the borrower staff, as well as inexpe- rienced Bank staff, has been increasingly noted. Parti- ANNEX D - 39 - Page 8 cularly where borrowers do not already have standard documents that have been used for international bidding, a good sample could reduce the time and costs involved for borrowers in the employment of consultants to prepare documents or review documents, the considerable work in Bank review of documents, and the number of unsatisfac- tory bid documents. The Bank is now drafting such a sample document Eor the purchase of goods and equipment, and the Asian Development Bank has undertaken a similar task for construction contracts, to be considered for use by the World Bank and the Regional Development Banks. (ii) The Bank should develop a procurement manual which will provide guidance in respect of the most commonly encountered procurement issues with illustrative case material. The Bank agrees that procurement manual would be useful but has not given this a high priority because a substantial amount of written guidance already exists in addition to the Procurement Guidelines. Operational Manual Statements and Central Projects Notes are available to Bank staff, and Supplemental Procurement Guides and training materials used in seminars are available to borrower staff as well. These will be supplemented significantly by the sample tender documents now being prepared. - 4- ANNEX D Page 9 In addition, an International Procurement Training Manual prepared by the United Nations Institute for Training and Research (UNITAR) is available which goes a long way towards meeting some of the needs in this area. The Bank is assisting in an expansion and revision of the current edition and this is expected to be available next year. While it will not be a Bank document, it will present good international procurement practice and draw heavily on Bank material. E. Additional Actions (i) The Bank should provide a "revolving fund" to facilitate initial reimbursement in those cases where borrowers are unable to provide such funds, to expedite project implementation. Revolving funds can be useful and the Bank is pre- pared to use them wherever appropriate. However, revolv- ing funds do not accelerate project implementation--they can only accelerate disbursements--unless accompanied by actions such as earlier preparation of bidding documents, prequalification of bidders, etc. (ii) Advance payment to local contractors should be encouraged. This is standard practice. The Bank encourages borrowers to include specific provisions in their bidding documents for advance payments to contractors and sup- pliers to assist in mobilization and for other initial - 41 - ANNEX D Page 10 expenses, against suitable securities. Advance payments are proposed by borrowers and the Bank alike, and are acceptable if in accordance with gEneral practice (15% to 20%) and secured by guarantees or liens on equipment. (iii) Information should be provided to borrowers on cost and terms of supply to assist them in limited tender- ing or negotiated purchases where, due to lack of competition, they are unable to gauge market prices. While agreeing with the recommendation, the Bank notes that the variety of goods and works financed makes it impossible to develop and main:ain cost information on many items with any assurance of accuracy. However, in selected areas, this information is being furnished. For example, with the concur- rence of borrowers, the Bank is circulating price details of awarded contracts for major railway equipment procurement. Other selective efforts are being initiated where items procured and conditions of purchase are sufficiently comparable to yield useful data at a reasonable staff cost, e.g., standard items in telecommunications. The Bank's review process itself affords an opportunity to advise borrowers if prices appear unduly high. (iv) The Bank should examine the system for bulk procure- ment and contracting where they exist, and with necessary improvement agree to their being used in common-user items required for Bank financing and other projects to reduce unit cost. The Bank agrees and in fact, where ICB is not warranted and there is a bulk procurement arrangement in a borrowing country, then it is used if the appraisal mission states that the system is fair, competitive and acceptable. - 42 - ANNEX D Page 11 III. SECTOR OPERATIONS REVIEW AGRICULTURAL AND RURAL DEVELOPMENT PROGRAM IN THE PHILIPPINES (SecM82-118, February 10, 1982) This report presents the findings of a study undertaken by the OED to review Bank/IDA lending strategy and operations in the agricultural sector in the Philippines and was considered by the JAC on April 14, 1982 (JAC/M82-4). The Philippines was selected for this review for several reasons: (i) it is one of the largest Bank borrowers in East Asia, preceded by Indonesia and Korea; (ii) Bank/IDA commitments to the Philippines have supported a broad range of agricultural activities through 33 projects; and (iii) from the time of the first Bank project in 1965, the Philippines has expe- rienced continuous economic growth mainly as a result of an impres- sive agricultural performance. While this report contains only one procedural recommenda- tion, it does point out several sector issues which merit early attention by Government and possibly by the Bank in future sector operations. The report also offers suggestions for improving the linkage between agricultural research stations and extension sub- ject-matter specialists and for avoiding inadvertent omission of important subjects in future sector work. Following are the summary of the one recommendation, main sector issues, and suggestions: A. Procedural Recommendation A number of the more recent projects were introduced into the Bank lending program rather late, i.e., only shortly before appraisal. This suggests that a Five-Year Planning horizon may be too short for certain agricultural projects. Considering the long lead time required to obtain results from agricultural research and the long periods required for assembling economically, financially and technically sound "packages" which farmers can be expected to take up, consideration should be given to support every five-year sector operations plan for a country with a longer period indicative plan, which would identify the pre-investment work needed to prepare future project options in timely fashion. - 43 - ANNEX D Page 12 B. Sector Issues (i) Rapid population growth, pressure on land, and the need to diversify agricultural production call for a land use study; (ii) Providing farmers with quality seeds is still an unresolved problem and the merits of a national seeds program deserve early examination; (iii) The socio-economic constraints hampering the develop- ment of farmer associations and cooperatives merit study; (iv) Development of agri-industries, including cottage industries, requires much more systematic attention than it has been given to date; and (v) The constraints hampering cost-effective rice exports point to the need for concentrating invest- ment in drying, storing and bulk handling facilities for rice exports in areas of substantial rice surpluses. C. Suggestions (i) To avoid inadvertent omissions of important subjects in sector work, all intrinsically or potentially important subsectors or activities should be listed in order to ensure that omissions in analysis are deliberate and not the result of oversight; and (ii) To systematize the linkages between the research stations and the subject-matter specialists, the following suggestions were made: (a) Training - It may be more effective if teams of subject-matter specialists undertake train- ing of technicians in smaller groups of 20 to 25 in the concerned districts, in rotation for a full day once every two weeks; - 44 ANNEX D Page 13 (b) Visits - Field technicians should provide guidance to contact leaders by visiting and working with them in their fields to ensure that the recommended practices are satisfacto- rily implemented by them; and (c) Linkage between research and extension - The Training and Visit approach requires ongoing contact between subject-matter special- ists and nearby research stations to ensure that the extension program will have a high level of technical content. 45 - ANNEX D Page 14 IV. REVIEW OF TRAINING IN BANK-FINANCED PROJECTS (SecM82-163, March 1, 1982) A summary of the principal recommendations of the above review is set forth. The report and management's response thereto are expected to form the basis of a Board seminar in due course. Meanwhile, the review was found by management to be substantially in accord with the Bank's own findings in its monitoring of training components and supports many of the developments in policies and procedures which Bank management has been pursuing. It is acknowl- edged that there is still scope for further improvement and the Bank needs to organize itself to cope more effectively with training components of projects. Where management has some reservations about the review's findings, these have been shown below each recommendation. Summary of Principal Recommendations I. Recommendation 1 In recognition of the crucial role of trained man- power in development, the Bank needs to raise the visibi- lity of human resource development in its operational work and should in particular: (i) Conduct more thorough reviews of human re- source development needs in its country economic and sector work, developing for each counzry an agreed strategy of investment which takes account of the human resource con- straints and the role of training in meeting the needs; (ii) Place increased emphasis upon long-term approaches to training, including (a) greater efforts to build up the borrowers' training capacities and (b) where advantageous, greater use of separate training projects; -46 - ANNEX D Page 15 (iii) Utilize more widely management review of the sectors and agencies concerned in the cause of assessing the diagnosis of training needs and make sure that the diagnosis of training needs is more systematically and thoroughly done, including analysis of achieve- ments/ deficiencies of prior investments. (iv) Make more effective use of its staff resources in training, particularly (a) the increased use of education divisions in the generation and supervision of training components/pro- jects; (b) the provision of detailed guide- lines to borrowers and Bank staff on the diagnosis of training needs and the prepara- tion and implementation of training compo- nents/projects; and (c) the reorganization of seminars for project staff on the application of these guidelines; (v) Require training components and training projects to be as advanced in their prepara- tion as possible before loan/credit approval; (vi) Ensure that local and regional training resources and institutions are examined with a view to their possible utilization in training efforts; (vii) Ensure that training investments are supported by evaluation arrangements to assess both internal and external efficiency of the programs and in particular where experts are used as trainers, their terms of reference include more specific and stronger references to the training role and arrangements made for periodic review of the experts' training performance; (viii) Ensure that, in the case of expert counterpart relationships in which the expert occupies a line position as distinct from an advisory role, the progressive transfer of responsibi- lities to the counterparts is built into the project design; ANNEX D - 47 - Page 16 (ix) Pay increased attention to the use of fellow- ships as a means of training and to separate provision of loan/credit funds for financing fellowships, instead of leaving such funding to contracts with consulting firms; (x) Provide adequate support for instructional materials for training programs as needed; and (xi) Pay due attention to the potential and role of women within the scope of training invest- ments which are being considered or imple- men-ed. The Bank management finds the above recommendations to be in line with what the Bank supports. Thus free standing training projects are currently being developed in Indonesia, Nigeria and Zimbabwe. The Bank acknowledges that there are areas in which current practice needs to be strengthened or improved. The Bank notes that progress is being made in developing borrowers' skills and specifi- cally the needs of the borrowers in drawing up sector training guidelines. In addition, EDI courses for bor- rowers' staff include training in the analysis of institu- tional organization and training needs. Seminars for Bank staff are being held by the Training Unit of the Education Department of OPS on sector specific training which help develop Bank skills. ANNEX D - 48 - Page 17 The recommendation that there be an increased use of fellowships has been noted by the Bank which considers that the success noted in the sample conducted in the survey does not appear to be reflected in general Bank experience. Each case needs to be judged on its merits. Accordingly, a decision to recommend fellowships for financing would depend upon the circumstances of each case. II. Recommendation 2 Certain Bank reports need to be more specific on training given the need to pay due attention to training in all lending activities. A. The Appriasal Report in each case (and President's Reports in the case of Technical Assistance Projects) should be required to include: (i) where manpower deficiencies exist in the sector or agency concerned, a state- ment showing what they are and how they are being addressed; (ii) in cases where foreign technical assis- tance experts are being proposed for project financing, a statement as to what steps are being included or taken to eliminate or reduce the future demand for the importation of such services; and if they are for studies under a project whether local institutions are being involved and if not, why and, where appropriate, why a local capacity is not being developed; and (iii) where training is being financed in a project, a separate Annex on training detailing the objectives and scope of the intended training and all the basic data 49 ANNEX D Page 18 (e.g. the agencies involved) essential to a full understanding of the component as agreed between the Bank and country. 1/ B. The Supervision Report should include specific reference to training in all cases where training is included Ln the project. If the training component was not reviewed in the course of the particular supervision visit this should be mentioned. In Section 2 of the Supervision Form (No. 590) the appropriate Performance Rating categories should be expanded to include training components. Bank management considers the recommendations on the reporting system as being to some extent a reinforce- ment of currenz procedures and practices, while others would sharpen the Bank's focus on training; the Bank supports the recommendations and expects them to be interpreted and applied in a practical manner by Bank staff. Bank also agrees that the Appraisal Report (or President's Report in the case of Technical Assistance Projects) should generally provide greater treatment of human resources and training issues, although selectivity would be necessary as the extent of such treatment depends upon the nature of the proposed project. The Bank also agrees that where foreign technical assistance is included in a project, there should be a statement about prospects for the reduction or elimination of the demand for a particular type of imported expertise. 1 The provision of such Annexes appears in practice to be optional at present. - 50 - ANNEX D Page 19 The inclusion of the separate Annex on training in the Appraisal Report is, as indicated in the footnote to this particular recommendation (sub-para. iii), optional and the Bank considers it should remain so with a clear expectation that an Annex would be included whenever the training component is complex or substantial. The Bank agrees that the Supervision Report should include reference to the training component of the project and that the present Form No. 590 should include a speci- fic category (training) for performance rating. The Bank also endorses the recommendation that supervision missions should monitor the use made in training investments which may be completed early during project implementation. III. Recommendation 3 As regards the organization, resources and staff use for training, the Bank should: (i) Make greater use of existing Cooperative Programs with international agencies. The Bank supports a wider use in training of Coopera- tive Programs and of the resources of other international agencies with which it has working relationships, with the clear understanding that (a) this involvement would not result in any increase in staff years to the existing Cooperative Programs but rather a change in staff utiliza- tion, and (b) any review of these Cooperative Programs would in no way be constrained by such use. - 51 ANNEX D Page 20 (ii) Bring the regional Education Divisions more into operations work, while maintaining a central operational support unit with sector training specialists [see also Recommendation l(iv') above]. Given that the Bank's involvement in training is expected to increase, management supports the idea impli- cit in the above recommendation of the regionalization of training although some lead time would be necessary to develop the necessary capacity in the Regions. Until this capacity is developed, the central Training Unit, which would be retained with present capacity, would thus have to continue to provide support to the Regions in this regard and later increasingly focus on training Bank staff in dealing with training components, preparing guidelines and policy statements and monitoring the effectiveness of the Bank's training work. The training support functions (including need for specialist staff, balance between categories of staff and their deployment) are under study by the Regional Offices and the Education Department; regional statements on this subject are being prepared. ANiNEX D Page 21 V. ANNUAL REVIEWS OF PROJECT PERFORMANCE AUDIT RESULTS This section sets down certain points which have emerged as of recurring concern in the Annual Reviews of Project Performance Audit Results. Most of them appeared in Annex III of the Seventh Annual Review and are reproduced here to provide a ready reference by which progress made in the more important deficiencies noted can be tracked in subsequent project performance audit reports. The Reviews themselves provide the major device for follow-up; the summary findings appearing at the end of each sector review (Chapter III) not only identify areas of continuing concern in the experience examined in that sector but also indicate subse- quent remedial actions already taken or planned by the Bank. Annex III of the Eighth Annual Review provides, in addition, a sampling of the lessons emerging from individual PPARs in the agriculture and rural development sector and the management response to them. What follows here is an abbreviated listing of the more generalized issues emerging from the recent Reviews. A. General (i) appropriate attention to the preparation and design of projects, with special reference to (a) policy environ- ment bearing on project success, (b) relative simplicity of project design in general, and (c) sociological factors involved; (ii) realism of implementation schedules; (iii) adequacy of sensitivity analysis of delay of projects; (iv) arrangements for the design, implementation and effec- tiveness of technical assistance for institutional support: in this connection, adequacy of (a) mutual agreement between the Bank and borrowers on detailed terms of reference of technical experts, (b) the use of short-term specialists and local experts where possible, and (c) fellowships for local personnel; (v) compliance with loan covenants by borrowers and reasons for non-compliance where this occurs; -53 - ANNEX D Page 22 B. Agriculture (vi) importance given, in project preparation and design and in policy understandings reached with the borrower, to appropriate producer prices; (vii) the cost effectiveness, economic viability and social equity of small farmer projects in supporting agricul- tural development; (viii) attention given to developing decentralized groundwater projects for irrigation in conjunction with surface water development; (ix) Bank actions to strengthen expertise in agro-industries and fisheries development; C. Public Utilities (x) in power and water supply, adequacy of utility demand forecasting guidelines and the procedure to adjust implementation schedules to minimize economic effects of divergence wherever feasible; (xi) guidelines on the determination and reduction of un- accounted-for water are in the process of development; (xii) implementation of the results of the research program to develop quantitative measures of management and opera- tional performance of public utilities. The program is under way and is expected to be completed in respect of the power subsector by mid-1983; (xiii) timeliness of dialogue between the Bank, the government and the project entity on pricing policies in the early stages of project preparation to establish a program of action on tariffs and other related matters; D. Transport (xiv) the realism of traffic projections in relation to the effective design of transport facilities provided. In this connection, review progress in developing rail traffic forecasts differentiated by major commodities, and designing projects to alternative future levels of traffic within certain ranges; - 54 - ANNEX D Page 23 (xv) progress made in promoting systematic approach to road maintenance, requiring a plan, appropriate funds, equip- ment and staff; (xvi) compliance with audit covenants for revenue earning entities and actions taken to overcome shortcomings within the organizations which inhibit regular audits; E. Education (xvii) progress made in establishing the correct correlations between the time given to project preparation, borrower understanding of and commitment to the broader objectives envisaged, and the need for a country-and-sector strategy for educational development as a framework for individual Bank-supported projects; (xviii) follow-up on the CPS study of the performance of project- financed educational institutions and research into various aspects cf the problem of introducing practical subjects into secondary schools; and F. DFCs (xix) arrangements for the monitoring and evaluation of sub- projects financed. - 5- ANNEX E Page 1 OED REPORTS ISSUED DURING FY82 Project Performance Audit Reports Public Utilities Ivory Coast Abidjan Sewerage & Drainage SecM81-716 08/12/81 (Loan 1076-IVC) Turkey Keban Transmission & TEK Power Transmission SecM81-1016 11/23/81 (Loans 568/763-TU) Cyprus Nicosia Sewerage (Loan 729-CY) SecM81-1007 11/24/81 Colombia Third Telecommunications (Loan 1073-CO) SecM81-959 11/25/81 Swaziland Water Supply & Sewerage (Loan 1058-SW) SecM81-960 11/25/81 Sri Lanka Fourth Power (Loan 636-CE) SecM81-1011 12/07/81 Sri Lanka Fifth Power (Credit .372-CE) SecM81-1012 12/07/81 Bolivia ENDE Fourth Power (Loan 1238-BO) SecN81-997 12/10/81 Colombia Guatape II Hydroelectric (Loan 874-CO) SecM81-998 12/14/81 Morocco Casablanca-Rabat Bulk Water Supply SecM81-999 12/16/81 (Loan 850-MOR) Burma First Telecommunications SecM81-1019 12/23/81 (Credit 551-BA) Thailand Bangkok Water Supply (Loan 1021-TH) SecM82-04 12/28/81 Jordan First and Second Hussein Thermal Power SecM82-264 03/25/82 (Credits 386/570-JO) Papua New Guinea Power (Loan 999-PNG) SecM82-465 04/30/82 Tunisia Third Water Supply (Loan 989-TUN) SecM82-464 05/04/82 Thailand Ban Chao Nen (Srinagarind) Hydroelectric SecM82-570 06/23/82 (Loan 977-TH) Morocco First Power (Loan 936-MOR) SecM82-656 06/30/82 Transportation Senegal Second Railway (Loan 835/Credit 314-SE) SecM81-715 08/24/81 Yugoslavia Third Railway (Loan 531-YU) SecM81-841 10/09/81 Kenya Highway Maintenance (Credit 224-KE) SecM81-887 10/26/81 Ivory Coast Second, Third and Fourth Highways (Loans 761/837/Credit 405-IVC) SecM81-888 10/28/81 Kenya Nairobi Airport (Loan 826-KE) SecM8l-1010 12/07/81 Dominican Republic Road Maintenance and Reconstruction (Loan 1316-T-DO) SecM81-987 12/09/81 ANNEX E - 56 - Page 2 Transportation - Continued Guyana First Highway (Credit 301-GUA) SecM82-39 12/28/81 Korea Fourth and Fifth Railways (Loan 863/1101-KO) SecM82-6 12/31/81 Cameroon Second and Third Highways (Loan 945/Credit 429/Loan 1515-CM) SecM82-555 05/24/82 Pakistan Ninth Railway (Loan 521-PAK) SecM82-575 06/21/82 Bangladesh Inland Water Transportation SecM82-636 06/30/82 (Credit 424-BD) Western Samoa First Highway (Credit 535) SecM82-637 06/30/82 Fiji First Highway (Loan 771-FIJ) SecM82-638 06/30/82 Nigeria Second Lagos Port (Loan 922-UNI) SecM82-644 06/30/82 Morocco Second Highway (Loan 955-MOR) SecM82-645 06/30/82 Tanzania Second Highway (Loan 586/Credit 142-TAN) SecM82-648 06/30/82 Tanzania Third Highway (Credit 265/Loan 586-TAN) SecM82-649 06/30/82 Benin Second Highway (Credit 415-DA) SecM82-650 06/30/82 Togo Second Highway (Credit 450-TO) SecM82-651 06/30/82 East African Community Second & Third Harbours SecM82-652 06/30/82 (Loan 638/865-EAC) DFCs, Industries, Program Loans and SALs Pakistan Multan Fertilizer Expansion SecM81-714 08/18/81 (Loan 988-PAK) Tunisia Banque de Developpment Economique SecM81-785 09/09/81 de Tunisie (Loans 648/798/881-TUN) Greece National Investment Bank for Industrial SecM81-1018 12/22/81 Development (NIBID) Fourth & Fifth (Loans 945/1135-GR) India Cochin Second Fertilizer SecM82-64 12/31/81 (Credit 264-IN) India Sindri Fertilizer (Credit 520-IN) SecM82-65 12/31/81 India Nangal Fertilizer Expansion (Credit 357-IN) SecM82-71 12/31/81 Indonesia Private Development Finance Company SecM82-243 03/16/82 of Indonesia (PDFCI) (Credit 436-IND) Zaire Second and Third Credits to Societe SecM82-245 03/22/82 Financiere de Developpement (SOFIDE) (Credits 271-463-ZR) Tanzania - Tanzania Investment Bank (TIB) I SecM82-272 03/30/82 (Credit 460-TA) Jordan Industrial Development Bank (Credit 646-JO) SecM82-436 04/23/82 ANNEX E 57 Page 3 DFCs, Industries, Program Loans and SALs - Continued Kenya Industrial Development Bank (Loan 946-KE) SecM82-532 05/28/82 Philippines Small and Medium Industries SecM82-574 06/16/82 (Loan 1120-PH) India Trombay IV Fertilizer Expansion & Plant SecM82-622 06/23/82 Operations Improvement (Credit 481-IN) Mauritius Coromandel Industrial Estate SecM82-639 06/28/82 (Credit 411-MAS) Turkey First Structural Adjustment Loan & SecM82-620 06/30/82 Supplement (Loans 1818/1915-TU) Agriculture Senegal Terres Neuves Second Resettlement and SecM81-860 10/27/81 Eastern Senegal Technical Assistance (Credit 578-SE) Argentina Agricultural Credit (Loan 1564-AR) SecM81-957 11/30/81 India Bihar Agricultural Credit (Credit 440-IN) SecM81-1022 11/30/81 Panama First Livestock Development (Loan 901-PAN) SecM81-1013 12/23/81 Ecuador Fisheries (Loan 555-EC) SecM81-1017 12/23/81 Argentina Balcarce Livestock Development SecM81-1023 12/23/81 (Loan 505-AR) Sri Lanka Mahaweli Ganga Development (Stage I) SecM81-1020 12/28/81 (Credit 174/Loan 653) Ivory Coast Fourth Oil Palm and Coconut SecM81-1024 12/28/81 (Loan 1382-IVC) Zambia Integrated Family Farming (Loan 882-ZA) SecM82-5 12/30/81 Tunisia First Fisheries (Credit 270-TUN) SecM82-26 12/31/81 Mauritius Tea Development Authority (Credit 239-MAS) SecM82-62 12/31/81 India Wheat Storage (Credit 267-IN) SecM82-62 12/31/81 People's Democratic Republic of Yemen: First Fisheries (Credit 370-YDR) SecM82-63 12/31/81 Spain Agricultural Research (Loan 768-SP) SecM82-66 12/31/81 Korea Rural Infrastructure (Loans 1216/1218-KO) SecM82-463 05/10/82 Morocco Rharb Sebou Irrigation (Loan 643-MOR) SecM82-556 05/27/82 Sangladesh Northwest Tubewell (Credit 341-BD) SecM82-557 05/28/82 Burma First Irrigation (Credit 483-BA) SecM82-531 05/28/82 Kenya Second Forestry (Credit 566/Loan 1132-KE) SecM82-534 06/03/82 Guinea Pineapple Development (Credit 569) SecM82-535 06/09/82 ASNEX E -5- Page 4 Agriculture - continued Indonesia North Sumatra Smalholiez Development SecM82-558 06/09/82 (Credit 358-IND) Nigeria Second Cocoa (Loan 1045-UNI) Se&Q2-533 06/02/82 Nigeria Funtua/Gusau/Combo Agricultural SecM82-595 06/16/82 Development Loans (Loans 1092/1099/1164-UNI) India Drought Prone Areas (Credit 526-IN) SecM82-623 06/16/82 Ghana Livestock (Credit 500-GH) SecM82-574 06/16/82 Ethiopia Lower Adiabo Development (Credit 516-ET) SecM82-579 06/21/82 Malaysia Third Jengka Triangle (Loan 885-MA) SecM82-578 06/18/82 Egypt Nile Delta First Drainage (Credit 181-EGT) SecM82-568 06/21/82 Mauritius Rural Development (Credit 419-MAS) SecM82-569 06/22/82 Jordan Northeast Ghor Irrigation and Rural SecM82-594 06/21/82 Development (Credit 498-JO) India Godavari Barrage (Credit 532-IN) SecM82-603 W621/82 Niger Maradi Rural Development (Credit 608-NIR) SecM82-621 06/21/82 Syrian Arab Republic Oilseed Processing SecM82-646 06/21/82 (Loan 1631-SYR) Ethiopia Coffee Processing (Credit 290-ET) SecM82-667 06/21/82 Ethiopia Second Livestock (Credit 365-ET) SecM82-671 06/21/82 Botswana First Livestock (Credit 325-BT) SecM82-604 06/22/82 Indonesia Beef Cattle Development (Credit 355-IND) SecM82-573 06/23/82 Mauritania SONADER Technical Assistance SecM82-605 06/23/82 (Credit 694-MAU) Bolivia Third Livestock & First Agricultural SecM82-655 06/23/82 Credit (Credits 261/561-B0) Ethiopia Addis Ababa Dairy Development SecM82-640 06/23/82 (Credit 269-ET) Uganda First Beef Ranching Developmemt SecM82-602 06/21/82 (Credit 130-UG) Papua New Guinea Smallholder Livestock SecMB2-647 06/23/82 (Credit 348-PNG) ANNEX E - 59 - Page 5 Education Nigeria Second Education (Loan 814-UNI) SecM8l-955 11/16/81 Upper Volta First Education (Credit 430-UV) SecM82-191 03/01/82 Mali First Education (Credit 420-MLI) SecM82-221 03/09/82 The Sultanate of Oman First Education SecM82-246 03/09/82 (Loan 980-OM) Honduras First Education (Loan. 954/Credit 452-HO) SecM82-554 05/21/82 Algeria First Education (Loan 913-AL) SecM82-577 06/18/82 Costa Rica First Education (Loan 915-CR) SecM82-593 06/21/82 Turkey First Education (Loan 748-TU) SecM82-675 06/28/82 Population Trinidad & Tobago Population (Loan 743-TR) SecM81-958 11/19/81 India First Population (Credit 312-IN) SecM82-25 12/31/81 Iran Population (Loan 928-IRN) SecM82-78 02/01/82 Tourism Yugoslavia Babin Kuk Tourism (Loan 782-YU) SecM81-786 09/18/81 Urban Development India Calcutta Urban Development (Credit 427-IN) Sec82-664 06/30/82 - 60 - ANNEX E Page 6 Annual Reports/Reviews Annual Report on Operations Evaluation R81-246 9/01/81 Concordance to Project Performance Audit Reports SecM81-830 9/30/81 Issued by Operations Evaluation Department March 1972 to June 30, 1981 Seventh Annual Review of Project Performance R81-265 10/09/81 Audit Results Sector Operations Review: Agricultural and Rural SecM82-118 2/10/82 Development Program in the Philippines Review of Training in Bank-Financed Projects SecM82-163 3/01/82 Special Studies An Interim Report on Procurement Issues in SecM8l-636 7/15/81 Bank-Financed Projects Impact Evaluation Report - Malaysia: Muda and SecM8l-713 8/24/81 Kemubu Irrigation (Loans 434/500-MA) Impact Evaluation Report - Madagascar: SecM81-736 8/31/81 Lake Alaotra Irrigation (Credit 214-MAG) Impact Evaluation Report - Kenya: First SecM81-806 9/22/81 Livestock Development (Credit 129-KE) Impact Evaluation Report - Peru: San Lorenzo SecM82-485 5/24/82 Irrigation and Land Settlement (Loan 418-PE) Impact Evaluation Report - Colombia: Atlantico SecM82-662 6/21/82 Irrigation (Loan 502-CO)