* h Report No. 10606 Bank Experience in Project Supervision April 30, 1992 Operations Evaluation Department I MICROFICHE COPY Report No. 10606-GLB Type: (IER) CLAVERIE, / X31753 / T9047/ OEDDI FOR OFFICIAL USE ONLY Document of the World Bank This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bankauthorization, ABBREVIATIONS AND ACRONYMS AR - Annual Review of Evaluation Results ARIS - Annual Review of Implementation and Supervision CAM - Country Assistance Management CIR - Country Implementation Review ERR - Internal Economir Rate of Return ESW - Economic and Sector Work Form 590 - Implementation Summary IAD - Internal Auditing Department ICB - International Competitive Bidding KNR - Korean National Railways M&E - Monitoring and Evaluation MIS - Management Information System MTR - Mid-Term Reviews OD - Operational Directive OED - Operations Evaluation Department OMS - Operational Manual Statements PCR - Project Completion Report PAR - Performance Audit Report PPIR - Problem Project Implementation Review RVP - Regional Vice President SA - Special Accounts SAL - Structural Adjustment Loan SAR - Staff Appraisal Report SOE - Statement of Expenditure SOD - Sector Operating Division SVP - Senior Vice President TRS - Time Recording System WPI - Working Paper I "The Evolution of Bank Project Supervision" WPII - Working Paper II "Report on Interviews with Bank Staff" THE WORLD BANK FOR OFFICIAL USE ONLY Washington. D.C. e0433 U.S.A. Office of Director-General Operations Evaluation April 30, 1992 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Bank Experience in Proiect Supervision Attached, for information, is a copy of a report entitled "Bank Experience in Project Supervision" prepared by the Operations Evaluation Department. Attachment This document has a restricted distribution and may be used by recipients only in the performance of their offRcial duties. Its contents may not otherwise be disclosed without World Bank authoriation.I FOR OFFICIAL USE ONLY BANK EXPERIENCE IN PROJECT SUPERVISION Table of Contents Page No. PREFACE ............................................... i EXECUTIVE SUMMARY ........................................... iii I. BACKGROUND ............................................ 1 M ethodology ........................................... 2 II. SUPERVISION: DEFINED, FUNDED AND IMPROVED ..................... 2 Supervision Defined ...................................... 2 Purpose of Sup rvision .................................... 3 A Fail-safe InformationSystem ............................... 3 Nuances of Responsibility .... 4 Supervision Resources in Perspe ive ..... The CAM (Country Assistance Management) and the Budgetary Process ..................................... 6 Comparing Supervision Intensities ............................. 7 Project Ratings .......................................... 10 Important Innovations ..................................... 12 III. BANK ATTITUDES AND PROCEDURES .............................. 17 The Measure of Success ................................... 17 Other Indicators ofOutcome ................................ 19 Cancellations ........................................... 21 Reasons for Slippage ...................................... 21 Expectations ........................................... 24 Mis-Match in Objectives ................................... 25 Risk Analysis and Legal Covenants ............................ 26 Procurement ............................................ 26 Marginal Changes and Evolution .............................. 27 Open-Ended ProjectDesign ................................. 28 Mis-Match of Supervisory Skills and Needs ...................... 29 Promotion on Appraisals . 30 Files, Records and Resource Use . 30 Lack of Objective Indicators ................................. 31 Adequacy of Resources .................................... 32 Financial Aspects ........................................ 32 BudgetsandAllocations ................................... 34 The Trouble With TrustFunds ............................... 34 Coordination of Supervision Amongst Donors ..................... 35 Staff Perspective ........................................ 35 Borrower Contribution ..................................... 39 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. IV. RECOMMENDATIONS ......................................... 40 A. Responsibilities for Implementation and Supervision ................. 41 Responsibilises .......................................... 41 Bank's Assistance Objectives ................................ 41 Reali.tic E::pectations ..................................... 42 Supervision by the Borrower and the Bank ....................... 42 B. Incentives and Resources ................................... 44 Promotion and Performance Criteria ............................ 44 Institutional Focus .......... ............................. 44 Resources for Supervision ................................... 46 C. Staff Capability .......................................... 48 Staff Profile ............................................ 48 Supervision Functions and Training ............................ 49 D. Procedures to Enhance the Supervision Process ................... 49 Implementation Plan ...................................... 49 Risk Analysis and "Flags" for Project Redesign ................... 50 Confirmation of Continued Feasibility ........................... 50 Supervision Mission Composition ............................. 51 Procurem ent ... ......................................... 51 Accounts andAudits ...................................... 52 Reliability of Supervision Ratings .............................. 53 External Review of Remedial Actions ........................... S3 Mid-Term Review ........................................ 54 Hand-over of Project Supervision ............................. 54 Evaluation Methodology .................................... 55 Project Records ......................................... 55 Portfolio Summaries ...................................... 55 Sharing of Form 590 ...................................... 56 Donor Coordination in Supervision ............................. 56 Additional Resources ...................................... 56 1 Staff Perceptions of Management Priorities ........................... 37 ioures 1 Supervision Per Project by Per Capita GNP Level ....................... 9 2 Supervision Per Project by Per Capita GNP Growth ...................... 9 3 Supervision Per Project by Status of Economic Adjustment ................ 11 4 Supervision Per Project by Indebtedness ............................. 11 5 Approved Value ofLoans ....................................... 18 6 Comparison of Success Rate of Evaluated Projects ..................... 18 7 Problem Projects ............................................. 20 8 Overall StatusRating .......................................... 20 9 Comparison of Ex-Post and Supervision Ratings for Investment Projects in their Fourth and Fifth Year of Implementation ............... 22 10 Profiles of Cancellations Relative to Principal ......................... 23 11 Profiles of Cumulative Cancellations (1985-91) ....................... 23 Tqxt Tables 1 Distribution of Project-Specific Supervision ........................... 6 2 Relative Contribution of Trust Funds to Bank Work ..................... 35 Annex Tables 1 Net Staffweeks Reported by Higher Level Staff in FY90 .................. 57 2 Allocation of Supervision Staff Time, FY74-90 ........................ 58 3 Operations Complex, Operational Stpffweeks (levels) .................... 59 4 Operations Complex, Operational Staffweeks (%) ...................... 60 5 "Active Project" Criteria used for Budgeting Supervision .................. 61 6 Budget and Actual Staff Years Used for Supervision FY85-91 .............. 62 7 Average Project Specific Supervision Staffweeks per $100 million by Sector and FiscalYear ...................................... 63 8 High and Low Per (Investment) Project Supervision in FY90 ............... 64 9 Ratings of Projects Reported by OED by Evaluation Year ................. 65 10 Evaluation Ratings of Projects per Year of Approval ................ :66 66 11 Project Implementation Status by ARIS Cohorts ....................... 67 BANK EXPERIENCE IN PROJECT SUPERVISION PREFACE "Successful implementation of Bank-financed operations far outweighs new commitments as the principal indicator of the Bank's development effectiveness. Lewis T. Preston February 5, 1992 Supervision of the implementation of Bank-financed projects uses about a quarter of the Bank's operational staff time, and about 60 percent of supervision time is used for investment projects. In 1980, the Operations Evaluation Department (OED) submitted its first report on the effectiveness of Bank supervision to the Board. This second report is a continuation of a series of Operational Policy Reviews by OED, and examines changes in the supervision function since the earlier report and its relationship with project implementation and outcome. It focuses on investment projects rather than adjustment operations, as OED has recently reviewed the supervision of adjustment lending operations. The study involved use of data from the Bank's Management Information System (MIS), Annual Reviews of Implementation and Supervision (ARIS), Annual Raviews of Evaluation Results (AR), Internal Audit Department (lAD) reports, and staff time allocation records as well as extensive inter- views with 50 experienced Bank staff and a review of 16 case study audits and of the supervision aspects of over 100 ex-post evaluation reports (PARs). It did not involve any participation in supervision missions or independent field reviews of projects under implementation and supervision. Successive ARs have noted a declining proportion of completed projects rated as "satisfactory". Supervision by the Bank is only one of many factors which can influence project outcome. Bank management, in cognizance of the role supervision can play, has introduced numerous improvements in the process over the last decade, but this has failed so far to enhance the overall performance of the portfolio. This suggests that the improved processes have not yet had time to work through to project outcomes, that further improvements in supervision are necessary, and/or that factors other than the quantity and quality of implementation assistance by the Bank are more important determinants of project outcome. The study was undertaken with the knowledge that the role of supervision in project performance is a very complex issue, and it was not anticipated that its relative importance would be easily identified or that iny "quick-fix" measures would be available to remedy any observed deficiencies in the supervision function. BANK EXPERIENrE IN PROJECT SUPERVISION EXECUTIVE SUMMARY SUPERVISION audits and the supervision aspects of over 100 ex-post evaluation reports (PARs). A study of 1. "Supervision" refers to any contact Opera- relevant Bank Operational Directives (ODs) and a tions has with the loan/credit portfolio aftur historical review of the evolution of Bank Board approval. It serves to inform management supervision practice were also undertaken. of progress in implementation; it identifies Further details on the Evolution of Bank Supervi- emerging problems and suggests possible rem- sion and a Summary of Staff Interviews are edies both to Bank management and the bor- provided for interested readers in two Working rower; through its interaction with the borrower Papers (WPI and WPII, respectively). The report it assists the borrower and/or its agencies to focuses on investment projects rather than address issues and resolve problems; it monitors adjustment operations. compliance with agreed conditionality and assures that the Bank's procurement guidelines THE PORTFOLIO PROBLEM are followed; and it recommends project cancella- tion or redesign in the event of unsatisfactory 3. Both the ex-post evaluation results (paras. implementation experience, which may some- 3.1 to 3.3) and reports by staff on implementa- times be attributed to exogenous and unforeseen tion progress (paras. 3.4 to 3.6) indicate that factors. This comprehensive iterpretation of there are serious project implementation problems supervision, which includes a s.gnificant degree in the Bank's portfolio. of implementation assistance, is in accordance with the Bank's OD 13.05, and the report 4. The Annual Reviews of Evaluation Results considers that this role is appropriate for the (AR) for the cohorts of projects evaluated Bank as a major development institution; this, annually over the last decade indicate a declining rather than the use of a narrow interpretation of trend in the reported portfolio performance, with supervision pertaining tc monitoring of progress a marked decline in the last few years. The con- and "policing" implementation to ensure compli- clusions concerning the performance of cohorts ance with loan conditionality (paras. 2.1 to 2.4, analyzed in recent years, however, has been 3.11, 3.12, 2.42 and 4.4). affected by a Bank-wide campaign to corplete overdue Ptoject Completion Reports (PCR). As THE STUDY the ratio of unsatisfactory performers in this group was disproportionately high, this has pulled 2. Supervision of the implementation of proj- down the overall satisfactory rating for the proj- ects uses about a quarter of the Bank's opera- ects reviewed in these years. However, when tional staff time. An earlier report on the annual evaluation data are examined on the basis effectiveness of Bank supervision was submitted of the year of approval to eliminate this influeace, by the Operations Evaluation Department (OED) average results for projects appraised prior to to the Board in 1980. Changes in the Bank's 1973 were distinctly superior to later projects; portfolio, an internal reorganization, expressed the performance for projects approved from 1974 concerns as to the "quality" of the portfolio, and to 1982 is irregular; and projects approved in the need to examine the impact of the measures subsequent years 1-ave a significant downward adopted following the 1980 report, suggested trend but do not represent the full cohort of that another review would be timely. This report projects approved in those years because many is based on use of data from the Bank's various are not yet evaluated. information systems and databases, as well as extensive interviews with 50 experienced Bank 5. The ratings of ongoing projects from super- staff (para. 1.2) and a review of 16 case study vision reports in 17 Annual Reviews of Implemen- tation and Supervision (ARIS) also reveal a in supervision performance as a significant con- declining performance trend, as reflected in the tributor to the unfavorable trend in project out- average status ratings and the percentage of comes. This should not be interpreted, however, problem projects. Of particular note is the to indicate that staff are less concerned about decline in ratings after FY88. project design failings; despite the supervision emphasis of the interviews, the need for better- THE CAUSES OF THE PROBLEM prepared projects was frequently highlighted. Indeed, the review and analysis of more than 100 6. The implementation and final Impact of a PARs as part of this study failed to establish any project can be affected by a large array of endo- meaningful relationship between level of super- genous and exogenous factors, and .. : '- vision and serious implementation problems or there a single cause of project success or fail, project outcomes, despite the numerous refer- Undoubtedly, a well conceived and appraised ences to inadequate or deficient supervision in project is the primary factor affecting positive projects which performed poorly; this is consis- project results. However, even with an appro- tent with the assertions in paragraph 6. Never- priate design which has realistically considered theless, staff were convinced that the Bank's the risks, the macroeconomic and sectoral ability to supervise projects had declined, that environments and the implementation capacity, inadequate attention and resources were allo- a project can be faced with unpredictable cated to supervision, and that there was insuffi- influences which impede implementation and cient recognition for staff performance in respect necessitate adjustments. This implies that the of supervision. The interviews provide some in- supervision functions noted in the first paragraph sight into the characteristics of the prcblem and have a critical role in facilitating a successful its underlying causes as perceived by staff outcome of Bank-assisted development projects. (paras. 3.47 to 3.70). The extent to which the amount and/or quality of supervision may have contributed to the apparent ACTONs TAKEN decline in performance of the portfolio, however, is difficult to determine. 9. Notwithstanding these criticisms, Bank management has frequently acknowledged the 7. The 1990 AR pointed out that the lower need to improve project performance and the role performance of Bank-supported projects that supervision can play in this improvement. approved after 1973 could be partially explained Measures taken since the 1980 supervision by the increasing diversity, complexity and riski- report include: the introduction of leaving an ness of Bank operations. Another major factor aide-memoire in the country at the completion of contributing to the decline was the significant supervision missions to reflect major findings, instability in the world economy in the late-1 970s agreements on action plans and recommenda- to mid-1 980s. This implies that project design in- tions; use of mid-term project reviews; the hiring adequacies and negative macroeconomic factors, of local staff to supervise from resident missions; rather than shortcomings in implementation not the introduction of project launch workshops, of adequately addressed by supervision, may have Country Implementation Reviews (CIR) involving been the major contributors to declining perform- senior officials of the Bank and the borrower, of ance. The AR90 analysis, however, concluded portfolio restructuring and, recently in one region, that, despite the critical nature of other variables, the appointment of country teams to review per- the ability of the borrower and the Bank (through formance ratings and the creation of regional its supervision function) to respond to rapidly networking groups with specialized interests. changing external circumstances also had a major Revised Operational Directives on Supervision impact on project outcome. (OD 13.05) and Monitoring and Evaluation (OD 10.70) have been issued, and specify that a 8. The staff interviews reported in WPII Supervision Plan must be included in the Staff (which records 277 statements in support of its Appraisal Report (SAR) and that a record of findings) were more direct in suggesting a decline actions taken on earlier recommendations be in- -luded in the Implementation Summary in the consideration is that tha Bank needs to be more Form 590. In themselves, these initiatives realistic in assessing the ability ^ the borrower's demonstrate an awareness of the problem and of institutions to implement programs, and to design the need to maka improvements, but actions projects accordingly. No amount of Bank assist- taken so fer have not yet resulted In improved ance can overcome a situation in which the performance. There remains an issue as to project management requirements are beyond the whether enough is being done by the Bank to capacity of the executing agency, even with enhance the chances for successful implements- liberal built-in technical assistance (R1). tion of Bank-supported projects and programs. 12. A recommendation is also developed con- RECOMMENDATIONS AND SUGGESTIONS cerning the need for the borrower to institute Its own independent supervision capacity (R2 and 10. The information from the f4atabases, SI). A significant proportion of the Bank's super- reports and anecdotal experiences reviewed in vision time is used inefficiently on data gathering this study provide an indication of those which should be the responsibility of the measures which could contribute to improved borrower. Bank supervision and project implementation. In Section 4 of the report, these are developid into 13. B. Incentives and Resources. This theme recommendations which are grouped according points out that unless there are incentives for to the following themes: staff performance in facilitating the successful implementation of projects, the required emphasis - responsibilities for project implementation by Bank staff in this function is unlikely to be and supervision, realized. This applies to both promotional and performance meri criteria (R3). These real incen- - incentives for performance in assisting tives could be complemented by a reaffirmation successful project implementation and of the importance the Bank places on successful provision of resources to carry out this implementation of projects in a revision of OD function, 13.05 (R4). - Improvement in the capability of staff to 14. The Bank's Boar I is primarily concerned supervise projects, and with the policies, priori:ies and direction of the Bank's development lenting program. However, - procedures which can enhance the implementation problems in the existing portfolio efficiency of the supervision process. and the outcome of con oleted projects should weigh heavily in the Board's deliberations. If the There are 19 reccmmendations (R) and 14 sug- Board were to give more time to consideration of gestions (S), the latter being considered of less both these analyses, this would further empha- significance than the former. The recommends- size to Bank staff (and management) that the tionrs on each theme are summarized below. successful outcome of its projects must be the primary concern of the Bank (R5). More explicit I 1. A. Resoonsibilitles. This theme confirms reporting by Bank staff on the performance of on- the responsibility of the borrower for project going projects and programs in Country Briefs implementation, but points out that this does not and Country Assistance Strategy discussions absolve the Bank of its responsibility in assisting would enable the Board to be better informed in the borrower and its executing agencies to its consideration of the lending directions and successfully implement the project. Although strategies for individual borrowers. Bank assistance can be directed through indi- vidual projects, the primary focus should be on 15. If the supervision function is to be im- developing the overall implementation capability proved, it is obvious that the resources must be of the borrower which demands a more compre- available to Division Chiefs and Task Managers to hensive assistance strategy. The other important provide the required services. Resources should - vi - be calculated in accordance with the individual stances which make successful completion of the project needs. The inadequacy of resources is project improbable (S4). often a matter of resource allocation within a budget rather than an inadequacy in the total 19. As supervision of the procurement process budget per se (R6 and S2). Appropriate occupies an inordinate amount of staff resources, decisions on allocation will relste to the percep- the report recommends that greater use be made tion of the importance of the supervision of provisions in OD 11.02 which allow for inde- function, which is partly dependent on how the pendent non-Bank review of procurement docu- recommendation referred to in the previous two ments. More frequent use of model bidding paragraphs are developed. documents and of a requirement that major bidding documents be available prior to loan 16. C.Stqlff Capability. This theme empha- processing are also recommended (R13). It is sizes the need to increase the average cap..ility considered that the Bank's procurement guide- of staff, especially in the implomentation assist- lines should be reviewed with the objective of ance aspects of supervision (see para. 1), by (a) reducing the demand on staff resources without placing greater weight on hands-on development losing the advantages of competitive bidding, and experience in staff recruitment, (b) introducing a that the use of specialized procurement staff supervision training series based on a case study should be expanded (R14 and S5). approach, and (c) updating the Supervision Hand- book for use by staff (R7 to R9 and S3). 20. The existing procedures for reviewing project accounts and statements of expenditures 17. D. Procedures to Enhance the Supervislon (SOE) do not appear to do justice to the process Process. Recommended procedures include the and, at the same time, oblige many supervision use of a detailed and sequential implementatfon staff to become involved in this activity in which plan in the SAR to encourage greater attention to they have little or no skills or interest. The report feasibility of proposals, to facilitate dialogue with recommends that the use of centralized regional the borrower regarding feasibility, and to provide review of accounts be instituted (R15), and that assistance for subsequent supervision missions the responsibilities of supervision staff for SOE (R10). An expansion of the risk analysis at accounts review be simplified (R16); it also project appraisal to consider the tolerable limits suggests that the Bank increase its participation for variations in key variables (rather than the in programs designed to improve the public commonly used "mechanical" calculation of the sector accounting and auditing capability of its impact of a percentage change in inputs and out- borrowers (S6). puts) is also recommended; this would provide "flags" to signal special attention during 21. Three recommendations are made relating implementation (R1 1). to the assessment and analysis of project imple- mentation itatus. To make ratings more com- 18. Provision of the required skills to identify parable and reliable, a formal review system for and address emerging and existing problems in a ratings is recommended until management is con- project would be facilitated by obliging the Task vinced that uniform standards are being practiced Manager to indicate after each supervision (R1 7). An independent Input into the review of mission the required skills he or she considers projects which are persistently rated as problem necessary to provide appropriate supervision projects (R18), and the adoption of a mid-term assistance in the near future; inexperienced staff review (MTR) for all projects except those rated would be expected to consult with the TD or as virtually problem-free (R19', complete this set other experienced personnel to determine this of recommendations. (R12). It is also suggested that the Task Manager should specify in the Form 590 whether 22. A number of suggestions are made pertain- the project should be subject to a major review in ing to the handover process between supervision consideration of current and anticipated circum- Tasl: Managers (S7); the need for improvements - vii - In evaluation methodologies (S8, S9 and S10); the Bank as a development institution) then this improved record keeping (511); increased access would likely represent an increased efficiency in to supervision data on the Bank'u information resource use. systems (S1 2); the possibility of sharing Form 590 with the borrower (S1 3); and the use of a LoOKINo AHEAD single stioervisory unit to act on behalf of all donors in comp'. multi-donor projects (S14). 25. The rather extensive list of recommenda- tions and suggestions should not be allowed to ADDITONAL RESOURCES detract from the r',cent considerable progress made in some sectirs and regions in institut'i[' 23. Some of the recommendations and sugges- an enhanced concern for the successful imple- tions are resource-neutral, others would require mentation of Bank-supported r.ojects (paras. more supervision resources, while still others 2.31-2.35, 2.41-2.44 and 2.47). An improved could reduce the resource requirements. The success rate in projects could be expected if the appeal in R6 to allocate sufficient supervision Iank expands on these initiatives and clearly resources implies an increased resource use for acknowledges that its performance can only be supervision, but this need not mean significantly measuied in the outcome of its programs and increased total operational resources if the rate of projects. This acknowledgement would have to expansion of the lending program is adjusted be reflected in internal policies, incentive systems accordingly. If the procurement guidelines can and procedures. The institutionalization of a be simplified (R14), ar f 'or the procurement docu- general "implementation culture" would ensure ment review by Bank staff can be reduced (R1i3), that the Bank is able and willing to provide the this should lessen the resources needed for this informed implementation advice which should be time-consuming component of supervision. The expected of a reputable international develop- use of a more specialized procurement review ment institution; just as importantly, however, process should be tesource-neutral, as should the this culture would encourage the upstream recommendations concerning modifications in the activities of economic and sector work and review of project accounts. project design to provide the best possible, controllable conditions for project imp!ementa- 24. Recommendations 17, 18 and 19 concern- tion. Any targeting of number or size of loans or ing increased attention to the assessment of net transfer of funds to a borrower would be project implementation status do imply a marginal subordinate to the primary objective of selection increase in the resources which would be applied and successful implementation and outcome of to supervision. If, however, these measures feasible projects which have established develop- resulted in a greater proportion of successfully ment priority. implemented projects (the ultimate objective of BANK EXPERIENCE IN PROJECT SUPERVISION I. BACKGROUND 1.1 Supervision uses about a quarter of the Bank's operational staff-time, and about 60 percent of this is used for supervision of specific investment projects. In 1980, the Operations Evaluation Department (OED) submitted its first report on the effectiveness of Bank supervision to the Board. Implementation of the key recommendations of this report together with changes in the Bank's portfolio, the reorganization, and concern as to the "quality" of the Bank's portfolio, suggested the need for another review of the effectiveness of project supervision. This report is based on data from the Bank's Management Information System (MIS), Annual Reviews of Implementation and Supervision (ARIS) and Evaluation Results (AR), Internal Audit Department (IAD) reports, and staff time allocation records as well as extensive interviews with 50 experienced Bank staff and a review of 16 case study audits and of the supervision aspects of over 100 ex-post evaluation reports (PARs). 1.2 A Working Paper "The Evolution of Bank Project Supervision" (WPI), shows how changes in the nature of the Bank's portfolio have prompted changes in supervision. These changes have been predominantly marginal improvements and adjustments, rather than revolu- tionary new beginnings. A second Working Paper "Report on Interviews with Bank Staff" (WPIl) summarizes and illustrates the views of Bank staff. Staff were selected to give a wide coverage of the regions, country departments and sectors existing at the time of the survey in February-March 1991; at least two divisions per country department were chosen; and the emphasis was on Task Managers from the sectoral operations divisions (76 percent of interviewees), most of whom were nominated by their Division Chiefs..1/ Expressed in their own words these quotations have a directness and incisiveness to which an edited and agreed report cannot realistically aspire, and provide witners to a body of staff struggling against the odds to make Bank projects work, often without the recognition due for a job well done. Although the sample was not large, the credibility of the results is enhanced by (a) the broad coverage of the staff category which is largely responsible for carrying out the supervision function, and (b) the in-depth interviewing technique by an ex-SOD Division Chief (consultant) which encouraged elaboration of actual experiences. In view of their length, these Working Papers are not presented here. 1.3 The next section describes the purposes of supervision, documents the resources devoted to it by the Bank, and summarizes the improvements made by the Bank in the last decade. A third section reviews current Bank expectations and approaches to supervision, summarizes the views of staff directly involved, and comments on supervision by the borrower. A final section makes a number of (major) recommendations and (minor) suggestions prompted by the study. The balance of this section briefly reviews how the study was conducted. J. Sample breakdown (numbers) Repion Classification Sector Exerience Africa 12 Task Mgr. (SOD) 38 Agriculture 15 < 5 years 5 Asia 16 Task Mgr. (TD) 4 Infrastructure 12 5-10 years 8 EMENA 12 Div. Chief (SOD) 4 Industry & Energy 15 11-15 years 18 LAC 9 Div. Chief (TD) 1 Human Resources 6 16-20 years 14 PRE 1 Project Adviser 1 Management (non-sector) 2 > 20 years 5 (ex-SOD) Director (TD) 1 PRE (ex-SOD) 1 TOTAL 50 50 60 50 -2- 1.4 Methodoloya. Never an exact science, evaluation of a process such as supervision presents problems additional to those normally encountered in evaluation of investment projects. Whilst precise numbers are often available, they are seldom for precisely the desired quantity. Thus, one would like to know the value to borrowers of completed projects; what is readily available is the approved amount of loans (and credits) extended to borrowers. These approved amounts provide at best a proxy for the value of the corresponding projects to borrowers; a proxy which can be improved by taking account of ex-post judgments as to which projects were "satisfactory". 1.5 The inevitable lag between changed supervision practices and the completion of proj- ects which have benefitted from the changes inhibit the definition of their impact. More than a decade would need to separate AR cohorts which had (a) none, and (b) all projects subjects to a changed policy. The change may be identified precisely, but the impact is inherently fuzzy. 1.6 Just as projects approved in one year will be completed two to ten years later (and, on past experience, yield PCR's two to fourteen years later), so the AR cohort for a year includes projects approved from two to fourteen years earlier. Precise inference is not possible; but major changes can be observed. 1.7 Observed changes seldom stem from a single cause; preparation, appraisal, implementa- tion and external shocks all influence project outcomes. This study focusses oa supervision, but this is not intended to imply that it is the only, or even the primary causal factor influencing project outcomes. An attempt can be made to link the quantity and quality of supervision to a particular project outcome, but the complexity of the potential factors affecting final outcome makes this extremely difficult. 1.8 Similarly, with large numbers of staff, and large numbers of projects, almost any trend or generalization is likely to be accompanied by "an exception which proves the rule". This is particularly evident in the staff views summarized in WPII, but it applies equally to project experience. 1.9 With such uncertain evidence, the study can only seek to see where different sources of information are mutually reinforcing (or contradictory); to see where observed changes can be plausibly explained by common sense; to be as clear as possible on what is known, and what inferred; and to recognize that more than one explanation may be consistent with what has been observed. II. SUPERVISION: DEFINED, FUNDED AND IMPROVED 2.1 Supervision Defined. "Supervision" refers to contact Operations has with any type of lending operation after Board approval. The sub-categories of supervision captured in the Time Recording System (TRS) are discussed in paras. 2.13 and 2.16. The present study, however, is concerned with supervision of investment projects rather than the adjustment project portfolio.2/ Z/ For a review of the supervision of adjustment operations see, "Effectiveness of SAL Supervision and Monitoring," OED, Report No. 9711, June 26, 1991. -3- 2.2 The above is a purely mechanistic definition of supervision, which obscures the fact that exactly the same f)inctions may be carried out by Bank staff in the course of project redesign (counted as "supervision" since it occurs after Board approval) as in project appraisal. A more analytical description of what the function of supervision should be is attempted in paras. 2.5 to 2.8. 2.3 Purpose of Supervision. The mandate for project supervision derives from the Bank's Articles of Agreement: "The Bank shall make arrangements to ensure that the proceeds of any Loan are used for the purpose for which the Loan was granted, with due attention to considerations of economy and efficiency...". Article III, Section 5(b). It is interesting to note that even though supervision has been carried out in one form or another for all Bank operations, the first use of the term in the Bank appears to have been in 1948, in connection with a power loan to Chile.2/ 2.4 The most recent description of the purposes of supervision is contained in OD 13.05, para. 1, which may be summarized as: i) to ensure implementation of the project as agreed, ii) to identify problems, and help the borrower to resolve them; iii) to cancel projects which cannot reach their objectives, and iv) to learn how to further improve project design. 2.5 A Fail-safe Information System. In contrast to the above, the purpose of supervision could also be defined as the provision of a fail-safe information system. In this sense, a primary civil-works contractor needs to supervise that staff are turning up on time; that full weight is being delivered; or that inferior materials are not being substituted. Despite this contractor level supervision, the government needs a resident engineer to check (a) that the contractors supervision arrangements are effective, and (b) that specified practice is being followed. Even where the government has employed a resident engineer to look after its interests, the Bank needs to super- vise to assure itself that the standards being used are appropriate, and that the aspects of project implementation of primary interest to the Bank are being properly addressed. This supervision structure provides a hierarchy of second opinions. If one level fails to perform (either due to malfeasance, incompetence or good faith misunderstanding of what is expected), this should be identified by the indeoendent supervision being provided. 2.6 In projects implemented by a single prime contractor, it is standard practice to also employ a resident engineer to supervise construction on behalf of the borrower. In such cases, the Bank merely needs to supervise the work of the site engineer, whose reports typically serve as a primary input into the Bank's supervision. 2.7 Where infrastructural projects are implemented by a borrower agency (i.e. by "force accounts"), this independent supervision is often lost. In civil works constructed by force account, j/ WPI, para. 3.05. -4- the implementing agency may, in effect, report on the quality of its own work. Similarly, in the delivery of services the implementing department has sole responsibility for reporting on the effectiveness of its work. This is a sophisticated point, since a properly staffed independent monitoring unit, say located in a university or planning ministry, could provide independent supervision where the same unit located in the implementing agency could not. (An in-house monitoring unit could well be indispensable to the agency in its management of the project; but it could not be expected to assure central ministries, or the Bank, that "there are no over-looked problems".) 2.8 Except in the case of major civil works, project design does not usually provide for independent supervision on behalf of the borrower. The result is that the Bank's supervision task becomes Herculean. Bank supervision has no lower-level independent supervision group to provide it with needed basic data which can be confirmed on a sample basis and, by default, it has to provide for the supervision needs of the borrower as well as the Bank. 2.9 Viewed from this perspective, there should be no complaint from borrowers that the Bank had failed to provide adequate supervision or technical assistance. Important though these deficiencies might be, the valid complaint would be that the project design had failed to provide these services. In a properly designed project, the borrower's supervision needs should be built into the project; a tautology which is nevertheless often honored in the breach. 2.10 A review of 349 SARs approved between April 2, 1989 and March 31, 1991, showed that 58 percent of them included specific reference to monitoring and evaluation in the table of contents, and 39 percent mentioned the provisions for progress reporting. In only one case (Rwanda Agricultural Services Project, Cr. 2026-RW) was there explicit reference to the existence of a monitoring group external to the project with responsibility for providing an independent assessment of project progress. The SAR proposed to by-pass this external monitoring group (on the grounds that it was understaffed and under-equipped) and provide for the usual M&E reporting to project management, implying that the independence needed to assure a fail-safe system would be missing. It thus appears that the common Bank practice is to rely on Bank supervision to meet the borrower's needs for independent supervision. 2.11 Nuances of Responsibility. Bank interest in project success is a major motivation for supervision; but legal responsibility for project outcome rests with the borrower.A/ The Bank acts in many ways as if it were responsible; extensive project appraisal, detailed legal covenants as to how the project will be implemented, supervision missions to follow progress and assist in problem resolution, and suspension of disbursements or even cancellation, if the prospects for a success are judged to be poor. The Annual Review of Implementation and Supervision (ARIS), and OED's activities all point to a major concern for project success. I/ For a staff view of how this responsibility may work out In practice see WPII, e.g. " Then you have the transfer of staff In the borrower's country; every six years they have elections and the out-going staff take the English reports with them so you have always to carry with you a handful of appraisal reports. We have enormous problems with the borrower knowing what he Is supposed to do. I have been on missions where the borrower hadn't the slIghtest clue as to what he was supposed to do. So you start all over again. " (WPII, para. 7.10, 213) or "As far as Implementation is concerned a lot of the time, and I find this increasingly so, the people who are Implementing are not sufficiently aware of what the Bank requires, of what In fact they are supposed to be trying to achieve. In other words, I mean, It's a sort of condemna- tion of both sides." (WPII, para. 7.10, 214), or ' This Is different to other developing countries to the extent that the Bank Is pushing something on those countries.' (WPII, para. 7.02, 193). -5- 2.12 Proper understanding of the Bank's expectations from supervision is crucial to its proper execution and funding. At one extreme, rigid acceptance of the borrower's legal responsibility for implementation of its project could lead to poor use of the Bank's development lending resources in many cases; at the other extreme, too activist an approach could lead to a de facto managerial role for Bank staff, with an undesirable degree of de facto responsibility for project outcome and missed opportunities to build local institutional capabilities. The proper role for the Bank can perhaps be expressed as "doing everything reasonable to assure project success, short of adopting a managerial role" ("reasonable," implying that likely benefits exceed costs). In practice, it is to be expected that the supervision requirements will vary according to the stage of institutional develop- ment in each borrowing country. Those borrowers with weak institutions will require more supervision to assist them realize the intended project outcome. A judgment has to be made, however, in approving only projects which are within the implementation capabilities of the borrower, so that excessive management intervention by the Bank is unnecessary. 2.13 Supervision Resources in Perspective. Of total Bank staff-time reported in the Time Recording System (TRS)5/ for FY91, 26.1 percent was "overhead" (i.e. leave, holidays, training, management, and general), 17.6 percent was "non-operational", leaving 56.3 percent for "operational" work. This latter was split between Lending 21.0 percent, Supervision 13.6 percent, Economic and Sector Work (ESW) 13.3 percent and Other 8.4 percent. Thus, about 14 percent of staff-time (or 24 percent of Operational staff-time) is spent on supervision, (see Annex Tables 1 and 2). 2.14 Of the 27,860 staffweeks reported as supervision in FY91, 86.3 percent was used for projects included in the ARIS (all open projects with an undisbursed balance of $100,000 or more). Split another way, 85.0 percent went for project-specific supervision, 7.6 percent for preparation of Project Completion Reports (PCRs), and the remaining 7.4 percent for non-project specific tasks principally preparation of the ARIS and Country Implementation Reviews (CIRs). Of the 23,676 staffweeks reported for project supervision, 14,292 (or 60.4 percent) went for supervision of investment projects, whilst the balance was used for supervision of policy-based and other lending. Thus only 12.4 percent of total operational staffweeks were used for proiect-soecific supervision of investment proiects. the primary interest of this study. 2.15 Total reported operational time has climbed fairly steadily from 1974 to date, more or less in line with the number of ARIS projects in the portfolio, except for FY83 and for FY87, following the reorganization, when operational time fell slightly (Annex Table 3). Changes over the last decade can be examined by comparing the staffweeks allocated in the years FY80-82 and FY89-91; whereas the total operational staffweeks increased by 36 percent, the resources allocated to supervision increased by 27 percent and to lending by only 7 percent. These changes coinc,Jed with an actual decline in the number of loans being processed, but with a rise of 20 percent in real volume of lending and an increase of 27 percent in the number of active projects in the portfolio._/ These developments are reflected in Annex Table 4 which shows the proportion of staff time allocation. The declines in relative allocation to lending and supervision were to the benefit of economic and sector work which increased by 66 percent over the same period (and II It should be realized that the TRS data must be interpreted with a degree of skepticism. Many staff treat this administrative task rather casually, and there may be even conscious "adjustment" of allocations to be more in line with expectations in some cases. J/ These data are consistent with the supervision per-project and per-loan value data in para. 2.22 which show a relatively constant per-project coefficient but a declining per-loan value supervision coefficient. -6- which most would consider was at least partly warranted), and by a more than doubling of the "other" category which is more difficult to explain. 2.16 Whilst the TRS identifies the project being supervised, project specific supervision (TRS Code 401) does not describe the activity (procurement, reading reports, reporting to management, preparation for ARIS or CIR, travel, on mission, site visits or the like) with the result that the relative time spent on these different activities cannot be analyzed. In FY90 26.7 percent of time reported on project-specific supervision was in the field. 2.17 Data from a sector division which has created a more rofined time reporting system showed that total "project specific supervision" was distributed as follows: Table 1: DISTRIBUTION OF PROJECT-SPECIFIC SUPERVISION (TRS Code 401) FY90 FY91 Total Used (SW) 271.7 250.6 of which Trust Fund (SW) 5.4 12.0 ARIS Projects (No.) 27 28 Used (SW/Project) 10.1 9.0 Used for % Procurement & Physical Improve. 35.9 37.3 Compliance/Audit 6.9 61 Gov't.Negotiations & Tech. Assist. 33.3 31.3 Internal Admin. & Reporting 9.0 13.3 Sub Project Administration 14.9 12.0 *SW = Staffweek A further analysis, separating physical implementation and site visits from procurement, and separating technical assistance from negotiations and administrative contacts with borrower agencies, would be de --nable. Nevertheless, this analysis allows us to see that about 80 percent of supervision time (40 perL int of which was on mission) was available for procurement and substantive project implementation. Without some such analysis, it is difficult for managers to know how supervision resources are being used; and hence to manage them in a meaningful sense. 2.18 The CAM (Country Assistance Manaoement) and the Budgetary Process. The CAM is a budgetary process designed to allocate resources to individual tasks subject to a total Bank-wide resource limit ("envelope") for operations. Under the CAM the available resources cascade down to Regions, Departments, Divisions, countries and Tasks, at each stage being divided in accordance with the relevant manager's view of priorities and relative work loads. Task Managers are then required to produce detailed budgets for the resources available to them. Scheduling conflicts for use of key staff, or to avoid too many missions being in the field simultaneously, are resolved; and the result is aggregated upwards to show how the available resources will be used. -7- 2.19 Supervision allocations in the divisional CAM are largely determined on the basis of historical coefficients,Z/ although managers can apply for higher allocations because of special requirements. This has permitted significant variation in the allocations between projects and even sectors (Annex Table 8), but the relative stagnation of the actual staffweeks reportedly used by supervision since FY84 (in the face of an increasing portfolio value and a declining performance measured by ex-post results) suggests that there is an informal "cap" on staff resources to be used for supervision. Such a ceiling would place a limit on the degree to which resource allocations can be truly responsive to individual project needs. This "cap" could be associated with a de fpct acceptance in the institution that lending must be expanded to increase the rate of development of borrowing countries, so that upstream activities and lending targets take precedence over the implementation phase of projects, within the total resources available. This is certainly the opinion of many staff in the Bank (para. 3.51), and, if a preoccupation with lending is affecting the attention that should be given to supervision, this is a very undesirable development. An insidious consequence of this impression by staff is that, even in nominating the requirements for supervision of specific projects, there is a tendency to be very conservative in requesting staff resources because of a knowledge that there are very real limits on what can be provided for the task. 2.20 It is interesting that the annual "contracts" which divisional and departmental managers make with regional management have usually included the upstream activities as number of appraisal departures, Board presentations and sectoral reports, but nothing on the implementation phase of the portfolio (except, perhaps, preparation of PCRs due to pressure from COD because of a back-log in deliveries). In November 1991, however, the Africa Region agreed to include commit- ments on mid-term reviews, CIRs, supervision intensity and other implementation matters as part of the negotiated "contract" exercise. 2.21 Divisional managers have considerable flexibility within their budgets to reallocate between the upstream tasks and supervision. However, transfer of resources to supervision is not usually appreciated if this is at the expense of meeting the contracted upstream targets. An analysis of budgeted and actual staff time for supervision (Annex Tables 5 and 6) indicates that in FY89, FY90 and FY91 there were more projects under supervision than budgeted for, but in FY89 and FY90 the actual supervision staff years used on supervision was slightly less than budgeted; this does not suggest that managers are using their prerogative to give additional resources to supervision. 2.22 Comparina Supervision Intensities. As shown in Annex Table 7, project-specific super- vision per project (last line) has stayed fairly constant from FY76 to FY90 when averaged over all operations, and figures f r the last few years have been at the higher end of the narrow range. However, if supervision .:; related to the value (in constant 1991 dollars) of the approved loan, a very different picture emerges with approximately a 36 percent decline in supervision resources per $100 million (in 1991 values). Whereas in FY76 the Bank was providing approximately 23.3 staff- weeks per year per $100 million lent, by FY90 this had declined to about 14.8 staffweeks. Despite the very different supervision intensities between sectors, 70 percent of the sectors dealing with investment projects have experienced a decline in coefficients over the last decade and especially in the latter part of the decade. Obviously, the required amount of supervision cannot be linked simply to loan size. Project complexity, the capacity of implementing agencies, and the incidence of exogenous factors which might prompt design modifications are very Z/ "Operations FY92 CAM Handbook", April 1991, page 3-13 "historical supervision coefficient ... should be used as guidelines in planning supervision input." -8- significant variables which would influence supervision requirements. However, the increase in project complexity and in negative exogenous factors which have occurredQ/ at the same time as the decline in supervision resources per $100 million lent (and a constant per-project supervision coefficient) support the contention that allocations have not been maintained at an appropriate level.a/ 2.23 To the extent that increasing loan value might have reflected a combination of factors which, on average, demanded more supervision resources, the decline shown in Annex Table 7 should not be read to imply that supervision intensities (per $100 million) were "right" in the early 1970s; merely that there has been a change. Indeed, It should be remembered that the relatively high number of projects judged satisfactory at that time were associated with still earlier, and unknown, supervision coefficients. One would probably have to go further back than 1974 to find "the golden age of supervision", if there ever was one. Nor should it be too quickly assumed that increased supervision resources would immediately resolve all quality problems; there may also have been changes in the type of project, the quality of appraisal and in staff skills. 2.24 The review of over 100 PARs in this study failed to yield any simple, meaningful relationship between supervision intensity and project outcomes. This is understandable considering the number of factors which can influence project results (para. 3.9).1Q/ The AR90 by OED found that countries with a higher GNP/capita and higher GNP growth rate tended to have more favorable project outcomes. An analysis of the staffweeks used per project from FY80 to FY91 in countries grouped according to GNP per capita (using 1991 IEC data) indicates that high- income countries tended to have less supervision attention than other borrowers (Figure 1). This suggests that supervision was not a critical factor in better project performance in these countries. There has been higher supervision coefficients for low-income borrowers (both small and large), implying that the Bank has been giving more supervision attention to these borrowers when implementation difficulties could be expected to be greater. On the other hand, an analysis of D/ August 30, 1991. Annual Review of Evaluation Results 1990, paras. 1.10-1.20. ./ The averages in Annex Table 7 disguise a vast variance in supervision time spent on individual projects, ranging from zero to 92.6 staffweeks (see Annex Table 8). In addition, it should be noted that several definitions of supervision intensity are possible (as illustrated by the several average supervision intensities given at the bottom of Annex Table 7). Supervision can be defined to include only "project specific supervision" (as in the body of Annex Table 7); or all "portfolio management" (specifically TRS Codes 401 through 406), or preparation of Project Completion Reports (TRS Code 402) can be dropped, or advisory activities (TRS Codes 1017, 1032 and 1034) can be added. The number of projects can include "all projects receiving, say, 0.1 weeks of supervision in the year" (as in the body of Annex Table 7), or only ARIS projects, or even for budgetary purposes, the expected number of projects to be supervised (Annex Table 5). J.&/ The report of the Task Force on the Relationship of Loan Processing to Project Ouality (in preparation) analyzed the OED ex-post evaluation outcomes of projects commenced after June 1971 and completed before July 1990 against a series of variables, including supervision inputs. This yielded a negative link between supervision intensity and performance, reflecting the responsiveness of Bank management to poor interim performance; poorly performing projects tend to be allotted more supervision input so that the simple relationship between a supervision index and project outcome is meaningless. The analysis of supervision response is very complex and difficult due to (a) the number of influencing variables, (b) the problem of identifying the outcome which would have been obtained with a different amount, type and/or frequency of supervision to that actually associated with a project, and (c) the crudeness of the measures currently used to assess outcome (paras. 4.43 and 4.44). -9- Figure 1: SUPERVISION PER PROJECT BY PER CAPITA GNP LEVEL 16- 14- *12- 10 4- 80 81 82 83 84 85 86 87 88 89 90 91 Fiscal Year H LOW (large) -W- LOW (small) -- MIDDLE (lower) MIDDLE (upper) By Income Levot DefinitionNExmelo Low Income Countries Per capita Income lees than or equal to $580 in 1989 of which *. Large Countries China and India * Small Countries All other countries Lower Middle Income Group Per capita income between US$581 and US$2,336 Upper Income Countries Per capita Income between US$2.336 and US$5,999 High Income Countries Per capita Income of US$6,000 or more (Cyprus. Bahamas, Barbados) Source: MIS Database. Figure 2: SUPERVISION PER PROJECT BY PER CAPITA GNP GROWTH 13 12.5 12 171 1.- I 10.5 10- 9- 8.5- 80 81 82 83 84 85 86 87 88 89 90 91 Fiscal Year Hig Moderate -U1- Negotive U Very Negotive Definitions: High growth Greater than 2% Moderate growth 0 to 2.0% Negative growth 0 to -2.0% Very negative growth Greater than *2.0% Source: MIS Database. supervision input per project in countries grouped according to GNP per capita growth rate indicates that more attention has been given to higher growth rate borrowers for most of the 1980s, although these differences were largely eliminated in FY90 and FY91 (Figure 2). It cannot be assumed that there is a cause and effect relationship between the higher supervision irput for borrowers with a higher growth rate and the reported better project performance, but this does suggest that operational divisions may have been giving more attention in supervision time to borrowers which were "on the move" and, therefore, more responsive.11/ 2.25 The FY91 ARIS noted better project implementation in countries which were making desirable adjustments in their economies and in countries which were less indebted. An analysis of staffweeks used per project from FY80 to FY91 in countries grouped according to adjustment status indicates that adjusting countries do receive more supervision attention (Figure 3). Here, again, cause and effect relationships cannot be assumed, but the findings do suggest that opera- tional divisions have tended to give some preference to borrowers where a positive response to supervision can be anticipated. An analysis of supervision input in accordance with countries grouped according to indebtedness does not reveal any striking differences (Figure 4), but, if any- thing, shows that borrowers with below average indebtedness at least were not favored in the allocation of supervision staff resources. 2.26 Project Ratings. The Implementation Summary (Form 590) requires the project to be rated on ten factors, the most important of which are Overall Status and Project Development Objectives, both are rated on 4 point scales. It is stipulated that "The OVERALL STATUS should never be given a better rating than that given to PROJECT DEVELOPMENT OBJECTIVES" (OD 13.05 Annex D2, para. 27).12/ 2.27 These two measures are supposed to provide information on three issues: (a) the current status of project implementation, (b) the expected achievement of project objectives, and (c) justification for continuation of the project with its present design. These are three separate dimensions of project performance, and they cannot satisfactorily be condensed into two. Consider an irrigation project which has been excellently implemented to date, justifying a "1" for implementation; but where world rice prices have halved, thus ensuring that the PCR Internal Economic Rate of Return (ERR) will be much less than 10 percent for the project as a whole, justifying a "4 for Development Objective (since project benefits will fall well short of project costs), which rating normally implies that "continuation of the project is no longer justified." But should the project be continued? That depends on the relation of additional benefits to additional costs. If all major investments have been made, even with a halving of rice price, project completion may yield a high ratio of extra benefits to extra costs. A rating of "3" under Develop- ment Objectives implies that the current design should be reconsidered, as there is "doubt about the continued justification for the project in its existing form." The need for redesign, however, is 1]/ The AR90 noted that, for the most part, the high and low growth rate countries were not the same as those in the high and low income level groups, so that the results indicated in Figures 1 and 2 in this analysis are not inconsistent. The level of development and recent country economic performance may have acted largely independently in terms of their possible effect on project performance. 12/ Interestingly, for the ARIS 91 the Africa Region explicitly decided to divorce these two ratings, so that "a project may be given a higher overall rating than the development objective rating ... (e.g., excellent per- formance at the project level in a dismal but expected-to-improve economic environment)," (memo, July 12, 1991). - 11. - Figure 3: SUPERVISION PER PROJECT BY STATUS OF ECONOMIC ADJUSTMENT 13 12- 10. ! 10 8- 80 81 82 83 84 85 86 87 88 89 g 91 Fiscal Year -4.- w dusto -0- ADJsTHo W/ aAe 44 ADJuSTIM8 W/O oAe Adjusting Countries: Undertaking a stabilization and an adjustment program * with Bank assistance With current Bank assistance program * without Bank assistance Following appropriate economic policies without current Bank assistance program Non*adjusting Countries: Leaking any adjustment program Source: Classification of adjustment status from ARIS FY91 MIS Database. Figure 4: SUPERVISION PER PROJECT BY INDEBTEDNESS 13 - 12.5 12 11.5 11 Q. V, 10.5 10 9.5 9. 8.5 80 81 82 83 84 85 86 87 88 89 -90 91 Fiscal Year SEVEPE -i- MODERATE BELOW AVERAGE By Level of Indebtednese (IEC, Classification of Economics. May 2. 1991) Severely Indebted Countries: Three of four key ratios above critical areas * Debt to GNP 50% * Debt to export of goods and services 275% * Accrued debt service to exports 30% * Accrued interest to exports 20% Moderately indebted Countries: Three of th6 four key ratios within a certain range * Debt to GNP 30-50% * Debt to export of goods and services 18*275% * Acorued debt service to exports 18*30% * Accrued interest to exports 12-20% Below Average Indebtedness: Key ratios below the lower point of the moderately Indebted countries Source: MIS database. - 12- muddled by reference to other factors. This dilemma couid be addressed by adding an additional mandatory rating on the desirability of continuing with the project with the present design, implying in the negative a priority need for an in-depth review of the project to decide on redesign or cancellation. Such a recommendation could also be included in Section 8 of the Form 590 ("Actions Recommended"), but this as yet is not recorded in the MIS. In essence, there is a need for the Task Manager to specify if the project circumstances are such that an in-depth review mission is necessary. 2.28 If the project has been redesigned, then project objectives are also likely to have been redefined. Rating, in this case, should refer to the likelihood of the revised objectives being achieved. 2.29 Imoortant Innovations. The key innovations which have been made since OED's 1980 report are discussed below. 2.30 AIde MemoIre. Used intermittently earlier, OED's 1980 report recommended that an Aide Memoire be left by every supervision mission. This recommendation was accepted and was eventually codified in March 1989. Properly us&d, the Aide Memoire can reinforce project manage- ment by recording the progress achieved, as well as serving as a reminder of what needs to be done, by when, and by whom.1a/ Routine use of Aide Memoires has augmented the authority of the supervision mission leader and his independence from Washington. Despite the occasional "clarification" which has to be sent, this authority has been contrasted favorably by borrowers with some other multi-lateral banks, whose missions are seen to serve primarily a reporting function for headquarters. 2.31 Local Hire. Starting in 1974, Resident Missions have employed local professionals to assist with operational work. In countries with insufficient work to justify full time appointments, part time appointments or the use of local consultants can be considered (OD 13.05, para. 24). Especially in (but not limited to) countries where the working language is non-european, local staff have proved to be an effective complement to regular staff. They (a) know-the language, (b) know the responsible officials, (c) know local customs, (d) are often technically, rather than economically, qualified, and (e) are able to provide continuity in following up on procurement and other aspects of supervision. In addition, their lower cost means that a division can devote more time to supervision without exhausting the budget. (Since FY90, the CAM allows four local hire staff to be substituted for one regular position, reflecting their lower cost.) This said, they can help essentialiy at the margin: they cannot be expected to provide the width of Bank and international experience which is characteristic of experienced regular staff, nor the authority that regular staff have in communi- cating with high level local officials. There is also a problem in lack of a carees path for locally employed professionals which can limit the ability to attract the best candidates in some countries.J./ 2.32 Despite the obvious advantages in using local-hire staff in supervision, there is need for some caution to avoid over-reliance on this resource. Exposure to implementation problems and j.J/ OD 13.05 recommends "concentrating mainly on implementation problems..." (para.56). Equal weight should, perhaps, be given to the recognition of achievements even at the expense of a somewhat longer document. 14/ Even when rewarded for increased experience, work in a Resident Mission provides a short career path. Transfer to regular staff, on the other hand, would remove the budgetary incentive to employ local staff. -13- experience in dealing with these problems are very important aspects in the professional develop- ment of Bank staff. Without a profound knowledge of implementation capacity and circumstances, there is a serious danger of processing projects which are unrealistic in their expectations. It is essential, therefore, to keep headquarters staff closely involved in supervision of projects in countries where the resident capacity is developed. 2.33 Project Launch. Especially in projects to be implemented by a variety of agencies, it has proved useful to organize a "project launch" immediately after effectiveness, or sometimes as a part of the strategy of achieving effectiveness. By ensuring that agencies agree between them- selves as to what each is expected to do, when they are expected complete their tasks, and how one agency's actions may facilitate or impede other agencies, a project launch can greatly reduce the time taken for a project to take-off (OD 13.05, para. 10). In addition, the 'social pressure" on an agency to deliver on what it has publicly undertaken to do, can be valuable. The project launch is a valuable tool in that it attempts to avoid the development of problem projects. which is a more efficient use of staff resources than addressing major problems after they have arisen. The oppor- tunity to use this venue to clearly establish SOE accounting and audit report requirements, however, has not always been realized. 2.34 Mid-Term Review. A mid-term review (recently made mandatory for all new investment projects in the Africa Region, para. 2.43) provides an opportunity for the Bank and borrower to assess progress, reflect on any changes in the external or political environment, and to agree on any desirable changes in project objectives or design, or invoke cancellation. From the Bank's viewpoint, it also provides an opportunity to augment the Task Manager's knowledge with other expertise drawn from across the Region. Whilst this may result in some debate on (and even change in) Bank priorities, it is important that the responsibility ot the Task Manager for reflecting Bank views be reinforced. The aim of a mid-term review is to strengthen implementation of the balance of the project; it should be forward looking except as experience to date should impact this perspective. 2.35 Project Supervision (OD 13.05). The new operational directive on supervision (OD 13.05) issued in March, 1989, made two changes which promise major benefits. The first was the requirement to automatically reproduce the recommendations from the previous Form 590, and to comment on how they were acted upon. The new OD also makes reference to a Supervision Plan which is to be included in the Staff Appraisal Report (SAR), so that the supervision requirements of the project will have to be appraised along with the rest of the project; and the borrower will know exante what level, timing and skills of supervision can be expected. (The OD for Processing of Investment Lending (OD 9.00) issued in June 1991 also specifies the requirement for a Supervision Plan.15/) Perhaps more importantly, Bank management will have told the Board the level of supervision resource to be provided to the project. This promises to lead to budgeting of super- vision resource based on the SAR determined requirements of the individual project, although this linkage does not appear to be developed yet. Due recognition of the Supervision Plan would obligate Bank management to agree to commit the indicated supervision resources, or insist that alternative arrangements be made to reduce the cost of supervision by the Bank without reducing the prospects for successful project implementation. Six projects presented to the Board in the year from April 1, 1989 (the issue date of OD 13.05), had Supervision Plans and in the next year 61 had a Plan. Thus only 67 of the 351 projects presented to the Board in the two years following the OD (19.1 percent) were in accordance with the directive, which was very low even though those SARs which were completed or close to Board review at the time of issue of the OD could j.J/ OD 9.00, footnote 24. - 14- have been excused from conforming. This reflects. perhaps, less priority to the supervision function than to other "obligatory" references in the SAR. 2.36 An innovation suggested but rejected in the preparation of OD 13.05 would have been to share the Form 590 with the borrower. Present practice means that the first time the borrower sees the Bank's project ratings is in the PCR. Whilst substantively the same information may have been conveyed in the aide memoire, nevertheless the shock of seeing, in cold print, that the project is being reported as a "4" (Major Problems - Not jeing addressed) could be salutary. 2.37 The arguments against sharing include a possible "chilling effect" on the frankness of the Task Manager's reporting, and possible borrower complaints in the event of apparently arbitrary changes in project ratings or perspectives. The latter complaint possibly can be viewed as a benefit, since it should reinforce the need to make such judgments carefully. The possibility of a "chilling effect" is moot, as Forms 590s seldom contain information which could not be shared. The most controversial issue is likely to be the rating for Project Management; a poor rating could lead to future difficulties in obtaining full cooperation from the executing agency, and perhaps lead to a less than candid recording of performance. Task Managers should, perhaps, be encouraged to share the Form 590 on an experimental basis, if they feel this would improve the borrowers under- standing of the Bank's perspective on implementation progress and priorities. 2.38 Issues Paper. If a project is experiencing complex problems, the OD indicates that an Issues Paper should be prepared prior to a supervision mission (OD 13.05, para. 50). This provides a useful artifact of the total preparatory work carried out prior to mission departure. If significant changes in project design are contemplated, then a formal Issues Meeting involving all interested departments should also be held (OD 13.05, para. 51). 2.39 Computers and Communicatons. Technological progress has transformed the infor- mation which can be conveniently available in the field. A Lap-Top computer can now contain the full text of the SAR..L/ and Legal Documents, project correspondence files, all previous Forms 590 and Aide Memoires, etrc. This takes organization and time to set up, but almost as much documentation can be available in the field, as in the office. Furthermore, word search programs can be used to immediately locate references (or reveal the absence of any reference) to key topics such as extension, project coordination, etc. Word processing programs allow rapid production and correction of key documents such as the Aide Memoire and Back to Office Report. Fax, and Electronic Mail from Resident Missions, provides for instantaneous transfer of documents to and from Washington. Rapid document production and transfer to Washington permit staff in the field to report their findings whilst still fresh, and having done so to concentrate on the next assignment. Individual Bank staff are rapidly and imaginatively capitalising on the new technological possibilities, but the Bank as a wholt has yet to provide guidelines for the less computer adept. The down-side to this improved communications capacity is that the very facility of communication may militate against the proper filing of communications; this should be addressed when guidelines are promulgated.17/ 1IA/ Ironically this is occurring just as the SAR itself has ceased to be an effective guide for project implemen- tation (para. 3.73). 17/ Currently the All-in-1 provides for "official" or "unofficial" communication; in the former case the author may choose to "file" or "not file" his message. It would be better for al official EMs to be filed; with a choice as to which "unofficial" communications would also be filed. - 15- 2.40 Since "one picture is worth a thousand words", the potential of video recording, should not be overlooked as a means of highlighting particular aspects of implementation on supervision within the Bank. 2.41 Implementation Plans. Education projects frequently include an implementation plan or matrix which ties project objectives to monitorable actions and outcomes through key performance indicators. Implementation plans vary in coverage and detail, the most elaborate being the imple- mentation matrix, which specifies the actions/investments to be made, who is responsible for taking action, exactly what will be monitored, by what date, and by whom./ Implementation plans are routinely used for education projects in the Asia region. The presentation of compre- hensive implementation plans in other sectors and other regions, however, is very limited. rhe matrix arrangement, in particular, has several good features, initer alia: i) It is a good preparation device, because it forces project designers to specify feasible objectives. Objectives that are unrealistic (because they cannot be achieved in the time or with the resources included in the project) or inappro- priate (because they do not relate to genuine needs) have a better chance to be detected and recast at an early stage. 19/ ii) It is a good supervision device, because the objective statement of what is to be measured provides the basis for any contracting out project supervision respon- sibilities (either on beh.if of the borrower or the Bank). ii) It is a good monitoring device, because it provides a clear link between project objectives and the actions proposed to meet them. This is in marked distinction to analysis by category of expenditure, which--while useful for monitoring disbursements--is ineffective for judging achievement of non-financable project items (such as changes in policy). iv) It is a good device to assist in mid-term evaluation, because it assigns specific responsibility for taking actions and monitoring outcomes to individuals and organizations, and allows for early detection of deviations. v) It is a good evaluation device, because it specifies what will be measured as evidence that an action has been taken. As noted, monitorable "artifacts" are not limited to hardware or contractual service, but can include such things as the 1J/ See SARs for: Brazil, "Skills Formation Project," Annex 7 (Report No. 6564-BR, dated April 8, 1987); China, "Agricultural Education and Research Project," Annex 4 (Report No. 3819, dated October 14, 1982'; Indonesia, "Second University Development Project," Annex 10, Table 11, (Report No. 5354, dated April 26, 1985); and Indonesia, "Higher Education Development Project," Annex 5 (Report No. 7085, dated April 12, 1988). ja/ An interesting example is provided by the case study audit where a key indicator of the improved quality of university staff was the number of staff sent abroad for training. Many more were sent overseas than anticipated, thus giving the impression of great success. However, at the time of audit, about one-third had actually returned to resume their teaching duties, delaying-perhaps indefinitely-the beneficial impact of their overseas experience for their host institution. In retrospect, the number trained and returned to 2ggt would have been a more appropriate indicator, and would have forced some consideration of factors affecting decisions to return. (Case Study Audit of "China: Agricultural Education and Research Project", Credit 1297-CHA, para. 4.06). - 16- convening of meetings, passage of legislation, or official announcement through publication in the public record. In order to optimize the usefulness of the imple- mentation plan, the Bank should study and develop planning frameworks which can be used as guidelines for staff in this exercise in SARs. 2.42 Implementation Culture. The staff interviews summarized in WPII, reveal a strong commitment amongst Sank staff to the successful completion of projects. A number of Country Departments have introduced initiatives which are designed to address mounting concern with implementation problems. Of particular note is the action recently taken by the management of the Af-ca Region; it has adopted an "implementation culture" meaning "an environment of incentives, processes and resources which encourages and supports the staff to strive for quality operations". Primary attention has been given to four main areas: (a) further enhancement of managerial attention to implementation, (b) dissemination of good practices and integration of lessons learnt in lending operations, (c) broadening of the implementation effort, and (d) improvements in auditing and procurement. 2.43 The result appears to be much greater attention to supervision, with the RVP, Directors and Senior (and Departmental) Operations Advisors, and Country and Sector Teams actively reviewing project ratings, problems and needed actions. Provision is to be made for a mid-term review of all new projects. This is expected to provide a needed demonstration of management's interest in supervision (see Box 1); to provide a "second opinion" on project ratings (para. 2.45), and the opdortunity to learn from others. Some additional resources have also been provided (see below). The Region has sought to stabilize the number of projects, so additional time can be freed to supervision. Planned (and actual) CAM supervision coefficients for the Africa Region have been 11.8 (11.1) in FY89, 13.2 (12.5) in FY90, 13.9 (12.8) in FY91, and 14.3 staffweeks per project are planned for FY92.2Q/ Another result of this new initiative is an expressed desire by the RVP to make good supervision an important criteria for promotion. A number of staff have been promoted primarily on the basis of good supervision and it has been a factor in other promotions. (It would be difficult, however, for a specific weight to be given to supervision performance in staff assessment due to the differing mix of responsibilities for individual staff, some of which may not be exposed to supervision.) 2.44 In agriculture, which has been one of the worst performing sectors in Africa, net- working arrangements have been introduced to promote thematic supervision (similar components of different projects are supervised by the same staff member), regional workshops of specialist Bank and borrower staff, and sectoral subgroups which develop a commonality of views on technical issues in the region. While emphasizing the role that improved supervision can play, agriculture staff accept that improved project preparation and design are essential if project implementation and impact are to be substantially improved.21/ 2.45 ARIS Review of Ratfngs. The Africa Region has also recently introduced the practice of having the Country Team review, as part of the ARIS process, the divisional Form 590 ratings of 2Q/ Exclusive of PCR preparation, with local staff discounted to a quarter of actual time. 21/ A quote from the Africa Agricultural Group Team memo to Africa Region Management Team, January 21, 1992: "Several staff observed that the purpose of project preparation is often distorted to the point that the preparation of a good appraisal report becomes the primary objective. Indeed, project design must be undertaken in a manner which looks to implementation. Design for implementation, not to satisfy appraisal reviewers, should be the focus." - 17 - "Overall Status" and "Project Development Objectives". This will improve comparability across divisions and also allows proper weighting of macro considerations. Any differences are first discussed between the division and country team. If a consensus cannot be reached, then the two judgments are ad, anced to the Director's level where debate and resolution occurs. This provides a useful check on what is otherwise an inherently subjective process.22/ 2.46 Monitoring and Evaluation (M&E). OD 10.70 requires that every investment project include plans for monitoring and evaluation (OD 10.70, para. 4). It also provides guidance as to the scope, management and use of M&E. In particular M&E is expected to reduce the time needed for data gathering for supervision missions, allowing them to concentrate on problem solving and technical assistance. Despite this expectation, few actual examples of effective M&E can be cited, and still fewer which have been used to complement Bank supervision (see, however, para. 3.76). 2.47 Country Implementation Review (CIR). Although CIRs were practiced by some divisions and departments prior to 1980, this activity which brings together senior officials of both the borrower and the Bank to address generic and selected specific implementation issues has been usefully expanded. In some countries, it has not been as successful as in others, usually due to lack of borrower commitment, but it continues to be a worthwhile technique in support of the supervision program. The country focus also promotes a more complete integration of sector and economic work with individual project operations. III. BANK ATTITUDES AND PROCEDURES 3.1 The Measure of Success. Board presentation of a lending operation is generally regarded as the high-point of the lending cycle. Although Board members do receive copies of PCRs, rarely does this generate any follow-up inquiry.2/ The result is that the Bank focusses on what is intended, not on what is actually being delivered. The value of loans approved in a year is readily available, and there has been an upward trend in the approved value of loans since 1974 (Figure 5). This should not, however, be considered as a measure of the Bank's performance. The ultimate measure is the value to the country of implemented projects. In the absence of an assess- ment of the present value of net benefits for each project, it is obvious that if the success rate for projects is declining, this must have serious implications not only for individual borrowers, but also for the performance of the Bank's portfolio2A/ as a whole. The decline in satisfactory projects indicated by ex-post evaluation (Figure 6) implies that the upward trend in approved loan value 2/ From a FY87-89 average of 18 percent of agricultural projects rated '3' or '4', in the Africa region, FY90 and FY91 proportion of these "problem" projects rose to 30 percent and 36 percent respectively. ... Nevertheless in preparation for ARIS 91, it was commented "The significant expansion in the proportion of problem projects reflects, in the opinion of the agricultural staff, more realistic ratings, rather than a deterioration of portfolio performance." There is even reference to "a better performing agricultural portfolio at present than was the case in the past," (files, October 15, 1991). ZJ/ On April 26, 1990, OED issued a PCR, "Indonesia: Nucleus Estate and Smallholder IV Project" (Ln. 1835- IND) Report No. 8511, in which approximately one-third of paragraphs stopped in mid-sentence. There has still to be a request from anyone outside OED for the full text. It is inconceivable that a 5imilar experiment with an SAR would pass apparently unnoticed. /AI "Portfolio" is used here to mean all projects which are supported by Bank loans and credits. - 18 - Figure 5: APPROVED VALUE OF LOANS 30 0 i 25 a20- 15- 10 O - CL 0- 74 76 78 80 82 84 86 88 90 CY of Approval TOTAL apoved Figure 6: COMPARISON OF SUCCESS RATE OF EVALUATED PROJECTS 100% X sawMAMW fa a 60% m 40% 74 76 78 80 82 84 86 88 90 Evaluation Year -19- would be considerably flattened if the actual net benefits generated by the Bank's projects were presented. 3.2 Excessive emphasis by the Bank's Board and senior management on the expansion of the lending program, without adequate attention to implementation problems and the evaluated outcome of projects, could lead to a false sense of institutional achievement. It could be postu- lated that Bank borrowers could have been better served by a slower increase in the level of project approvals, if this had allowed the earlier (higher) proportion of satisfactory projects to be maintained. 3.3 The foregoing indications of ex-post evaluation results are based on project performance analyzed by year of evaluation. However, the marked decline in performance in the most recent years has been affected by a program in Operations which cleared most of the backlog of PCRs. In line with a correlation between projects with a long lag from completion to evaluation and poorer performance, the proportion of these overdue evaluations which were unsatisfactory was unusually high. If project results are analyzed on the basis of years of approval some caution must be intro- duced with respect to the most recent years. There is a substantial decline in performance for those projects approved after 1973 compared to earlier projects; for project approved from 1974 to 1982, the annual results are irregular but have no significant trend; and for projects approved after 1982, although a significant decline is registered, the numbers of projects completed and evaluated per year naturally decrease, and the cohort is also affected by an abnormally high proportion of quick-disbursing loans. Therefore, this may not be indicative of the eventual results of the complete portfolios approved in these later years (Annex Tables 9 and 10). 3.4 Other Indicators of Outcome. While project outcomes must be the real indicator of success of the development portfolio, the project ratings made by supervision missions (paras. 2.26 and 2.27) can indicate the trend in performance. An analysis of ratings reported in the ARIS series from 1975 indicates that there has been a declining trend in the overall performance ratings and an upward trend in the percentage of problem projects (Annex Table 11 and Figures 7 and 8). The very poor performance in AR-1 990 and ARIS-91 supports the contention that there is a serious implementation problem. 3.5 A project with a persistent supervision rating as 3 and 4 (a problem project) could be expected to have an unsatisfactory outcome. Similarly, the percentage of problem projects in the portfolio could be expected to correlate with the percentage of unsatisfactory projects. This is not the case (see Annex Tables 9 and 11), as the percentage of problem projects is consistently lower than the percentage of unsatisfactory projects. This suggests that staff are rating projects too favorably during implementation. The percentage of problem projects, however, is based on all projects under supervision, which obviously includes those in their Initial years of implementation as well as more mature projects. Records show that the incidence of problem projects rating is low in the early years of projects and increases as the projects mature. This could be anticipated, as it would be natural for most supervising staff to give project management and/or the borrower some time to address any emerging problems. As the project progresses, if action is not taken the ratings reflect a declining performance. 3.6 A study undertaken by the Africa Region on its troublesome agricultural portfolio in 199025/ examined the differences between implementation and ex-post ratings in some detail. Z/ The Performance of Agricultural Projects in Africa - A Review of Project Ratings and Risks. September 12, 1990. - 20 - Figure 7: PROBLEM PROJECTS 0.21 0.2 0.19- 0.18- 0.17- C 0.16 - 0.15- 0.14- 90.13 - E 0.12 0.11 0.1 0.08 0.07.97 7.777.7.808 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 FISCAL YEAR Soure: Annex Table 11. Figure 8: OVERALL STATUS RATING 2.1 2- 8 1.9 o 1.8 1.7 - 1.6- 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 FISCAL YEAR Sources Annex Table 11. - 21- While 49 percent of projects evaluated between 1974 and 1988 were assessed as being unsatis- factory, the average percentage of problem projects in the ARIS from 1981 to 1988 ranged from only 15 percent to 21 percent. In an examination of individual projects, there was a 31 percent inconsistency between OED findings and the last supervision rating, and a higher 43 percent incon- sistency with ratings intermediate in the implementation period. A comparison of ratings of projects in the fourth and fifth year of implementation in the annual ARIS cohorts for investment projects from FY80 to FY91 is made against ex-post AR results in Figure 9. The AR cohort for a particular year is compared with the implementation ratings for the ARIS cohort three years earlier, as an approximation of comparable project cohorts; this also reveals a persistent gap between the final evaluation ratings and the implementation in these years, even though it is less than the gap between ex-post and average portfolio implementation status ratings (the average ratings for the comparable nine years are 27.8, 15.8 and 12 percent for the ex-post, years 4 and 5, and total portfolio respectively). 3.7 Cancellations. With a declining portfolio performance, it would be expected that cancellations would increase under management which resorted to this measure (as per OD 13.05 c.f. para. 2.4) to minimize economic costs as projects become untenable. Accessible annual data on cancellations were available from 1985. For the cohorts of loans/credits which had cancella- tions at or before closing date in these years (about 36 percent of operations), cancellations as a percentage of principal were variable (Figure 10), and variations are likely to be random znd do not indicate any change over time. When the stage in project implementation at which cancellations occurs is examined (Figure 11), there is some indication that in 1991 the cancellations tended to occur slightly earlier in their respective projects;2&/ if this tendency persists in the near-term for investment projects, it would suggest management is giving closer and earlier attention to projects with serious implementation problems. At this stage, however, no trend is obvious and the occurrence could be random. There is no firm evidence, therefore, to indicate any significant change in the use of cancellations as a means to make corrections in a portfolio with declining performance, at least not for the time frame examined here. 3.8 Reasons for Slionaae. The ARs by OED have documented the downward trend in project results (Figure 6 and Annex Table 9). The AR90 result of only 63.5 percent satisfactory projects, although admittedly the results of this cohort were affected by the abnormal volume of long-overdue PCRs (para. 3.3), was especially disquieting. Successive ARs have carefully described the origins of the ratings; over 95 percent of ratings represent a consensus between Operations and OED as to project performance; both OED and Operations treat this rating exercise very seriously and professionally. 3.9 Since every project presented to the Board is expected (in a statistical sense) to be "satisfactory", the above ratings represent a substantial slippage. This slippage can be due to one of more of the following factors in the SAR: i) poor technical design; ii) unrealistic price, cost or output projections; iii) unrealistic expectations of institutional performance; iv) unrealistic expectations about the evolution of other key variables influencing project success; 21 The LAC Region was prominent in FY91 in restructuring portfolios and cancelling loans, especially in Brazil, Colombia and Mexico. - 22 - 0 Figure 9: COMPARISON OF EX-POST AND SUPERVISION RATINGS FOR INVESTMENT PROJECTS IN THEIR FOURTH AND FIFTH YEAR OF IMPLEMENTATION 40%- CC 35%- 30%- CKC L 25%- 0 20%-AN CY 5 Y CY 8a 20.CY 3 F 82 C6 15%- J 84RR M/ FY85 FY80 Fya1 10%- 1 11 I1I FY8O/CY83 FY82/CY85 FY84/CY87 FY86/CY89 FY88/CY91 FY90 FY81/CY84 FY83/CY86 FY85/CY88 FY87/CY90 FY89 FY91 CALENDAR YEAR OR FISCAL YEAR ARIS PROBLEM PROJ. -1- OED UNSAT. PROJECTS OED Unsatisfactory Proiects - Projects reported as unsatisfactory, according to OED Annual Review evaluation (calendar) year. ARIS Problem Proiects - Projects in fourth and fifth year of implementation (at the beginning of the ARIS FY) which are rated problem projects. Source: MIS ARIS Database and OED Annual Review Database. - 23 - Figure 10: PRORLES CF CANCELLA1ONS RELATIVE TO PRINCIPAL Cohort • AN Operations with cancelons during perod 12% 10% 2 %- 0% 5 ~-4 - 2 - Years prior to Closing Date -~-85-86 -- 87-88 - 89-90 -- 91 Source: CTRW* Rnansial Database (FDB). Figure 11: PROFILES OF CUMULATIVE CANCELLATIONS (1985-41) Cohort - AD Opeations with canco~tions during period 6 0% 50% 40% 30% 20% Years prior to Closing 85-86 87-88 -~ 89-90 -91 83-9 1 Souros: CTRWs Finanal Databae (FDB). - 24 - during implementation: v) exogenous factors; vi) government "macro" policy; vii) poor project management; or after implementation: viii) changed conditions after completion; or ix) ex-post rating errors, by OED and Operations. 3.10 It should be noted that the OD 9.00 requires that the SAR discuss any perceived risks under items (iv) to (vii). The case study audits, and a review of PARs indicates that examples can be found where each of these items (i) to (vii) has been the major cause of slippage. However, information on the relative importance of these factors in explaining slippages is not generated routinely. At least on a sample basis, it would be interesting for PCRs/PARs to attempt to partition total ERR slippage amongst items (i) to (vii), and to calculate the present value of the associated costs. There is also a need for PCRs, PARs and the AR to distinguish between "new mistakes" where there is indeed something for the Bank to learn, and repetition of known mistakes; and to judge whether the causes were unforeseeable, or simply unforeseen. 3.11 Expectations. Expectations, at all levels, play a key (probably, 1f1 key) role in explaining the performance of large organizations. With respect to the role of supervision, there is no doubt as to the legal responsibility for implementation, which rests with the borrower, and that the Bank's role in supervision is only to support the borrower in implementing the project. In comments from the Bank's mana2ement on the approach paper for this study, it was emphasized that the borrower applies for a loan on a project it proposes and is then responsible for its imple- mentation; the Bank "ensures that the project is satisfactorily implemented by the borrower." There appears to be some divergence, however, in how to interpret the latter responsibility. 3.12 In the FY89 ARIS it was stated: "The decline (in regional supervision inputs per project) was anticipated given the need to strengthen the pipeline of lending operations during the year."2Z/ While there has been a reduction In the absolute priority rating for supervision from the OMS 3.50 to the OD 13.05 issued in March 1989 (para. 4.6), this need not be of much signifi- cance as it is unlikely that supervision was ever given "the highest priority" relative to other Bank functions in any case. Of more significance, however, is the preponderance of opinion amongst the interviewed Bank staff that supervision was not being given the attention and resources needed to fulfill the Bank's objective to "ensure the project is satisfactorily implemented by the borrower."2/ An exception to the general view, however, was an appreciation of the "imp!ementation culture" which has been initiated in the Africa region (paras. 2.42 to 2.44). Apart from the evidence from the database indicating that there has not been a full response in the allocation of staff resources to supervision in the face of an expanding and more complex portfolio, the anecdotal evidence suggests that there is a need for management to reiterate and demonstrate its intention to meet project supervision requirements as perceived by operational staff. The increase in average supervision staffweeks per projects reported in all regions in FY91 (para. 2.15) 2Z/ Report R90-17, page 10, para. 32. 211 See WPII, quotations 57, 108, 109, 110, 115, 117, 125, 126 and 127. -25 - should lend support to any program by management to verify its commitment to and preoccupation with project implementation. 3.13 In practice, it is logical that the needed amount of supervision will vary considerably amongst borrowers, and even amongst sectors within a country, In accordance with the stage of development of the involved institutions. Despite the emphasis given in this report to the desir- ability of an increased supervision input by the Bank, this does not imply that it should make an open-ended commitment for a relatively high rate of supervision "assistance* without any time limit. It would seem appropriate that if the institutional capacity of a borrower is very poor, then a program should be designed and undertaken to build up the borrower's institutional capability; this, with a view to reducing the supervision input to more normal levels as the institutions improve. 3.14 Mis-Match in Obiectives. Borrower compliance with project conditionality depends crucially on the degree of concurrence between the objectives of the borrower and the Bank.22/ This implies that it is at project identification and preparation that the degree of borrower compliance with conditionality is determined. Two series of railway transport projects provide pertinent examples of divergence in objectives and priorities which led to a repeated lack of compliance on loan/credit conditionality. The first is the case of the Pakistan Railways which has had Bank support in eleven projects, the last three of which spanned the period from 1969 to 19852Q/ and were audited by OED. Repeatedly, conditionalities on productivity ratios, rationali- zation of services, and financial measures were largely ignored. The Borrower was more concerned with the acquiring of new assets to meet present and future demand, while the Bank's condii:on- ality was geared to increased productivity of existing assets and labor, and to enhanced efficiency in the management and organization of the railway system and the transport sector. The lack of improvement in efficiency resulted in the estimated ERR on the tenth and eleventh projects being negative, and the losses of Pakistan Railways amounting to about US$400,000 per day at the end of the 1980s.31/ 3.15 The second example concerns the Korean National Railroad (KNR) which has received eight investment loans.32/ This case is markedly different from the situation in Pakistan, in that KNR is a relatively efficient railroad and the investment and operational aspects of the projects were well implemented. However, the agency and the borrower repeatedly violated the condition- ality on tariff and institutional issues. The ultimate objectives of a reasonable return on assets and the operation of KNR as a semi-autonomous public sector institution were the same for the Bank and the borrower, but expectations differed on the rate at which these could and should be achieved. Recently, progress has been made in both the tariff and institutional issues, but the 2/ The existence of "Bank Objectives" implies the possibility that the borrower will have other objectives. As one respondent said in the staff interviews "Our agenda Is not necessarily our client's agenda' (WPII para. 6.23, 183). One interpretation of the PCR *satisfactory/unsatisfactoryo rating is the judgment: knowing all that we now know, would the Bank have wished to lend for this project? The Bank's PCR does not consider the parallel question: knowing this would the borrower have wished to borrow? In principle, this question should be answered in the boirower's PCR, if it is prepared or approved by a responsible agency which i-s able to represent the borrowers' views. A2/ Ninth Railway Project (Ln. 621-PAK); Tenth Railway Project (Cr. 684, Ln. 1372-PAK); and Eleventh Railway Project (Cr. 1278-PAK). 11 / October 18, 1991. OED Report No. 10015. 22/ Credits 0025 and 0110, Credit 0183/Loan 0669, Loans 0863, 1101, 1542, 1836 and 2267-KOR. -26 - frustration of supervision staff and the perceived loss of rigor in the Bank's loan covenant system could have been avoided if the Bank had been realistic in its expectations considering the existing political realities. Projects would have been judged on their merits within achievable scenarios, and any longer-term objective could have been an important item in the continuous dialogue with the borrower. 3.16 This emphasis on putting project objectives, where appropriate, in a wide institution- building context, emphasizes the importance of ESW in improving project definition. The process of explaining Bank objectives and perspectives, and illustrating borrower constraints, incentives and perspectives may be at least as important as the projects or policy analyses nominally being reviewed. For the Bank to reap the full benefits from ESW, continuity of staffing and an approach which emphasizes coming to understand borrower views, rather than relaying Bank positions, are desirable. For a proper understanding of the likely support for policy or institutional changes, ESW should be extended, where possible, to yield inputs from non-governmental decision-makers including politicians, academics, and business leaders. 3.17 Risk Analysis and Legal Covenants. Standard Bank Risk Analyses as included in the SAR amount to little more than the switching values of important parameters used in the calcula- tion of the ERR. At best, this represents a mechanical assessment of some risk variables which could affect implementation. It is not used to set "flags" for subsequent supervision at which action to cancel, redesign or re-evaluate the project should be undertaken.U/ The covenants included in the legal agreements are designed to reduce risk by ensuring that actions deemed necessary for project success are undertaken. They reflect the substantive risk analysis undertaken at appraisal. Violation gives the Task Manager the occasion to recommend project redesign or cancellation. It would be helpful if the SAR indicated the expected adverse effects in the event of non-compliance with critical covenants. The current study on the methodologies for economic analysis of project being led by COD should also contribute to an improvement in the Bank's performance in the area of risk analysis. 3.18 Procurement. Though there are no comprehensive data, it appears that procurement takes between 30 percent and 40 percent of project specific supervision time (c.f. para. 2.17 and WPII paras. 6.02-6.07). Much of this time is taken up with procurement proposals which initially may be inconsistent with Bank guidelines. Even where documents are finally judged to be consis- tent with the Bank's guidelines, very considerable amounts of time may have to be devoted to detailed examination and discussions with a procurement advisor. Documents which are inconsis- tent with the guidelines generate even more work, as Bank staff clarify for borrowers the options open to them under the guidelines. Though the procurement guidelines provide considerable discre- tion to Bank staff, many are reluctant to use it. The very substantial hassle which follows from any suggestion of misprocurement by an unsuccessful bidder, or firm, intervention by officials (including Board members) to protest the exclusion of a party from a short list or other potential contract award, even if the charge or inquiry is completely unfounded, gives staff a little incentive to take responsibility. There is, on the other hand little, if any, penalty for delaying procurement, in order to get a second opinion. 2/ Properly done, a risk analysis should ensure a minimum performance level for supervision by identifying parameters to be watched, and the type of remedial action to be taken. Moreover, the risk analysis is primarily for the borrower since passing one of the switching values does not affect the borrower's responsibility to repay the loan. - 27 - 3.19 In principle the question of non-compliance should not occur. OD 11.02, paras. 9 to 12 clearly require appraisal of the borrower's procurement capacity, and that the project includes provisions which will ensure its ability to comply with Bank guidelines. Given the persistence and importance of procurement problems, despite OD 11.02, It would appear that provision could profitably be made for independent review of documents on the borrower's behalf, before their submission to the Bank, or use of a professional purchasing and Inspection service to do the procurement (as provided for in OD 11.02, para. 11(b)). Total elapsed time could well be reduced if procurement problems were resolved by the borrower, rather than interactively between the Bank and borrower. The more general use of model bidding documents should be able to help in this regard. However, as procurement matters take such an inordinate amount of supervision time (over one-third - para. 2.17), this raises the ouestion as to the practicality of current arrangements and suggests that another critical review of procurement guidelines could be undertaken.24/ With the advent of the ex-communist block countries into the Bank and the initiation of investment projects in these countries, this issue will require special attention. 3.20 Marginal Chanaes and Evolution. The few loans made in the early phase of the Bank's lending for post-war recovery were supervised by a staff member resident in the recipient country. Later, little real supervision was required for international contractors charged with a single major coostruction project. They had their own supervisors ind Bank needs could be largely met from progra;s raports, written/phone consultation on any unforseen problems, the verification which came from disbursement requests, and occasional field visits to confirm what had previously been reported. Later, the re-emergence of Europe and Japan as major suppliers led the Bank to introduce International Competitive Bidding (ICB), and the addition of procurement as a major supervision activity. 3.21 The expansion of the Bank during the 1960s and carrying on with the McNamara era required the codification of Bank practice starting with the creation of the Form 590 in 1971 and followed in 1973 by OMS 3.50 on supervision procedures, and the 1974 publication ')f the Super- vision Handbook. New style projects also transformed the challenge of supervision. Project implementation often required the coordination of several weak public sector institutions operating in diverse geographical locations. The smaller scale of individual construction projects, together with a pro-active policy of encouraging local contractors, often led to a large number of small contracts. Progress reporting became unreliable and erratic. Inspection of more than a small sample of project investments was often physically impossible. Despite a stated clear priority for supervision and the provision of double the usual supervision resources for new style projects, there was no way that the staff could keep themselves as well informed of the progress of new style, as compared to old style, projects. Even a doubling of supervision resources did not allow the same intensit4 of supervision as had been effectively achieved for simpler projects. Under- standably, at this time the Problem Project Implementation Review (PPIR) and CIR were added to management's supervision activities in an attempt to (a) get the attention of high level decision makers on implementation problems,2/ and (b) to resolve systemic problems at the appropriate level. Subsequently, the Board was brought into the process as the PPIR was renamed more 4/ As part of its series on Operational Policy Reviews, OED produced an Interim Report on Procurement Issues in Bank-financed Projects in July 1981 (Report No. 3557). Another report on Procurement and Construction of Highway Civil Works in Small Countries was issued in June 1989 (Report No. 7911). 2/ For a summary of the types of issues raised through FY86, see "Rural Development: World Bank Experience, 1965-86", World Bank, 1988, Box 6.1. - 28- comprehensively as the Annual Review of Implementation and Supervision (ARIS), which starting at the divisional level culminated in an annual report to the Board, and Board discussion. 3.22 OED's 1980 "Operational Policy Review. The Supervision of Bank Projects" Report No. 2858, identified a number of problems with then existing Bank practice, notably the need to Improve the planning of supervision (both staffing and notification of borrowers of timing and substance to be reviewed),f/ to provide better ieedoack on borrower reports, to leave an Aide- Memoire, and an increased role for programs staff and the use of CIRs. It did not question the Bank's basic approach to supervision, but rather sought to make suggestions which would make it work better. In particular it tacitly accepted the shift of the focus of supervision from headquarters to the field.,17/ Whereas originally, the purpose of field visits was to confirm what was already known, by 1980 regular reporting had largely degenerated to the point where field visits were necessary to find out what was going on. This is symbolized by the tying of Form 590s to the completion of a mission, rather than on a regular calindar basis. The ARIS provided a vestige of the earlier supervision approach, in that if a Form 590 had not been completed in the last six months, then it had to be completed as an ARIS input whether or not a mission had taken place. Against all the evidence in the body of the OED report, the report concluded that the resources available for supervision were basically adequate. However, it should be pointed out that the ex- post performance ratings which partly contributed to the analysis in the 1980 report would have largely been for projects approved before 1974, which would have registered a significantly better performance than the later portfolio included in the current study (para. 3.3). 3.23 The introduction of Structuisal Adjustment Lending (SALs) did not affect the require- ments of project supervision, though it did coincide with a change in the mix of staff available to carry it out (paras. 3.28 to 3.30). 3.24 The 1987 reorganization resulted in a flatter organizational structure. With project and country issues now meeting at the Director, rather than the Vice President level, there are clear operational advantages. The reorganization did not change the way supervision is done beyond a slight reduction in the available resources from FY87 to FY89 after the peak allocation in FY86 (Annex Table 3). The reorganization did not have a supervision focus, but the continued deterior- ation of the portfolio since 1987 has led to a number of initiatives to respond to the situation. This is especially so in the Africa region (pares. 2.42 to 2.44) which has had the poorest performance record, but the implementation problem warrants greater attention in all regions. 3.25 The picture which emerges from this brief historical review, is of new project designs being supported by marginal changes in existing supervision practices, rather than by a parallel redesign of the supervision process itself. 3.26 Open-Ended Proiect Desian. In addition to the new style projects requiring coordination of several implementing agencies, and delivery in diverse geographic sites, there was also a trend towards incompletely designed projects (usually with the caveat that extra Bank supervision would JA/ OD 13.06 now provides for a Supervision Plan. 2/ In FY90 24 percent of project-specific supervision was in the field. Data cannot be reconstructed for earlier years. I -29 - be provided and/or that studies would provide the required guidance for Implementation) ./ However, the Bank has a clear policy that the preparation phase must be completed prior to Board presentation: "At the time of Board presentation, as a general rule, project preparation should be sufficiently advanced that procurement and other important phases of project implemen- tation can start shortly after loan approval, and with cost estimates that are reasonably reliable and contain only modest physical contingencies ..." (OMS 2.28, October 1978). 3.27 In recent years this has been increasingly disregarded. In the softer areas of policy and institutional development vital design components were often left to be decided during implemen- tation, either by default, or explicitly as employing a "process approach".aI Thus, a line item would be included for agricultural extension, without any attempt to ascertain whether there was a suitable technology to extend. A component of 970 km of rural roads would be included without specific roads being nominated; it follows that the roads to be financed had been "appraised" at best on a sample or illustrative basis. In many developed countries, new social programs are initially on a pilot scale. This is because scarcity of resources for such programs, lack of know- ledge of their likely effects, and the myriad possible design alternatives, dictate some social experi- mentation to reassure legislators that the money spent will yield the intended benefits. Whereas a degree of the "process approach" is legitimate in pilot or experimental and innovative projects, the Bank by skipping this pilot phase for many of its new style projects enormously increased (a) the risk of failure, and (b) the difficulties of project supervision.4Q/ Here, as for procurement, the advent of the ex-communist block countries into the Bank will warrant special attention to the packaging of pajects which will face all the unknowns of an environment in the process of change to a market economy. The role and responsibility of the supervision function in relation to the degree of preparation of approved projects will have to be defined and provided with resources accordingly. 3.28 Mis-Match of Supervisory Skills and Needs. When the Bank was financing single site, large scale, infrastructural projects, supervision required well trained and experienced professionals. The Bank met this need by hiring staff in mid-career, with demonstrated professional ability and substantial experience. Such staff were often able to give invaluable advice on site, based upon their own considerable experience. U/ Functionally, these resources would be required to complete project appraisal; however, the Bank's mechanistic definition of supervision (para. 2.1) means that these extra appraisal resources are described as "supervision". 2/ Performance Audit Reports on: Nepal-Sunsari Morang Irrigation and Drainage Project (Cr. 812-NEP), Renort No. 8670, paras. 36-41; Kenya-Bura Irrigation Settlement Project (Cr. 722-KE), Report No. 8865, paras. 4.20-4.21; Lao Peoples' Democratic Republic-First, Second and Third Agriculture Rehabilitation Projects (Credits 760, 924, 1021-LA), Report No. 8884, paras. 51-53. 42/ For a discussion of pilot projects used in a developmental context see "Organizational Change as a Development Strategy", J. Haye and K. Sinsterbasch, Rienner, London, 1978, and "Managing Develop- ment Programs: Lessons of Success", S. Paul, Westview Press, Boulder, Co. 1982. In the Bank's case, as many as a third of audited rural development projected Intended as pilot projects were expanded during preparation and appraisal to full scale multi-component projects. ("Rural Development: World Bank Experience, 1965-86,' World Bank, 1988, p. 32.) - 30 - 3.29 From the early 1970s, the increased poverty and equity focus by the Bank led to the introduction of many "new-style" projects, particularly in the "soft sectors" of agriculture (especially rural development), education, urban development and family planning. Many of the existing experienced staff were uncomfortable with these large-scale, usually complex projects with unproven designs and reliance on many implementing entities with unknown capabilities. About this time, the Bank adopted a policy of increased recruitment of professionals with broad rather than specific technical skills, including bright young professionals with excellent academic qualifications but without much hands-on implementation experience. This policy has continued, so that since FY76 about one-quarter of staff in the operational grades 22-25 have been professionals who were recruited at age 30 years or less; another outcome of this policy has been an increasing proportion of divisional managers who were recruited at age 30 years or less (21 percent and 31 percent at 26 level in FY76 and FY91 respectively). 3.30 This recruitment policy allowed the Bank to expand its lending in the poverty alleviation projects, permitted it to increase the economic sector work (which was needed), and, more recently in the 1980s, enabled the Bank to quickly respond to the needs for increased structural adjustment lending. However, this change has occurred at some cost. In the heyday of rural development projects, many staff without hands-on experience presented projects for approval which had little chance of success considering the risks involved. A good example was the way in which the perennial problem of coordination of the numerous implementing agencies was com- monly resolved - "a Project Coordination Committee made up of five Permanent Secretaries would meet monthly to ... ." In many cases, no such meeting ever occurred, nor was it ever likely to occur. These types of over-ambitious projects posed high demands on the supervision function, which, due to the gradual change in staff mix, was often performed by staff who had little experi- ence dealing with practical implementation issues. For investment projects today, many staff consider that the skill mix has moved too far away from professionals with practical hands-on experience (para. 3.53), and that the use of consultants to fill this requirement is not a practical alternative (para. 3.69).41/ 3.31 Promotion on Appraisals. The appraisal of projects has become, for most project officers, the most important factor affecting promotion in the Bank. Although guidelines for promo- tion include experience in a wide range of Bank activities, there is strong bias towards achieve- ments in new lending rather than in the project implementation process in assessments for progression to senior levels 24 and 25. A start has been made, however, in breaking down this bias in at least one region (para. 2.45). 3.32 Files. Records and Resource Use. The central filing system typically does not contain all documents vital to effective supervision. Replete with multiple copies of travel arrangements and details of long redundant information on consultant availabilities, yet substantive reports, aide memoirs and other bases for decisions often cannot be found.A2/ In part this may be explained by the maintenance of a duplicate file, the Black Book, in the responsible division during imple- 41/ The changing skill mix and style of projects in the Bank from the early 1970s are also elaborated in WPI para. 7.05. The WPI then argues (paras. 7.06-7.11) that the highly centralized nature of Bank supervision was unsuited to the implementation assistance demands of the new-style projects. It makes a strong case for the benefits of supervision from resident missions (WPI para. 7.21). 42/ The problem is not with the records section D.ras, but with what is sent to it. See also "Report on an Audit of Borrowers' Accountability for the Use of Bank Funds Through the Receipt of Audit and Progress Reports", IAD, March 30, 1990, Report FY90-17 especially paras. 20, 36 and 61. - 31 - mentation, however it makes effective ex-post evaluation and heuristics extremely difficult. Also, the Bank's management style and loose, informal filing system leaves little track of how critical decisions are taken and by whom.43/ It should also be noted that it is not Bank practice to maintain a sequential diary or log of project progress, problems and decisions. It is thus only signed "hard copy" documents that tend to enter the filing system. Telephone calls, conversations and All-in-1 Notes vanish without a trace.44/ Even when conscientiously briefed by the out- going Task Manager, it is thus virtually impossible for a new Task Manager to get a full appreci- ation of project experience to date.A/ 3.33 Lack of Objective Indicators. The Bank's supervision procedures are characterized by an absence of automatic "flags" which would signal the need for some form of action. OD 13.05 broke new ground in requiring systematic reporting of actions taken in response to earlier recom- mendations (para. 2.35). Certainly, the decision to rate Overall Status on the Form 590 as a "3" or "4" will trigger management review and inclusion in the ARIS, but the decision as to project rating is highly discretionary, and primarily in the hands of the Task Manager, or mission leader (see, however, para. 2.45). The recent study of African agricultural projects (para. 3.6), showed that many projects rated as "1" or "2" ("Problem Free"; or "Moderate Problems") were nevertheless expected to be classed as "unsatisfactory" on completion. 3.34 The Form 590 includes quantitative estimates of implementation and disbursement delays and cost overruns and a series of subjective ratings on factors which are generally considered to influence implementation and project outcome (e.g. compliance with covenants, management performance, training, procurement and technical assistance). However, a series of project-specific and objective indicators can be determined at the time of appraisal to trigger special attention to the project or its supervision. These could originate from the SAR's risk analysis (para. 4/ For example, the Bank does not serially number file items or log them, such that the contents of a file can be changed, ex-post, with impunity. Similarly, unlike systems found in.some of the Bank's major stock- holding agencies, no routine record is kept on file of how correspondence is processed, by whom, when and of who participated in decisions (for example, on important procurement matters, about which borrowers repeatedly complain in PCRs that the Bank is slow to process documents). Two recent audits of contentious projects found, largely fortuitously, that key documents on major issues were never formally filed, but were kept separately. (Narok Agricultural Development Project (Phase I), Cr. 858-KE, Report No. 8795, June 29, 1990, footnote 12; and India: Fourth Agricultural Refinance and Development Corporation Credit Project, Ln. 2095-IN; c.f. following footnote. A/ A memo in connection with Loan 2095 (June 18, 1984) includes the following illustrations of the prob- lem: "The entries in the general file ... 'appear to contain a complete record of original qprrespondence but the responses-which in many cases have been telephone calls, discussions, or meetings with mana- gers-are not well documented...'" "(a) In March 1983, Mr. X reported to me on the findings of his super- vision of ARDC IV. Together we went to DEA and had at times acrimonious discussions with Mr. Y on, inter alia, not only recovery rates but also on the report showing misuse of credit. The outcome was clear; we continued to insist on delaying NABARD I. But the meeting was not recorded in our files. (b) Similarly, there is no file record about the discussions of agricultural credit when you and I and Mr. Z went to the Ministry of Agriculture in October 1983. (c) There was a 'milestone' meeting in your office on April 19, 1984. There was no record of the decisions and even the discussion paper ... which was an informal document and not filed." 41/ The IAD has commented similarly. "Often staff had changed several times since the inception of the project. ... In general, we found that the new Task Manager had little knowledge of the previous history of their projects and had ... received only limited, if any, briefing from their predecessors." Audit of Procurement Monitoring in Investment Lending - Africa and Asia Regions, June 1989, para. 13. - 32 - 3.17) and be included in an implementation plan (para. 2.41). Project-specifc indicators can refer to staff availability, recurrent funding, consistency of bids with SAR projections, key appointments, schedules for critical contracts, changes in crucial world prices, and so on. 3.35 It is also notable that the Bank does not solicit any feedback from the borrower during implementation, as to the quality and amount of the supervision being received. The CIRs usually present a suitable forum to elicit this reaction, but are seldom used for this purpose. 3.36 Adeauacy of Resources. A close reading of Chapter 3.of OED's 1980 report on super- vision shows a pervasive concern that staff were being asked to do more than could be achieved in the time available. Supervision missions which were too short, multiple countries or projects on the sarne mission, mixing supervision with other responsibilities, and under-staffed missions so that requir, . expertise was not provided, were all noted as practical problems. Internal Audit (lAD) reports seldom, if ever, identify wasteful use of supervision resources; rather they call for additional tasks, or existing assignments to be allocated more time. Staff continue to report these same difficulties.46/ This does not imply that staff consider that improved supervision will cure all the problems in the portfolio. Indeed, as reported in para. 2.24 the analysis of over 100 PARs was unable to establish a meaningful relationship between poor supervision and unsatisfactory projects, despite the numerous references to inadequate supervision in association with *Ie latter. A complex array of exogenous and endogenous factors can affect project outcome.47/ Never- theless, staff experiences led to firm opinions on the inadequacy of supervision in general, and the lack of resources to provide it with the warranted attention. 3.37 The above and the comment in para. 3.27 should not be construed to imply that costs can be disregarded. Genuine increases in efficiency which reduce costs without jeopardizing benefits are to be welcomed. Notable examples include the use of local hire staff in resident missions, leaving an aide memoire, and taking advantage of improved telecommunications. On balance, however, it implies increased resource allocations for supervision of poorly performing projects. If the proportion of such projects can be reduced, the need for additional resources will decline to match. 3.38 Financial Asoects. IAD reports have drawn attention to (a) overdue audited accounts, (b) unchecked Statements of Expenditure (SOEs) and (c) problems with Special Accounts/Revolving Funds (SA).48/ One possibility would be to train Task Managers to identify situations where the specialized financial (audit) skills may be needed; OD 13.10 is a good start in this direction. In theory, one or two days per mission should be sufficient to (a) sample the SOEs to check for evident anomalies, and (b) check that the basic organization and reconciliation of a projects SA is in order. If anomalies were found, they should be referred to an expert, rather than expecting the Task Manager himself to deal with them. 46/ WPII, paras. 4.01 to 4.08; however also see para. 4.09. 47/ The AR90 concluded that the observed deterioration in the performance of the portfolio could be partly explained by adverse macroeconomic conditions and by increasing project complexity. The report of the COD task force on the Relationship of Loan Processing to Project Ouality also indicates that macro- economic factors and project complexity can be statistically linked to this deterioration, and that the importance of specific external variables changes according to the sector in which the project is located. AJ/ "Report on an Audit of Procurement Monitoring in Investment Lending - Africa and Asia Region", June 30, 1989. - 33 - 3.39 The intention of SOEsA2I was that (a) disbursements would be made on the basis of SOEs, but (b) these would have to be refunded (or deducted from a subsequent payment) if subse- quent supervision missions found irregularities in the documentation of such expenditures. The pressure of other work and staff members' assessment of priorities have led to many cases where there has been no sampling of the SOEs. Many operational staff who are particularly concerned about implementation, and who even go to extraordinary lengths to assist the borrower to address implementation issues, do not consider that a financial audit function is an appropriate use of their time. Despite a few faulty disbursements having been identified and remedied, the theoretical SOE supervision system (which normally obliges non-specialized staff to undertake a specialized task) is thus not working. However, periodic "crash" programs to address the problem have been under- taken by some departments and regions.5/ Instructions were issued in March 1986 requiring suspension of the SOE procedure if the SOE audit is not received within 90 days of the due date. Some suspensions have been imposed, but as noted in the 1989 report on the subject "there has been a general reluctance in Operations (and LOA) to suspend the use of the SOE procedure knowing that by doing so disbursements would slow or halt since in many cases the SOE procedure is the only practical method of disbursement.".i/ 3.40 If fraudulent or undocumented SOEs are being paid, this is primarily a problem for the borrower. It is borrower supervision (probably in connection with routine financial auditing) which should identify the problem. Consequently, unless wholesale and intentional misrepresentation of SOEs is identified, it should be sufficient for Bank supervision to check that the borrower has adequate control and auditing mechanisms in place. For those projects where this assurance cannot be provided, the option of disbursing against SOEs should be withdrawn.52/ 3.41 This raises the issue of the extent to which the Bank has assisted borrowers in develop- ing their auditing capability. In essence, the Bank has insisted on the production of acceptable audits of project accounts and often financed consultancy contracts to provide the required audits, but it has done little to improve the institutional capability of the borrower in this area. This is likely due to the nature of the investment, which is largely of a technical assistance nature more suited to bilateral or multilateral donors which are heavily involved in this type of support. In addition, the Bank may have inadvertently contributed to the audit problem by not conveying a clear message as to the scope of SOE audits and the form of reporting required. If the Bank is to continue to expect the borrower to meet its auditing requirements (which seems rational) it may have to increase its participation in programs designed to improve the institutional auditing capabil- ity of borrowers. Wider use of pre-investment activities financed by UNDP is an obvious possibility. 42/ Use of SOEs as a basis for disbursement evolved from the Nurick report in 1966 and was codified in 1976. Report of the Committee on Disbursement Policies, June 21, 1966, and Central Projects Memorandum 7.15. 5Q/ ... OWe must face the fact that it (sampling during supervision) has not been effective to date, and is not likely to be unless much more managerial and staff attention is devoted to it, and the necessary resources are allocated. In the meantime, billions of dollars are being disbursed without adequate controls." Revolving Funds and Statements of Expenditures, Discussion paper, July 6, 1988, para. 26. (Approximately $7 billion was disbursed under SOEs in FY91). .1/ Op. cit., para. 24. I/ As directed in OD 13.10 "Borrower Compliance with Audit Covenants, October 29, 1991, para. 18(a). - 34- 3.42 Budaets and Allocations. In almost all dimensions, staff reported that inadequate re- sources prevented them from providing the level of supervision that borrowers have a right to expect. In practice, PBD (Planning & Budgeting Department) data comparing the 1980-82 and 1989-91 budget years indicate a real increase of 36 percent in the total operating budget compared with a rise of 20 percent in the volume of lending, an actual decline in the number of loans made, and an increase of only 27 percent in the number of active projects in the portfolio. This suggests that the perceived resource constraint for supervision may be more an issue of allocation within the budget rather than the total budget per se. 3.43 Regional managements do have full fungibility in the resources allocated by PBD. The relative allocation of staff resources (as a reflection of budget resources) has been discussed in para. 2.15 and showed that the proportion of staff time allocated to both supervision and lending has declined, to the benefit of economic, sector and other Bank activities. The less-than-full use of even the budgeted supervision resources reported in para. 2.21 also raises questions on the priority given by managers to the supervision function. Managers have to make decisions on how to make the most efficient use of the resources available to them, and supervision is only one of the import- ant functions the Bank has to perform. Decisions are undoubtedly influenced by the criteria against which resource use efficiency is measured. If the measure is the proportion of successfully imple- mented projects, it is likely that more weight would be given to the supervision function; this, despite the realization that appropriate project design is still probably the most important factor affecting project outcome. 3.44 The Trouble With Trust Funds. Use of consultants paid for under Trust Fund arrange- ments has been rising rapidly, from 6,677 staffweeks in FY90 to 18,781 staffweeks in FY91. In FY91 Trust Funds provided 17,444 staffweeks of operational time, or 11.9 percent of total opera- tional staff-time.5/ Tight budgets have obliged staff to rely increasingly on Trust Fund arrangements to complete their normal operational tasks. In practice, however, Trust Funds are not used substantially for supervision (Table 2). On the one hand, many Funds discourage or even exclude the use of their consultant resources for supervision for reasons which vary from a wish to have an influence on the shape or direction of the Bank projects (hence, upstream involvement) to a realization that participation in supervision is hardly worthwhile as it is very short-term and is unlikely to lead to any further substantial contracts. On the other hand, staff generally do not favor the use of Trust Funds to boost supervision manpower. This is due to (a) a large time input in negotiating for access to a Trust Fund for a relatively short supervision task, (b) lack of ability to select individuals who are known to have the specific skills and knowledge required for the project conditions, and (c) an inability to ensure continuity of supervision service by the individual. The latter is important in supervision in minimizing the cost of the consultant in becoming familiar with the project, in maximizing the consistency of advice given, in knowing the support needed by the Task Manager, and in building professional respect with the implementing agency. Consequently, Trust Funds have been more effective in relieving the resource constraints on Lending and Economic and Sector Work than for supervision. This also implies that some preference should be given to supervision in allocating actual Bank staff and paid consultant time amongst operational activities, with greater use of Trust Funds for upstream work. IV These 17,441 staffweeks were more than the 14,620 operational staffweeks represented by the 382 higher level staff retired under the 1987 reorganization. - 35 - Table 2: RELATIVE CONTRIBUTION OF TRUST FUNDS TO BANK WORK (% of total operational time contributed) FY90 FY91 Lending 5.0 10.1 Economic & Sector Work 9.0 11.9 Supervision 2.2 3.7 3.45 Coordination of Supervision Amonast Donors. Multi-donor projects can lead to super- vision difficulties for the Bank and for the borrower. When the Bank has primary responsibility for project supervision (on behalf of itself and other donors), serious problems are uncommon, especially if the Bank Task Manager maintains courteous relations and communications with the representatives of any cofinanciers and provides them promptly with supervision mission reports and relevant documents. Where cofinanciers maintain responsibility for supervising their own component, difficulties can arise if there is a divergence of views on items which influence components supported by both parties. Where the Bank agrees that a cofinancier has the primary responsibility (as occurs with the Asian Development Bank in most Pacific island loans and credits), frequent problems occur due to differences of opinion on project and sometimes on sectoral issues. In addition, separate supervisions by more than one financier can result in a serious and unnecessary drain or, the reporting requirements of the implementing agency. Different reporting requirements as well as uncoordinated supervision visits exacerbate the problem. 3.46 A recent project in Bangladesh provides an unusual example of effective donor colla- boration in supervision of a multi-donor project. The Fourth Population and Health project (Cr. 2259-BD) has used a special supervision grant fund of US$6.0 million to cover the cost of super- vision by a single unit which is located in the office of the Bank's Resident Representative in Bangladesh. This unit, which traditionally has supervised Bank projects and components in this sector, was expanded to assume the supervision responsibility for each of the 12 donors, with each donor contributing to the supervision fund (including the Bank at US$0.6 million). Due to the specificity of this unit, it goes well beyond a monitoring function and has a day-to-day interest in project activities. 3.47 Staff Persoective. Fifty staff interviews covered all regions and all sectors, but focussed primarily on staff who had been supervising projects for at least five years; three managers with similar experience prior to their promotion were included. As would be expected, quite diverse views and experience were reported from different respondents. Basically, however, the staff interviews support the analysis developed so far; not unsurprisingly, since much of the analysis derives from a careful reading of the interviews. A wider diversity of views and depth of experience is reported by direct quotation and summary comments in WPII than in the following summary, which nevertheless does capture most of the major views expressed. 3.48 Staff were found to be aware of and in sympathy with the Bank's supervision oblocves. Some tended to feel that the objective of policing the loan covenants was more important than trying to help borrowers improve project implementation,but all recognized that both objectives were important. There was a divergence of views on the extent to which supervision should be used to provide technical assistance, some feeling that it was not a function of the Bank, - 36 - others that it was an essential part of supervision. Some staff obtained great professional and personal satisfaction from supervision as they considered this was the Bank's function which allowed them to have a tangible development impact, though this appreciation was by no means unanimous. 3.49 There was ambivalence about the extent to which staff felt they should get involved in project management. All recognized that they should not do so, but at the same time some felt if they did not do so, implementation would suffer. 3.50 The majority of staff interviewed felt that the nature of supervision had changed over the last decade. Most attributed this to the change in the nature of the projects the Bank was financing, increased complexity in both the number of components and the number of objectives sought to be attained, with an increase in the attention paid to institutional and policy objectives, and just increased size. Many projects now had to rely more on supervision by the borrower, with the Bank supervising the supervisors. There was also a feeling that the Bank was moving back towards more traditional style projects so that the nature of supervision might again be changing. 3.51 Most staff thought that the Bank's ability to supervise had deteriorated and was still deteriorating. The primary reason for this was the inadequacy of the budgetary resources allocated for supervision and the low level of staffweeks which were supposed to be used on it. 3.52 The use of trust funds to supplement divisional resources was working badly when it came to supervision. 3.53 A secondary reason for the perception that the Bank's ability to supervise had deterio- rated was the change In the mix of staff skfis. Staff felt that the Bank's ability to provide technical supervision and wise judgment during implementation has been weakened and its credibility with borrowers has suffered as a result (paras. 3.28-3.30).54/ 3.54 The 1987 reorganization, through the creation of technical departments, was believed to have made technical supervision even more difficult. The new Sector Operating Divisions were seen as too small, and forced supervision missions to be too small to be effective. The new position of Task Manager was seen as beneficial in many ways and has given individuals greater professional satisfaction. 3.55 The majority of staff interviewed believed that, whatever management said about the priority of supervislon, this was not reflected in their actions. The promotion and reward system in the Bank was biased towards new lending; supervision came nowhere. Division Chiefs were given lending targets, not supervision targets. If there was ever a clash for resources, new lending came before supervision (see Box 1). 3.56 The most time-consuming part of supervision was procurement, described as the night- mare of most Task Managers, where any errors created huge problems. There was usually inadequate time to do a thorough job and staff considered borrowers were getting away with much §A/ The FY91 ARIS also expressed concern about the skill mix used in supervision. As the Bank's database does not provide useful information on this, a special review was proposed. - 37 - Box 1. STAFF PERCEPTIONS OF MANAGEMENT PRIORITIES* The majority of the staff interviewed believed that, whatever management might say about the priority of supervision, this was not shown by the way they acted. Supervision had a lower priority than new lending and sector work. The only exception to this was in the Africa Region where the new 'implementation culture' had been promulgated even if not yet fully taken on board by staff. The general perception was that this (lack of priority) was the result of the reward system in the Bank. The main problem is that the division chiefs in the sector divisions are judged by the number of projects they deliver per year. (108) * * Most division chiefs will earn their promotions and their recognition from the preparation of new operations... (109) You certainly don't feel the pressure from Bank management to do as good a job on supervision as you do on appraisals. It's my impression that management tends to pay lip service to supervision when in fact they are really interested in new lending. (110) When are the onlj times in the Bank when we break open a bottle of champagne? When we take something to the Board. (115) Since the reorganization most of the country directors are people with programs backgrounds. People who feel very comfortable talking general macro-policy issues, very frustrated in dealing with micro, project-level implementation difficulties and so on. (117) I was talking to a colleague recently when there was all this concern about travel (related to the Gulf war) and she said all 'non-essential' travel had been cancelled for the time being. I asked what is 'essential'; answer 'appraisal departures'. (57) I don't think there are any incentives in the Bank to do good supervision. (125) You never get rewarded for supervision at the Bank. (126) I think for a long time people have recognized the fact that there is no reward in supervision. (127) but I was struck last month when a copy of a supervision report came back to me with notes and questions scribbled on it. It came back to me from the director. And I said 'at least he read it.' (121) Only in three interviews was promotion as the result of good supervision mentioned, all in connection with the Africa Region's implementation culture, although one division chief thought someone had once got a merit increase for restructuring a project superbly well. * Extracted from WP II, Section 5. " Refers to the numbered quotations in WPII. - 38 - misprocurement, particularly on smaller contracts. The Bank's own guidelines were considered part of the problem. Despite these views most staff felt that the organizational arrangements for super- vision of procurement that most departments had made were working reasonably well. 3.57 Another area where many Task Managers felt inadequate was accounting and auditing. Most felt that they had little knowledge and less interest in this key area. Most admitted that the Bank lacked adequate expertise in this area yet financial analysts were rarely promoted to higher level positions. Most technical staff ranked audits low in their supervision priorities. The Bank's current drive to monitoring the receipt of audits was viewed skeptically as yet one more quantitative gimmick while the quality of the audit received little attention. 3.58 As regards the role of supervision in facIlitating implementaton, many staff felt frustrated that they were able to spend so little time in the teld on supervision and that they were less effective in this role than previously, when they had more time. Although staff agreed that the borrower should shoulder more supervision responsibility, many felt that it was unrealistic to expect this in countries with very poor development of the public sector. 3.59 There was a strong view that implementation was more effective in well-prepared projects. Staff felt there was more effort now in trying to build into new projects the experiences gained from on-going ones. 3.60 Many staff felt that what the Bank sometimes regarded as project failure was in reality unrealistic appraisal expectations about the time required to bring about behavioral and institutional change. 3.61 The adequacy of the Staff Appraisal Report as an implementation guide came in for criticism. Although some staff felt nothing had changed when the length of the SAR was cut, many thought that the Bank now provided the borrower with inadequate documentation for implementation purposes, and that borrowers were not getting this guidance from staff in the way they used to. Many felt that SARs were less thoroughly prepared now. They now contained more analysis of the policy environment and mandatory paragraphs on the environment, role of women, etc., but less on what is to be financed and how the project was to be implemented. 3.62 Reporting systems through which the borrower keeps the Bank informed during imple- mentation were thought to be mechanical but useful, although staff paid inadequate attention to designing well thought out systems. 3.63 The Bank's system of keeping borrowers informed of supervision mission findings, the aide memoire, was thought to be an excellent innovation and working well for that purpose. 3.64 The aide memoire also serves, together with the Form 590, as the principal source of management Information on supervision. Most staff felt this provided management with a1 they needed to know on implementation. They complained about lack of feedback from management on this documentation, feeling that quality was judged more by the quality of the report than the quality of what was done in the field. 3.65 The rating system of the Form 590 was regarded as too subjective to be much guide to performance. The extent to which staff rated projects as a problem "3" or 4 was felt to be less than was justified because the amount of extra work involved in doing so deterred them. Some felt a reluctance to rate projects a "3" or "4" because they had observed that management tended, - 39 - often unfairly, to blame staff for poor project performance. This view raises the vexed question of how much responsibility, if any, the Bank and Bank staff should bear if a project has problems. 3.66 There was considerable agreement that the Bank's records were deteriorating and that trying to trace a project's history had become very difficult. This was partly becavse the Form 590 itself inhibited more than cursory reporting, partly due to lack of time and partly that it was no longer required. 3.67 There is no Bank trainin9 in supervision other than on-the-job training (c.f. appraisal, negotiation) with an experienced Task Manager, and there is less on-the-job training as Division Chiefs can no longer afford the luxury of sending someone along on a supervision mission to be trained, and there are fewer experienced Task Managers. There is thus a lack of consistency in supervision practices and greater superficiality. 3.68 Supervision from resident missions was regarded as generally very effective and staff thought it should be done more often. Locally recruited professional staff, for instance in Delhi and Islamabad, were singled out as exceptionally effective. The many small African Resident Missions staffed with headquarters staff were also thought to be much more effective than missions from Washington, especially in agriculture. 3.69 Use of consultants for Bank supervision, other than for specific tasks, was not thought to be effective, unless they were accompanied by a staffmember. This was partly because consultants can rarely provide continuity, partly because few consultants, other than ex-Bank staff, understand what the Bank wants out of supervision. 3.70 Modern technology, such as the use of computers in the field and the All-in-1 system, have brought about many improvements in supervision effectiveness but some staff thought that the Bank could make better use of available technologies. 3.71 Borrower Contribution. If the need for a fail-safe and independent system is taken seriously in the design of project supervision (paras. 2.5 to 2.10), then project success should not be dependent on the effectiveness of Bank supervision; borrower supervision should be capable of supplying all information needed by the borrower to monitor implementation (and eventually to evaluate impact, unless this is the responsibility of a separate borrower entity), and ideally it should even be able to supply or organize any necessary technical assistance, including visits by specialized experts. The organization of this capacity should be an integral part of the project as appraised. As discussed earlier, independent supervision is not provided by the usual in-house monitoring arrangements, which are designed to tell the implementing agency about project problems. The implementing agency in turn reports to the borrower on the progress it, itself, is making. Examples can be cited from PCRs and elsewhere of borrower disappointment at the level of Bank supervision and technical assistance provided. Viewed from the perspective of agreed borrower responsibilities, this is a mis-directed complaint. The disappointment should be with the appraised project design which failed to include in the project all the supervision and technical assistance needed. Depending on the sophistication of existing borrower institutions, the borrower supervision could be provided by a specialized unit in a central ministry, a university, on a local or international consulting company. 3.72 Recognizing that current project designs implicitly assume some supervision support, in accordance with OD 13.05, during implementation, it should be noted that where a project involves many geographical sites there may be no possibility of Bank staff visiting all sites in the course of supervision. In such cases effective (Bank) supervision is dependent on supervision carried out on - 40- the borrower's behalf, which may demand orderly reporting to the borrower by the implementing organization. First level supervision by the borrower is likely to be very much cheaper than by the Bank, and should generate much of the basic information needed for Bank supervision, and allow Bank staff to make site visits on a sample basis. This, in turn, will allow them to judge the effectiveness of the supervision being carried out for the borrower, and hence a judgement as to how well the project is being implemented. Fostering in-country supervision requires appraisal of the supervision arrangements as an integral part of the project. As one of the first points of call for Bank missions, the supervision unit should be able to draw any problem that it itself is facing (staffing, salaries, mobility, managerial attention, etc.) to the mission's attention. 3.73 Examples exist where individual Task Managers, and indigenous institutions, have created this in-country supervision capability,U/ however, Bank practice in this regard has not been codified. If the supervision handbook is re-issued, a section on borrower supervision, as distinct from M&E, could well be added. 3.74 The 1980 study of supervision reported favorable borrower attitudes to supervision. A majority of respondents (to the 1980 questionnaire) rated the Bank's implementation assistance as "good" (4 on a 4 point scale); however 40 percent rated this as only "fair" (3 on a 4 point scale). And, for resolution of managerial problems the majority of respondents rated the Bank's performance as "fair". Where criticism was expressed it reflected a view that supervision staff did not always fully understand country problems, policies, organizations, legal systems and regulations (1980 study, paras. 27 and 28 in conclusions). 3.75 Poor response, five out of 16, to the same questionnaire in the context of the present study, does not allow similar quantitative conclusions to be drawn; however various aspects of implementation assistance and problem resolution were rated "good", "fair" and "poor" in the ratio 29:61:11. IV. RECOMMENDATIONS 4.1 The major findings of this study are that (a) the portfolio is confronted with increasing project implementation problems which are being reflected in poor project outcome, (b) the Bank's staff and management are aware of this problem, (c) Bank management has introduced a number of measures in an attempt to address the problem, but (d) there is a widespread perception amongst staff that the Bank's management and Board are more concerned with processing new loans than with the successful implementation of ongoing projects.5/ Supervision is only one U/ A good example is in the Indonesian power sector (Loans 2214, 2300, 2443, 2778, 3097 and 3098), where the Bank has helped PLN (the State Electricity Corporation) set up its own project monitoring system, which documents the progress of all sub-contracts, and produces a quarterly report on the status of all projects under implementation. Essential to PLN's own management of its investment program, this system automatically flags delays, and allows Bank staff to focus on substantive problems. Despite the vastly larger scale and greater complexity than older style single site infrastructure projects implemented by an international contractor; this information system still allows much of the Bank's supervision to concentrate on more substantive issues rather than spending time in collecting and reconciling factual information. f/ The several initiatives in the Africa region and stronger emphasis on supervision in the context of ARIS FY91, may be beginning to modify this perception. See also the quotation of Mr. Preston in the Preface. - 41 - of the factors affecting project outcome, and due to the diversity of these factors the study was unable to quantify the impact of supervision relative to other factors such as project design, exogenous infloences, etc. Nevertheless, there was a general feeling amongst operational staff that improvements in the supervision function could significantly enhance the average performance of projects in the portfolio; this would be through, inter alia, improved identification of emerging and actual problems, provision of sound technical and managerial advice to address issues and problems, and timely intervention to redesign or modify projects to adjust to circumstances which make an original design untenable. The authors of the study believe there is ample evidence to support the contention that beneficial improvements can be made in the Bank's supervision process. A number of recommendations are made which would clearly illustrate the importance that the institution places on the successful implementation of the projects it helps to finance, and which should improve the effectiveness of the supervision function. 4.2 The recommendations are grouped according to the following themes: (a) responsibilities for project implementation a;nd supervision; (b) incentives for performance by Bank management and staff in facilitating successful project implementation, and provision of resources to carry out this function; (c) improvement in the capability of staff to supervise projects; and (d) procedures which can enhance the efficiency of the supervision process. 4.3 Recommendations vary in their significance and potential impact. In recognition of this, an arbitrary division has been made between recommendations and suggestions, the latter implying a measure is of less significance than the former. A. Resonsibilities for Implementation and Supervision 4.4 Responsibilities. The legal responsibility for implementation undoubtedly rests with the borrower (para. 2.4), and it is the borrower who bears the costs and realizes the benefits (large, small or negligible) of the project investments. The Bank is clearly responsible for monitoring progress in project implementation and compliance with the legal conditionality imposed through project legal documents, and for taking action in the event of delinquency to ensure appropriate use is made of the Bank's funds. However, the Bank, as an international development institution, has a broadee role In assisting borrowers to obtain the greatest possible benefit from the use of its loan resources. The legsl responsibility of the borrower for implementation, therefore, does not absolve the Bank of a responsibility to actively assist the borrower in implementing projects by expanding the narrow monitoring/policing aspects of supervision into an assistance function (para. 2.12). This section of the report devis with means to improve this broader supervision function, but three aspects of this relationship between the Bank and the borrower should be stressed at the outset. 4.5 Bank's Assistance Obiectives. Although the Bank has a responsibility to assist borrowers in the implementation of individual projects, the Bank's primary objective in this assis- tance should be the broader and longer term one of developing the overall implementation capacity of the borrower and its executing agencies. Assistance provided in individual projects does contri- bute to the improvement in institutional capability, but there is an array of interventions which can substantially complement this input in the form of (a) direct technical assistance loans, (b. public sector management components of adjustment loans which improve siiary and staffing structures, - 42 - and (c) training and technical assistance components in investment projects which are coordinated as part of an overall Bank assistance strategy. Bank management and staff are generally aware of the importance of the Bank's role in institution building, but there is room for improvement in coordination of the programs by the Bank's sectoral departments and in the emphasis given to this aspect of borrower development. 4.6 Realistic Expectations. The second aspect also concerns the project implementation capacity of the borrower. A common criticism in ex-post evaluations is that the project design was beyond the implementation capability of the borrower. Excessive optimism in this area on the part of those preparing and/or appraising a project is a major contribution to project failures. The Bank cannot provide sufficient implementation assistance to overcome incompetent project management and attain a satisfactory project outcome. RECOMMENDATION 1. That, while the Bank has an overall objective of improving the management capabilities of the implementing institutions of its borrowers, and while measures to improve management are legitimately included in investment projects, care must be taken in project design to be realistic in assessing the ability of the implementing agency to coordinate and implement projects (especially those with high complexity), even with the programmed assistance.57/ In practice, this implies that in countries with weakly developed institutions, simpler and more modest objectives in project design will be necessary until the institutional capacity improves. 4.7 Supervision by the Borrower and the Bank. The third aspect concerns the extent to which the Bank should be expected to carry the burden of monitoring implementation progress. Normally, a borrower uses one or more of its executing agencies to implemeit projects. As the borrower bears final responsibility for loan repayment and for ensuring that the project is imple- mented as intended, it is obviously in the interest of the borrower to have up-to-date information on the performance of its institutions. Unfortunately, the borrower's critical role in project super- vision often fails to receive the prominence it deserves, so that the borrower becomes dependent on Bank supervision to provide an independent view of project performance. This is inconsistent with borrower responsibility for effective implementation (paras. 2.5 to 2.10) and can also lead to an excessive load on the Bank's supervision function. Having an independent local supervision capability should not only service the borrower's monitoring/supervision needs,_/ but also would necessitate built-in progress reporting by the implementing agency which, in itself, should be a key element in upgrading project management performance. A decline in the quality and regularity of project progress reporting has frequently transformed the Bank's supervision missions to fact-finding rather than an analytical and result-oriented exercise. This is an inefficient use of Bank resources. While it is unlikely that every borrower will give priority to the establishment of its 57/ Of at least equal importance to successful project implementation and outcome is a macroeconomic environment which is conductive to project operations (see footnote 47). This is an essential considera- tion in project conception and design, but is not dealt with as a recommendation in this report which concentrates on the supervision/implementation aspects of the project cycle. Mg/ Borrower supervision in this context refers to a system of monitoring project progress and observing con- straints to implementation, so that the borrower is aware of implementation status and has the oppor- tunity to intervene, if circumstances warrant, to assist the implementing agency meet the development objectives of a project. Although the borrower has the ultimate responsibility for the project vis-a-vis the Bank, an independent supervision function must not be allowed to detract from the responsibility and accountability of the implementing agency. - 43- own supervision capability, and while its impact will vary according to project type and circum- stance, this should receive special promotion ; vd attention by the Bank (pares. 3.71 to 3.75). RECOMMENDATION 2. That priority be given to institu- tionalizing a borrower capacity for periodic monitoring of projects independently of the project management. This is especially necessary for projects implemented by force account involving services and infrastructure (which typically are not subject to supervision by independent engineering firms) and projects with numerous contracts and many components. Proj- ect loan/credit funds could be used to strengthen this institution-building activity, but the key tasks of Bank staff should be to promote the concept and assist the borrower in establishing a system (before project effectiveness) which would provide the basis for prompt intet vention when warranted.Ug/ In many cases, independence would be assured by giving supervision responsibility to a central ministry or agency, but the location of the unit would vary with the institutional circumstances of the borrower. Where a major executing ministry has a series of development projects in its program, the ministry would also benefit from having its own supervision capacity. 4.8 The CIR exercise involving senior officials of the Bank and the borrower offers an oppor- tunity for fruitful dialogue on the development of the borrower's independent supervision capacity, as well as to focus attention on systemic problems of poor project management. Suggestion 1. That the CIR exercise (currently adopted by most country departments) be utilized (a) to enter into a dialogut with the borrower concE:ning the respective roles of Bank and borrower supervision, (b) to promote the creation of an independent supervision capacity by the borrower, and (c) to highlight the implications of any general inadequacies in project management and elaborate proposals to gradually overcome these deficiencies. A borrower's supervision unit could take advantage of the increased implementation emphasis recommended for the SAR in this report. The implementation plan and the expanded risk analysis irscluded as Recommendations 10 and 11 would equally service both the Bank's and the borrower's needs. With appropriate training by the Bank, the unit could also undertake a monitoring and even a guidance role in procurement matters (para. 4.34). A capacity to inspect project accounts and SOE (para. 4.37) would depend upon the relationship between the supervision unit and the i?/ Although ex-post evaluation is not the subject of this study, it should be mentioned that the presence of a national supervision agency would not only enhance the borrower's commitment to projects under imple- mentation, but also ensure that the necessary data for impact assessment and ex-post evaluation are collected. Alternatively, where an ex-post evaluation capacity is developed, this would greatly facilitate the establishment of an independent supervision capability. A distinction is made here between monitoring/supervision, which pertains to an information system relevant to the project during implemen- tation, and the ex-post evaluation function which measures the impact or relative success of the invest- ment project. - 44- borrower's internal and external auditing entities. In addition, the existence of a supervision unit would enhance the likely benefits from sharing the Form 590 with the borrower (para. 4.47). B. Incentives and Resources 4.9 Promotion and Performance Criteria. Tho previous paragraphs in this section have paid considerable attention to the responsibilities of the borrower. The remainder of the report pertains to means by which the Bank may improve its performance in the supervision function. Human nature dictates that performance will generally be obtained when there is a significant incentive for the individual to perform. Unfortunately, a common impression amongst staff is that rewards are not obtained from effective inputs into the project implementation phase, but rather In the up- stream activities of economic and sector work and project appraisal (paras. 3.31 and 3.55). Recently, steps have been taken to reverse this impression (para. 2.43) at least in one Bank region. 4.10 It is well recognized that it is difficult to objectively measure the short-term performance of institutions involved in the process of national development; this is due to the long-term nature of the output from most investments, the diffuse nature of many benefits, and the frequent absence of a market-determined valuation of output. A similar problem occurs within these institu- tions in objective assessment of performance of operating units and their individual staff. Some quantitative criteria can be selected on the input side for individual performance - number of appraisals led, sector reports presented, mid-term reviews (MTR) completed, supervision missions led, etc. Use of the real measure of performance, however, which is the successful implementa- tion of the operation resulting from the intervention, is rarely feasible. Umited continuity of staff and a significant degree of collective or institutional responsibility further detract from identifying the linkage between an individual's input and real output. Consequently, less-than-perfect and relatively subjective assessments must be made of individual performance, but this does not detract from the advantages of having assessment guidelines which ensure due weight is given to all the key functions of the Bank's operations. 4.11 Progression through professional grades is a reflection of performance of a staff member in the Bank. Currently, there is little emphasis given to constructive and effective super- vision of projects and programs in promotional criteria. An important step in the career path of most Bank operational staff is the passage from a level 23 to 24, for which the staff have to have displayed competence in their selected professional field, experience in an array of Bank functions, and leadership qualities. In practice, this is usually interpreted to mean successful leadership of appraisal missions, without adequate recognition to the equal importance of other Bank functions, including supervision, or the eventual outcome of the projects appraised. A similar bias arises in the promotion to 25 level. While it is true that the output from the supervision input is difficult to objectively quantify, it is no more so than for real accomplishments in the upstream phases of the project cycle. RECOMMENDATION 3. That promotion and performance cri- teria for Bank operational staff be adjusted to reflect the concern that the Bank holds for successful project Implemen- tation. Increased weight should be given to performance in the supervision function and in its management at the divisional and departmental levels. 4.12 Institutional Focus. The OMS on supervision before 1979 stated "Supervision has the highest priority in the Bank's work.* This clear policy was subsequently modified to avoid such an open-ended commitment to supervision. Undoubtedly, a well conceived and appraisk I project is -45- the primary factor affecting project results in most cases, so it Is difficult to justify the "highest priority" classification for supervision es only one of the functions affecting project outcome. Nevertheless, frequent failings in project design and unforeseen endogenous and exogenous factors which commonly occur during implementation demand an ability and a willingness by the Bank to closely monitor project progress and provide assistance to address issues which can detract from the project meeting its objectives. The new OD on supervision should be reinforced to clearly indicate a commitment to implementation assistance. While, in itself, such a reinforcement could not be expected to significantly affect the attitudes of staff and management, it could be expected to effectively complement other measures designed to enhance the concern for project implementation. RECOMMENDATION 4. That the new OD 13.05 be revised to indicate the importance the Bank places on the successful implementation of the projects it supports, the critical role that supervision must play, and the priority that must be given in allocating resources to supervision to "ensure (the Bank's) loans make the greatest possible contribution to increasing produc- tion, raising living standards and opening opportunities for further investments" (Eugene R. Black). 4.13 This means, inter alia, that (a) priority should be given to providing assistance to project management to enable successful project launching, and (b) should a project be rated 3 or 4, precedence would be given to supervision to address major implementation problems, to redesign the project in order to attain the original or modified objectives, or to cancel the loan/credit. It does not imply that there should be a reduction in the quality of upstream activities in Identification, appraisal and economic/sector work; rather, these activities may have to be rephased if intensified supervision becomes necessary. 4.14 The staff perception of the Bank's concern for project implementation would be further enhanced by an awareness that the Bank's Board is particularly interested in project development impact. The Board is primarily concerned with the policies, priorities and direction of the Bank's development lending program. It is currently seen by staff as focussing on the SAR and President, Memorandum in the context of approving new loans. This concentration on loan approval has a number of unfavorable characteristics: (a) it is too late for any intervention to influence project design; (b) it consumes an inordinate amount of the Board's time to the detriment of monitoring of the outcome of the existing lending program and of considering plans for development assistance to countries in earlier stages of project preparation/processing; and (c) it diverts the SAR from being r -uide for project implementation to being a rationalization for Bank support.gw As there is a je slippage from appraisal intentions to achievements, this gives the impression that intentions, not achievements, are the primary focus (para. 3.1). 4.15 The ARIS series reports to the Board on implementation of the portfolio. Following aggregation of departmental and regional submissions, it has traditionally been weak on substantive information so that it has failed to stimulate any In-depth inquiry by the Board. The most recent ARIS (ARIS FY91), however, presented a more in-depth analysis of implementation performance and problems. At the Board presentation of this report, many comments were received expressing N/ "Another view expressed was that the SAR was now largely a document for the Board and that it was tailored to whpt they wanted to hear and was no longer realistic about implementation and much more concerned with macro-economic matters and the new Initiatives." (WPII, para. 7.09). -46- serious concern for the status of the portfolio. At least one executive director made his own critical review of the report and sent copies to his colleagues. It is hoped that this improved inter- action with the Board on implementation of the portfolio is further developed. The AR series by OED provides the Board with an independent ex-post review of the Bank's development experience. It provides key information on project outcome. Although it reports on projects which have completed their investment phase and which were mostly approved from five to ten years earlier, it demonstrates the degree of success or failure in the Bank's lending program and analyzes various aspects of performance. These results should be critical for the Board in carrying out its functions of guiding the Bank's policies, priorities and direction. In practice, however, after some review by the Joint Audit Committee of the Board, the discussion of the portfolio results at the Board usually consume only one to two hours; very little, considering the impact that the experience of completed projects should have on future projects and programs. 4.16 To dispel the impression that the Board is mainly interested in new lending, and to convey to middle-level management and staff that project outcome is paramount, greater attention by the Board to the review of ongoing projects (mainly through the ARIS series, usually in February) and analysis of completed projects (mainly through the AR series, usually in October/November) would be very beneficial. To support this, the ARIS should be comprehensive and analytical (the ARIS FY91 is an improvement on previous reports), and the AR series should continue with its systematic and transparent approach in reviewing project outcomes. 4.17 The size and scope of the ARIS demands a degree of aggregation which causes it to lose more of its potential impact. To expose the Board more directly to implementation problems and their potential impact on projects and programs, greater use could be made of Country Briefs and of Country Policies and Bank Group's Assistance Strategy for IDA countries. These country presentations give the Board the opportunity to show its concern for implementation issues as they might affect proposals under consideration. RECOMMENDATION 5. That, at the portfolio level, manage- ment continues to enhance the format of the ARIS to provide greater analysis of portfolio performance by sector and region and to define actions being taken to address problem areas. At the country level, guidelines for reporting to the Board on existing projects and programs and proposed assistance strategies should specify that actual performance levels and any major constraints to implementation are frankly elaborated. 4.18 Resources for Supervision. Staff weeks used for project-specific supervision oer groiect have been in the range 9.1 to 12.1 for fifteen years (12.1 in FY91, 9.1 in FY83 and 10.5 in FY77). However, the size and complexity of Bank projects has escalated over this time. Larger projects typically have many more components to be implemented. In FY90 the average investment project was 67 percent larger in real terms than in FY76. The decline in the proportion of "satisfactory" projects has thus been accompanied by a decline in the supervision resources used per $100 million (1991 constant dollars) being supervised. Although increased loan size does not always imply a greater supervision requirement, the concurrent increased complexity of projects would suggest an increased supervision requirement in most cases. 4.19 In the regional budgetary processes, historical coefficients which reflect average inputs are used to allocate resources without sufficient regard to the severity of problems being encountered by individual projects (paras. 2.18 to 2.21). Cases are successfully made by Task Managers and Division Chiefs for exceptional treatment of particular projects, but the general - 47 - process is clouded by an understanding by staff that the overall supervision coefficient must be kept near the guideline level, and not encroach on the resources allocated to upstream activities. The recent introduction of a Supervision Plan in the SAR has meant the specification of supervision requirements well in excess of historical averages in many cases; this is not yet being reflected in the annual Country Assistance Management (CAM) process. Due recognition of the role of the Supervision Plan in the SAR would obligate Bank management to agree to commit the indicated supervision resources, or insist that alternative arrangements be made to reduce the cost of super- vision by the Bank (such as by the establishment of an enhanced borrower capacity for supervision, or, in some instances, a simplified project). 4.20 Divisional managers have significant flexibility during the year in allocating the resources previously specified in the CAM amongst sector work, lending and supervision, but, traditionally, any substantial increase in the supervision allocation has not been well received by regional managements unless the "contracted" number of projects to the Board and appraisal departures have been fully met. The less-than-complete use of.the staff years budgeted for supervision in FY89 and FY90 for Bank operations as a whole reflects this attitude (para. 2.21). If delinquent projects are to receive their due attention, adequate resources for supervision should be identified and justified in the planning process. Provided the needed expertise is provided, this could be expected to result in an improved performance rating in the portfolio of each division and some portfolio restructuring in the near term, both of which can be monitored to introduce accountability at successive management levels for use of staff resources. RECOMMENDATION 6. That supervision requirements should be built up from a realistic assessment of individual project requirements, and sufficient staff resources should be budaeted and allocated to permit all the supervision functions to be prop- erly performed; special emphasis should be given to emerging or existing "problem" projects which require additional assis- tance, redesign and/or consideration of cancellation. 4.21 In most cases, after a project is completed, follow-on projects and sectoral dialogue enable the Bank to monitor the continued adherence by the borrower to project covenants of a long-term nature, and to react accordingly. However, in the rare cases of cessation of Bank activities in a sector, staff time should be allocated to monitor such compliance so that any untoward developments can be addressed in the Bank-borrower macroeconomic dialogue. 4.22 An increased average resource allocation to the implementation phase of projects could be expected to result in an equivalent increase in overall operating costs per project or a reduction in the budgetary support for one or more of other Bank activities. However, if increased costs or a reduction in the rate of portfolio expansion resulted in significantly improved results in project outcome, this would represent an increase in operational efficiency (if the denominator is comple- tion of satisfactory projects). One measure which has allowed increased attention to supervision without increasing costs is the use of local staff on resident missions to carry a greater share of the supervision load (paras 2.31 and 2.32), and this practice can be expanded, with certain precautions. Suggestion 2. That, to enable an increased supervision input at reasonable unit cost, the Bank expand the use of local staff in resident missions; this, however, only on the condition that the local staff supervision function is closely integrated with regular Bank staff to maintain the authority and - 48 - impact potential of Bank supervision missions, and to ensure Bank staff con- tinue to have maximum exposure to implementation issues and problems. In order to maintain the authority and impact potential of supervision using locally-recruited staff, this will normally imply the adequate use of headquarters-recruited staff on resident missions. The latter category of staff have always been considered to be especially effective in super- vision,f1/ but their unit costs are higher than headquarters-based staff. The need for this balance in resident missions would normally reduce but not eliminate the savings enabled by local saff recruitment. In addition, headquarters-recruited staff should also be especially effective in developing the capacity of the borrower to supervise (paras. 4.7 and 4.8). C. Staff Caoability 4.23 Staff Profile. There has been a substantial change in the career profile of Bank staff and managers over the last 20 years. Traditionally the Bank relied on the recruitment of mid-career professionals with extensive field experience to process and supervise investment projects. Starting in the early 1970s, greater (but by no means exclusive) attention was given to recruiting generalists without the experience in a particular technical profession, including young professionals with excellent academic qualifications and other qualities considered necessary for successful participation in development assistance. This, combined with some losses of experienced technical personnel during the 1987 reorganization, led to a reduced proportion of staff with hands-on implementation experience in their pre-Bank careers. Since the reorganization, the proportion of economists to staff classified as technical specialists has actually remained relatively stable. However, the recruitment of incremental technical specialists to maintain this proportion has been weighted towards the "new skill" areas that the Bank needs (e.g. environment, forestry, public sector reform and private sector development), so that the more traditional technical skill areas have not had the same attention. Consequently, the changing staff profile of the Bank over the last 20 years has allowed it to adapt to the changing emphasis in its lending program (and has essentially been in response to an assessed need); the net effect, however, appears to have been a reduced ability to make discerning and timely judgments in project supervision. This is certainly the impression of many Bank staff (para. 3.53). RECOMMENDATION 7. That the Bank put greater weight on the pre-Bank responsibilities and hands-on development experi- ence of professionals in recruitment. 4.24 One difficulty experienced by managers and personnel staff in recruiting highly qualified and experienced staff is the firm policy that external recruitment should be at a salary level up to the middle of the 23 grade range. Although it is sometimes possible to recruit at a level above this, the process is extremely demanding of staff resources and time so that few managers are willing to attempt such a justification process. Suggestion 3. That the salary level at which highly experienced and qualified professionals can be recruited be made more flexible so as to attract such staff, with due safeguards to avoid other-than-merit considera- tions in recruitment and also inequity with respect to similarly experienced staff in the Bank. Ll1/ See WPI paras. 7.06-7.11. - 49 - 4.25 Supervision Guidance and Trainina. There Is no Bank training program for supervision, other than on-the-job training through exposure, with or without the assistance of Bank staff with considerable implementation experience (para. 3.67). The 1974 Supervision Handbook is not readily available, even though it is considered a useful guide by staff who are aware of it and use it. Considering the existing staff profile, there is a need to introduce practical training courses in supervision to complement on-the-joib training. A case study approach is likely to be most suitable. RECOMMENDATION 8. That training for supervi*'n be improved by introducing a course series on supervisik . and obliging new staff to serve on at least three supervision missions with staff experienced in effective supervision prior to leading a (possibly one person) supervision mission. RECOMMENDATION 9. That the 1974 Supervision Handboui should be edited to reflect organizational and other changes (including any changes which may result from this study), and then be re-issued to staff. 4.26 An appointment to a resident mission, especially one with substantial supervision responsibilities, is an ideal way for less-experienced staff to gain exposure to project implemen- tation conditions and problems.§/ Some Bank staff, however, are reticent to locate outside headquarters to this type of assignment, as inter alia, they (a) consider the implementation emphasis to be less mainstream and less "glamorous* than the upstream activities, and/or (b) have the perception (despite statements to the contrary by Bank management) that one has to be at headquarters "to be noticed." Such perceptions are only likely to be overcome if a much Improved implementation culture is developed in the Bank (paras. 4.9, 4.11, and 4.12-4.17). D. Procedures to Enhance the SuDervision Process 4.27 Imolementation Plan. All SARs provide a description of project components. Few, however, contain a detailed and sequential implementation plan. The preparation and inclusion of such a plan (a) encourages greater attention to the feasibility of proposed implementation and disbursement schedules, (b) facilitates dialogue with the borrower and implementing agency in con- firming this feasibility (including counterpart funding requirements), and (c) provides assistance for subsequent supervision missions (para. 2.41). 4.28 The difficulty of identifying and assigning monetary value to education, research, extension and teci nical assistance projects means that they have been exempted from presenting an ERR in the SAR. This does not remove the necessity for the borrower and the Bank to evaluate §Z/ It should be noted that, although all resident missions become involved to some extent in project imple- mentation issues due to their presence in a country, very few of them (usually only the large ones) have a substantive supervision function. The Bank's Field Office Task Force in 1989 identified the core functions of resident missions as maintaining communications, Improving public understanding, assisting head- quarters missions, participating in country implementation reviews, and following up on debt service payments. It also mentioned their broader functions of enhancing the quality of Bank assistance, either by substituting for or supplementing the functions being carried out by headquarters staff. The report did not recommend a uniform approach to supervision from resident missions due to the major variations in circumstances amongst missions. It suggested a *role-defining matrix" to be used by country directors for defining the precise terms of reference for supervision from resident missions and for delineating the lines of authority between headquarters and resident staff (WPI paras. 5.37 and 7.1 '-7.14). - 50- the impact of the investment. In these cases, an implementation plan would not only articulate specific actions, sequences, and responsibilities, but also specify a clear and monitorable set of expected intermediate results (often provided by beneficiary survey) which could be used to assess progress and predict impact. RECOMMENDATION 10. An implementation plan should be mandatory for all SARs. 4.29 This would go further than the requirement for a monitoring and evaluation plan for investment projects as specified in OD 10.70. In order to optimize the usefulness of the implemen- tation plan, the Bank should review alternatives and develop a planning framework which can be used as a guideline for staff in this exercise in SARs. Approaches used by other agencies to track the attainment of specific project objectives, particularly those difficult to quantify, and to monitor the means to achieve them during implementation should be evaluated../ 4.30 Risk Analysis and "Flaas" for Proiect Redesian. For large unitary investment projects for which a redesign would normally not be feasible in the event of major implementation problems, it is essential that the preparation and appraisal processes examine all relevant development models and assess the risks associated with each before deciding on the most robust solution; this may not be the one with the highest NPV under optimum conditions. For projects in general, the risk analysis currently provided in the SAR needs to be made more operational and compatible with the Operational Directive on addressing project risks at the Executive Project Summary (EPS) phase (OD 9.00, Annex C). The current, relatively "mechanical" calculation of outcomes with percentage changes in costs, benefits and benefit lags should be replaced by more realistic assessments of the possible range in key variables taking into account the type of investments, the project outputs, and previous borrower performance (paras. 3.17, 3.33 and 3.34). RECOMMENDATION 11. That the preparation and appraisal processes for Bank projects examine all relevant alternative models for development components, including the risk factors in each, before deciding on the most robust alternative; in addition, the SAR risk analysis should be expanded to include the acceptable range in key variables influencing project success, critical price levels or ratios, performance levels and implementation lags, beyond which the project should be re- evaluated for possible redesign or cancellation. These "flags" should be monitored by Task Managers responsible for supervision, and could also be used to assist in country, divisional or sectoral reviews of projects within regions. 4.31 Confirmation of Continued Feasibility. The Form 590 (used for reporting the findings of supervision missions) requires the rating of implementation problems and likely overall project developmental impact, but it does not require an explicit judgment as to whether, in the light of sunk costs or the evolution of key variables that Influence project success, the project should be continued with its current design and baic implementation arrangements (implying, in the negative, the need for a review to enable redesign, major actions on controllable endogenous or exogenous I/ For a useful description of this type of approach and the software to make it operational see EDI Review, January 1992. -51 - factors, or cancellation). Such a recommendation could be included in Section 8 of Form 590, but this would not enter the MIS (para. 2.26). Suggestion 4. That a new mandatory rating be added to the Form 590 to reflect the desirability of continuing the project with its present design in current and anticipated circumstances. 4.32 This obliges the Task Manager to take account of the impact of any significant endogenous or exogenous factors which are likely to affect project outcome, and to present a judg- ment as to whether or not the current design and program are likely to lead to the most desirable project result under the prevailing circumstances. Monitoring of "flags" (Recommendation 11) would assist in this exercise. 4.33 Supervision Mission Composition. There has been a gradual change in the staff profile of the Bank (see para. 4.23) so that there are fewer professionals who have had hands-on experience in development projects. This, combined with increased project complexity, makes it more difficult for many Bank staff to be able to identify emerging and existing constraints to imple- mentation and to be capable of addressing these constraints in investment projects without additional professional back-up (paras. 3.28-3.30). The Supervision Plan specifies the expected skill requirements at the consecutive stages of supervision. This must be a tentative prediction due to the likely emergence of various unforeseen circumstances. In cognizance of the Plan and current developments, at the completion of a supervision mission the Task Manager should be obliged to indicate the skill areas which he/she considers the Bank should allocate to supervision of the project in the near future. In cases where the Task Manager is not widely experienced on the tech- nical aspects involved in the project, he or she would be expected to consult with appropriate tech- nical staff in the division or the TD to confirm likely requirements. In preparing terms of reference for a subsequent supervision mission, the Task Manager should indicate how these requirements are to be met (or why they will not be met). This would also allow the Division Chief to balance resources against requirements, and an inability to provide the required resources would alert other levels of management about possible sub-optimal results from supervision. RECOMMENDATION 12. That the leader of each supervision mission specify the expertise he/she considers necessary for the subsequent mission and for any specific tasks prior to the mission (after appropriate consultation, as necessary), and that the terms of reference for the subsequent mission be related to this prior recommendation. 4.34 Procurement. Greater attention should be given to OD 11.02, paras. 9-12 during appraisal, to ensure that procurement problems will not cause implementation delays and will not make undue demands on Bank staff supervision resources (paras. 3.18 and 3.19). RECOMMENDATION 13. That, when substantial procurement is involved in a project and the borrower's capacity to comply with the Bank's procurement guidelines is in doubt, the project should either (a) provide for independent review of documents before presentation to the Bank, (b) use a professional purchasing and inspection service, or (c) finalize major bidding documents prior to presentation of the project for Board approval. The more extensive use of model bidding documents should also facilitate the procurement process. - 52 - 4.35 The inordinate amount of supervision time consumed by procurement raises questions as to the practicality and appropriateness of the procedures dictated by the Bank's guidelines and legal covenants under its loans (para. 3.19). RECOMMENDATION 14. That the Bank undertake a review of its procurement guidelines with a view to reducing the staff time required to oversee this process without losing the advantages for borrowers and member countries associated with competitive bidding. 4.36 All regions except EMN have procurement advisers, while EMN has a procurement unit in its TD. In general, TDs tend to have more staff trained and experienced in procurement than operating sectoral divisions. Suggestion 5. That consideration be given to the use of specialized procure- ment staff in each region to take responsibility for a major part of procurement, so as to release more staff time for other supervision duties. This would require an expansion of procurement staff, but reallocation of staffing responsibilities could avoid any incremental staffing impact. 4.37 Accounts and Audits. Several staff observed that whilst the Bank has succeeded in getting more complete and more timely delivery of project accounts, little is done with them, and most Task Managers are unqualified either to judge their quality, or to make use of them. It is also obvious that most Task Managers do not do random inspections of SOE, and are not trained or interested in this activity. In addition, the Bank has done little to develop the public sector accounting and auditing capability of borrowers, and any progress in more timely presentation of accounts is usually a one-off exercise in reaction to Bank pressure, rather than profound institu- tional improvement (paras. 3.38-3.41). RECOMMENDATION 15. That better use be made of project accounts, by provision for central review of project accounts in each region, to check that accounts are of adequate quality, and to suggest to Task Managers issues which may need looking into on the basis of the accounts. RECOMMENDATION 16. That Bank supervision of SOEs be reduced to ascertaining that the borrower has a control and auditing procedure in place to limit widespread or systematic fraud. Where this is not in place, the facility of disbursing against SOEs should not be provided.i4/ Suggestion 6. That the Bank make greater efforts to identify the require- ments for improved public sector accounting and auditing by its borrowers, and facilitate significant institutional improvements with the help of bilateral or multilateral technical assistance programs or its own technical assistance loans. I4/ There is also the opportunity for trained local staff on resident missions to take responsibility for periodic SOE checks. - 53 - 4.38 Reliability of Supervision Ratings. Project ratings are an important component in the supervision process, but they inevitably contain an element of subjectivity and there is concern that the ratings may not always represent a candid assessment of the extent of implementation problems. The average portfolio performance as indicated by supervision ratings has consistently been much better than that indicated by the annual ex-post evaluation cohort. The average super- vision ratings for projects under implementation decline, however, as the projects mature. Conse- quently, the rating in the latter years of a project should be more indicative of the Task Manager's assessment of project outcome. Data analysis shows that the average ratings for these later years do reduce the gap between the percentage of projects "with major implementation problems" and the percentage "unsatisfactory" in ex-post evaluation, but the difference persists (paras. 3.5 and 3.6). An independent review of supervision practices and ratings would both ensure comparability of standards and assist Task Managers to improve their supervision practices. If the reviewing staff were drawn from other units, it would encourage transparency, reduce subjectivity and assist each unit to improve its own practices._U/ RECOMMENDATION 17. That, until regional managements are convinced of the reliability of ratings, formal review systems would be instituted to ensure relatively uniform standards in allocating project ratings. Possibilities include the use of regional subsectorai teams to desk review all projects within the sub- sector (as done with livestock projects in Africa), or the use of sectoral teams to desk review a random sample of projects (say 20 percent). Project Advisers and staff of the TDs would be prominent in these reviews. 4.39 While this would not ensure comparability across all sectors and regions, it would act as an incentive for staff to be realistic in assessments and should improve the situation within a region. Organized, cross-sectoral sessions by review team staff from within the TDs would further enhance comparability. 4.40 External Review of Remedial Actigns. Since the dismantling of the Operations Policy Staff (OPS) in the 1987 reorganization, the Bank and Regions have lacked a central operations policy group with oversight and technical review capacity. In certain sectors and regions, the Regional Technical Departments fulfill this function on an a h.h basis. PCR records indicate many ultimately "unsatisfactory" projects have had problem project ratings for three or four years. Admittedly, it is often difficult to quickly turn a problem project around, but persistent serious problems should be subject to redesign (and improvement in performance) or cancellation. The cancellation data for the years 1985 to 1991 (para. 3.7) do not provide evidence of any increased use of this medium to address problems in the portfolio, which is declining in performance, but there is some indication of increased rationalization of country portfolios through major revisions and earlier cancellations starting in 1991. There is a need for review of the remedial actions being taken for persistent problem projects (i.e., those rated "3" or "4"). One or two experienced sectoral staff could be provided from strengthened Technical Departments (or from OSP - previously PRE) to be involved in such reviews, possibly through participation in a supervision or major review mission. Apart from providing an additional and independent opinion on prdject imple- mentation, the external participant would be expected to be aware of cross-sectoral and exogenous .U/ A recent review of livestock Forms 590 by Africa Region in connection with ARIS 91, illustrated how problems can be identified, and improvements suggested. Interestingly, they proposed that their subsector team carry out a somewhat analogous rolling review of the regional livestock portfolio. - 54- issues which could impinge on the project; this implies interaction with relevant staff in concerned CODs and SODs. Care would have to be taken not to undermine the responsibility and account- ability of the Task Manager for the project subject to external review. RECOMMENDATION 18. That the remedial actions proposed for any project which has been rated 3 or 4 for more than a year (i.e., in three consecutive supervisions at intervals of six months) should be subject to participation by independent reviewers (as appropriate technical staff) in the next supervision/review mission. In the event of disagreement between the Task Manager and the external reviewer(s) on the appropriate course of action, the latter would provide a separate report to the Departmental Director, which would also be included as part of the supervision report; agreement on action to be taken (which would be expected in most instances) would generate a note of concurrence. 4.41 Mid-Term Review. This study has already recommended (a) that project ratings be progressively reviewed for comparability in standards, and (b) that projects persistently rated "3" or "4" be subject to independent review. It has also suggested that a new rating be added to the Form 590 to explicitly flag projects which should be cancelled or redesigned. At least until a significant improvement in project outcomes has been achieved (above the current 63 percent "satisfactory"), a mid-term review should be instituted Bank-wide to check whether the above recommendations are proving to be effective. For problem projects, an independent representative of the same group responsible for the external reviews of persistent problem projects could be beneficially included in the mid-term review. RECOMMENDATION 19. That, pending an improvement in the proportion of "satisfactory" projects, all projects without a "1" rating for overall status should be subject to a mid-term review. If the mid-term review results in project redesign, then the proposed changes should be documented in as much detail as in the original design. For problem projects, the institutional judgment could be strengthened by participation of a relevant expert external to the division to which the project pertains. 4.42 Hand-over of Proiect Supervision. Continuity in the Task Manager responsible for super- vision of a project is highly desirable, but change-overs do occur, at least once and frequently twice during implementation. Transfer of supervision responsibility between Task Managers runs the risk of discontinuity in the dialogue with the borrower on the status of the project and needed actions. In addition, a project considered satisfactory by the out-going manager may be thought to be unsatisfactory by the incoming manager; or the incoming manager may believe a quite different input is needed to maintain a satisfactory level of supervision. Most managers have installed satis- factory transfer procedures, but this is not always the case. Suggestion 7. Wherever possible, a joint supervision should be carried out as part of the hand-over of supervision responsibility. When a joint mission is not undertaken, a Form 590 and hand-over report, highlighting major issues, should be completed and submitted to the Division Chief. The new Task Manager should particularly indicate the level and type of supervision - 55 - resources felt necessary to satisfactorily meet supervision requirements after his or her first in-depth exposure to the project. 4.43 Evaluation Methodology. The present system of classifying projects as "unsatisfactory" or "satisfactory" disguises as much as it reveals. It gives as much weight to a small, marginally unsuccessful project in a small country as to a large, highly successful project in assessing the performance of the Bank's portfolio. The same limitations apply when assessing the impact of the portfolio in a region or a country. Ideally, the present value of net benefits should be calculated for each project to obtain a real measure of value and impact. This would require the estimation of monetary values for social benefits such as lives saved, unwanted pregnancies avoided, high school graduations, potable water deliveries, etc. Suggestion 8. At present, the complexity and cost of introducing the calculation of present value of net benefits to the project assessment and evaluation process make such a proposition impractical in many cases. This should encourage the Bank, however, to continue to support work, such as that being carried out by the Economic Analysis work group (ECON), to develop practical methodologies which would more correctly measure performance. 4.44 With an increased emphasis on measuring the real impact of the Bank's development assistance, it would be advantageous to have clear guidelines for analysis of the ex-post results in the PCR and PAR, that would provide more complete and reliable information on project perfor- mance and permit analysis of reasons for poor performance. Suggestion 9. That staff be given guidance in practical methodologies to estimate the relative contribution of factors such as project design, exogenous influences, government policies and project management in shortfalls between performance and outcome (para. 3.10). Suggestion 10. In addition to identifying causes of slippage, the PCR should note whether these causes constituted new lessons, or repetition of well known causes of poor performance. 4.45 Proiect Records. In ex-post evaluation of projects (as PCRs and PARs), it is common to find that key documents and communications are missing from official files. Although in only very rare cases is it suspected that documents are deliberately excluded from the filing system, care- lessness or reliance on a separate duplicate filing system without real concern for central files are frequent causes of this problem. The situation seems to have been exacerbated by the advent of correspondence electronic mail, but this need not be so under proper management. Suggestion 11. That Division Chiefs make a greater effort to instill in staff an appreciation of the importance of official (as opposed to personal) filing of all correspondence, documents and information pertaining to the performance of a project, and arrange for periodic checks to ensure compliance. 4.46 Portfolio Summaries. Form 590 (past and present), disbursement and staff-time information is available on the SPNF-ARIS database for an individual project. This information should also be made available in summary form so that Bank managers can move between levels of aggregation to inspect the status of an individual project, a divisional, departmental, regional or - 56 - sector portfolio, or the Bank's portfolio as a whole. This would assist managers to (a) diagnose the source of problems or (b) compare their performance to other units. Suggestion 12. That information on Forms 590, disbursements, and staff- time for individual projects be made available on-line to allow user-selected aggregations. 4.47 Sharina of Form 590. There are arguments for and against sharing the Form 590 with the implementing agency and the borrower (paras. 2.36 and 2.37). Although OD 13.05 is silent on whether the Form 590 should be distributed outside the Bank, there appears to be a case for this where the Task Manager and Division Chief believe that sharing it with the borrower would impro, e the borrower's understanding of the Bank's perspective on implementation progress and priorities. Its use in CIR proceedings could also be very useful once the borrower became accustomed to the format. Suggestion 13. That Task Managers be encouraged to share the Form 590 with the borrower, with the approval of the Division Chief, where they consider it will contribute to improved project implementation. 4.48 Donor Coordination in Suoervision. In development programs supported by numerous donors, supervision requirements by individual donors are often a drain on the borrowers capacity, make progress assessments difficult because of interrelationships between the various donor- financed components, and usually represent a very inefficient use of supervision resources. The third and fourth Bangladesh population projects have pioneered a new approach to overcome these problems. A special unit in the Bank's resident office was expanded to provide supervision on behalf of all donors (para. 3.46), with each contributing to the cost of the unit. Suggestion 14. That experience with the Bangladesh population projects be followed carefully to assess its potential for wider use for efficient super- vision of large, multi-donor programs in which the Bank is involved. 4.49 Additional Resources. Some of the recommendations and suggestions are resource- neutral, others would require more supervision resources, while still others could reduce the resource requirements. The appeal in R6 to allocate sufficient supervision resources implies an increased resource use on supervision, but this need not mean significantly increased total operational resources if the rate of expansion of the lending program is adjusted accordingly. If the procurement guidelines can be simplified (R14), and/or the procurement document review by Bank staff can be reduced (R13), this should lessen the resources needed for this time-consuming component of supervision. The use of a more centralized procurement review process should be resource-neutral, as should the recommendations concerning modifications in the review of project accounts. 4.50 Recommendations 17, 18 and 19 concerning increased attention to the assessment of project implementation status do imply a marginal increase in the resources which would be applied to supervision. If, however, these measures resulted in a greater proportion of successfully imple- mented projects (the ultimate objective of the Bank as a development institution) then this would likely represent an increased efficiency in resource use. - 57 - Annex Table 1: NET STAFFWEEKS REPORTED BY HIGHER LEVEL STAFF IN FY9 Al Staffweeks % GRAND TOTAL 204,319 100.00 UNALLOCATED 207 0.10 OVERHEAD TASKS 53,237 26.06 LEAVE 18,949 9.27 HOLIDAYS 7,055 3.45 TRAINING 3,962 1.94 MANAGEMENT 11,920 5.83 GENERAL 11,350 5.56 PRODUCT & PROCESS TASKS 150,875 73.84 NON-OPERATIONAL 35,860 17.55 OPERATIONAL 115,015 56.29 LENDING t/ 42,862 20.98 ESWS _/ 27,078 13.25 OTHERS 17,215 8.42 SUPERVISION d/ 27,860 13.64 SUPERVISION 27,860 13.64 SPN 23,676 11.59 Specific Investment 14,292 6.99 Sector & Structural Adjustment 2,550 1.25 Olher q/ 6,834 3.34 PCR 2,126 1.04 Non-project specific .,058 1.01 SUPERVISION 27,860 13.64 ARIS projects t/ 24,045 11.77 Non-ARIS projects 1/ 1,757 0.86 Non-project specific 2,058 1.01 Notes: Al Net staffweeks for all types of higher level staff (higher level, international consultant, Research Support Budget consultant, Young Professional, special positions, FAO/CP, unidentified, assistant, local consultant, local higher level, and trust fund consultant). bj Lending staffweeks consist of time on tasks: LENP, LENN, LENA, LENR, LDV, LDVP, LOP, OARD. OARU, ORDL and ORDN. rI Economic sector work staffweeks include time on tasks: ERA, SRA, EWO, SWO, PFP, OARA, OARP, ORDE and ORDP. d/ Supervision staffweeks encompass time spent on nine tasks: SPN, PCR, CIR, PIR, ARI, PAD, OARS, ORDS and ORDO. Al Debt Reduction, Emergency Reconstruction, Financial Intermediary, Not Available, Sector Investment & Maintenance, and Technical Assistance. I/ Supervision staffweeks for product tasks (SPN and PCR) included or not included in the ARIS portfolio, FY91. Source: World Bank Time Recording System - 58 - Annex Table 2: ALLOCATION OF SUPERVISION TIME, FY74-91 a/ (Operation Complex, Staff weeks) FY SPN PCR CIR PIR/ARI PAD OARS ORDS ORDO Total (401) (402) (403) (404/408) (407) (1017) (1032) (1034) 74 8,470 0 0 0 655 0 0 0 9,125 75 10,134 0 0 0 732 0 0 0 10,866 76 12,387 316 0 0 796 0 0 0 13,500 77 14,066 1,105 0 0 1,008 0 0 0 16,179 78 14,228 1,109 0 0 962 0 0 0 16,299 79 15,267 1,170 0 0 971 0 0 0 17,409 80 16,894 980 0 0 1,108 0 0 0 18,982 81 17,538 1,720 0 0 1,107 0 0 0 20,365 82 19,113 1,801 0 0 1,130 0 0 0 22,044 83 19,033 1,733 0 0 1,155 0 0 0 21,920 84 21,617 1,936 0 0 1,422 70 0 0 25,046 8" 20,224 1,849 0 0 1,145 91 0 0 23,309 86 20,806 1,949 148 275 1,884 185 1 0 25,248 87 20,698 1,752 127 319 1,746 59 0 0 24,701 88 20,562 1,513 41 223 1,828 118 0 0 24,285 89 19,448 2,415 133 319 1,506 137 0 0 23,958 90 21,790 2,226 134 354 1,334 146 0 0 25,982 91 23,676 2,126 267 335 1,298 157 0 0 27,860 Notes: a/ Net staff weeks for all types of higher level staff (higher level, international consultant, Research Support Budget consultant, Young Professional, special positions, FAO/CP, unidentified, assistant, local consultant, local higher level, and trust fund consultant). * Project Supervision (SPN) (A product task): A general supervision task including staff input for supervision field time, report writing, direct follow-up work resulting from a supervision mission, and supervision desk work. * Project Completion Report (PCR) (A product task): This task accounts for the specific PCRs. * Country Implementation Review (CIR) (A process task): The task relating to work associated with country implementation reviews. * Proiect Implementation & Review/Annual Review of Implementation & Supervision (PIRARI) (Process tasks): A combined category of the outdated PIR task and currently used ARI task. * Proiect Administration (PAD) (A process task): This task accounts for the time spent on general procurement issues, disbursement, and other monitoring activities not specifically project related. * Advice & Review (OARS, ORDS, ORDO) (Process tasks): The tasks OARS, ORDS, and ORDO reflect time spent on supervision A&R, general operations review (supervision), and environment review (supervision), respectively. Source: World Bank Time Recording System - 59 - Annex Table 3: OPERATIONS COMPLEX, OPERATIONAL STAFFWEEKS (levels) a/ FY LENDING b/ ESWS c/ SUPERV. d/ OTHER TOTAL 74 26,861 8,842 9,125 3,852 48,680 75 29,491 10,365 10,866 3,061 53,783 76 31,656 10,677 13,500 3,494 59,326 77 32,899 11,736 16,179 3,673 64,487 78 38,318 12,124 16,299 3,883 70,623 79 36,877 11,811 17,409 4,717 70,814 80 38,245 12,766 18,982 4,559 74,553 81 37,357 14,208 20,365 5,644 77,575 82 36,261 18,548 22,044 6,504 83,356 83 32,490 18,650 21,920 6,835 79,895 84 32,458 18,906 25,046 6,497 82,907 85 35,251 19,429 23,309 7,307 85,296 86 36,002 19,871 25,248 11,179 92,300 87 34,856 20,760 24,701 11,562 91,879 88 34,978 22,211 24,285 12,148 93,621 89 36,122 23,638 23,958 13,343 97,061 90 41,015 24,831 25r982 15,379 107,207 91 42,862 27,078 27,860 17,215 115,015 ,/ Net staff weeks for all types of higher level staff (higher level, international consultant, Research Support Budget consultant, Young Professional, special positions, FAO/CP, unidentified, assistant, local consultant, local higher level, and trust fund consultant) with non-overhead task types. 2/ Lending staff weeks consist of time on tasks: LENP, LENN, LENA, LENR, LDV, LDVP, LOP, OARD, OARU, ORDL and ORDN. Q/ Economic sector work staff weeks include time on tasks: ERA, SRA, EWO, SWO, PFP, OARA, OARP, ORDE and ORDP. 4/ Supervision staff weeks encompass time spent on nine tasks: SPN, PCR, CIR, PIR, ARI, PAD, OARS, ORDS and ORDO. Source: World Bank Time Recording System - 60 - Annex Table 4: OPERATIONS COMPLEX. OPERATIONAL STAFFWEEKS (%) a/ FY LENDING b/ ESWS c/ SUPERV. d/ OTHER TOTAL 74 55.18% 18.16% 18.74% 7.91% 100.00% 75 54.83% 19.27% 20.20% 5.69% 100.00% 76 53.36% 18.00% 22.76% 5.89% 100.00% 77 51.02% 18.20% 25.09% 5.70% 100.00% 78 54.26% 17.17% 23.08% 5.50% 100.00% 79 52.08% 16.68% 24.58% 6.66% 100.00% 80 51.30% 17.12% 25.46% 6.12% 100.00% 81 48.16% 18.32% 26.25% 7.28% 100.00% 82 43.50% 22.25% 26.44% 7.80% 100.00% 83 40.67% 23.34% 27.44% 8.55% 100.00% 84 39.15% 22.80% 30.21% 7.84% 100.00% 85 41.33% 22.78% 27.33% 8.57% 100.00% 86 39.01% 21.53% 27.35% 12.11% 100.00% 87 37.94% 22.59% 26.88% 12.58% 100.00% 88 37.36% 23.72% 25.94% 12.98% 100.00% 89 37.22% 24.35% 24.68% 13.75% 100.00% 90 38.26% 23.16% 24.24% 14.34% 100.00% 91 37.27% 23.54% 24.22% 14.97% 100.00% I/ Net staff weeks for all types of higher level staff (higher level, international consultant, Research Support Budget consultant, Young Professional, special positions, FAO/CP, unidentified, assistant, local consultant, local higher level, and trust fund consultant) with non-overhead task types. k/ Lending staff weeks consist of time on tasks: LENP, LENN, LENA, LENR, LDV, LDVP, LOP, OARD, OARU, ORDL and ORDN. g/ Economic sector work staff weeks include time on tasks: ERA, SRA, EWO, SWO, PFP, OARA, OARc, ORDE and ORDP. d/ Supervision staff weeks encompass time spent on nine tasks: SPN, PCR, CIR, PIR, ARI, PAD, OARS, ORDS and ORDO. Source: World Bank Time Recording System - 61 - Annex Table 5: "ACTIVE PROJECT" CRITERIA USED FOR BUDGETING SUPERVISION Actual Under Actual- Criteria FY Budget Supervision p/ Budget % -T- 1,867 1,76 -141 -7.6_ All projects signed in last eight years 86 1,853 1,776 - 4.1 = 1,858 1,808 _____ ___ 88 1,921 1,798 -123 -6.4 All projects with undisbursed balances 89 1,770 1,908 138 7.8 of over $100,000 at the beginning of 90 1,783 1,890 107 6.0 FY, and all new loans signed in FY Tf ____8__ 1,855 12 53Yk/ 2.12 All investment projects signed in the last 8 years plus all adjustment projects 92 1,825 signed in the last 4 years I I I I _ I ./ Projects approved by Board during FY, and with more than $100,000 of undisbursed balance at beginning of FY. ]/ Estimate. Source: Review of World Bank Programs and Fiscal Year Budgets, various years. * - 62 - Annex Table 6: BUDGET AND ACTUAL STAFF YEARS USED FOR SUPERVISION FY85 TO FY91 BUDGET FY85 FY86 FY87 FY88 FY89 FY90 FY91 FY92 TOTAL Super. StaffYears 424 425 463 475 475 475 452 478 No. Active Projects 4/ 1,867 1,853 1,858 1,921 No. Active Projects k/ 1,770 1,783 1,802 1,825 Aq StaffWeeks/Project g/ 11.3 11.3 12.6 12.2 StaffWeeks/Project 1/ 13.0 12.8 11.8 12.4 StaffWeeks/StaffYears q/ 49.7 49.3 50.5 51.7 StaffWeeks/StaffYears q/ 48.4 48.0 47.0 47.3 ACTUAL TOTAL Super. StaffYears 444 479 471 458 455 480 TOTAL Super. StaffYears d/ 429 446 (471) No. Active Projects 1/ 1,726 1,776 1,808 1,798 1,908 1,890 (1,855) StaffWeeks/Project Q/ 12.3 12.9 12.6 12.5 10.4 11.2 (12.2) StaffWeeks/StaffYears p/ 47.8 47.8 48.4 49.1 46.3 47.5 (48.0) Al Projects within an eight year implementation period after effectiveness. k/ Active projects with over US$100,000 in undisbursed balances. p/ Staffweeks used as a basis for calculation of staff years in particular years. g/ With local supervision Staff Years discounted. I/ Estimate based on targeted acceleration of closings of old and inactive projects. ) Numbers in parenthesis are estimates of actual at time of FY92 review. Source: Review of World Bank Programs and Fiscal Year Budgets, various years. AnnexTable 1: AVERAGE PROJECT SPECIFIC SUPERVISION STAFFWEEKS PER $100 MILLION BY SECTOR AND FISCAL YEAR Ia (SPN/401, All Regions) 1k (In constant US$, FY91= 100) Sector 74 75 76 77 78 79 80 81 82 83 84 8s 86 87 88 89 90 Agriculture 22.4 24.8 31.0 32.4 28.0 27.2 24.8 22.6 '22.6 19.8 21.1 19.9 20.9 21.4 20.4 28.8 21.7 Communication 12.7 15.2 12.8 19.5 16.2 15.0 11.5 11.0 12.1 13.6 12.2 10.3 12.7 13.7 17.0 16.0 18.5 Education 46.8 46.8 49.4 52.6 44.9 32.8 34.3 22.8 25.7 24.6 23.5 19.9 20.0 23.4 19.8 18.2 18.9 Energy 13.4 8.2 19.9 14.8 6.2 9.5 35.5 21.0 21.3 16.9 21.7 20.2 21.9 22.6 18.4 11.2 11.6 IDP 26.0 180 18.6 Ii.9 19.9 16.9 14.1 13.0 12.7 11.3 123 12.8 13.6 9.5 8.8 7.6 9.4 Industry 20.9 35.2 12.2 14.1 9.4 12.8 9.5 8.4 9.3 8.9 10.3 10.8 9.7 11.2 9.0 9.4 10.5 PHN 81.1 IS6.0 145.1 342.1 140.3 130.1 101.7 91.4 69.0 53.1 45.5 43.0 63.4 45.7 39.8 38.1 34.7 Power 7.1 8.5 7.2 8.0 7.8 6.6 6.0 5.7 5.6 5.8 6.6 5.8 5.9 6.6 7.6 7.4 8.1 Public Management 0 0 0 0 0 0 0 0 0 0 0 74.5 15I.3 31.5 7.7 74.1 17.3 SAL 9.6 14.2 15.0 25.2 16.2 13.0 15.5 10.9 32.8 7.6 10.4 56 8.0 11.9 15.4 11.1 13.4 Technical Assistance 41.8 231.3 183.8 128.2 200.6 163.1 183.6 39.3 62.5 82.2 100.0 89.1 75.8 86.2 94.0 96.7 112.0 Tourism 238.2 312.5 219.4 355.6 140.0 79.0 77.1 405 29.7 28.0 25.8 19.7 14.6 16.4 12.7 49.7 28.5 Transportation 64.7 61.1 61.2 51.7 44.2 34.8 30.4 13.1 13.9 12.8 II.9 11'5 12.2 12.3 I1.4 1.8 12.2 Urbanizaion 28.9 373.6 243.9 186.8 98.2 102.1 94.0 40.5 27.6 18.1 19.9 18.8 16.4 14.0 12.9 13.9 15.7 Water Supply & Sewerage 22.2 30.8 25.9 27.7 22.8 17.8 17.9 17.0 16.3 15.4 16.2 14.6 14.8 16.2 14.6 14.4 13.0 All Prjects All Supervision4$I00M lc 18.26 21.14 2334 25.99 23.11 21.41 19.97 18.62 17.85 15.90 17.77 16.10 15.62 11s.8 14.81 13.98 34.81 All but PCR/$I00M 18.26 21.14 24.15 17.47 23.61 21.84 20.20 18.69 17.96 15.79 17.84 15.93 15.63 15.61 14.79 13.68 14.78 Project Speifricts0loM 18.22 20.56 22.28 24.02 21.72 19.98 18.77 16.99 16.70 14.95 15.94 14.85 15.32 15.50 14.78 13.80 15.03 EEclding SAL & TA All SupetvisionO$100M I 18.90 21.82 24.09 26.70 23.53 21.74 20.17 18.76 17.90 15.97 17.61 16.17 15.58 15.81 14.37 13.67 14.40 AII but PCR/I100M 19.90 21.82 24.93 27.99 24.03 22.03 20.39 18.74 18.01 15.78 17.67 15.94 35.58 15.51 14.33 13.31 14.36 Project Specificl$ooM 18.16 20.34 22.04 23.67 21.35 19.62 18.37 16.50 15.97 14.22 14.90 14.08 k4.52 14.41 13.40 12.51 13.53 Average per Project 8.33 9.33 10.11 10.58 9.83 9.68 9.82 9.58 9.84 9.31 9J9 969 995 30.35 30.39 30.05 11.37 la Excludes those projects which can not be assigned to any of the four regions, for example, one relating to a past borrower country like Finland. Number of Projects = all proje:ts receiving at least .001 SW of Supervision during the FY. Average staftweeks per dollar is computed in the following way: total amount of staffveeks dividing by the amounts of commitment for those projects for which their supervision time is greater than .001 staffwcks. Commitmeni figures ma n real US$ millions using the PAC Commitment Deflator, distribution date May 29, 1991. /h Supervision staffwcks recorded on task ID SPN in TRS. lp Includes sstf wceks recorded on task-IDa SPN. PCR, ClI, ARI, PIR, PAD. OARS. ORDS, and ORDO. SOURCE: Word Bank MIS files. - 64 - Annex Table 8S RISE AND LOW PER (INVESTMENT) PROJECT SUPERVISION IN FY90 (Staff weeks per year - more than US$1 million disbursed) Project Country Loan Amount Amount Disbursed Supervision (US$ million) (US$ million) Staff Weeks LOW Education V Morocco 50 1.4 0.10 Education IV Tunisia 26 2.6 0.13 PIDER III RD Mexico 175 4.1 0.13 Krishna-Godavari Exp. India 165 6.6 0.15 Bukit Asam Coal Dev. Indonesia 185 9.6 0.17 IGH Multi-State ADP Nigeria 162 19.3 92.59 Segba V-A Argentina 276 26.7 70.74 Narmada (Gujarat) CA India 150 20.0 64.15 Narmada (Gujarat) DA India 100 11.8 63.24 Power Trans III Yugoslavia 115 8.1 61.38 Source: MIS ARIS database. - 65 - Annex Table 9: RATINGS OF PROJECTS REPORTED BY OED BY EVALUATION YEAR Unsuccessful Evaluation Sucessful Projects Projects as a % of Difference Year as a % of all Rated all Rated 1974 86.0 14.0% 72.0% 1975 86.81 13.2% 73.62 1976 85.5% 14.51 71.0% 1977 89.2% 10.8% 78.4% 1978 89.51 10.5% 79.1% 1979 87.61 12.4% 75.2% 1980 88.1% 11.9% 76.2% 1981 84.9% 15.1% 69.8% 1982 78.91 21.1% 57.91 1983 84.8% 15.21 69.6% 1984 72.8% 27.21 45.61 1985 69.31 30.7% 38.5% 1986 82.4% 17.6% 64.7% 1987 70.1% 29.91 40.2% 1988 75.0% 25.01 50.01 1989 69.51 30.5% 39.0% 1990 63.61 36.41 27.31 1991 62.5% 37.5% 25.0% Source: OED AR Database 8/23/91 Notes: "Projects" exludes SAL, ASAL, and TA operations. -66- Annex Table 10: EVALUATIOn RATINGS OF PROJECTS PER YEAR OF APPROVAL Approval No. Projects Evaluated % Satisfactory Year for Year of Approval 1968 69 84 1969 96 90 1970 116 82 1971 118 84 1972 142 85 1973 170 85 1974 177 74 1975 181 77 1976 206 79 1977 208 76 1978 220 68 1979 196 72 1980 206 77 1981 153 74 1982 121 73 1983 67 68 1984 21 55 1985 17 59 1986 11 55 1987 7 100 Source: OED AR database. - 67 - Annex Table 11: PROJECT IMPLEMENTATION STATUS BY ARIS COHORTS ARIS # ARIS FY General Status Problem Rating Projects 1 75 1.69 14.0% 2 76 1.61 10.0% 3 77 1.61 8.0% 4 78 1.64 9.0% 5 79 1.68 9.0% 6 80 1.72 8.8% 7 81 1.80 11.1% 8 82 1.82 11.8% 9 83 1.81 13.0% 10 84 1.79 11.0% 11 85 1.76 11.0% 12* 86 1.90 15.0% 13 87 1.80 14.0% 14 88 1.80 12.0% 15 89 1.80 13.0% 16 90 1.90 17.0% 17 91 2.00 20.0% Source: ARIS in FY 75, 76, 79, 82, 84, 89, 91. * In FY86, a status rating 4 was added to reflect a greater degree of implementation difficulty. This would, therefore, slightly increase the status ratings for the period after this date (compared to the earlier period), but would not be expected to affect the problem project percentages.