48032 v4 IMPROVING DEVELOPMENT RESULTS THROUGH EXCELLENCE IN EVALUATION Review of IDA Internal Controls An Evaluation of Management’s Assessment and the IAD Review Volume IV Report on the Completion of Part I Incorporating Compliance Testing of Key Controls (Part IB) 2009 The World Bank This paper is available upon request from IEG-World Bank. Washington, D.C. ©2009 The Independent Evaluation Group, The World Bank Group 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved This volume, except for the elements contributed by group and institutions outside the Independent Evaluation Group, is a product of the staff of the Independent Evaluation Group of the World Bank Group. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. This volume does not support any general inferences beyond the scope of this evaluation, including any references about the World Bank Group’s past, current, or prospective overall performance. The World Bank Group does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the World Bank Group concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The Independent Evaluation Group encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request to the Independent Evaluation Group. ISBN: 978-60244-113-2 Independent Evaluation Group Knowledge Programs and Evaluation Capacity Development (IEGKE) E-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 Printed on recycled paper Acronyms and Abbreviations AAA Analytical and advisory activities INT Department of Institutional AC Audit Committee Integrity ARPP Annual Review of Portfolio IRMF Integrated Risk Management Performance Framework AS2 Audit Standard No. 2 (as proposed ISR(R) Implementation Status (and Results) by the PCAOB under the Sarbanes- Report Oxley legislation) ICB International competitive bidding BP Bank Procedure IFI International financial institution BPM Business Process Module IT Information technology CAS Country Assistance Strategy LAS Loan Administration System CD Country Director LEG Legal CODE Committee on Development LOA Loan Department Effectiveness N/A Not applicable COSO Committee of Sponsoring OD Operational Directive Organizations of the Treadway OED Operations Evaluations Department Commission (now IEG) CPIA Country Policy and Institutional OP Operational Policy Assessment OPCQC Quality Assurance Compliance Unit CTR Controller’s (formerly QACU) DPL Development Policy Loan OPCS Operations Policy and Country DR Deficiency rates Services ER Exceptions rates ROSC Report on Observance of Standards ERL Emergency Recovery Loan and Codes ESW Economic and sector work PCAOB Public Company Accounting GAO U.S. Government Accountability Oversight Board Office PCPI Post-Conflict Performance GPN General Procurement Notice Indicators FM Financial Management PFM Public financial management FMA Fiduciary Monitoring Agent PMT Project Management Team FMS Financial Management Specialist QAG Quality Assurance Group FMSR Financial Management Status QEA Quality at Entry Assessment Report QSA Quality of Supervision Assessment FO Financial officer RSC Regional safeguards coordinator IAD Internal Audit Department RMFM Regional Manager, Financial ICFR Internal control over financial Management reporting RPM Regional Procurement Manager ICR Implementation Completion Report SIL Specific Investment Loan IDA International Development SM Sector Manager Association SOE Statement of Expenditure IEG Independent Evaluation Group SOX Sarbanes-Oxley Legislation (formerly OED) SWAP Sector Wide Adjustment Program IL Investment Lending TTL Task Team Leader Contents PREFACE.................................................................................................................. VII EVALUATION SUMMARY .........................................................................................IX 1. BACKGROUND AND STATUS AFTER COMPLETION OF PART IA .........1 Background, Recapitulation and Present Status ................................................ 1 Scope and Method for the IEG Evaluation of Part I............................................. 3 2. MANAGEMENT’S ASSESSMENT ...............................................................9 Introduction ........................................................................................................... 9 Issues of Approach and Method for Part IB Testing ......................................... 10 Summary of Major Findings................................................................................ 12 Deficiencies Identified During Part IB ................................................................ 14 Resolving Outstanding Issues from Part IA ...................................................... 17 3. THE IAD REVIEW .......................................................................................21 Introduction ......................................................................................................... 21 Approach and Method......................................................................................... 21 Major Findings..................................................................................................... 23 IAD’s Conclusions............................................................................................... 27 4. THE IEG EVALUATION ..............................................................................29 Management’s Approach and Method ............................................................... 29 Internal Controls at the Transactions Level....................................................... 31 Reading Management’s Test Results................................................................. 33 Material Weaknesses and Significant Deficiencies........................................... 36 Other Process Issues Uncovered in Parts IA and IB......................................... 37 The IEG Advisory Panel ...................................................................................... 39 5. CONCLUSIONS AND RECOMMENDATIONS...........................................41 iii Annexes ANNEX A: RECAPITULATION OF MAIN FINDINGS AND SUMMARY OF CONCLUSIONS FROM PART IA..............................................................................47 ANNEX B: SUMMARY OF THE IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING OF KEY CONTROLS IN PART IB.....50 ANNEX C: SUMMARY ACCOUNT OF THE DISPOSITION OF ALL REPORTED INTERNAL CONTROL ISSUES UNCOVERED DURING PART I ............................61 ANNEX D: A DESCRIPTION OF THE QUALITY RATING PROCESS USED BY IEG IN EVALUATING THE APPROACH AND METHODS IN MANAGEMENT’S ASSESSMENT AND THE IAD REVIEW OF PART I.................................................69 ANNEX E: STATISTICAL APPENDIX .....................................................................73 Boxes Box 1. Stages in the Study of IDA Internal Controls.................................................... 2 Box 2. Summary of the Key Elements in the Approach and Methods of Management, IAD and IEG in Completing the Work In Part IB ................................... 4 Box 3. Listing of Review Activities Performed by IAD in Part IB.................................22 Box 4. Scorecard and Status of IAD Issues and Recommendations From Part IA ....27 Box 5. Depiction of Management’s Testing Method...................................................32 Tables Table 1. Summary of IEG Ratings of Management’s Approach and Part IB Testing Methods.........................................................................................................31 Table 2. Distribution of Noncompliances by Random Occurrence and Concentrations ...........................................................................................................35 Figures Figure 1. Distribution of Control Steps with and Without Noncompliances ................34 Figure 2. Noncompliance Rates and Risk ..................................................................36 Figure 3. Reasons for Noncompliance .......................................................................37 Figure 4. Linkages between Key Controls and the COSO Components....................38 Attachments Attachment 1. Management Report on its Review of Internal Controls Attachment 2. IAD’s Review of Management’s Assessment Attachment 3. Statement of the External Advisory Panel iv Key Technical Terms Internal Controls: Controls, individually or in collective fashion, are structured means within an organization to enable it to achieve its busi- ness objectives while addressing risk. Control instruments include the control framework (in IDA’s case, the COSO (Committee of Sponsoring Organizations of the Treadway Commission) framework), organiza- tional checks and balances, published policies, and required procedures, among others. COSO Integrated Framework: A framework of management principles (“COSO components”) in an organization that, when collectively operat- ing as intended, will ensure the attainment of three key organizational goals (“COSO objectives”): reliable financial reporting; operational effec- tiveness and efficiency; and compliance with laws and regulations (in IDA’s case, with its charter and internal policies and procedures). The COSO components are: Control Environment, Risk Assessment, Control Activities, Monitoring and Learning, Information and Communications. Risk Focal Points: In the adaptation of the COSO framework by the Bank and IDA to meet their own needs, management has defined and added to the framework four key points of risk that face the mission of the Bank Group and are especially relevant to IDA . These are: Strategy Effectiveness; Operational Efficiency; Financial Soundness; and Stake- holder Support. Audit Standards: Criteria established by recognized accounting and au- dit bodies for conducting audits and reviews of internal controls that of- fer a basis for providing assurance that controls are well designed and working as intended, and for identifying deficiencies, significant defi- ciencies, and material weaknesses. Business Process Modules (BPMs): Management chose to conduct this review of internal controls by identifying the main business processes in which IDA is engaged on a daily basis in the course of its operations. These processes, 32 in all, covering IDA allocation; the Country Assis- tance Strategy (CAS) process; the main lending products ( Specific In- vestment Loans or SILs and Development Policy Loans or DPLs); and the fiduciary, contractual, safeguards, and quality assurance processes that support lending, were each mapped and described as separate busi- ness process modules, each containing the key internal controls that are the subject of the review. Process Map: The flow chart that graphically depicts all steps in a busi- ness process module. Key Control: A gateway and decision point, involving key units and IDA staff, in a given business process module, through which a business transaction being processed must pass. It is the effectiveness in design of these controls and the subsequent testing of the effectiveness of their op- eration that is at the center of this review. Business Process Template: A standardized assessment questionnaire and rating system used by IEG to provide quality ratings of manage- ment’s method and approach in identifying, describing, and mapping the business processes, and of its method in assessing the effectiveness of control design and of control operation. Evaluation Panels: In applying its Business Process Template, IEG as- sembled 3-4 person panels, including both controls specialists, and with experts in the particular discipline covered by the given BPM. The panels arrived at consensus judgments on the ratings that should be applied to each section of the module, according to their evaluation of the materials presented by management. Entity-Level Controls: The control framework that governs an organiza- tion at its aggregate level, emanating from central management down to the operating or business process level. In IDA’s case, the reference is to the elements of the COSO framework. Doing a controls review that started with an examination of entity-level controls could be described as a “top -down” approach. Bottom-up Approach: The approach adopted by management in its as- sessment did not begin with a top-down, entity-level review, but focused first on business processes at the transactions or operating level. Hence, it has been described as a bottom-up approach. Walk-through: An interactive interview and review of process docu- mentation conducted by management with relevant teams of IDA staff knowledgeable in a particular business process and its associated con- trols, with a view to verifying that controls are designed in the way de- scribed and operate as intended. Deficiencies, Significant Deficiencies, Material Weaknesses: Design flaws, omissions, or noncompliant operation of controls, discovered in the course of a controls review, denoting an ascending order of serious- ness. The precise criteria by which the three categories of materiality are distinguished are explained in annex B of the Part IA Report. NonCompliances: Controls or control steps found, during testing, to be not operating in conformity with the design of the control. Noncompli- ances include both exceptions and deficiencies (see below). Exceptions: Noncompliances deemed to be of a less serious or material nature than deficiencies. Exceptions/Deficiency Rates: The number of exceptions/deficiencies found during the Part IB testing of key controls, divided by the number of control steps in the sample. vi Preface In the IDA14 Replenishment Report Bank management committed to carrying out an independent comprehensive assessment of IDA’s con- trol framework, including internal controls over IDA operations and compliance with its charter and policies. Each part of this review was to be done in a three-phase approach: the first phase would be a self assessment by management, to be followed by an Internal Audit De- partment (IAD) review and report on management’s self assessment, and an IEG independent evaluation of both management and IAD work. Part IA of the review was completed in the fall of 2006 (IEG re- port dated October 18, 2006). The present report contains IEG’s evaluation of the assessment made by management and the review provided by IAD at the completion of Part I, incorporating the results of the testing done in Part IB. The basis for the work done by IEG in the current evaluation included the report prepared by management detailing its assessment (attach- ment 1); access to all the underlying materials that management gen- erated in its process based descriptions, definitions of controls, and the documentation of its testing of controls operation; and the report presented by IAD (attachment 2). Under the task management of Nils Fostvedt, this report was pre- pared by Ian Hume, with the assistance of a core consultant team, in- cluding Dexter Peach (strategic advisor, formerly assistant comptrol- ler general for planning and reporting, U.S. Government Accountability Office (GAO)), James Campbell and Rosemary Jellish (consultants, both former assistant directors, GAO) and Rachid Laajaj. The core team was assisted on selected topics by Jed Shilling, Tribhu- wan Narain, David Goldberg, and Mohammed Farhandi. Evaluation Summary Review of IDA Internal Controls: An Evaluation of Management’s Assessment and the IAD Review This report has its origins in a commitment that IDA management made as part of the IDA14 Re- plenishment process, in which it undertook “to carry out an independent, comprehensive assessment of IDA’s internal control framework, including internal controls over IDA operations and compli- ance with its charter and policies.” Management proposed, and the Board agreed, that Management would make an assessment of the controls, to be followed by an IAD review of the assessment and an IEG independent evaluation of both the management and IAD reports. Management conducted its assessment within Management subsequently divided Part I of the as- the COSO (Committee of Sponsoring Organi- sessment into two stages: Part IA, completed in Oc- zations of the Treadway Commission) integrated tober 2006, covered management’s approach and controls framework, and divided its study into method in identifying and mapping the business two parts: Part I dealt with compliance issues, processes that represent IDA operations and as- and was focused on controls at the level of 30 sessed the effectiveness of the design of controls (later increased to 32) business processes. These within these processes. Part IB, the subject of the were identified as representing IDA allocation; present report, and which concludes Part I, deals Country Assistance Strategy (CAS) and IDA with the testing of how these controls actually oper- lending products; supporting contractual, fiduci- ate, compared with their design. Part II is intended ary and safeguard processes; and quality assur- for completion by Management at the end of calen- ance. Part II will deal with issues of operational dar 2007, with the full IEG evaluation then expected efficiency and effectiveness, and will include an in early 2008. examination of entity-level controls, within the full COSO framework. This report contains IEG’s evaluation of the work completed by both management and IAD in their re- Management’s decision to start the review with spective assessments and review of Part I, incorporat- a bottom-up approach was different to what ing the testing done in Part IB. would be most often done for such reviews. While management presented reasons for this IEG conducted its evaluation using a combination decision it also unavoidably incurred certain of approaches, including reviewing all management scope limitations, including the need to post- and IAD materials and test results, using an IEG- pone final conclusions regarding the effective- created template to evaluate the quality of Manage- ness of controls until Part II and the entity-level ment’s approach to, and method of, testing, and of review have also been completed. the IAD review. ix Summary Findings: Approach and Method icy/Bank Procedures (OP/BPs)). Given the importance of the latter issue —as the yard- Management decided that the best way to track stick for compliance—and given its perva- the use of IDA resources was to focus its as- siveness, IEG regards it to be a potential ma- sessment at the transactions level on business terial weakness, though a final determination processes. Management used 32 Business Proc- will be made once Part II has been com- ess Modules (BPMs) to represent IDA opera- pleted. tions; 29 of these were subjected to testing. Management also elaborated on how 126 inter- Overall, management tested 115 key controls nal control issues raised in Part IA, includ- across 466 control steps. This represents signifi- ing those raised by IAD and IEG, have cant progress in developing an understanding of been addressed and disposed of, in some IDA’s internal controls at the transactions level, cases by deferring them to Part II. Man- and the test results matrixes reveal an unprece- agement could not test the process dealing dented view of these controls in operation. with safeguards and Corporate Risk list, be- cause testable guidelines did not exist. Findings: Controls Testing IEG evaluated management’s testing ap- Noting the significant deficiencies and other is- proach and method as being satisfactory, ro- sue uncovered by the assessment in Part IB bust, and credible, but with some quali- and its own review, and subject to the work fications relating mainly to sampling still to be done in Part II, IAD expressed methods. the opinion that management’s qualified conclusions regarding the compliance phase Management presented its testing results in of the controls review was “fairly stated”. the form of an aggregate pass rate of 93 percent, measured as the total number IEG’s evaluation considered a number of alter- of control steps in the sampled projects native “pass rates” but confirmed manage- found to be compliant. On this basis, it ment’s assessment results which showed concluded that IDA processes and con- that controls at the transactions level were in trols, and their associated fiduciary, compliance with required policy and proce- contractual, and safeguards “umbrella” dures at a rate exceeding the 90th percentile processes were adequate with some ex- level. ceptions. Overall, IEG finds that this result provides a IEG found that most noncompliances oc- reasonable level of assurance that the com- curred in random fashion, as isolated pliance aspect of internal controls for the cases of failure to observe control re- business processes are working as intended, quirements by those processing pro- with some notable exceptions. These excep- jects. Of the 466 control steps, only 6 tions relate to the potential material weak- percent had concentrations of control ness and significant deficiencies identified, failures (of three or more failures in a and the work still to be done in Part II, given control). This means that, in which will assess the operational effective- terms of remedies, the need for super- ness and efficiency of internal controls and vision by management may be more IDA entity-level controls. important than the need for re-design of controls. The overall accomplishments of Part I include: At the end of Part I, management has iden- Satisfactory mapping and verification of design tified three significant deficiencies (timely effectiveness of 31 out of the 32 principal accessibility of relevant documents; business processes variances in regional application of cer- tain financial and procurement proce- Establishing of a credible method for testing key dures; and failure to keep pace with controls and processes needed updates of the Operational Pol- x Summary Conducting compliance testing to assess the management oversight may be called overall operational effectiveness of the for. key controls in the business processes. o Documentation Retention and Accessibil- Taking remedial actions on the majority of ity: This significant deficiency sug- transaction control issues surfaced to gests the need to draw links with date. both the Control Environment and the Information and Communications com- Review of entity-level controls is to be com- ponent at the entity level, where pleted in Part II. improved IT systems will be part of the solution. Advisory Panel o Dated OP/BPs: This potential ma- As part of its evaluation, IEG obtained the ser- terial weakness is an essential ele- vices of an international Advisory Panel, which ment of the Control Activities com- visited Washington in early March. The Panel ponent, which it would be well to prepared a statement that is supportive of the accelerate and complete (as much approach, method, and conclusions reached af- as possible) in time for the comple- ter the work done so far. A copy of the state- tion of Part II; ment is appended to this report. o Managing the Risk Framework and Summary Extending COSO: As IEG recom- Except for the qualifications and deficiencies mended on Part IA, management should consider extending the noted in this report, some of which are sig- COSO framework by adding a nificant, and subject to the outcome of the fourth objective (strategy—high level planned follow-on work, the internal controls goals) and three new components included in the scope of the work completed (objective setting, event identification and to date provided reasonable demonstration risk response). This suggestion was that relevant IDA policies and procedures also made by the Advisory Panel. were being followed in IDA’s daily lending transactions. Where deficiencies have been o Efficiency and Effectiveness: As the noted, management is undertaking remedial overall review moves from the actions to address them. transactions level to the entity level, and from compliance to ef- Recommendations fectiveness and efficiency, a chal- lenge in Part II will be to build on IEG makes six sets of recommendations, as fol- the results from Part I linking lows: these to IDA’s Monitoring and Learning activities (including the Completion of the Entity-Level Assessment Quality Assurance Group (QAG) (Part II): The challenge for Part II will be to re- and IEG), in order to provide the dress the scope deficiencies in Part I and to element of effectiveness and effi- draw linkages between actual findings from the ciency testing that was lacking in transactions-level assessment and the COSO Part I, and is needed before final framework elements at the entity level. IEG conclusions can be drawn regard- recommends that the following topics be the ing the overall effectiveness of subject of specific focus in completing Part II: IDA’s internal controls. o Controls and Project Processing: Part I Issues Deferred to Part II: A number More frequent noncompliance of issues have been deferred to Part II, in in project processing, not con- addition to those such as information tech- trols flaws, suggests a need to nology (IT) controls, field offices, fraud, link to the Control Environment at and corruption, which management had the entity level, where greater xi Summary stated at the outset would be dealt with some of these control steps. This should be in Part II. The notable additions, which done on a risk-weighted basis, in cases IEG is recommending be given special where it is deemed important to have the focus during completion of Part II, are: evidence from testing. assessing the IDA Analytical and advi- sory activities (AAA) products (eco- Streamlining: The process mapping and con- nomic and sector work (ESW) and trols identification (and testing) has been technical assistance); and the 44 issues one of the important contributions man- remaining from the 126 identified agement has made as part of its transac- process issues from Part IA, issues tions-level assessment. IEG believes there is identified in Part IB that are deferred, merit in systemizing these materials, giving and progress on the management ac- them widespread visibility within the Bank tion plans developed during Part I. and IDA, and possibly incorporating them as part of the revised and updated corpus of Remedies for Problem Modules: Part I OP/BPs. These materials could also be in- revealed a number of modules with valuable tools in the effort to simplify and more prevalent controls failure, includ- streamline Bank/IDA processes. ing the fiduciary modules highlighted by IAD in its report: Core Specific In- vestment Loans or SILs; Financial Management in SILs; Procurement SILs; Procurement Complaints; Loan Management SILs; Loan Suspensions; and Loan Closings. These become clear candidates for review and, where rele- vant, remedies, and management has recognized this in its report. Safeguards and Corporate Risk: Man- agement is addressing this issue, which concerns the fact that procedures for placing projects on the Risk List are not standardized and documented. How- ever, given the sensitive nature of this topic and the potential risks to the reputation of IDA at stake in managing the Corporate Risk List, IEG recom- mends that management consider sup- porting the strengthening of this moni- toring device by the introduction of specific controls, standardized across all regions, to more fully integrate these processes into the internal control sys- tem. Testing Inoperative Control Steps: In Part IB a number of control steps were not actually tested, because these were often conditional control steps, not ap- plicable in the actual sample of projects tested. Most of these control steps may rarely be needed, but IEG does rec- ommend that management should make a selected, targeted sample to test xii CHAPTER 1 BACKGROUND AND STATUS AFTER COMPLETION OF PART IA Evaluation Essentials ™ This report focuses on IDA’s compliance controls and their operating effectiveness 1. Background and Status After ™ IEG evaluated Completion of Part IA Management and IAD outputs from testing of compliance controls ™ The report links the Background, Recapitulation and Present Status assessments of controls 1.1 In the IDA14 Replenishment Report, 1 Bank management to the five COSO components “has committed to carry out an independent comprehensive as- sessment of its control framework including controls over IDA op- erations and compliance with its charter and policies” (para. 39 of that document). Table 3 annex B of the document stipulated that this assessment should be undertaken by the Independent Evalua- tion Group (IEG, formerly Operations Evaluation Department or OED). That document has been approved by the Executive Direc- tors. 1.2 Management subsequently confirmed that the review was to be conducted within the context of the Committee of Sponsoring Organizations of the Treadway Commission (COSO) framework, 2 though it was to deal with only two of the COSO objectives: compli- ance with IDA’s charter, policies, and procedures and the efficiency and effectiveness of IDA operations. The objective of assuring reliable financial reporting is already routinely dealt with in the Bank Group’s annual financial reporting cycle. IEG has verified these procedures and has no issue with the exclusion during this review. 1.3 Management decided that it would divide its assessment IDA14 into two parts (I and II). Part I was to deal with internal controls committed to a over IDA’s compliance with its charter and internal policies and pro- review of cedures, and Part II with internal controls over IDA’s operational ef- internal controls ficiency and effectiveness. Each part was to have three phases: first, a management assessment of internal controls; second, an Internal Audit Department (IAD) review of management’s assessment; and third, IEG’s independent evaluation of both the assessment and the review. 1.4 Following certain delays in completion of Part I, and to bet- ter manage the reporting of its progress to the Board, management decided to divide Part I into two stages (Part IA and Part IB), each dealing with distinct components of the assessment of the internal controls. How the overall review has been divided, and what topics are covered in each part, is described in Box 1. 1 CHAPTER 1 BACKGROUND AND STATUS AFTER COMPLETION OF PART IA Box 1. Stages in the Study of IDA Internal Controls Management has divided its assessment into the following parts: Part I—Compliance with IDA’s Articles and Policies: This part, conducted at the transactions level, has been split into (A) the identification of key busi- ness processes and controls and assessment of the design effectiveness of the identified key controls and (B) the assessment of the operating effectiveness of the identified key controls through compliance testing. A. This portion of the overall assessment identified and mapped the Business Process Modules (BPMs) and the key controls in each proc- ess. Then management reviewed the design effectiveness of the busi- ness processes and key controls involved to identify any deficiencies in the design of the key controls. Management recommended reme- dial actions to address the design deficiencies. B. Management has conducted tests on a representative sample of products/transactions to determine whether the key controls were applied to the sample items as designed. Management has thus as- sessed and determined whether there are significant deficiencies or material weaknesses in the operating effectiveness of the key con- trols, and, where this has been found, management has recom- mended measures to address these deficiencies. Part II—Efficiency and Effectiveness of Operations: Management plans to assess whether the existing internal control framework, including corporate governance and entity-level controls, provides reasonable assurance that IDA’s operations are carried out efficiently and effectively, focusing on the processes and controls identified in Part I. In Part II, management also plans to address the other scope limitations in Part I, such as information technol- ogy controls and the Bank’s operations in the decentralized field offices. This report 1.5 The work on all three phases of Part IA (Management As- builds on the sessment, IAD Review, IEG Evaluation) was completed in the fall of conclusions of 2006, and IEG sent its report to the Audit Committee (AC) of the Part 1A of the Board on October 18, 2006, for a meeting that was held on October overall 30, 2006. 3 The principal subject matter of Part IA was the assessment assessment of the quality of process mapping and of the design effectiveness of key controls in the business processes that constitute and provide inter- nal controls over IDA operations. The main conclusions that emerged from the AC meeting were the following: y The progress made during the completion of Part IA was ac- knowledged; y The Committee was only partially satisfied with the scope and methodology of Management’s Part IA assessment, and was concerned at the omission of analytical and advisory activities (AAA); y The Committee expressed surprise at the differences of view regarding scope and method, and called for a more definitive 2 CHAPTER 1 BACKGROUND AND STATUS AFTER COMPLETION OF PART IA assessment, saying that this might have been better done up- front at the start of the review; y The issue of disclosure was also discussed, and it was decided that, given the interim nature of the report, it would not be disclosed at this time; y The Committee concluded that the list of deficiencies and weaknesses should be fully discussed among all the parties to assess their salience and materiality. 1.6 The subject of the present report is the (Part IB) testing of the This review operation of the key controls to verify that in practice they work as de- covers only signed, the resolution of some issues outstanding from Part IA, and compliance the deferment of some issues to Part II. Completing Part I both con- controls and cludes the assessment of controls at the transactions level, and com- their operating pletes the assessment of internal controls over compliance, the sec- effectiveness ond COSO objective. As background to the controls testing in Part IB, Annex A provides a summary of the main findings, conclusions and pending issues at the completion of Part IA. Scope and Method for the IEG Evaluation of Part I 1.7 Objective of the IEG Evaluation: Consistent with the man- IEG’s objective date that the Board has given to IEG, its objective is to provide an in the analysis independent evaluation of both management assessment and the is to provide an IAD review, covering scope, method and approach, findings, and independent the quality of conclusions drawn by both parties. In addition to evaluation of evaluating—and rating where appropriate—the work of the other Management two parties, IEG is also to draw its own conclusions regarding the and IAD reviews evident state of IDA’s internal controls over compliance with its ar- ticles and internal policies and procedures. 1.8 Ultimately, the anticipated outcome of the review (Parts I and II) will be a judgment of whether the IDA internal control framework, as designed and as found to be operating, provides rea- sonable assurance that objectives relating to accurate financial re- porting, compliance with IDA charters and related policies and pro- cedures, and efficient and effective operations are being achieved. 1.9 Completion of Part IB Completes the Whole of Part I -- Compliance Testing: The significance of completing the testing of controls in Part IB is threefold. First, it brings to a close the scope limitation that had applied to Part IA, where a judgment on the ef- fectiveness of the design of key controls could not be made until the operation of the controls had been tested. Second, it brings to com- pletion management’s assessment of IDA’s internal controls over compliance with its charter and internal policies and procedures, so far as this can be done without reference to efficiency and effective- 3 CHAPTER 1 BACKGROUND AND STATUS AFTER COMPLETION OF PART IA ness issues and the other issues (decentralization, information tech- nology (IT) issues) to be addressed in Part II. And third, it will con- clude the transactions level (bottom-up) portion of the assessment, which now awaits linkage to the entity-level (top-down) phase of the review, which will be completed during Part II. 1.10 In its evaluation approach, IEG has therefore maintained a focus on both the testing elements pertaining to Part IB itself and on the broader issues that are marked by the completion of Part IB: how to evaluate the testing process itself; what conclusions can be drawn regarding the completion of the compliance phase of the re- view; and what linkages can be said to have been established to the entity -level issues that will follow in Part II. IEG evaluated 1.11 Evaluation Methods: The basic materials available to IEG in the reports making its evaluation were: a report from management on its as- produced, raw sessment and a report from IAD on its review of management’s as- data, and sessment; methodological documents (sampling criteria, testing summary plan); raw data from management’s testing, in the form of results findings of the sheets and summary findings and conclusions for each module testing tested; various review sheets and working tools from IAD; and a data room that housed the documentation collected to match each control step tested. A summary of both management and IAD re- ports, giving an account of the approach, method and findings and conclusions, is presented in the two chapters that follow. The ap- proach and methods of each of the reviewing parties in addressing their respective tasks were explained in the Part IA report, and these have not changed. 1.12 A summary of the key elements that constituted the respec- tive tasks of IEG, management, and IAD is contained in Box 2. Box 2. Summary of the Key Elements in the Approach and Methods of Management, IAD and IEG in Completing the Work In Part IB MANAGEMENT IAD IEG Test Operation of Controls Review Testing of Controls Evaluate Quality of Controls Design controls test plan Review test methodology Tests Define sampling method Review process for Provide ratings for: Conduct testing documenting results • Testing of key control Assess process to detect compliance fraud • Linkage to COSO Tabulate Findings Review deficiencies Framework Test results matrix Conduct Independent analysis of results Form Conclusions Form Conclusions Form Conclusions Statement and Report and Opinion Overall Evaluation, Recommendations Recommendations 4 CHAPTER 1 BACKGROUND AND STATUS AFTER COMPLETION OF PART IA 1.13 As it did for Part IA, IEG has applied a combination of The IEG method methods in making its evaluation of Part IB: used critical evaluation, y IEG reviewed all methodological materials presented by man- independent agement, including those describing the sampling method for analysis of projects whose processes were to be the subject of the controls’ testing data testing, as well as the management report on Part IB (shown in attachment 1). y IEG attended a series of management briefing meetings held during the conduct of Part IB, in which the sampling methods and the actual process of testing were described. y Using the template that IEG had created for this purpose (de- scribed in annex D), it evaluated and rated the quality of man- agement’s approach and methods of testing for each of the 29 Business Process Modules (BPMs) that were tested. 4 y IEG conducted a statistical analysis of management’s test re- sults (that is, revealed noncompliances, by control steps, key controls, and modules) as a basis for evaluating the overall noncompliance rates revealed by the tests. Analysis was com- pleted to identify the different types of noncompliance and to show project- and control-related concentrations of noncom- pliance. Noncompliance was also examined on a broadly risk- weighted basis in an effort to establish whether controls have operated more or less effectively in higher-risk modules. y IEG evaluated all materials presented by IAD during the course of its review for soundness of method, findings, and quality of conclusions. This included those dealing with meth- odological and sampling issues, together with the report writ- ten by IAD on the overall completion of Part IB (shown in at- tachment 2). y With regard to the issues and potential deficiencies that were outstanding from Part IA (see para. 8 in annex A), IEG partici- pated in meetings with both Management and IAD that sorted these issues, resolved some, sent some for testing during Part IB, and identified those that were to be dealt with during the completion of Part II. 1.14 The completion of Part IB constitutes the end of the compli- The report ance portion of the internal controls review and marks the comple- seeks to link the tion of the transactions-level phase. IEG has therefore given some assessments attention to how this phase should be linked to the entity-level as- done to the five sessment that will be conducted in Part II. To this end IEG has used COSO its template to discern the degree to which management’s assess- components ment has drawn links between individual key controls within the BPMs and the five COSO components, whose effective operation will constitute the main framework of enquiry under Part II. 5 CHAPTER 1 BACKGROUND AND STATUS AFTER COMPLETION OF PART IA 1.15 Advance Summary of Findings: Using the evaluation meth- ods just described, IEG has completed its evaluation of manage- ment’s assessment and the IAD review of Part IB. Its complete evaluation is presented in chapter 4. The key overall conclusions reached by management, IAD and IEG can be summarized as fol- lows: y At the completion Part I, IEG concludes that it has been able to achieve the objective of its evaluation as laid out in para. 1.7 above, by evaluating both management’s assessment and the IAD review and arriving at a qualified conclusion. IEG found the approach and method used by management in its controls testing to be concrete, robust, and credible, and it found the review by IAD to be compre- hensive and rigorous. y On the basis of its own assessment, which showed a 93% compliance rate over all control steps tested, management came to the conclusion that both the design and the evidence of operation of IDA’s controls over compliance with its charter and internal policies and procedures were adequate with certain exceptions to ensure that IDA funds were used for the purposes intended. The exceptions related to the three significant deficiencies it identified and the need to further re- view fiduciary controls in the areas of financial management, pro- curement, and safeguards(Corporate Risk) during Part II. y In its report, IAD emphasized certain methodological issues relating to management’s testing program, and it focused on the less positive results generated by pass rate concepts that differed from the one used by management. On balance, and subject to the work that has still to be performed, IAD stated that, in its opinion, Management’s qualified conclusions were fairly stated. y In making its overall evaluation, IEG believes that management cor- rectly qualified its conclusion regarding the evidence of compliance with processes and controls. IEG believes the need for management to qualify its conclusion should relate more to the number of defi- ciencies revealed by its assessment, including several significant de- ficiencies (one of which IEG still considers a potential material weak- nesses). y Management, IEG, and IAD have together identified three signifi- cant deficiencies, one of which IEG considers to be a potential mate- rial weakness, and six deficiencies identified by Management. Where deficiencies have been noted, management is undertaking remedial actions. NOTES 1. See “Report from the Executive Directors of the International Development Association to the Board of Governors, Additions to the IDA Resources: Fourteenth Replenishment, Working together to Achieve the Millennium Development Goals” (approved by the Executive Directors of IDA on March 10, 2005). 6 CHAPTER 1 BACKGROUND AND STATUS AFTER COMPLETION OF PART IA 2.The Committee of Sponsoring Organizations of the Treadway Commission which published its report in 1992: Internal Control-Integrated Framework. The World Bank Group adopted the COSO principles of internal control in 1995. 3. See “Review of IDA Internal Controls: An Evaluation of Management’s Assessment and the IAD Review” (AC 2006-0099), October 18, 2006. 4. There were a total of 32 modules developed by management, but three were not tested: Module 2 (Country Performance and Institutions Assess- ment [CPIA]) and Module 3 (Post-Conflict Performance Indicators [PCPI]) are sub-processes of the Allocation Module (Module 1), which was fully tested and IEG saw no major reason to test them separately. Module 29 (Corporate Risk—OPCQC/ QACU) was not tested because the Bank has dif- ferent regional practices (mostly undocumented) whereby high-risk projects are placed on the Corporate Risk List, and these were basically not testable. Both Management and IEG regard this as a deficiency that needs to be ad- dressed (see Management report para 18 E). 7 Evaluation Essentials ™ The testing and sampling methods used for Management’s 2. Management’s Assessment assessment were generally acceptable ™ The Testing Plan ensured transparency in the Introduction testing of each control step 2.1 Management’s report at the conclusion of its assessment of Part I (including Part IB) is attached as attachment 1. What follows ™ Management’s qualified is a summary description of the major topics covered, each accom- conclusion is appropriate panied by IEG observations on key aspects of the assessment. How- ™ The deficiencies identified ever, IEG’s overall evaluation, also taking into account IAD’s re- are the right ones and view, is contained in chapter 4. appear to be concentrated in fiduciary 2.2 This chapter describes some changes that were made by and safeguard processes management in the basic universe of modules and controls to be ™ Management has taken tested. It discusses management’s method and approach in conduct- action on issues and ing the compliance testing, summarizes management’s main find- recommendations made ings from the testing, and describes management’s main conclu- in the earlier analysis by sions in completing not just Part IB but Part I as a whole. IAD and IEG 2.3 Some Adjustments to the Module Base: As a result of the ex- change of views that took place during the completion of Part IA, management has made adjustments to some modules. 1 Further, the Debt Sustainability BPM (now renamed Debt Reporting) which was still being mapped and assessed at the time of Part IA, has now been finished, and has been subjected to the same set of walk- through and controls testing as the other modules. In addition, re- sponding to comments made during Part IA by IEG and IAD, man- agement has added two new modules, one dealing with the Post- The Conflict Performance Indicators (PCPI) and the other with the Management Country Performance and Institutional Assessment (CPIA) proc- assessment esses. As a result, the number of modules has now grown to 32. 2 completed Responding to comments made principally by IEG, management assessment of has also begun work on mapping and locating key controls in a the Debt BPM representing AAA work, which will be completed in time to Reporting be assessed and evaluated during the completion of Part II. module and added the PCPI 2.4 IEG Comment: IEG views the addition of these modules to and CPIA be useful and appropriate. However, the Debt Reporting Module modules should be extended to contain a growth, export, and debt manage- ment analysis to assess the debt sustainability of the relevant coun- tries. This aside, these additions mean that, with exception of the AAA products (to be added during Part II), the BPMs now give an adequate representation of the universe of IDA operations. CHAPTER 2 MANAGEMENT’S ASSESSMENT Issues of Approach and Method for Part IB Testing 2.5 Key Elements in Management’s Approach and Method: Other than the cases mentioned in para. 2.3 above, the universe of BPMs and their internal controls to be tested were identified and defined in part IA. The key elements of method that were different or additional in Part IB included the sampling methods, the Testing Plan, 3 and the actual testing process. 2.6 Sampling (Lending Products): The essence of management’s sampling method was, first, to separate the allocation and other non-lending processes from the main lending operations (SIL, DPL), with the latter chosen from a universe of projects approved or made effective between July 2005, and February 2006. For these lending operations, the project universe was sorted by region; the projects were sorted by approval date. The regional sample size was fed into the random sequence generator (www.random.org), and from this random sample, the number of projects established separately by management as being required(i.e. 15 SILs and 7 DPLs) was identi- fied. The sampling 2.7 According to management, “the required number of projects methods and to be selected for testing of Investment Lending operations was de- the resulting termined by management to be 15 each for pre-supervision and su- sample sizes pervision activities. This provided a sample size equivalent to the used by median between the applicable audit requirements applied by Management IDA’s external auditor….used for the review of internal controls re- were generally lated to financial reporting for Weekly and Monthly frequency of acceptable activity.” 4 For the sample SIL projects, two groups of 15 were needed, because it would have been impractical to follow the same projects throughout their lives, a span of several years. So, two groups were selected: one set from project inception to Board ap- proval; and a second set from effectiveness to completion. From the 15 SILs, 10 were core SIL operations and 5 were other investment lending operations (responding to the IEG request made in Part IA, to test non core SILs as part of the sample). 2.8 IEG Comment: IEG finds the sampling methods and the re- sulting sample sizes to be generally acceptable. However, there were some modules in which not all the projects chosen by the random sample technique exhibited the business process being tested (for ex- ample, loan suspensions, or loan cancellations), so the active sample was smaller than the chosen sample. IEG observed that in the DPL sample, not all regions were represented, and management accord- ingly has undertaken to conduct an additional test with at least one DPL from a region not in the original sample. In those future tests where the sample size would be small by the nature of the case (that is, rarely operated controls), IEG would recommend, where feasible, 10 CHAPTER 2 MANGEMENT’S ASSESSMENT expanding the sample sizes (or make targeted samples) to ensure all regions are represented. 5 IEG also notes that the sample period did not cover the full operational year, so it missed the year-end “bunch- ing” season, which is when project processing is under most stress. 2.9 Sampling (non-Lending): The allocation and programming processes were treated separately from the lending operations, since these did not involve individual projects. The IDA allocation proc- ess required no project sampling, since this is a unique annual event in itself, but front loading was tested in two countries. Similarly, for the debt sustainability, PCPI, and CPIA modules, several countries were tested through these processes. For CAS processes, five coun- try cases were chosen. QAG processes occur on an annual cycle, and samples were collected by management during the Part IA walk- through process, and the same was intended in the case of the Qual- ity Assurance Compliance Unit (OPCQC, formerly QACU) and the Corporate Risk List (though in the end testing was found not to be possible, because criteria used by the regions have not been stan- dardized and documented). 2.10 IEG Comment: IEG agrees with the different approaches fol- lowed in the case of these non-project modules, though it observes that in the case of the CAS processes the sampling was not really random. At IEG’s request, management is conducting an additional test on one other CAS cycle. 2.11 Controls’ Audit and Testing Plan: In 2006, under Part IA, Management conducted a very detailed examination of each key con- trol in each module, to identify the actual, documentable control steps that constitute the key controls. This inventory became the Test- ing Plan, which was a description of each testing step management’s Project Management Team (PMT) used to verify the operation of the key control. This provided the basis for the call for documentation, during Part IB, from the Bank’s operating departments to test the ac- The Testing tual working of each control. Plan was a useful addition 2.12 IEG Comment: The Testing Plan was a useful addition to the and made the BPM materials, and made transparent exactly what was being tested testing of each at each control step. control step transparent 2.13 Documentation of Controls Steps —The Testing Process: This differed from the walk-through processes that were conducted in Part IA to verify the accuracy of mapping and effectiveness of control design. In testing for the operation of controls, management requested departments to supply the relevant documentation for each of the sampled projects, for each control step or attribute in each key control. (In the case of the non lending processes, man- agement asked the respective Bank units responsible for executing 11 CHAPTER 2 MANAGEMENT’S ASSESSMENT these processes to supply the required documentation, to verify that the processes operated as designed). 2.14 IEG Comment: This method appears to have worked well, and the documentation for each step is contained in one or more binders for each module, stored in a data room that IEG has visited and where it has inspected several binders, on a self-selected basis. IEG inspected binders for six business process modules (SIL project cycle, DPL project cycle, Financial Management SIL, Procurement Complaints, Application Review, and Safeguards SIL) and looked at documentation related to two to four projects in each module. IEG found that the binders displayed the relevant documentation of the controls tested and it found that the management team staff were very knowledgeable in explaining the meaning of the documenta- tion that showed the operation of the controls. 2.15 This concludes the summary description of management’s ap- proach and methods used in Part IB; what follows is an account of the results of management’s testing, and the major findings that emerged. 6 Summary of Major Findings Management 2.16 Management’s Overall Assessment: In presenting its find- appropriately ings and conclusions on the completion of Part IB, management qualified its provided an overall summary of its assessment (paras. 17 and 18), conclusion that which states as its view that “the design and operational effective- compliance is ness of identified processes and associated key controls are ade- adequate with quate to ensure compliance with IDA’s policies and procedures that certain funds are used for the purposes intended, except for” three signifi- exceptions cant deficiencies and the need to further review the fiduciary con- trols in Part II. 2.17 IEG Observation: IEG believes that management correctly qualified its conclusion regarding the evidence of compliance with processes and controls because of the number of deficiencies re- vealed by its assessment, including several significant deficiencies (one of which IEG still considers a potential material weaknesses). 2.18 Testing for Operating Effectiveness of Key Controls: Man- agement tested 115 key controls in 29 modules. Many key controls have several control steps, or (management’s term) “attributes.” Thus, the full test of a control often involved testing several indi- vidual steps or attributes. Since there were 466 individual control steps and 115 key controls, the average control had around four control steps, with a range from 1 to 15. (In annex B IEG provides a detailed description of management’s testing approach and method.) 12 CHAPTER 2 MANGEMENT’S ASSESSMENT 2.19 Summary of Results and Management Conclusions: Each module was tested by documenting the processing of a number of sample projects—with samples varying from 30 projects for some modules, to 3 to 5 or fewer for others (345 projects in total). The total number of control steps tested (including the N/As) was 3706, or 3603 net of not applicables. From this total, management found 246 noncompliances, —that is, control steps where the control failed— a failure rate of 7 percent. (IEG Note: It should be stressed that this “failure rate” does not mean that 7percent of all tested projects “failed.” The rate is an aggregate measure, across all projects tested, of the gross number of control steps that were not complied with, as a percentage of the total number of control steps tested (for exam- ple, 246/3603). Further, as part of its assessment, management found that about half of all noncompliances were, in its judgment, “exceptions” rather than “deficiencies.” Exceptions were less seri- ous cases of noncompliance, frequently involving other mitigating controls, which therefore offset the negative impact of their failure. IEG examined management’s criteria in making these distinctions, module by module, and found them to be justified. 2.20 Management took these findings—based on the 93 percent The finding of a pass rate—as the overall salient feature of its testing program. On this 7% failure rate, basis it drew the following conclusions: 7 (The following list appears about half of in para. 24, page 9 of management’s report) which were “exceptions” y The performance-based allocation model is being imple- rather than mented in a manner that directs scarce IDA resources in sup- “deficiencies,” port of priority development activities in the poorest eligible is based on member countries. credible and y The complementary use of the three primary instruments for car- transparent rying out IDA operations (CAS, ILs and DPL), and evidence re- testing lating to the application of the processes and controls that apply to them (from identification to completion), confirm that:  IDA financing is being provided in support of develop- ment priorities and is focused on matters that appropri- ately fall within IDA’s mandate; and  Consistent with the provisions of IDA’s Articles of Agree- ment, IDA financing is made available for specific projects as well as other “special circumstances” operations, where appropriate.” 2.21 Management also goes on to say, in the same paragraph, that the test results show that the “umbrella processes” 8 and associated fiduciary, contractual, safeguards, and other processes are adhered to as well as that the procurement provisions and controls also ensure compliance with OP/BP11.00 and the Bank Guidelines: Procurement under IBRD Loans and IDA Credits and Guidelines: Selection and Em- 13 CHAPTER 2 MANAGEMENT’S ASSESSMENT ployment of Consultants by World Bank Borrowers, except for some defi- ciencies and exceptions noted in para 25B. (Management Report, page 9) Deficiencies 2.22 IEG Observations: IEG observes that management has con- appear to be ducted credible and transparent testing of controls, which provides concentrated in evidence that tends to confirm that process controls mapped and the fiduciary, assessed for design effectiveness in Part IA have broadly operated contractual, and as designed, with a pass rate of 93 compliance. There are no ready- safeguards made criteria against which to measure the “acceptability” of such a processes pass rate, but in chapter 4, IEG develops perspectives by which this rate can be assessed. Regarding the claim that the tests also show that the Bank’s “umbrella processes” have operated as designed, IEG would agree that, since several of the 29 BPMs chosen to repre- sent IDA operational processes covered these umbrella areas, it would be fair to conclude that the overall pass rate also generally covered these areas. However, based on its own review of Man- agement’s test results, IEG would emphasize that a number of defi- ciencies appear to have been concentrated in the fiduciary and safe- guards processes (see Table B.4 in Annex B) so these processes do not escape the qualification of Management’s overall conclusion. Also, these testing methods are testing only for compliance, and do not yet speak to how the internal controls may assure effective or ef- ficient the tested actions may have been. Deficiencies Identified During Part IB 2.23 Significant Deficiencies: Management identifies three sig- nificant deficiencies, as follows: y Timely Accessibility of Relevant Documents: Management The three explains that while it found a 93 percent overall pass rate in its deficiencies controls testing, the lack of timely access to relevant docu- identified are ments remained the most common source of noncompliance. the right ones, Management has already instituted a remedial program, and particularly the recommendations are due to be issued by June 30, 2007. inclusion of y Variances in Regional Implementation of Institutionally En- documentation dorsed Financial Management and Procurement Guidelines: accessibility The testing revealed that financial and procurement review processes were not always implemented in accordance with the most recent guidance issued by the appropriate sector boards. Management has called for the boards to review the relevant documentation, and for a harmonization of Imple- mentation Status and Result (ISR) Report reporting across the regions, together with improvements in the procurement com- plaints database. 14 CHAPTER 2 ASSESSMENT MANGEMENT’S N o t Keeping Pace w i t h Needed Updates of the OP/BPs: A s i d e n t i f i e d during P a r t IA, management repeats t h a t this i s a sigruficant deficiency, but argues t h a t it does not rise to t h e l e v e l of a m a t e r i a l weakness because i n t e r i m procedures h a v e been issued w h e r e gaps h a v e b e e n i d e n t i f i e d i n t h e OP/BPs. It outlines a 24-month p r o g r a m to address this issue, with rec- ommendations to b e issued by end-2007, a n d management will r e v i e w t h i s issue as p a r t of i t s e n t i t y l e v e l assessment in Part 1 1 . 2.24 I E G Observations: IEG agrees t h a t all three issues w e r e rightly i d e n t i f i e d by management. It also agrees t h a t the issue of documentation accessibility (revising i t s opinion stated i n Part IA) can now b e treated as a sigruficant deficiency, along with t h e issue of regional variances. However, g i v e n the centrality of OP/BPs as a pil- l a r of p o l i c y compliance, IEG retains its earlier opinion t h a t t h e OP/BPs issue remains a potential material weakness, to b e revisited during Part 1 1. Also, IEG notes the tests conducted by management also revealed variation in i m p l e m e n t a t i o n in institutionally endorsed guidelines with regard to safeguard processes as w e l l as financial a n d p r o c u r e m e n t r e v i e w processes. 2.25 Exceptions and Deficiencies: From a l o n g e r l i s t of a n u m b e r of exceptions and deficiencies i d e n t i f i e d during P a r t IB, manage- m e n t h i g h l i g h t s the following six: streamlining of Investment Lending Operations: Manage- ment’s assessment i d e n t i f i e d t h a t existing processes and documentary requirements are inefficient and onerous. For example, t h e presupervision phase of a S I L comprises 20 con- trols and 95 attributes. Streamlining recommendations will b e m a d e following f u r t h e r r e v i e w of these issues during P a r t 1 1. Frequency o f Corporate Reviews of ILs: T h e d i s p a r i t y w a s n o t e d b e t w e e n t h e frequency of I L s and D P L s b e i n g sent for corporate review, and Operations Policy and Country Services (OPCS) has already issued a n e w Guidance Note to address t h i s issue. Credit Information Update and Loan Department Clearance Six highlighted Processes: Testing revealed certain weaknesses i nt h e processes deficiencies are for u p d a t i n g the Loan A d m i n i s t r a t i o n System (LAS) and docu- in accord with m e n t i n g t h e clearance processes, including: L A S set up re- IEG’s findings quirements w e r e not fully implemented; Loan Department (LOA) approvals of notices of loan suspensions w e r e sometimes not filed, and not a l l suspension notices w e r e sent to LOA for clearance; and a historical t r a i l of non-payment related suspen- sions w e r e not m a i n t a i n e d in the L A S because of system l i m i t a - tions. These will b e issues addressed in a n e w system to b e im- 15 CHAPTER 2 MANAGEMENT’S ASSESSMENT plemented by the third quarter of FY08, and these new controls are to be periodically tested by the Controllers. y Providing Clearances from Reviewers: Management discov- ered a number of cases where clearance of project documenta- tion had been given, conditional on certain changes being made. However, there was often no follow-up by reviewers to ensure the changes were in fact made, and there was some- times confusion between Task Team Leaders (TTLs) and re- viewers as to whose responsibility this was. Management is reviewing the need for further guidance on this issue. y Procedures Related to Safeguards Corporate Risk List: Man- agement discovered that the existing procedures for OPCQC to determine the inclusion of high-risk projects on the Corpo- rate Risk List were drafted as an advisory system, not a con- trol. Therefore, regions were often found to differ in their ap- plication of these processes, and this lack of a standard template meant there was no basis for the testing of this busi- ness process module (Module 29). Management is redressing this deficiency by setting up a standardized system, and it will attempt to test Module 29 under the new system (if sufficient data has accumulated) during Part II. y Debt Reporting Process: Management’s review found that borrowing countries, which are required by OP/BP 14.10 to report quarterly and annually on their external debt, were complying with the annual requirement, but not the quarterly requirement. Management did not review the process relating to the broader debt sustainability analyses required under IDA 14, including the grant allocation framework based on the level of debt distress. 2.26 IEG Observations: IEG observes that these issues have been correctly identified by management as deficiencies to be high- lighted, and these accord with its own review of management’s re- sults. In the case of the procedures for corporate risk review, IEG is not aware of any evidence suggesting that these issues have not been properly managed. However, given the sensitive nature of the Corporate Risk List 9 and the implications it carries, IEG believes that management’s remedial actions should also include a standard- ized approach to these processes across all regions. IEG has also pointed out that debt sustainability analysis is an important element of the IDA allocation process and needs to be further developed. It should be noted, however, that management has begun, and in many instances completed, significant steps toward remedying not only these issues but the full array of issues and deficiencies identi- fied by the three reviewing parties to date. 16 CHAPTER 2 MANGEMENT’S ASSESSMENT Resolving Outstanding Issues from Part IA 2.27 Management saw it as a second objective of the testing in Part IB to resolve the issues that had been raised during Part IA, by management’s own work, and by the IAD review and IEG evalua- tion of management’s assessment. 2.28 Addressing Issues Raised by IAD: Management listed eight Management took actions on issues that had been raised by IAD during Part IA, and described eight issues raised by IAD in the actions it had taken to address them, as summarized below: the earlier analysis y Issue 1: IDA Processes Selected. Management justified the trans- actions-based first stage of the overall review as being dictated by the unprecedented nature of the review, time limitations, and the fact that the prime focus of the mandate given to man- agement was to assess the extent to which IDA resources were being used for the purposes intended. IEG commented extensively on management’s approach in Part IA. y Issue 2: Information Technology Controls. Management has always maintained that it would address the IT controls as part of the entity-level assessment to be done in Part II, but IAD had sug- gested that some computer controls were also relevant to the transactions-level assessment. Management has undertaken to document all IT systems impacting operations with automated compliance controls and report on them in its Part II report. IEG accepts management’s explanation as reasonable. y Issue 3: Fraud and Corruption Controls. Management responded to IAD’s claim that fraud and corruption should have been explicitly addressed at the transactions level by pointing out that this is inherent in the nature of the controls tested for country governance (CAS process), and also in the fiduciary and other aspects of testing project controls. Also, manage- ment identified fiduciary controls (such as separation of duties for loan applications and withdrawals within LOA) that have been specifically designed to prevent fraud and corruption. Management will use the results of the Bank’s Internal Control Over Financial Reporting (ICFR) as an input to its entity-level assessment of these issues in Part II. IEG accepts management’s explanation as being reasonable. y Issue 4: Outdated Operational Policies (OPs) and Bank Procedures (BPs). Management explains that a program to update and streamline the OP/BPs for investment operations is under- way, and will be completed within 24 months, and that the re- sults of the present controls review at the end of Part II is likely to be an important input to that program. IEG had also raised a similar point; however, pending completion of Part II, IEG still regards this issue as a potential material weakness. 17 2 CHAPTER ASSESSMENT MANAGEMENT’S Issue 5: Categorization and Remediation of Deficiencies. Manage- m e n t explains that it has evaluated the m a g n i t u d e o f the con- trol deficiencies identified by itself, o r by I A D or IEG, a n d it has shared these evaluations with b o t h I A D a n d IEG. Issue 6: Document Retention and Accessibility. Management re- peated the explanation it gave o f i t s o w n assessment of this is- sue: that i t regards t h i s as less an issue of compliance than as an issue of ensuring adequate accessibility to relevant documenta- tion. IEG agrees with this explanation, but notes that (as explained in chapter 4) this issue remains a significant deficiency. Issue 7: Assessment o f Entity-Level Controls. The r e v i e w o f entity- l e v e l controls will b e carried out during Part 1 1. Management has consulted closely o n the scope of the activities t o b e in- c l u d e d in t h i s r e v i e w with I A D a n d IEG. IEG accepts manage- ment’s explanation for deferring review o f these controls to Part II. Issue 8: Walk-throughs o f Process Documentation. Based o n discus- sions with IAD, management believes that the compliance- test- ing methodology that included testing i n d i v i d u a l credits t h r o u g h all the controls i n a procedure h a d alleviated IAD’s concerns about the design effectiveness by the completion of Part IB, and d o not believe this i s an issue any longer. IEG ac- cepts this explanation, and shares the view that this i s no longer an is- sue. 2.29 Addressing IEG Recommendations: Management dealt with each of the s i x recommendations m a d e by IEG in i t s Part I A report, as follows: Management also took actions to deal with IEG’s earlier Recommendation 1: Confirm the Validity of the Business Process recommendations Mapping Cluster. IEG h a d raised two issues u n d e r this head- ing. The f i r s t deals with the fact that the BPM f o r SILs m a y n o t fully represent the controls relating t o a l l ILs. IEG suggested t h a t during Part IB, management include a range of I L s in ad- n the sample, and management did this. The d i t i o n t o the SILs i second issue was the exclusion o f AAA products from the cluster. Management explained that the exclusion w a s consis- tent with i t s focus and m a i n objective of assessing the internal controls in place for ensuring how borrowers use IDA re- sources. During Part I1management will r e v i e w the A A A ac- tivities t o determine if the k e y controls for the majority o f these activities are consistent and can b e easily documented a n d tested. IEG explained its position i n Part IA: that the review is examining controls over IDA operations (all operations, not just lending), but it believes that management’s undertaking to examine the AAA issues during Part 1 1can settle this issue (especially since many A A A aspects relate to eficiency and eflectiveness). 18 CHAPTER 2 MANGEMENT’S ASSESSMENT y Recommendation 2: OP/BP Status: Management repeats the same explanations made in the case of the similar point raised by IAD. y Recommendation 3: Complete the Remaining Stages of the IDA Re- view. Management repeats the explanation given in the case of the same point raised by IAD. y Recommendation 4: Resolve Issues and Potential Deficiencies with IAD. The potential issues identified by Management during its assessment and by IAD’s review have been evaluated to de- termine their impact on IDA’s internal controls and the reme- dial actions, if any, that may be required to mitigate risks. Management’s listing of the identified deficiencies and their resolution has been shared with IAD and IEG. IEG comments on these issues in more detail in the sections that follow. y Recommendation 5: Manage the Risk Framework. In response to IEG’s Part IA recommendation, management will review the COSO Enterprise Risk Management framework in the second half of FY 07, for potential strengthening and/or adaptation into COSO of the Bank’s (including IDA) existing Integrated Risk Management Framework. This decision is not expected to impact Part II of the review. IEG accepts this explanation as ade- quately addressing its concern. y Recommendation 6: Mainstream Internal Controls Reviews. Man- agement will commence discussions with the Audit Commit- tee to consider the value to the Board and shareholders of adopting a process for periodic or ongoing monitoring and re- porting on internal controls, in addition to the review of inter- nal control over financial reporting. IEG regards this as a positive step and encourages serious consideration of its recommendation. 2.30 Addressing other specific issues raised in Part IA: During Part IA, management, IAD, and IEG raised a total of 126 other is- sues. Management reported that 45 of these were resolved, 44 were deferred to Part II, and 37 were included with other issues raised in Part IB. 2.31 This concludes the summary of management’s assessment. Further details of management’s work and the conclusions drawn from the completion of Part IB can be found in the management re- port itself, at Attachment 1. NOTES 1. A principal example: QAG processes were added into the core SIL process map. 2. As explained in paragraph 1.13iii above (and footnote 5 in Chapter 1) man- agement only tested 29 of the modules. 19 CHAPTER 2 MANAGEMENT’S ASSESSMENT 3. The Testing Plan was a listing of steps the project management team took to evaluate the components of each key control, with a view to verifying that each control step operated as designed. 4. The applicable audit standard is described in Annex 3 to the Management report (attachment 1): Standard No.2 (AS2), An Audit of Internal Control over Financial Reporting Performed in Conjunction with an Audit of Financial State- ments, issued by the U.S. Public Company Accounting Oversight Board (PCAOB) in response to the provision of Article 404 of the Sarbanes-Oxley Act. As explained in the Part IA Report (see annex B), management has cho- sen to use concepts similar to AS2 (though not the standard itself) as guiding standards for the present review. Management made the choice of a sample size of 15 based on general guidance in paragraph 105 of AS2 that manual controls that operate more frequently should have more operations of the control tested. Other guidance in AS2, such as that controls that are rela- tively more important should be tested more extensively— for example, by evaluating whether the judgments made were appropriate and adequately supported, may be used in Part II. 5. IEG also recognizes that in the cases where rarely used controls are to be tested, often these may have to be individually targeted, so the strict princi- ple of random selection may not be achievable. 6. A segment of management’s report also deals with a number of the issues that had arisen during Part IA (all of which have either been mentioned in chapter 1, or are dealt with in chapter 4). These include: Management’s methodology; treatment of IT controls; fraud and corruption issues; the need for improved OP/BPs; documentation issues; walk-through concepts; ex- tending the content of the BPM cluster; treatment of risk; and the resolution of issues uncovered during Part IA. 7. Management makes clear in its report that this pass rate does not imply that 93 percent of all projects were broadly compliant, because the rate is cal- culated as a broad aggregate on the overall incidence of noncompliance, compared with the total number of tested control steps, not by individual projects. 8. The Bank’s Flagship OP/BPs contain four umbrella statements covering: i. Eligibility for IDA financing; ii. Country Assistance Strategies; iii. Investment Lending; and iv. Development Policy Lending. See Part IA Report Box 7, page 13. 9. The Corporate Risk List is a document compiled to record the most highly sensitive projects, from a safeguards risk perspective, to allow senior man- agement to internally monitor progress in maintaining their safeguards pro- visions. At present, therefore, it is really only a monitoring device, and not strictly a control mechanism. 20 Evaluation Essentials ™ IAD’s approach and method were rigorous, detailed, systematic, and 3. The IAD Review comprehensive ™ IEG agrees with most of the issues raised by the IAD review Introduction ™ IEG does not share the 3.1 IEG has evaluated the report presented by IAD at the con- view that the three clusion of its review of management’s Part IB assessment, which is fiduciary processes included as Attachment 2. What follows is a summary description identified by IAD are the “most critical” in ensuring of the major topics covered in the IAD review, each accompanied by intended use of IDA IEG observations on key aspects of the assessment. However, IEG’s resources, though they overall evaluation is contained in chapter 4. are important 3.2 Objectives of the Review: IAD describes the objectives of its Part IB review, as laid out in its Terms of Reference, 1 as follows: “IAD’s objective was to review the basis of management's assess- ment and express an opinion on whether Management's assessment of the effectiveness of internal controls over IDA operations is fairly stated based on the criteria established in Internal Control— Integrated Framework issued by COSO.” 2 IAD emphasizes that Part IB relates to assessing and reviewing “the operating effectiveness of key controls currently in place to ensure compliance with the relevant Articles provisions and policies governing IDA’s operations.” 3 Approach and Method 3.3 IAD provided a description of the approach and method it would adopt in conducting its review of Part I (Part IA and Part IB) in its 2006 Terms of Reference which were issued prior to the start of its Part IA review. (See also the summary contained in Box 2, above). In its Part IB report IAD defined the scope of its review and described the IDA processes that management had excluded from its assessment. This was followed by a description of the various re- IAD reviewed 31 view activities that constituted its overall method and approach. of 32 modules 3.4 Scope Limitations: IAD explained that it conducted a review that of 31 of the 32 BPMs that Management had presented for assess- Management presented for ment, which included the 2 newly added submodules of the IDA assessment Allocation Process Module. 4 The excluded module was the Safe- guards Corporate Risk module which management had been unable to test. CHAPTER 3 THE IAD REVIEW 3.5 IAD explained that its review did not include several areas that management had deemed outside the scope of the review, in- cluding: y The overarching control framework (governance and effec- tiveness and efficiency); y Specific processes deemed by management to be outside the scope of this assessment: Economic and sector work (ESW), Re- port on Observance of Standards and Codes (ROSC), Inde- pendent Evaluation Group (IEG) processes, Internal Auditing Department (IAD) processes, Annual Report on Portfolio Per- formance (ARPP), the Inspection Panel, the Department of Insti- tutional Integrity (INT), and the Results Assessment Frame- work of IDA; y Certain OPs and BPs excluded by Management, as outlined in its Part IA report; and, y Compliance controls embedded in automated applications used in IDA operations. Box 3. Listing of Review Activities Performed by IAD in Part IB Management Test Plans: IAD reviewed man- and management’s methodology for determin- agement’s test plans for assessing the operating ing compliance rates and identifying exceptions effectiveness of the key controls identified for and deficiencies. the in-scope processes. Transmittals for Communicating IAD’s Re- Management Sampling Methodology: IAD re- view Results: IAD issued 32 transmittals to viewed the sampling methodology note pro- communicate review results and to solicit man- vided by management, provided specific com- agement responses. Issues covered adequacy of ments, and followed up with management on testing and sampling as well as process-specific all comments. issues. Workshops/Review Sessions: IAD attended Management Drafts: IAD reviewed drafts of process walk-through sessions convened by management’s report and provided specific Management for the three processes added in comments. Part IB, to validate process flow charts and nar- Significance of Deficiencies Identified in Part rative descriptions of individual key controls IA: IAD reviewed management’s evaluation of provided by management, challenging, seeking the significance of deficiencies identified by clarification, and identifying potential deficien- management, IAD, and IEG during Part IA. cies as appropriate. Representation of Compliance Rates: IAD re- Revised Process Documentation: IAD re- viewed the consistency of test results with the viewed revised process documentation incor- candor of the representation of those results by porating changes identified in the three addi- Management in their draft report. tional workshops/review sessions, and based on testing in Part IB. Consideration of Related IAD Audit Work: IAD reviewed relevant results of independent Management Test Results and Supporting internal audit projects to assess consistency Documentation: IAD reviewed and validated with and reasonableness of management’s test the test results, including the documentary evi- results and conclusions. dence supporting process-level conclusions, 22 CHAPTER 3 THE IAD REVIEW 3.6 Approach: The IAD report states that, as agreed with man- agement, IAD has applied the Audit Standards 2 (AS2) concepts to its review of Part IB. The report also describes the 10 sets of activi- ties it performed in completing its Part IB review. These are summa- rized in Box 3. 3.7 IEG Observations: IEG observes that IAD’s approach and IAD’s approach method for this review have been rigorous, very detailed, and com- and method prehensive. The IAD Transmittals instrument provides for an open were rigorous, and clear statement of issues as these were identified, and this was detailed, and followed, module by module, by extensive discussions between comprehensive IAD and management as a means to resolve outstanding issues. IEG believes that these methods, in their focus and detail, were particu- larly well suited to a review of the specifics of management’s trans- actions level assessment approach. Major Findings 3.8 IAD summarizes its findings by first making a general obser- vation; it then continues by listing and reviewing a series of specific observations that arose during Part IB or were outstanding from Part IA; and it ends by making an overall conclusion. 3.9 General Observation: The report mentions that not only was this assessment of IDA controls a first of its kind within the Bank, but is unique among the multilateral international financial institu- tions (IFIs). Commenting on the overall review, the report states: While the effort underlying the commitment was clearly underesti- mated at the outset, substantial commensurate benefits are antici- pated: its results will provide a compelling baseline to identify oppor- tunities for streamlining IDA’s (and concurrently IBRD’s) operations and internal controls while significantly improving con- sistency and efficiency.(page 4) 3.10 IEG Observation: IEG agrees with this vision of the overall The review of review of IDA controls, which has indeed been more taxing than IDA controls has initially envisaged, but which should bring significant benefits to been more the organization in improved knowledge, more focused controls, taxing than and an improved basis for streamlining. envisioned but will bring 3.11 Summary of Specific Observations: The report highlights 7 significant specific observations that IAD regards as the significant findings benefits that have emerged from its review, including some issues out- standing from Part 1A. These are summarized with IEG’s observa- tions in paras 3.12-3.27. 23 CHAPTER 3 THE IAD REVIEW 3.12 Observation 1: Key Fiduciary Control Compliance Rate. IAD states that of the 115 key controls, 29 are key controls related to the fiduciary processes (procurement, financial management, loan management) “as being most critical to ensuring that IDA funds are used for the purposes intended.” (page 5). IAD points out that 21 per- cent of these key fiduciary controls do not operate effectively, ac- cording to management’s results, and it further states that manage- ment has concluded that significant deficiencies exist in the procurement and financial management processes. IEG does not 3.13 IEG Observations: IEG does not share the view that the agree that three fiduciary processes, while obviously important, are the “most ensuring critical” in ensuring the intended use of IDA resources. IDA alloca- intended use of tion, the CAS process, and the two lending vehicles (SIL/DPL) are IDA resources the main drivers of IDA operations. The fiduciary processes— depends critically together with legal and safeguard processes (not mentioned by on the four IAD)—are supporting processes. IEG did not calculate separate processes compliance rates for these fiduciary process key controls, but, as identified as part of its overall key controls and other compliance analysis, it “most critical” by found (see Table B-7 in Annex B) that the fiduciary processes were IAD among those with the most concentrations of control failures, and they represent the most important qualification to management’s overall aggregate pass rate. Management also observed this fact, so all parties are in agreement that this is an area where attention is needed. 3.14 Observation 2: Representation of Compliance Rates. IAD re- viewed the way in which management had represented the compli- ance rates found in its Part IB testing. Its report notes that manage- ment’s testing was not designed to assess compliance of a single project with all applicable controls and points out that the interde- pendence of key controls is “particularly critical in IDA processes during the supervision phase.” The overall pass 3.15 IEG Observation: IEG agrees with this judgment. IEG un- rate used by dertook an extensive examination of the various pass rate concepts management is presented by management, IAD and IEG itself (see Annex B paras. a fair 10 and 11). While different pass rates may be used to suggest differ- representation ent outcomes, and while the treatment of controls that were inop- of IDA’s overall erative during these may be an issue for the future (see annex B compliance with para 6) IEG is of the view that IAD has correctly confirmed that the its policies and overall pass rates presented by management give a fair representa- procedures tion of the general state of compliance of IDA’s internal controls. 3.16 Observation 3: Operating Effectiveness of the Key Controls within Fiduciary Processes. IAD states that although “management has verified that documentary evidence exists to support the opera- tion of key fiduciary controls, the quality of the underlying special- 24 CHAPTER 3 THE IAD REVIEW ist input…has not been examined.“ (page 5). IAD also cites ISR evi- dence from other IAD audits that supports the finding that fiduci- ary controls “do not consistently operate effectively.” 3.17 IEG Observation: IEG believes these aspects of operating compliance would be appropriate for review in Part II of the re- view, along with the effectiveness and efficiency aspects of the con- trols. But it is important also to acknowledge that Management has addressed certain qualitative aspects of control effectiveness in Part I. Management reported a significant deficiency relating to vari- ances in regional implementation of institutionally endorsed finan- cial management and procurement guidelines and has called for the appropriate sector boards to review the relevant documentation. It also reported a deficiency in providing clearances from reviewers based on the discovery that in number of cases where clearance of project documentation had been given, conditional on certain changes being made but there was often no follow-up by reviewers to ensure the changes were in fact made, and there was sometimes confusion between TTLs and reviewers as to whose responsibility this was. Management is reviewing the need for further guidance on this issue. IEG would also point out that when examining the quality of controls, it should be recognized that the Bank already has well-established quality monitoring units and processes (in- cluding those of QAG, IAD, IEG) that examine both projects and programs, and it will be the proper subject of Part II to examine the efficiency and effectiveness aspects of the present review, also using the work of these units. 3.18 Observation 4: Statement of Expenditure (SOE) Reviews and Audit Arrangements: IAD states that it regards SOE reviews as criti- cal to ensuring eligibility of expenditures, but management did not include such reviews in its testing, because these reviews are not mandatory where auditing arrangements are considered adequate. Noting that management will be covering audit practices during Part II, IAD asks if SOEs will also be covered. 3.19 IEG Observations: SOEs are an important disbursement tool which should be subjected to periodic controls but, as Management explains, it is not mandatory that these be checked, if audit proce- dures are acceptable. Therefore, it is not clear why IAD did not sug- gest that the Part IB assessment would have been better conducted had it tested audit processes (which is the primary control function), rather than argue that an SOE review should have been conducted. However, IEG also notes that Management intends to examine audit practices during Part II. 25 CHAPTER 3 THE IAD REVIEW 3.20 Observation 5: Debt Reporting Process: IAD points to some inconsistencies between Management’s testing results and state- ments regarding full compliance with OP/BP 14.10 on External Debt Reporting. Management acknowledges that compliance may not be complete, but asserts that debt reporting control objectives were being achieved. IAD also points to other issues not covered by management’s testing— the debt sustainability analysis needed in some countries and the free rider risk to IDA that arises in some cases — that were not tested by management. 3.21 IEG Observations: IEG has also raised this matter in its own evaluation, but regards the lack of any debt sustainability analysis in the process module to be the most important missing element. 3.22 Observation 6: Status of Issues from IAD’s Part IB Transmit- tals. IAD commented on management’s testing methods by sending comments on the test plans (44 specific comments). Of these, 40 have been actioned or resolved with management, and the remain- ing 4 have been deferred to Part II. IAD also sent comments on management’s test results in the form of Transmittals, of which it sent 32 in all, with a total of 58 issues. Of these, 45 have been re- solved, 12 were deferred to Part II and one remains open. 3.23 IEG Observations: IEG observes that the number of issues identified, and the fact that almost all issues have been either re- solved or deferred to Part II, is a reflection of the thoroughness of IAD’s approach, and the substantial interaction with management that this involved. 3.24 Observation 7: Status of Recommendations and Potential De- ficiencies from Part IA. IAD points out that of the eight recommen- dations it made to management in Part IA, six are still outstanding and open. Since IEG had also taken a position on all of these issues, it makes sense to summarize these in a common table, which is shown in Box 4. IAD also states that it found 59 potential documen- tation and design deficiencies in Part IA, of which 18 remain open , many because OP/BPs require updating. 3.25 IEG Observations: IEG notes that the six “open” recom- mendations all relate to matters that are to be addressed or are in process of being remedied. They do not signify disagreement at this stage between the reviewing parties, and these issues will all be ad- dressed during Part II. Regarding the 18 open issues awaiting re- form of the OP/BPs, IEG takes this as one piece of evidence (among others) that the OP/BP issue is a potential material weakness. 26 CHAPTER 3 THE IAD REVIEW IAD’s Conclusions 3.26 In making its concluding remarks at the end of its review re- port, IAD states: In our opinion, subject to the outcome of the assessment of entity- level controls and other relevant outstanding assessments to be com- pleted in Part II, Management’s view that the design and operational effectiveness of identified processes and associated key controls are adequate to provide reasonable assurance of compliance with IDA’s policies and procedures to ensure the use of funds for the purposes intended is fairly stated, taking into account the exceptions noted by Management. Box 4. Scorecard and Status of IAD Issues and Recommendations From Part IA STATUS IAD ISSUE IEG POSITION Open Closed 1. IDA Processes IEG agrees there are scope limitations in Selected management’s approach, but has made X Management was given its evaluation contingent on these limita- no mandate to introduce tions being addressed before the review is scope limitations completed. 2. IT Controls IEG accepts management’s position that These should have been these will be assessed in Part II, with a X assessed at the transac- return to assessing transactions proc- tions level esses where these may be specifically re- quired. 3. Fraud and Corruption General remedies for significant deficien- Controls cies would cover fraud and corruption These should have been but the latter do not require additional X tested explicitly in Part I remedies. 4.Outdated OP/BPs IEG regards the status of OP/BPs as a These are key elements potential Material Weakness X in compliance 5. Categorization of De- Management has satisfactorily resolved ficiencies these issues, with appropriate inter- X Management to decide actions with IAD and IEG. materiality 6. Documentation Re- All parties agree this is a significant defi- tention and Accessibil- ciency but IEG questions IAD view that X ity this is a “design deficiency.” For IEG failure to retain or access documents ap- pears to be largely an operating failure, not a design failure. However, there may be cases where OP/BPs need to state that retention should be maintained. 7. Assessment of Entity Broad scope of Part II is to be reviewed Level Controls with all three parties. X 8. Walk-through of After extensive debate, this issue has now X Process Documentation been closed. 27 CHAPTER 3 THE IAD REVIEW 3.27 IEG’s Summary Observations: IEG takes note of the de- IAD’s review was detailed, tailed and systematic approach to the review outlined by IAD, and systematic, rigorous, and notes also the extensive review activities that IAD performed in appropriate completing its work on Part IB. In IEG’s opinion, the method and approach taken by IAD in its review was appropriate, suitably rig- orous, and was generally well explained. IEG notes the range of is- sues that have emerged as the findings of this process, and a num- ber of these are shared by IEG as findings it has come to in its own evaluation. Most importantly: IEG takes note that IAD has stated its opinion that, subject to the number of important issues that remain to be addressed in Part II, the conclusion arrived at by management that IDA’s internal controls were found to be operating in adequate compliance with its policies and procedures, was fairly stated, tak- ing into account the exceptions noted by management. NOTES 1. Terms of Reference for a Review of Management’s Assessment of Internal Con- trols over IDA Operations, IAD, May 16, 2006, issued as a memorandum to the Vice Presidents of CTR and OPCS, which IAD also sent to AC and CODE. 2. IAD Report page 1. 3. IAD Report page 1. 4. IEG excluded the two submodules from its count since it viewed them as part of the parent module, so it has stated that 29 modules were tested, whereas IAD has kept these within the testing sample, therefore speaking of 31 modules. 28 Evaluation Essentials ™ IEG rates Management’s assessment exercises as satisfactory with some 4. The IEG Evaluation qualifications and judges its approach and method for testing compliance to have been transparent, 4.1 Introduction: The work on Part I has now been completed concrete, comprehensive, conclusive, and durable by both management and IAD. Their two reports are attached. This allows IEG to make its overall evaluation of the completion of Part I, ™ IDA controls have been which dealt with compliance with IDA’s articles and internal policies shown to operate and procedures (COSO objective 2), and which was tackled by man- generally as designed, agement in a bottom-up, process-level approach. The evaluation but disaggregated analysis is also will incorporate the results of the controls compliance testing that necessary was completed by management as Part IB. Surveying all the mate- rial that has been generated during Part I, by both Management and ™ Most control failures IAD, IEG has organized its evaluation around the following five appear to have been the main topics: result of flaws in project processing rather than in y Management’s Approach and Method control design y Internal Controls at the Transactions Level ™ Noncompliance rates y Material Weaknesses and Significant Deficiencies appear to have been y Other Process Issues Uncovered in Parts IA and IB somewhat higher in y Overall Evaluation modules judged to carry the most risk for IDA ™ Management has found Management’s Approach and Method three significant deficiencies; IAD agrees 4.2 Scope Limitations: IEG’s Part IA report commented that with Management’s Management’s decision to open the overall review with a transac- position on these; and tions-level assessment was contrary to the normal approach to such IEG considers one of reviews, and, coupled with the phasing of the review, led inevitably these to be a potential to a number of scope limitations. As a result, until the entity-level material weakness and controls have been assessed and reviewed (in Part II), it will not be the other two are possible to make definitive conclusions regarding the overall qual- significant deficiencies ity of the internal controls framework. It is also possible that some conclusions made during Part I regarding the quality of the transac- tions-level controls may need to be revisited once Part II has been completed. That said, there has clearly been significant progress made in the completion of Part I, as described below. 4.3 Representing IDA Operations—the Cluster of BPMs: Both IEG and IAD had raised issues relating to certain business processes that were excluded from the cluster of 30 BPMs used to represent IDA operations in Part IA (AAA products; non-SIL Investment Loan products). Management has responded to these observations: it tested non-SIL operations in Part IB, it is preparing a BPM for CHAPTER 4 THE IEG EVALUATION The Debt AAA for testing in Part II, and it has extended the cluster of BPMs Reporting to 32, by adding modules to capture the CPIA and PCPI processes. module needs to Management also completed the BPM on Debt Reporting. However, be extended to in IEG’s opinion that module as now described needs to be ex- include some tended to include some controls on the assessment of debt sustain- controls for the ability (for example, encompassing country growth and export assessment of prognosis, and government debt management policies). With this debt qualification, and with the note that it remains important that AAA sustainability products are assessed during Part II, IEG is satisfied that the cluster of BPMs as now constituted gives a fully adequate representation of the processes underlying IDA operations. IEG rated 4.4 Mapping and Design Effectiveness of Key Controls: IEG has Management’s already acknowledged the major contribution to the Bank’s knowl- mapping and edge of its controls system from the mapping of the Business Process assessment of Modules, now 32 in number. IEG used the rating template that it cre- design ated for this purpose to rate management’s mapping methods. (Rat- effectiveness as ing scale: 1-4 with 1 = highly satisfactory). 1 Based on this scale, IEG satisfactory with rated management’s mapping and assessment of design effectiveness some at 2.5 (that is, satisfactory with some qualifications). qualifications 4.5 Testing Methods for Compliance: As to the methods for testing the controls, IEG is also satisfied that management’s approach is credible, transparent, and concrete, as described in chapter 2. To understand management’s testing methods, Box 5 presents a depic- tion of the different elements involved. (More detail is presented in annex B). The main reason 4.6 IEG Ratings of Management’s Testing Methods: As it did in for slightly Part IA, for management’s approach to process mapping and controls lower ratings for design, IEG systematically rated management’s approach and presentation of method in the testing of controls. On the basis of the questions con- results was that tained in the IEG Business Process Template, the testing method for descriptions of each module was rated, and aggregated into an overall average for all findings were 29 modules. This average rating was 2.4 overall. not always written clearly 4.7 The overall rating, as summarized in Table 1, was comprised and of ratings for individual components, as follows: for testing methods unambiguously per se, a rating of 2.1—very close to being fully satisfactory; for sam- pling methods and quality of conclusions a ratings of 2.4; and for presen- tation of results a rating of 2.7—close to being satisfactory with qualifica- tion. The rating for sampling was mainly the product of small sample sizes in a number of cases, which were often further diluted by the presence of nonoperative controls in the specific random projects chosen in the sample. In the case of presentation of results, the main reason for slightly lower ratings was that descriptions of the findings were not always written clearly and unambiguously. However, taken overall, IEG regards the testing methods to be credible and robust for 30 CHAPTER 4 THE IEG EVALUATION the purposes of the review. The overall rating of 2.4 implies that management’s general approach to the testing process was rated sat- isfactory but with some qualifications. Annex D presents more details of the rating process. Table 1. Summary of IEG Ratings of Management’s Approach and Part IB Testing Methods Average Overall Rating = 2.4 Function Rated Rating Function Rated Rating Choosing the Sample 2.4 Testing Results 2.7 Testing Methods 2.1 Quality of Conclusions 2.4 Rating System: 1= Highly Satisfactory; 2 = Satisfactory; 3 = Satisfactory with Qualification; 4 = Less than Satisfactory; N/A = Not Applicable 4.8 Overall Evaluation of Management’s Method and Approach: Management’s IEG finds management’s approach and method for the compliance approach and testing to have been generally transparent, in the sense of being easy method for to follow; it was concrete, in the sense of being based on very specific testing and documented control steps; it was broadly comprehensive, in the compliance was sense of not leaving out any key controls within the universe of IDA transparent, operations contained in the 32 BPMs; 2 it was generally conclusive, in concrete, the sense of leaving little doubt whether controls were working or comprehensive, not working as designed; it was durable, in the sense of providing an conclusive, and information platform which can be used again in the future, includ- durable ing as a basis for comparison; and it was helpful from the perspective of the COSO framework, in the sense that it provides at least the be- ginning of a basis to draw links between controls deficiencies ob- served at the process level and what may be broader causes at the en- tity level. The building blocks of Management’s approach are depicted in Box 5. Internal Controls at the Transactions Level 4.9 How Effective are IDA’s Controls? Matters relating to man- agement’s approach and method have their place in this evaluation. However, in the end, the central question to be answered is: what are the status and effectiveness of the internal controls that govern IDA op- erations, as revealed by management’s test results? Based on aggregate “pass 4.10 IEG observes, as stated in chapter 2 that, based on the aggre- rates” IDA’s gate “pass rates” presented by management the evidence suggests controls have that IDA’s controls have been shown to operate generally as de- been shown to signed. However, there are several possible aggregate pass rates operate and their results differ. Also, any aggregate measure must be com- generally as plemented by other, disaggregated evidence that shows where con- designed, but disaggregated analysis is also needed 31 CHAPTER 4 THE IEG EVALUATION trols have clearly not operated as intended. IEG has conducted a systematic analysis of management’s results (shown in annex B), Box 5. Depiction of Management’s Testing Method The model test in the table shows (hypothetically) the BPM for a procurement process, which has three key controls, with between two and four control steps or “attributes”(A,B,C, etc). The proc- ess is being tested with four sample projects. In all, therefore, the process involves 32 individually tested control steps or “cells.” In some cases (e.g. Control 2C), the control was not applicable in two of the four cases, so these are marked “N/A”. BUSINESS PROCESS MODULE XX: PROCUREMENT SAMPLE PROJECTS CONTROL 1 CONTROL 2 CONTROL 3 A B A B C D A B Project W xxxxxxx xxxxxxx xxxxxxx Project X xxxxxxx N/A Project Y xxxxxxx N/A Project Z Legend: xxxxxxx Noncompliance. Management tested each step by asking the relevant operational units to present documents to show that the actions were taken by the staff at each control step. Where the document is pre- sented, the control step is deemed fulfilled; where not, or where the document may have been in- accurate, or shows the action was not timely, or was not in conformity with the design of the con- trol, management labeled this a noncompliance. Management’s Pass Rate (Aggregate Count): For the purposes of calculation the “N/As” have been excluded from both the numerator and denominator. In this model case the results show five noncompliances. Out of a total of 32 control steps (management’s method of calculating the pass rate), this would signify a pass rate of 83 percent. When, in the actual tests, the pass rates were aver- aged across all 29 modules that were tested, Management calculated its overall pass rate as 93 per- cent Alternative Pass Rate: An alternative concept of pass rate (IAD suggested this be considered) is to establish how many sample projects passed through the tested controls with a full pass on all control steps. In this model, only one project (Project Z) would have achieved this, a pass rate of only 25 per- cent. Concentrations: IEG has posited the notion that certain “concentrations” of control noncompli- ance have been found. IEG suggests that this can be said to have occurred when there is a se- quence of three or more noncompliances, in either of the following two ways: • Controls-Related Concentrations (“Vertical Cut”): When three or more noncompliances are found in any one control step (that is, viewed vertically, as in Control 2B in the model above), which shows that several projects failed to comply with that control, suggesting there is a problem with the control design; and/or • Projects-Related Concentrations (“Horizontal Cut”): When three or more noncompliances are found in any one of the sample projects being tested (viewed horizontally, as in Project W in the model, where Controls 2A, B, and D were found to be noncompliant), suggesting that, in the processing of that project, there was laxity in complying with some controls. In using concentrations some bias against large, complicated processes is introduced, since, for example, a process with many control steps is more likely to have a concentration of three or more noncompliances than a process with few control steps. 32 CHAPTER 4 THE IEG EVALUATION and in the following sections presents its evaluation of the various elements that entered into its overall judgment: the different aggregate pass rates; a more disaggregated view of the results data, showing where controls failures were concentrated; the principal reasons for controls failure; statements regarding material weaknesses and significant deficiencies; and other controls issues that have arisen as a result of management’s assessment, IAD’s review, and IEG’s evaluation. READING MANAGEMENT’S TEST RESULTS 4.11 Aggregate “Pass Rates”: In addition to the aggregate pass rate IEG believes the of 93 percent presented by Management, which was based on the most number of control steps that passed the tests without noncompliance, appropriate four other concepts have been applied. Management also used a pass rate concept based on key controls (pass rate 91 percent), IEG suggested us- indicators are ing business process modules at risk as a measure (pass rate 76 per- those based on cent), and —in the working discussions—both IEG and IAD sug- control steps gested alternative 100 percent compliance concepts, one based on and key controls control steps (pass rate 71 percent) and one on sampled projects (pass rate 64 percent). Since this is a review of IDA’s internal con- trols, IEG believes that the most appropriate pass rate indicators are those based on control steps and key controls. A full description of these different concepts and their respective pass rates is given in Annex B, and summarized in Table B.1. 4.12 Summary: What do the “Pass Rates” Convey? Of the differ- IEG could find ent pass rates presented in annex B, IEG favors the aggregate pass no criteria in the rates based on control steps (93 percent) and on key controls (91 per- literature by cent) because these most closely conform to the nature of the con- which to judge trols assessment being undertaken. Since no other IFIs have com- the acceptability pleted controls reviews of this kind, there are no benchmarks for of the pass rates comparison, though within the Bank itself these rates are within the found by range found in quality assurance measurements of Bank products, Management for example by QAG. IEG therefore expresses the judgment that, in its opinion, these pass rates demonstrate with a reasonable level of confidence that management is justified in stating that, at the trans- actions level, and subject to the qualifications already mentioned, controls have operated to an adequate level of compliance. 4.13 Considering the complexity of IDA, the fact that this was a first of its kind review, and considering also the rigorous nature of the controls testing, IEG finds it is not surprising that some failure rate was observed. Also, management has indicated that about half of the noncompliances were less material “exceptions”, not “deficiencies”. IEG is of the view that – subject to the other qualifications mentioned – this is an acceptable outcome. 33 CHAPTER 4 THE IEG EVALUATION 4.14 Disaggregating the Test Results: If the aggregate pass rates suggest that controls have been found to operate in a generally ade- quate fashion, it is also important to view the results below the ag- gregate, and to focus on those cases where controls clearly did not operate as designed. In this regard, it is important that control failures appear to have occurred as much in scattered fashion, reflecting no particular pattern, being isolated cases of failure to observe control requirements during the processing of projects, as in concentrations. As shown on Figure 1 be- low, noncompliances occurring in concentrations—where three or more failures occurred in a given control or a given project — accounted for slightly less than half of all noncompliances. (See also Box 5). The presence of concentrations is important, because they suggest areas of more intense control failure, and also whether these were design-related, or project-related failures. As Management found, IAD emphasized and IEG has also shown (see Table B-2 in Annex B), some of the concentrations of control failures occurred in the fiduciary processes, which is a concern, because the essential function of these processes is to add fiduciary discipline. Figure 1. Distribution of Control Steps with and Without Noncompliances Individual Control Steps Individual Sampled Projects “Vertical” “Horizon- tal” 29% 36% 71% 64% 100% Compliance Less than 100% 100% Compliance Less than 100% % Distribution of proje cts/controls a ccording to the ir num be r of non com plia nce s 80 71 70 64 Controls Cut or Percentage of Projects Cut 60 50 40 30 23 26 20 6 10 10 0 Clear (0) Random (1 - 2) Concentrated ( 3 or > ) Numbe r of Non-Compliance s pe r Cut Controls Cut Projects Cut * Random Occurrences: Control steps or projects with 1-2 noncompliances; Concentrations: Control steps or projects with 3 or more noncompliances 34 CHAPTER 4 THE IEG EVALUATION 4.15 From the analysis conducted in Annex B (summarized in Figure 1 and Table 2), IEG also observes that most control failures Control failures appear to have been the result of flaws in project processing, rather appear to have than control design problems. Of all the 246 noncompliances ob- been more the served, these fell about evenly between apparently random occur- result of flaws in rences (129) and concentrations (117), but fewer projects than con- project processing trols achieved 100 percent compliance. These results suggest that in than control seeking remedies for failures, the need for management oversight of design problems project processing may be greater than the need to re-design con- trols, though clearly both may be needed. Table 2. Distribution of Noncompliances by Random Occurrence and Concentrations Total N-Cs 1 2 3 4 5 >5 246 Controls 85 44 45 16 20 36 Projects 67 46 57 32 25 18 % Random 129 Concentrations 117 Controls 52 48 Projects 46 54 Source: IEG Calculations from Management test results data. 4.16 Are IDA controls stronger where risks are greater? IEG made Noncompliance an attempt to answer this question, based on a relatively crude rates appear to methodology, which it developed during Part IA. (See Annex B, have been para. 18). The results tend to suggest that noncompliance rates (av- somewhat eraged across modules and samples, to remove the effects of differ- higher in ent sized modules in the sample) have been somewhat higher in that modules judged group of modules judged by IEG to carry the most risk for IDA. (See by IEG to carry Figure 2) This is consistent with the other findings already de- the most risk for scribed, which showed that the core SIL and several of the fiduciary IDA modules (all of which are in the higher-risk group), were where the main concentrations of control failures were shown to occur. 4.17 It should be stated, however, that IEG regards this analysis as more indicative than substantive at this stage, and that it represents only a first attempt at making an analysis of control failures linked to business processes differentiated by risk. Such an approach is best done during a review of entity-level controls, so IEG is suggesting that a risk-based approach to the controls assessment should be possible as a component in the completion of Part II. 35 CHAPTER 4 THE IEG EVALUATION Figure 2. Noncompliance Rates and Risk Non-Complaince rate and Risk 7 7 Non-Compliance rat 6.8 6.8 6.7 6.6 6.4 6.4 6.2 6 Group 1 Group 2 Group 3 All Groups Risk Group Group 1 = higher risk; Group 2 = medium risk; Group 3 = lower risk Source: See Table B.6, annex B. Material Weaknesses and Significant Deficiencies 4.18 In its summary of outcomes at the completion of Part IA, IEG considered that the issues of documentation retention and accessi- bility and the state of the OP/BPs were both potential material weak- The retrieval of nesses. Management took issue with this view, as expressed in its documentation was footnote to the Part IA report. 3 IEG used the word “potential” mate- substantially rial weakness, because at the stage of the review then reached, there improved compared was not sufficient evidence to make a conclusive determination. to the preliminary This was particularly so in the case of the documentation issue, testing—it no longer which was to be tested in Part IB. Now that Part I has been com- is a material pleted, IEG is ready to return to this issue. weakness 4.19 Documentation Retention and Accessibility: In light of the findings from the testing in Part IB, IEG concludes that the retrieval Based on the of documentation was substantially improved compared with the role of OPs and preliminary round of testing (attempted in Part IA, but abandoned), BPs in a number and therefore concludes that this issue no longer rises to the level of of non- a material weakness. However, as evidenced in the testing results, compliant failure to find or access documents remains a significant cause of control steps, noncompliance (see Figure 3) and management has initiated a re- IEG considers medial program (to be completed by end June 2007). In manage- their status a ment’s report, this issue has been stated to be a significant deficiency. potential IEG agrees with this assessment. material weakness 4.20 Status and Currency of the OP/BPs: Since IEG made its as- sertion in Part IA, management has compiled an inventory of OP/BPs, showing the dates of their last revision, which allows a 36 CHAPTER 4 THE IEG EVALUATION somewhat more considered judgment on the extent of their possible out-datedness. Management has also made progress in updating some OP/BPs in a program that is due to be completed by end- 2007. But IEG also observed the important part played by OP/BPs deficiencies in a number of noncompliant control steps, which were revealed in the Part IB testing process (for example, there is no re- quirement for a procurement complaints database; there are no structured verification procedures for OPCQC compliance). On bal- ance, at this stage IEG prefers to retain its view that this issue re- mains a potential material weakness. This issue will be revisited dur- ing the completion of Part II, when a final determination will be made. Figure 3. Reasons for Noncompliance Deficiencies 43% 57% Documentation Other Reasons * “Other Reasons” include: Inaccuracies in documentation; failure to document in timely manner; failure of Borrowers to deliver documents, failure of staff and borrowers to act on issues. Source: IEG calculations from Management Data 4.21 In summary, therefore, management has found three signifi- cant deficiencies. IAD agrees with Management’s position on these, and IEG has found one of them to be a potential material weakness and the other two to be significant deficiencies. Other Process Issues Uncovered in Parts IA and IB 4.22 At the completion of Part IA management, IAD, and IEG to- gether had uncovered 126 process issues of various kinds that had arisen during the review of controls’ mapping and design effective- ness. In itself, this number of issues is a reflection of the thorough- ness of the overall review. IEG discusses in Annex C how these is- sues were reviewed, assessed, and disposed of, and explains that 44 out of the 126 have been deferred to be addressed during Part II, and 37 were to be addressed under the testing in Part IB. By the 37 CHAPTER 4 THE IEG EVALUATION conclusion of Part IB, 25 other issues had been identified, for a total of 62 in all. Annex C describes how management applied explicit criteria to these issues, collectively and individually, and assessed that none were material weaknesses, three were significant deficien- cies, and six were deficiencies. As described elsewhere in this report, IAD and IEG were in general agreement with these judgments, but IEG regards the major issue of the status of OP/BPs to be a potential material weakness. However, as described in annex B, management has nevertheless given a full accounting and disposal of all the is- sues uncovered during Part I by all three reviewing parties. One exception was the fact that IAD recommended an alternative pass rate methodology, based on individual projects, about which man- agement reported that its approach was not designed to address pass rates of individual projects. Figure 4. Linkages between Key Controls and the COSO Components Information & Communication Monitoring & Learning Control Activities Risk Assessment Control Environment 0 20 40 60 80 100 120 140 COSO LINKS Source: Management results data, see Statistical Appendix Table SA.11. 4.23 Links to the COSO Components: Management had ex- plained at the start of Part I that during its assessment of compli- The major ance at the transactions level, it would be focusing mainly on two of COSO linkage of the five COSO components: Control Activities and Risk Assessment. the key controls During the compilation of its process maps and key control designs, has been to the management listed which COSO component was linked to each key Control control, but did not aggregate these linkages as part of its Part IA Activities report. component 4.24 IEG did make such an aggregation which is shown in Table SA.11 in the Statistical Appendix. The linkages are summarized in Figure 4. It is clear that the major linkage of the key controls has 38 CHAPTER 4 THE IEG EVALUATION been to the Control Activities component and (to a much lesser ex- tent) to the Risk Assessment component. This would be expected in a compliance assessment at the transactions level, where controls are linked to individual risks and the policies (OP/BPs) addressing those risks. The OP/BPs are the critical element of the Control Ac- tivities component. IEG takes note of these linkages and expects to return to this topic when all COSO components are addressed dur- ing the Part II assessment and evaluation. The IEG Advisory Panel 4.25 As is common practice for its major evaluations, IEG drew on the services of an international advisory panel of experts to re- view and comment on its work at the conclusion of Part I. The Panel consisted of three distinguished former Auditor-Generals, from Australia (Mr. Patrick Barrett), India (Mr. Vijay Shunglu), and Nor- way (Mr. Bjarne MØrk-Eidem). The Panel visited Washington in early March 2007, reviewed the key materials and contents of the Part IA report as well as this draft Part I report, and met with man- agement, the IAD review team, and the chairman of the Audit Committee. The Panel has written a brief statement, which is con- tained in attachment 3, and is excerpted below: 4 y The Panel understands the pragmatic approach taken for Part I, while recognizing that final conclusions will be made on the review after the completion of Part II. In fact it may be neces- sary to vary some of the conclusions of Part I as the result of the work on Part II. y The strength of the approach in Part II is the top level strate- gic focus reflecting decisions made about the application of the overall integrated COSO risk management framework and as- sociated entity-wide controls within the governance arrange- ments that reflect both ‘tone at the top’ and the authority and accountability which is assigned to the review and any agreed outcomes. The necessary leadership is reinforced by both top down and bottom up approaches with the resultant commit- ment at all levels of the organization. y The Panel notes some concerns that were expressed (during Part IA), particularly by IAD, about a lack of documenta- tion…. The Panel agrees that documentation is a common problem across both the private and public sectors and is gen- erally being given a higher priority both to facilitate decision- making and to enhance accountability. y The Panel supports the use of suitable standards against which accomplishment/performance can sensibly be assessed. In the Panel’s view, this is what makes Part II of the review so important in building on the extensive investment in Part I. 39 CHAPTER 4 THE IEG EVALUATION The latter has been largely about compliance or conformance, the former is now a challenge to focus more on what is to be achieved both efficiently and effectively. y The Panel observes that there are different skills required for financial statement, assurance and performance audits. The Auditor General, not surprisingly, has a strong focus on the former and the related controls and risk factors. The Panel ac- cepts that IAD may also have some capacity to undertake re- views of administrative effectiveness. While noting the issue of ‘independence’ in reporting in the relationships with the President and the Board which gave IEG the review role in this exercise, the Panel also notes that Part II may well involve issues of policy effectiveness which are more within the prov- ince and scope as well as the competence of IEG. y The Panel is impressed by the professional approach and commitment of the three major parties to the review— Management, IAD and IEG. Considerable effort and resources have been dedicated to the task. The Panel is not in a position to conclude on the cost/benefit of what has been done to date. However, it notes the paucity of available information on simi- lar exercises elsewhere in areas where many organizations are dealing with demands for better controls, greater assurance, and better outcomes—whether more cost effective programs or more shareholder value. There has been a greater focus on governance frameworks and on associated organization cul- ture that reflects high level values and an ethical approach with greater transparency and associated accountability. NOTES 1. 1 = highly satisfactory; 2 = satisfactory; 3 = satisfactory with qualification; 4= less than satisfactory. 2. The qualifications regarding the comprehensiveness of the review were the following: 3 of the 32 BPMs were not tested (although two were subprocesses of the overall allocation process that did not have key controls), some indi- vidual control steps that were not applicable to the specific projects in the sample were not tested, and AAA products are to be addressed during Part II. 3. See chapter 2, page 30, foot note 9. 4. For reasons related to the joint availability of the panelists, their visit to Washington had to be timed somewhat before all the materials were ready in final form. However, this did not detract from the materiality of their conclu- sions. 40 5. Conclusions and Recommendations 5.1 IEG concludes that, subject to a number of qualifications, the substantial body of work carried out in the completion of Part I, in its breadth, penetration, and specificity of results, can indeed pro- vide general confirmation that internal controls at the transactions level are operating generally as intended. IEG has concluded that Part I has uncovered one potential material weakness and two sig- nificant deficiencies, and all three are already being addressed by management. Poorly operating controls in some individual proc- esses have been identified and are also being addressed. For an agency of the size and complexity of IDA, the scale of these revealed weaknesses, deficiencies, and noncompliances—averaging at less than one non-compliant control step per project tested—would ap- pear to IEG to be within an acceptable level of tolerance. 5.2 Concluding Evidence: As the key elements of evidence to support this conclusion, IEG would point to the following: y Management made substantial progress in Part IA by map- ping and verifying the effectiveness of design controls at the level of business processes. y In Part IA, both IAD and IEG observed certain deficiencies in the extent to which the cluster of 30 BPMs fully captured all IDA operations. Management corrected for this by expanding its testing sample for investment lending in Part IB, and it will examine AAA products and processes during Part II. y Management completed its compliance testing in Part IB in a convincing, detailed, and concrete manner. The results are highly transparent and can be verified or questioned. By its own methodology, management has achieved a stated pass rate of 93 percent, which lies within the range of acceptability achieved in other Bank quality reviews, such as QAG assess- ments of Bank and IDA operational products. This general pass rate has also been broadly supported by calculations of alternative pass rate concepts. y IEG finds that the revealed noncompliances appear to result more from laxity in controls processing than from flawed con- trol designs. This would imply that enhanced management CHAPTER 5 CONCLUSIONS AND RECOMMENDATIONS oversight is needed more than redesign of control steps, though some of the latter is certainly also needed, and some streamlining can also likely be achieved; y Part I has uncovered one potential material weakness and two significant deficiencies. For an agency as complex as the Bank, and for a historic first detailed review of internal controls processes, this would seem a benign outcome. Part IA had also led to some 126 specific process issues, but most of these were resolved during Part IB, or appropriately deferred for assess- ment during Part II, since they involved efficiency and effec- tiveness issues. Part IB disclosed 68 specific issues that are currently being address by management, 43 carried forward from Part IA and 24 issues arising in the testing; y IEG sees the findings in Part I regarding the origins of controls failure as giving important guideposts to the work to be done in Part II, whether in terms of the need for enhanced manage- ment oversight over project processing, the possible streamlin- ing of processing controls, or the need to redesign certain con- trols. 5.3 Qualifications to Be Considered: The affirmative conclusion described above is subject to some qualifications: y Within the COSO framework, definitive conclusions regarding the effectiveness of internal controls at the transaction level cannot be made in isolation from the framework as a whole. Any assurance must be based on a verification that all ele- ments of the control framework (all five COSO components) are present. As was made clear in the Part IA report, manage- ment’s transactions level approach to Part I did not encompass all COSO elements (see figure 4, para. 23 in chapter 4). y Definitive conclusions in the form of a general assurance will therefore have to await the completion of the entity-level con- trols during Part II. It is also possible (as the Advisory Panel also pointed out) that it may be necessary to revisit some con- clusions made during Part I after completing Part II. The af- firmative conclusions drawn above simply imply that Part I has been successfully accomplished as the first step in a two- step process. y Management has itself made a qualified statement, saying it concluded that controls were adequate with some qualifica- tions, and it has not given a general assurance, saying only that evidence has been amassed that is consistent with, and could provide the basis for, such an assurance in due course. y IAD has issued an opinion that management’s qualified con- clusion at this stage is fairly stated. 42 CHAPTER 5 CONCLUSIONS AND RECOMMENDATIONS 5.4 Recommendations: IEG makes recommendations for six sets of actions to be taken, as described in the following paragraphs. 5.5 Recommendation 1—Completion of the Entity-Level As- sessment in Part II: Clearly, the overriding need now is for the ex- peditious and effective completion of the review of internal controls at the entity level. The challenge will be to address those findings from Part I that suggest linkages to entity level issues, within the COSO framework, including the following: ™ Controls and Project Processing: The finding that noncompliances have seemed to arise somewhat more through lapses in project processing, rather than through controls flaws, offers an impor- tant insight: it suggests that enhanced management oversight may be needed as the remedy for this, and thus suggests the need for a linkage to the Control Environment at the entity-level, where “tone at the top“ and governance may need to be emphasized. ™ Documentation Retention and Accessibility: This significant defi- ciency, which has arisen at least in part as a result of the migration from manual to electronic systems, suggests a link needs to be made with both the Control Environment (management oversight) and the Information and Communications component at the entity level where improved IT systems will be part of the solution. ™ Dated OP/BPs: This is an essential element of the Control Activities component. As revealed in Part I, this remains a potential mate- rial weakness, whose remedial program has already been launched. It would be well to accelerate and complete this to the extent possible in time for the completion of Part II. Having a cur- rent and well-maintained body of policy documents is a bedrock element of the entity-level control system, and also serves both compliance and efficiency and effectiveness objectives at the transactions level. ™ Risk Management and Extending the COSO Framework: One of the findings that emerged from Part I is that management did not treat risk issues in this review in a way that allowed risk differen- tiation and prioritization. This was partly the result of not having had the benefit of a prior entity-level assessment, where risk dif- ferentiation could have been made, allowing a risk-based controls assessment to be completed. This should be done during Part II. IEG already recommended in Part IA that management consider extending the COSO framework to enhance the focus on risk by adding a fourth COSO objective (Strategy—high level goals aligning with supporting mission) and three new components (Objective Set- ting, Event Identification and Risk Response—see Part IA report, page 41). ™ Efficiency and Effectiveness Evaluation: One of the characteristics of the approaches in Part I was the more or less exclusive focus on 43 CHAPTER 5 CONCLUSIONS AND RECOMMENDATIONS compliance issues, while all issues of efficiency and effectiveness were deferred until Part II. As the overall review moves from the transactions level to the entity level, and from compliance to effec- tiveness and efficiency, the challenge in Part II will be to build on the results from Part I, linking these to IDA’s Monitoring and Learn- ing activities (including QAG and IEG), in order to provide the element of effectiveness and efficiency testing that was lacking in Part I, and which is needed before final conclusions can be drawn regarding the overall effectiveness of IDA’s internal controls. 5.6 Recommendation 2—Issues Deferred to Part II: A number of issues have been deferred to Part II, in addition to those concerning IT controls, field offices, fraud, and corruption, which management stated at the outset would be dealt with in Part II. The notable addi- tions, which need to be given due attention during Part II, are: ™ AAA and Other Knowledge Products: In Part IA, IEG and IAD had regarded these as significant omissions from the cluster of BPMs representing IDA operations, and management agreed that these would be addressed during Part II. ™ Debt Sustainability: IEG believes that the Debt Reporting Module (Module 30) needs to be further elaborated, beyond debt data col- lection, to encompass growth, export and other dimensions, to underpin the sustainability analysis of the module. ™ Other Unresolved Process Issues: These are the 44 issues still unre- solved from Part IA, which were part of the 126 additional process issues uncovered by management, IAD and IEG in Part IA, but thought to relate to efficiency and effectiveness issues, and were therefore deferred to Part II and the other issues from Part IB that relate to efficiency and effectiveness and thus were deferred to Part II. 5.7 Recommendation 3—Remedies for Problem Modules: The test results emerging from Part I show that many noncompliant control step tests were randomly and quite widely spread among all modules. However, they also showed that there were some concen- trations around seven modules in particular (see table B.7 in annex B): Core SILs 1; FM SIL; Procurement SIL; Procurement Complaints; Loan Management SILs; Loan Suspensions; and Loan Closings. Noncompliance in these modules also occurred for a combination of project processing and controls-related reasons. These concentra- tions of control failures—where there may be issues of control design, not just processing laxity—therefore become clear candidates for management review and, where relevant, remedies. 5.8 Recommendation 4—Safeguards and Corporate Risk: Man- agement is addressing this issue and will attempt to seek evidence to test Module 29. However, given the sensitive nature of this topic, 44 CHAPTER 5 CONCLUSIONS AND RECOMMENDATIONS and the potential risks to the reputation of IDA at stake in managing the Corporate Risk List, IEG recommends that management con- sider whether this monitoring device should not be supported by the introduction of specific controls, standardized across all regions, to more fully integrate this process into the internal control system. 5.9 Recommendation 5—Testing Control Steps which were In- operative in Part IB: A significant number of control steps in the sample of projects in Part IB were not actually tested, because these were often conditional control steps, not applicable in the actual sample of projects tested. IAD has raised this issue as a detraction from management’s overall pass rate. Most of these control steps may rarely be needed and may therefore be of less importance, but IEG does recommend that management should make a selected, targeted sample to test some of these control steps, in cases where it is deemed important to have the evidence from testing. 5.10 Recommendation 6—Streamlining: IEG believes there is in- stitutional merit in systematizing the process maps and related test- ing information, giving them widespread visibility within the or- ganization, and possibly incorporating them as part of the revised and updated corpus of OP/BPs. These could have clear applications in the Bank’s ongoing effort to streamline project processes. 5.11 This concludes IEG’s evaluation of Part I. While much has been accomplished in this phase of the review, and a major invest- ment of time and resources has been made, it is necessary to repeat that, under the principles of the COSO framework, definitive con- clusions on the health of the internal controls system governing IDA’s operations will have to wait until the framework issues have been assessed, and matched to the transactions issues so far ana- lyzed. Only then will the real return on this major investment be forthcoming. NOTES 1While the Core SIL had a relatively high rate of noncompliances, the bulk of these were less serious noncompliances (exceptions rather than deficiencies). This could be an indication of the need for streamlining these processes more than tightening of controls, consistent with the Part IA finding that many staff complained of SIL processing requirements as unduly complex and on- erous. 45 Annex A Recapitulation of Main Findings and Summary of Conclusions from Part IA 1. This annex provides a summary of the key findings and conclusions that emerged from completion of Part IA. The four principal topic areas covered during that Part were: issues re- lating to approach and method; actual findings from the assessment of business process mapping and design effectiveness; major conclusions; and issues still pending at the completion of Part IA. 2. Issues Relating to Approach and Method: Management decided to start its assessment by tracking internal controls at the level of individual business processes (a bottom-up ap- proach), rather than by first examining entity-level controls (a top-down approach). With this phasing of the review the entity-level issues would be examined in Part II. IEG was of the view (shared by IAD) that it would have been better to start with a top-down approach, but agreed that management’s approach did permit progress in several areas. However, IEG also raised some caveats concerning scope limitations and some omissions from the business processes as- sessed, as discussed in the following paragraphs. 3. Evident Progress: Management had made a credible linkage between IDA’s Articles, its internal policies and procedures, and the individual business processes as a basis to test inter- nal controls. The mapping of the business processes has made a major contribution to under- standing the Bank’s system of internal controls; the mapping was generally of a high quality, with some notable qualifications; and the walk-through processes of verifying the design of controls was rigorous, comprehensive, transparent, and documented, with a few qualifica- tions, to a high standard. The ensuing business process maps, now part of the Bank’s internal controls architecture, can be used again in similar reviews and may also prove to be useful in- struments in designing more streamlined and efficient control measures going forward. 4. Scope Limitations: Both the bottom-up approach and the phasing of management’s approach led to certain inescapable scope limitations of the assessment made in Part IA. These included: y The transactions-level BPM-based assessment dealt mainly with only two of the five COSO components, which rendered it essentially incomplete as an assessment of the controls framework. 1 y The separation of compliance from efficiency and effectiveness issues was difficult to sustain in some cases. y The partial treatment under COSO necessarily implied that definitive conclusions about the overall state of internal controls would have to be postponed until completion of the entity-level review in Part II. 47 ANNEX A RECAPITUATION OF MAIN FINDINGS AND SUMMARY OF CONCLUSIONS FROM PART IA y Postponing to Part II other systemic issues—the assessment of fraud and corruption con- trols and IT controls and the examination of decentralized field offices— also meant that definitive conclusions on the overall controls framework would have to await comple- tion of the second part of the review. 5. Omissions from the Business Processes: Two principal omissions were noted from the 30 business process modules (BPMs) that management had chosen to represent IDA opera- tions. One was the exclusion of any analytical and advisory activities (AAA) products, which are important since “IDA operations” should be taken to mean both lending and nonlending operations. The second omission stemmed from an assumption made that the internal controls that governed core Specific Investment Loan (SIL) operations could be taken to represent all investment lending (IL) operations (including Sector-wide Adjustment Programs [SWAPS], Emergency Recovery Loans [ERLs] and others). This assumption needed to be tested, because core SILs account for no more than 60+percent of all investment lending, and if other IL opera- tions were not fully the same, this could comprise a significant gap in the extent to which the BPM cluster fully represented IDA lending operations. 6. The Main Findings from Part IA: The main finding from Part IA was that, using its bot- tom-up approach, management had made satisfactory progress in defining, locating and assess- ing key internal controls at the level of some 30 individual business processes. As a result of the combined processes of verifying the integrity of the mapping processes and assessing the design effectiveness of the key controls, module by module, management had also revealed a number of deficiencies and potential weaknesses; the following were seen to be salient: y Potential Material Weaknesses: IEG found that in two areas the preliminary results from Part IA suggested that potential material weaknesses had been uncovered. 2 One was in the state of the Bank’s OPs/BPs, which in a number of cases had not kept pace with and had not been updated to reflect the structural and other changes in the Bank Group. A second related to difficulties with retention of and accessibility to documenta- tion needed to verify the effective operation of key controls (a finding that emerged from the preliminary testing of controls conducted during Part IA, and whose problems led to restructuring of this segment of the review into Parts IA and IB). y Other Deficiencies and Issues: In addition to the two major findings mentioned above, management also identified three other apparent deficiencies: the policy framework for processing SILs was seen as too complex and cumbersome; project processing require- ments were seen as onerous and inefficient; and there appeared to be a disparity in the Corporate Review frequency between SILs and DPLs. In addition, IEG and IAD identi- fied other issues that needed clarification of their nature and significance. (These con- sisted of a variety of controls issues. Some required actions that did not seem to be man- dated, some pertained to a lack of clarity in control design, and others involved weaknesses of some descriptions in the documentation. These were described in more detail in annex C of the Part IA Report). 7. Major Conclusions: IEG arrived at a mixed conclusion regarding the outcome of Part IA. It saw that important contributions had been made to the Bank’s knowledge of its internal controls, and a major addition had been made to the Bank’s documented business processes. Satisfactory progress had also been made in identifying and defining the key controls that governed IDA business processes at the transactions level and, as described above, manage- 48 ANNEX A RECAPITUATION OF MAIN FINDINGS AND SUMMARY OF CONCLUSIONS FROM PART IA ment’s assessment did lead to the uncovering of a number of revealed deficiencies. At the same time, as already described, there were some notable weaknesses implicit in Manage- ment’s approach, stemming from the scope limitations mentioned above, and the phasing of the overall review. 8. Issues Pending at the Completion of Part IA: On completion of Part IA, some issues had been postponed for treatment during Part II, and others remained to be resolved during Part IB: y Scope Limitations: The following issues were postponed until Part II: The full treatment of all COSO components; efficiency and effectiveness issues; addressing fraud and cor- ruption; IT and Bank decentralization; and addressing the exclusion of AAA products from the BPM cluster. Also, IEG had asked that the issue of whether core SILs ade- quately represented all investment lending be tested in Part IB, by also testing controls in some noncore SIL operations by adding some SWAPs, ERLs, and other ILs to the overall sample to be tested. y Potential Material Weaknesses: Management has already responded to both the docu- mentation retention and the OP/BPs issue by initiating further examination of both is- sues and by launching remedial efforts. The documentation issue has been the central topic of focus during Part IB, because it is precisely the testing of controls (by tracking documentation) that was the substance of Part IB, and the evidence now in hand (in comparison with that obtained during the preliminary testing during Part IA) has pro- vided the basis for a much firmer conclusion to be drawn, as is in this report. Manage- ment has stated that the bulk of the issues relating to the OP/BPs will be dealt with dur- ing the completion of Part II. y Resolution of Other Identified Issues and Deficiencies: Management undertook to ad- dress the issues and potential deficiencies described in para. 6 as part of the work to be completed during Part IB, with the following objectives: to clarify their nature as con- trols issues or other types of issues; to establish their materiality and propose remedial actions; to resolve these issues where possible; for those that required testing or other verification, to include these as part of the Part IB testing process; and to postpone issues to Part II that related more to entity-level controls or to efficiency and effectiveness ob- jectives. NOTES 1. Part I focused mainly on the Control Activities and Risk Assessment components, and very little, if at all on the remaining three: Control Environment; Monitoring and Learning; and Information and Communications. 2. Management itself had identified these issues, though it did not share the view that they were potential material weaknesses. Annex B Summary of the IEG Analysis of Results from Management’s Compliance Testing of Key Controls in Part IB 1. This annex describes the statistical analysis conducted by IEG on the Part IB test results presented by management. It opens with a detailed description of the specific method used by management in conducting its tests, and it then portrays the main findings that emerged from the analysis. 2. Many of the terms used in this review are new to the Bank, since this is the first review of this kind of internal controls. To assist readers in understanding the precise nature of the methods and processes used by management in its assessment, IEG has constructed a set of Management materials, simplified for illustration, which duplicate those used by management in its actual testing. Table B.1 provides brief descriptions of the key elements that entered into the testing process. Table B.1. Key Elements in Testing Internal Controls at the Transactions Level Business process module The main business processes in which IDA is engaged on a daily basis in the course of its (BPM) operations, as identified by management. There are 32 in all. They cover IDA allocation; the CAS process; the main lending products (SILs and DPLs); and the fiduciary, contractual, safeguards and quality assurance processes that support lending. Each were mapped and described as separate Business Process Modules, each containing the key internal controls that are the subject of the review. Figure B.1 shows the BPM for Module 15, Procurement Complaints. Key control A gateway and decision point, involving key units and IDA staff, in a given BPM, through which a business transaction being processed must pass. It is the effectiveness in design of these controls and the subsequent testing of the effectiveness of their operation that is at the center of this review. Control steps/attributes Individual elements of a key control, consisting of the substeps that collectively comprise the fulfillment of the key control. Description of Management’s Testing Method 3. From the 32 modules contained in the process maps, management conducted controls testing on 29, 1 and it was this number that formed the basis for the IEG analysis. To show how management conducted its testing, IEG has constructed a prototype from an actual module (Module 15, Procurement Complaints, see figure B.2 below), from which the elements involved in the testing can be explained. 50 ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING Table B.2. Prototype Depiction of Module 15—Procurement Complaints Control 1 Control 2 Region A B C D A 1 2 3 4 5 1 2 3 1 2 LCR N/A N/A N/A SAR N/A N/A N/A N/A AFR N/A N/A N/A N/A N/A AFR N/A N/A N/A N/A EAP N/A N/A N/A N/A N/A ECA N/A N/A N/A N/A EAP N/A N/A N/A N/A N/A EAP N/A N/A N/A N/A N/A AFR N/A N/A N/A N/A N/A SAR N/A N/A N/A N/A N/A 1 10 10 9 4 4 7 8 7 Legend: LCR =Latin America and Caribbean; SAR=South Asia; AFR = Africa; EAP= Easy Asia and Pacific; ECA = Europe and Central Asia Key Control: Go, No-Go gateway through which all projects must be processed. Key Controls are the essential elements of the internal controls of IDA business processes at the transactions level. Key Controls have steps Control Steps or Attributes 1A Key Control with one Control Step or Attribute 2A Control Step with 5 Attributes or control steps Control Step/Attribute which could have applied but were not applicable in the case of this sample project. Many N/A control steps/Attributes are conditional on project requirements, and can be quite rare: e.g. Control 2A5 requires the “entry of the date of the OPRC review, if required”. In none of the ten sampled projects was there an OPRC review Controls steps that were tested and found to be in compliance with the control design “Exceptions”: Noncompliant control step, either mitigated by another control, or less serious for other reasons “Deficiencies”: Noncompliant control steps unmitigated, and more material. Table B.2A: Summary Sheet : Test Results of Key Controls* Key Control Number and Test Number of Steps Number Required Passed Failed % Passed Overall Control 1—Refer Complaint to INT 1A1 1 1 0 100% PASSED Control 2—Determine Validity of Complaint 2A1 10 10 0 100% 2A2 10 10 0 100% 2A3 10 9 1 90% 2A4 4 4 0 100% 2A5 FAILED 2B 10 4 6 40% 2C1 2C2 10 7 3 70% 2C3 2D1 10 8 2 80% 2D2 10 7 3 70% Total 75 60 15 80% 52 ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING tributes”), which represent the actions to be taken by various parties to fulfill the requirements of the key control. Control 1 has only 1 control step or attribute (control step A), whereas Con- trol 2, which has control steps A (5 attributes), B, C (3 attributes), and D (2 attributes) has 11 at- tributes in all. The table also shows that, in this module, there were 10 projects selected in the testing sample. (The number of projects sampled varied from module to module, because some modules tested controls that were more esoteric and did not apply to all projects). This means that in the case of this module (Module 15), with 10 projects and 12 attributes, there were 120 attributes or cells to be tested in all, although 9 attributes and 75 cells were tested because the other three attributes and 45 cells did not apply in the sampled projects. (Note: Control 1 ap- plied to only one country, because Honduras was the only case in which the complaint was re- ferred to INT.) 5. Not Applicable Control Steps/Attributes (N/A in the Table): Since the internal con- trols framework provides for all eventualities in the processing of projects, quite frequently some control steps are conditional on requirements that may not apply to all projects. For ex- ample, in Module 15 shown in Table B.2, there were four control steps shown partly or fully as N/A. In the first two (Control Steps 2A4 and 2A5) the dates were required of the Regional Procurement Manager (RPM) and OPRC reviews, if these were applicable in the specific case. As the table shows, there were only four cases in which RPM reviews were held, and none which required OPRC review. Control C1 covers cases of non-ICB (international competitive bidding) procurement, and this applied to none of the projects in the sample. Control C3 cov- ered cases that required OPRC review, which did not apply in any of the cases. For a general description of the implications of these “not applicable” controls, see Box B.1. 6. IEG Note: Where control steps are rarely needed, IEG does not see the need for possible retesting at this stage, but this should be considered selectively going forward, in cases where such controls would be seen as of particular importance to risk management. 7. “Passing” the Controls test: The testing was done by management asking the relevant operational departments to present the documentation that showed that each control step had been taken as required by the design of the control. Where the documentation was shown, and was complete and accurate, that control step was marked with a certain color code to show that the control step was compliant, or, where it was not compliant, it was marked as an excep- tion or deficiency (see below). Management also used the results to make a judgment over whether each key control had “passed” or failed, based on how many and which control steps within each control had failed. This is explained in more detailed in the following section, where the actual results are shown. 8. NonCompliance, Exceptions, and Deficiencies: Where management found that a con- trol step was not documented, or was documented inaccurately, or later than required by the design of the control, or for some other reason did not conform to the control step as designed, it was labeled noncompliant. Management distinguished between two forms of noncompli- ance: There were cases in which controls, while not being adhered to, were mitigated by other controls, which made their noncompliance less serious in terms of jeopardizing the business process. Management labeled these as “exceptions.” Those noncompliant controls that were not so mitigated were labeled “deficiencies”, and were seen as somewhat more material than the exceptions, though management regarded both as clearly noncompliant. Table B.2 distin- guishes both forms of noncompliance, and it also shows the cases (which occurred fairly fre- 52 ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING quently) where individual controls steps did not apply (N/A in the figure) to the specific pro- ject in the sample. 9. Testing Period: Management chose projects for its testing sample, all of which were ac- tive in the respective elements being tested during the period between July 2005, and February 2006. Choosing this window had certain implications, one being that it did not cover a full fis- cal year cycle and, therefore, did not cover the so-called “bunching” season. Given that proc- essing errors may occur more frequently during the high-pressure bunching season at the end of each fiscal year (i.e. April-June), it is possible that this may have biased the findings some- what in a positive direction. Box B.1: Reading the Results “Vertically” for a Controls Approach, and “Horizontally” for a Projects Approach • The Controls Approach (“Vertical Cut”): The controls approach views the matrix vertically, to count the number of “passes” or “fails” at each control or control step. In Module 15, for any given control step, since there were 10 sample projects, 10 passes were required for a 100 percent pass rate, fewer if there were any N/As. One pass rate concept is to ask how many controls were without any fails: in Module 15, four (Control 1, and control steps 2A1, 2A2, and 2A4) were 100 percent compliant, a 44 percent pass rate, by this concept. • Management used an average of the pass rates across all control steps taken vertically (12 in Table B.2). If a given control step had 2 out of 10 failures, this would be a pass rate of 80%. In the case of Module 15, the overall pass rate, measured in this way was 80 percent; using a similar method across all 29 modules, Management’s pass rate was 93 percent. • Management emphasized that the pass rate in this example does not mean that 80 percent of all tested projects in this example passed the test without noncompliance. It only means that the aggregate count of total noncompliance in the module as a whole was 20 percent out of the total number of control steps tested. • The Projects Approach (“Horizontal Cut”): The projects view is to take each sample project and view it horizontally, counting the number of passes and fails. In this approach a pass rate could be based on how many projects achieved a 100 percent pass rate. In Module 15 this was 3 out of 10 projects (Montenegro, Madagascar, and Vietnam), a pass rate of 30 percent. IEG regards this as a legitimate approach, but it tends to suggest that it is the projects that are be- ing tested, whereas it is really the controls which are the subject of the tests. NOTES: The horizontal cut views each project in a given module. Some projects were used in more than one module in the testing. IEG has regarded these as separate projects, since the “horizontal” cut views them across a different set of controls in each module. IEG also points out that, in calculating pass rates the control steps marked N/A were excluded in both nu- merator and denominator. Concentrations: IEG has posited the notion that certain “concentrations” of control noncompli- ance have been found. IEG suggests that this can be said to have occurred when there is a se- quence of three or more noncompliances, in either of the following two ways (Note that these concentrations are more likely to occur in BPMs with more control steps and/or more projects): • Controls-Related Concentrations (Vertical Cut): When three or more noncompliances are found in any one control step (i.e., viewed vertically, as in Control 2B in the model above); and/or • Projects-Related Concentrations (Horizontal Cut): When three or more noncompliances are found in any one of the sample projects being tested (viewed horizontally, as in the Hondu- ras project above, where Controls 2B, 2D1 and 2D2 were found to be non-compliant). ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING Reading the Test Results 10. Aggregate Pass Rates: One purpose of the review was to find a basis to provide an overall assurance to management concerning IDA’s internal controls. In this regard, Manage- ment has presented two overall indicators of the testing results (based respectively on the con- trol attributes and on key controls). During the working level discussions, IEG and IAD have suggested two additional concepts (based on full compliance rates), and IEG also presents a fifth indicator based on business process modules. These concepts are presented below: 2 a. Control Attributes: As discussed above, each control has one or more control steps, which has been tested for a number of projects. Each test of a control step for a project gives rise to a control cell (as per Table B.2). Management tested a total of 115 key controls with a total of 466 control steps, across a total of 345 sampled projects, 3 for a total of 3603 control steps (tested control step cells, excluding those not applicable). For its control at- tribute measurement, management calculated the aggregate number across all modules of the attributes that showed noncompliances as a percentage of the total number of at- tributes. Management found (and IEG has verified) a total of 246 noncompliances, giv- ing an overall pass rate of 93 percent. b. Key Controls: Management examined all control steps in the controls, and where these were found to have any noncompliances in any attributes, a judgment was made as to whether that meant that the control in question was or was not working. These judg- ments were based primarily on whether the noncompliances were exceptions or defi- ciencies, as explained in Table B.2 and para. 8 above. IEG reviewed each of these judg- ments, and found them to be appropriate. Using management’s results for each module, IEG has calculated that under this concept 91 percent of all controls were judged by Management to have passed (meaning with no deficiencies). c. Business Process Modules: Using the Key Controls measurement described above, IEG finds that aggregating the failed controls by BPMs may also be useful, since this will al- low judgments as to which major IDA processes may have been placed at risk by the failure to comply with controls. Management also implicitly used this approach in iden- tifying the modules where it observed that the most serious problems had arisen. As shown in Table B.3, 22 out of the 29 modules were judged by management to have passed and 7 to have failed or been at risk (in the sense that at least 1 control was judged to have failed). This outcome was checked and verified by IEG. d. Tests with Full Compliance Rate (Controls): IEG also counted how many control steps passed the tests without any failures (i.e. with 100 percent compliance). It found that of the total 466 control steps, 71 percent passed the tests without any noncompliances. e. Tests with Full Compliance Rate (Projects): IAD had also suggested a pass rate concept based on viewing the number of sampled projects that passed the tests with no non- compliances. IEG calculated this result and found that of the total of 345 sampled pro- jects, 64 percent were tested at 100 percent compliance. 54 ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING Table B.3. Aggregate Pass Rate Concepts: Summary of Results Pass Rate Concept Total Units Tested Passed Failed % Pass Rate Control Steps 3603 3357 246 93 Key Controls 115 105 10 91 Business Process 29 22 7 76 Full Compliance Controls Cut 466 331 135 71 Projects Cut 345 221 124 64 11. Summary: What Do the Pass Rates Convey? This analysis (summarized in table B.3 above) has shown that different indicators convey different signals as to the overall effective- ness of IDA’s controls. IEG could find no established criteria in the controls literature by which to judge whether any of these pass rates fall within a range of acceptability. However, in IEG’s opinion, management’s aggregate controls steps pass rate of 93%, as with the key controls con- cept with a pass rate of 91 percent, would seem to provide a quite satisfactory level of assur- ance, at the transactions level, that controls have operated generally as intended. Consider this: while the 7 percent fail rate may sound significant, it does not mean that 7 percent of all pro- jects tested failed. It is an aggregate indicator measuring the rate of observed noncompliances. Since there were 345 projects in the testing sample, and 246 observed noncompliances, this means less than one noncompliance per project. 12. Also in a positive light, the full compliance pass rate presented in Table B.3 shows that of all controls tested nearly three-quarters (71 percent) tested with no noncompliances, the same was true for nearly two-thirds (64 percent) of all projects tested. Given the random nature of how noncompliances occur, these are quite high “perfect score” results. 13. While these aggregate indicators are a source of comfort, and while it is true that control failures have occurred more on a random pattern (see Table 2 in Chapter 4), the results also show that there are certain concentrations of control failure which have to be acknowledged and addressed. These concentrations are significant because, contrary to the generally adequate op- eration of controls, they suggest areas of control weakness. Also, the concentrations have oc- curred in the fiduciary processes (see Table B.4 below), whose principal function is to ensure tight controls over resource use. This is an issue of concern, which management itself has recog- nized, and which must be seen as a significant qualification to Management’s overall 93 percent aggregate pass rate. Differentiating the Noncompliance Data 14. Patterns, Causes and Concentrations in Noncompliance: IEG conducted additional analysis of the test results data to explore five sets of questions: what patterns and concentra- tions of noncompliances were apparent; what were the principal reasons for noncompliance— lack of documentation or other reasons; what patterns emerged as to control failures between types of units, or responsible officers in the Bank; what appeared to be the business modules with the most prevalent patterns of noncompliance; and what can be said about whether the ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING test results showed that controls appear to be weaker or stronger in areas of greater risk to IDA. Table B.4: Summary of Management's Results for Key Controls in Seven Most Problematic Business Processes Key Control Number Title Total Tested Passed Failed 12 FM-SIL 4 2 2 14 Procurement—SIL 8 6 2 15 Procurement Complaints 2 1 1 17 Loan Management—SIL 5 4 1 24 Loan Management—Suspensions 6 4 2 25 Loan Closing: Standard Procedures 2 1 1 26 Loan Closing: Special Procedures 2 1 1 Total Key Controls 29 19 10 Percentages of Controls Passed and Failed 100.0% 65.5% 34.5% 15. Project Processing versus Control Design: Based on the criteria for “concentrations” described in Box B.1 above, IEG constructed Table B.5 which provides a view of the overall pattern of noncompliances as they occurred across both projects and controls. Overall, out of the 246 noncompliances these have been split roughly evenly between those 129 which have occurred in randomly scattered pattern (one or two, but less than three per control) and the remaining 117 which have occurred in concentrated noncompliances (three or more failures). These are shown in Table B.5 below. Scattered or randomly occurring noncompliances do not necessarily signify problems with specific control steps – because, being random, they suggest only that slips have occurred in project processing on a widespread, or scattered basis. These random occurrences have arisen in many more control steps (23% of the total) than have con- centrated noncompliances (only 6% of the total), but they still only signify that project process- ing, not controls issues have been the problem. Controls issues (design or operation) arise only where there are concentrations of control step failures, with three or more noncompliances oc- curring in a given control step. On balance, therefore, project processing laxity has been more widespread, specific control design or operation issues more focused on a fewer number of specific controls. Table B.5: Distribution of Noncompliances by Control Steps and by Projects. Totals Clear ( 0 ) Random (1 -2) Concentrated (3 or >) Control Steps 466 331 107 28 Projects 345 221 90 34 % Distribution Control Steps 100 71 23 6 Projects 100 64 26 10 56 ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING 16. These results suggest that where remedies are sought these may lie more with man- agement oversight and the Control Environment (management attention to excellence, tone at the top, discipline and accountability) than in the need to revisit the design of specific controls, though the latter is also needed in certain cases, as is discussed below. 17. Documentation versus other factors: Table B.6 shows that a lack of documentation re- mains the cause of noncompliance in more than half the cases tested. The “other reasons” for noncompliance included inaccuracies in documentation, documentation that was filed late, borrowers who failed to deliver documents, and staff that did not act on required control step. IEG regards it as important that a significant portion of the total noncompliances were the re- sult of a failure to find adequate documentation for the controls steps. This relates to the im- portant issue that arose during Part IA, where IEG found that document retention and accessi- bility in the Bank was a potential material weakness. Now, with the data available from the Part IB testing, IEG has revisited this issue, and observes that the picture is clearly less dire than initially thought. Table B.6. Reasons for Noncompliance Lack of Documentation Other Reasons Deficiencies/Exceptions 57 43 Source: IEG calculations from management data 18. Key Responsibilities for Controls Observance: Consistent with the finding described above, that project processing seems to generally outweigh control design problems as a source of noncompliance, it would be expected that the operations officers most involved with the processing of projects would be those most likely to be linked to cases where noncompli- ance was found. The evidence in Figure B.1 below clearly bears this out. Task Team Leaders (TTLs) and Financial Officers (FOs) were those most frequently associated with controls that were not complied with (around 35 percent each of all cases). Country Directors were also in- volved in close to 10 percent of all noncompliances. Figure B .1: R esponsibilities for non-compliances 40% 35% deficiencies as % of non compliances Exceptions and 30% 25% 20% Deficiencies 15% Exceptions 10% 5% 0% TTL CD FO SM Leg FMS RSC CMU LO A R esponsible offices TTL = task team leader; CD = country director; FO= financial officer; SM = sector manager; Leg = country lawyer; FMS = financial manager; RSC= regional safeguards coordinator. ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING 19. Which Processes are More Problematic? IEG used the concentration analysis also to identify the business processes that appeared to be the most problematic in the sense of having the highest concentrations of both project and controls-related noncompliances. Table B.7 shows the key modules where concentrations of both types were evident. It also lists the mod- ules that management found to be the most problematic (shown in Table B.4 above). IEG agrees with Management in all but one case (Module 5, Core SIL). IEG finds this module to be problematic in that it had a large number of noncompliances (35) (and significant concentra- tions), though this was a relatively small percentage, because of the large number of control steps in the module. Management points out, and IEG agrees, that many of these non- compliances were “exceptions” rather than “deficiencies”. Table B.7. Modules With the Highest Incidence of Noncompliance MODULES 5 12 14 15 17 24 25 Core FM SIL Procurement Proc. LM LM LM SIL SIL Complaints SIL Suspensions Closing Management X X X X X X IEG X X X X X X X Total 35 41 23 15 28 20 11 noncompliances As % of Control 6 17 5 20 11 33 23 Steps (cells) in the Module Concentrations Controls 5 6 2 3 3 3 3 Projects 4 6 2 3 5 3 3 Source: Management test results data, see also Table B.2 above. 20. IEG regards these findings as significant in several respects: a. The results echo the walk-through findings in Part IA, where processing requirements of SILs were seen by the staff to be onerous and cumbersome (and, apparently, are not be- ing fully adhered to); some specific controls in Module 5 may need to be revisited as a candidate for streamlining as much as for tightening of controls. b. Test results showed some significant procurement issues (Module 14)—mainly with post- procurement reviews not being timely, and contract details not being sent in timely fash- ion to LOA. Management also highlighted these issues in its summary (see para. 17 ii of the Management Report) c. Certain areas of loan management and project closing are in need of possible revision and procedural tightening, specifically those relating to LAS set-ups with thresholds for pre- and post review, and retroactive financing cases. Two controls steps in Module 17 (Loan Management SILs) had almost 100 percent failure rates. 21. Are Controls Stronger where IDA Risks are Greater? IEG has taken a first step in dif- ferentiating risks among different modules, then comparing the incidence of noncompliance 58 ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING between the different risk-grouped modules. The risk-grouping was actually done during Part IA, based on a relatively crude methodology, 4 (the results of which are shown in Table SA.12 in the Statistical Appendix to the present report.) In calculating the non-compliant rates by risk group, there is a need to standardize by group size, because Group 1 (higher risk) has both many more control steps and more sample projects than the other two groups. It would follow that most noncompliances would occur in Group 1, as was the case, shown in Table B.8. To offset the sample size issue, IEG calculated the noncompliance rate for each of the three risk groups. As column 6 in the table shows, the average noncompliance rate was slightly higher in Group 1 (7.0 percent), the higher risk group, than in the other two groups (6.4 percent and 6.7 percent respectively). This suggests that the controls appear not to have worked as well in the modules in the higher-risk group. However, given the relative crudeness of this analysis, more work is needed in the area of matching controls strengths and compliance with some form of more rigorous differentiated risk analysis. Table B.8. Risk Grouping* of Modules, Controls and Noncompliances 1 2 3 4 5 6 Risk Controls Noncompliances Average groups Modules Attributes A/M Intensity #% Noncompliance Rates Group 1 12 272 23 1.44 168 68 7.0 Group 2 10 146 15 0.94 58 24 6.4 Group 3 7 48 7 0.44 20 8 6.7 TOTAL 29 466 16 1.00 246 100 6.8 * Risk groupings as given in the Part IA report, Statistical Appendix, Table G-7, page 78, reproduced in this report in the Statistical Appendix, Table SA.12 page 79: Group 1 = higher-risk; Group 2 = medium-risk; Group 3 = lower-rosk Notes: 1. Modules grouped by Risk Group (see table SA.12 Statistical Appendix). 2. Attributes = Control Steps 3. A/M = Average attributes per module (2/1) 4. Controls Intensity = A/M expressed as percentage of the average 5. Noncompliances in absolute numbers and by % share in each group. 6. ANC % = Average noncompliance Rates Summary 22. In summary, this IEG analysis has revealed some interesting insights into the testing results that were presented by management. IEG broadly agrees with management’s analysis of its pass rate and believes this is a reasonably satisfactory outcome, given the complexity of the Bank and that this is a first-of-its kind review. This overall result is also broadly corrobo- rated by the calculated alternative pass rates. Different perspectives were offered on the pat- terns and concentrations that were found in the noncompliance rates: IEG found that project processing was evidently a greater cause of noncompliance than was any problem with indi- vidual controls design; and documentation retention is a major cause of noncompliance, though it has improved considerably compared with the early findings in Part IA. As ex- pected, it was the officers such as TTLs and FOs that were most frequently involved in cases where controls were not fully observed. IEG made a simple first attempt at risk differentiation, and found that there was some evidence that higher rates of noncompliance correlated with ANNEX B SUMMARY OF IEG ANALYSIS OF RESULTS FROM MANAGEMENT’S COMPLIANCE TESTING the higher-risk business processes. However, with the crudeness of the methodology used, IEG would take these results as no more than an indication that more thorough analysis is needed, which should be done in Part II. NOTES 1. Management found itself unable to test Module 29 (Safeguards OPCQC) because of problems locating adequate data. Different regions use different clearance procedures, and there is apparently no Bank- wide codification for these processes. It also did not test Modules 2 and 3, because these were essentially once-a-year technical subprocesses of the IDA Allocation model (which was tested). So the total number of modules tested was 29. IEG takes no issue with the latter two cases, but believes that lack of a codified process for OPCQC is a deficiency and should be addressed. 2. General Note: All concepts calculate a pass rate that excludes—as is reasonable—the N/A from both the numerator and, as applicable, to the denominator. 3. The number of unique projects in the sample was smaller, because the same projects were in some cases used to test different controls. Where this happened, IEG regards these as separate projects. 4. Modules were grouped according to (i) the degree to which the business processes they contain involve mainstream allocation and lending operations and major financial and reputational risk is thus involved; (ii) the size of the processes; and (ii) their frequency. 60 Annex C Summary Account of the Disposition of All Reported Internal Control Issues Uncovered During Part I 1. This annex provides a summary account of all the issues reported by management, IAD, and IEG during Part IA and Part IB of the IDA14 framework review, including their status or disposition as of May 15, 2007, and highlights the significance the issues using agreed upon terms for their classification as material weaknesses, significant deficiencies, or deficiencies. Included also is summary information on the actions taken or planned by management to ad- dress the deficiencies. 2. During Part IA of the review, management, IAD, and IEG identified a total of 126 is- sues involving a broad range of items that required resolution and closure, attention in Part IB of the review, or deferral to Part II. Management reported the status or disposition of the 126 issues in para. 32-34 of its IB report, which are recapped in table C.1. Table C.1. Status and Disposition of Part IA Issues Number Percentage Addressed in Part IB 37 29 Deferred to Part II 44 35 Resolved and closed 40 32 Review documentation updated 5 4 Total 126 100 3. We have tracked all IEG issues raised in Part IA and agree with the treatment of these issues as shown by management in its report. In addition, we compared the issues reported in the Part IA report with those being formally tracked to their resolution or implementation, and agreement has been reached on the items that require attention and thus should be formally tracked by management, IAD, and IEG. 4. As indicated in the table C.1 above, of the 126 Part IA issues, 37 have been addressed in Part IB. Along with these 37 issues, management and IAD identified in Part IB an additional 25 issues for a total of 62, which provided the starting point for an aggregation and for judg- ments to be made as to the deficiencies, significant deficiencies, or material weaknesses arising from all of Part I. 1 61 ANNEX C SUMMARY ACCOUNT OF THE DISPOSTION OF ALL REPORTED INTERNAL CONTROL ISSUES Definitions and Criteria 5. Management’s report, which is included as attachment 1 to this report, describes the methodology used for assessing the results of the work completed to date in terms of deficien- cies, significant deficiencies, and material weaknesses. Definitions of these terms and the crite- ria for categorizing the review results are also included in management’s annex 3. The precise criteria for distinguishing the three categories of materiality are explained in annex B of IEG’s Part IA Report. The Aggregation Process 6. The identification, assessment, and reporting of results in the three categories referred to above involve an iterative, building-block approach. The process started in Part IA (control design) and continued in Part IB (control operation) with the identification of specific issues and exceptions for the overall purpose of assessing the design and operating effectiveness of 115 key controls in 32 separate IDA business processes. 7. Management has systematically tracked and reported on the issues and exceptions sur- faced by its own work, as well as that of IAD and IEG, from the start of the review in Part IA through the conclusion of Part IB. Management did not classify any of the issues and exceptions identified during Part IA as deficiencies, significant deficiencies, or material weaknesses. Instead, management made this classification after completing Part IB, as shown in Figure C.1. Figure C.1. Management’s Results by Categories of Significance Material Weaknesses: 0 Significant Deficiencies: 3 Deficiencies: 6 Issues and Exceptions: 62 Part IA and IB Descriptive Materials, Work Flow Mappings, Walkthroughs, and Transaction Testing for 115 Key Controls in 32 business processes Material Weaknesses 8. Management did not report any material weaknesses in the work performed during Part I. However, IEG reported at the conclusion of Part IA that there were two potential mate- rial weaknesses: documentation retention and accessibility, and the status of OPs and BPs. As ex- plained in the main text, IEG has revised its view on the documentation issue, and now agrees that it constitutes a significant deficiency (see below). However, it has retained its view that the status of OP/BPs remains a potential material weakness. 62 ANNEX C SUMMARY ACCOUNT OF THE DISPOSTION OF ALL REPORTED INTERNAL CONTROL ISSUES Significant Deficiencies 9. In its report (para. 25 A-C) management identified three significant deficiencies: y The inability to provide timely access to documents.(This is a revised formulation of what was earlier labeled a documentation retention and accessibility issue, framed after an improved documentation record and a 93 percent pass rate during the Part IB test- ing.) y Variances in regional implementation of institutionally endorsed financial management and procurement guidelines . y The status of OPs and BPs. This is the same issue mentioned above that IEG previously reported as a potential material weakness. 10. In summary, therefore, the judgments made respectively by management, IAD, and IEG regarding material weaknesses and significant deficiencies are as shown in table C.2. Table C.2. Designation of Potential Material Weaknesses and Significant Deficiencies Potential Material Weakness Significant Deficiency Management 0 3 IAD (See note) (See note) IEG 1 2 Note: IAD reported deficiencies but did not classify them as Potential Material Weakness or Significant Deficiency. Deficiencies 11. In addition to the significant deficiencies described above, management identified in its report (para. 26) six deficiencies that merit a closer investigation to assess causes and impact and identify appropriate remedial actions, which generally were in the following areas: y Need to streamline IL operations y Disparity in corporate review between IL and DPL y Lack of timely updates to the LAS y Inconsistency and lack of follow-up in clearing review comments y Need for improved controls over the safeguards Corporate Risk List y Non-compliance of IDA countries with quarterly debt reporting requirements and plans to assess debt sustainability in Part II. Issues and Exceptions 12. As stated in its report (para. 27), management provided IEG and IAD detailed informa- tion on the total 62 issues and exceptions, together with management’s response and action for each. For each of these 62 items listed, management provided information that included: y The business process module or modules in which the items were noted. y A description of the issue, exception, or deficiency ANNEX C SUMMARY ACCOUNT OF THE DISPOSTION OF ALL REPORTED INTERNAL CONTROL ISSUES y An evaluation of the significance of each of the items listed that considered whether the issue, exception, or deficiency would meet one of more of the following conditions:  Impair achievement of IDA’s objectives?  Violate IDA’s charter or contractual agreements?  Weaken safeguards—waste, loss unauthorized use of funds, property or assets?  Conflicts of interest?  Systemic problems in country assistance partnerships and project lending?  Require attention of senior management, the Board, or external stakeholders? 13. The above questions are the same as those listed in annex B, page 48, of IEG’s report on Part IA, which are measures to be used as guides by each of the three groups in determining whether identified internal control deficiencies in compliance constitute significant deficiencies or material weaknesses. Based on answers to the above questions and its judgment of the sig- nificance of the items listed, management indicated on the listing whether it believed any of the items, alone or in combination with other items on the list, could constitute a deficiency or significant deficiency. Management did not indicate that any one or combination of the items listed would constitute a material weakness. 14. Two ways of considering the issues and exceptions are by the way they were aggre- gated into the significant deficiencies and deficiencies included in management’s report and by the IDA business processes where they were found. First, the issues and exceptions can be summarized according to whether they were judged to be significant deficiencies, deficiencies, and issues/exceptions being reported and tracked by management until corrected and whether they were identified in Part IA (control design issues) or Part IB (control operation is- sues revealed by testing), as shown in Table C.3. 64 ANNEX C SUMMARY ACCOUNT OF THE DISPOSTION OF ALL REPORTED INTERNAL CONTROL ISSUES Table C.3. Issues and Exceptions as Aggregated and Tracked by Management Number of Issues in Deficiency Tracker Issue From Part From Part Total IA IB Significant Deficiencies 1. The inability to provide timely access to documents 3 1 2 2. The extent of variances in regional implementation of institutionally 19 7 12 endorsed financial management and procurement guidelines 3. The status of OPs and BPs. This is the same issue mentioned above 10 9 1 that IEG previously reported as a potential material weakness Deficiencies 1. The need to streamline IL operations 2 2 - 2. The disparity in corporate review between IL and DPL 1 - 1 3. The lack of timely updates to the LAS 7 4 3 4. The inconsistency and lack of follow-up in clearing review comments 5 4 1 5. The need for improved controls over the safeguards Corporate Risk 2 1 1 List 6. Non-compliance of IDA countries with quarterly debt reporting 2 - 2 requirements Other Issues and Exceptions In Management’s Tracker 1. Allocation procedures 2 2 - 2. Project changes 2 2 - 3. QAG 4 2 2 4. Refunds 1 1 - 5. Safeguards 2 2 - Total 62 37 25 ANNEX C SUMMARY ACCOUNT OF THE DISPOSTION OF ALL REPORTED INTERNAL CONTROL ISSUES 15. Second, the issues and exceptions can be viewed by IDA business process—where among the IDA business processes were the issues and exceptions found? Of the 32 processes, management reported no issues or exceptions for 12. As can be seen from Table C.4, two of the remaining 20 business processes accounted for 36.8 percent. Financial Management-SIL (11.8 percent) and Procurement-SIL (25 percent). (See Table C.4.) Table C.4. Distribution of Issues and Exceptions by Business Process Issue and Exceptions Business Process Number Percentage FRM Allocation—Main 2 2.9 SIL—Product Cycle 4 5.9 DPL—Project Cycle 3 4.4 Corporate Review (ROC/OC) 2 2.9 Contractual Remedies 1 1.5 FM—Specific Investment Loans 8 11.8 SIL—Procurement Regime 17 25.0 Procurement Complaints 4 5.9 Procurement Noncompliance 1 1.5 Loan Management—SIL 3 4.4 Amendments & Extensions 1 1.5 Refund Process 2 2.9 Cancellation Process 1 1.5 Loan Mgmt Suspensions 4 5.9 Loan Closing (Standard Procedures) 3 4.4 Loan Closing (Special Procedures) 1 1.5 QAG: QEA and QSA 4 5.9 Safeguards—SIL 3 4.4 Safeguards—Corporate Risk (QACU) 2 2.9 Debt Reporting 2 2.9 Total (See note) 68 100 Note —The total number of issues/exceptions does not equal to 62 because some were found in more than one process. 16. Management plans to update IAD and IEG as to the status of the identified actions for all of the items in the Tracker at the completion of Part II and plans to include their status in its final report. 66 ANNEX C SUMMARY ACCOUNT OF THE DISPOSTION OF ALL REPORTED INTERNAL CONTROL ISSUES Management’s Actions to Address Deficiencies 17. Management provided IEG with information from its Tracker on the actions taken or planned to address the deficiencies and issues that had surfaced in Part I, as summarized in Table C.5 below. Table C.5. Status of Issues and Exceptions as of May 15, 2007 Status Number Action taken 7 Action planned or underway 33 Deferred to Part II 11 No action planned by management 11 Total 62 18. In the cases where no action was taken or planned as of May 15, 2007, management in- dicated in the Tracker that clarification of the matter had been obtained or for other reasons management stated that it would not be taking action. Examples of completed actions as de- scribed in the Tracker follow: • Management took steps to standardize the approach used by OPCQC and the regions in screening and tracking projects on the safeguards corporate risk list and also provided related guidance to the regions. • The Executive Directors approved on July 18, 2006, Management's proposed revisions reflected in updates to BP 13.05, Project Supervision, covering Restructuring and Other Project changes. The revisions dealt with a potential deficiency by clarifying the roles and responsibilities of regional staff and Management, including the CDs, RVPs, and the Board. Management also provided staff with accompanying guidelines for document- ing certain project changes as they are first identified by team leaders and country direc- tors. • On March 25, 2007, OPCS issued a new Guidance Note on Management Review of In- vestment Operations intended to put in place appropriate measures to ensure that the level of management review that applies (including criteria for corporate review) is aligned with the level of risks involved in specific investment operations. The Tracker information provides a valuable source of data on areas of IDA operations requir- ing attention. The information has been jointly developed and shared by the three review groups and thus has eliminated the time and expense of maintaining three separate and poten- tially conflicting sets of data. As the review moves to Phase II and final, overall reporting, Man- agement intends to address some additional elements in the Tracker which could help in pin- pointing responsibility and tracking deficiencies to their final disposition, such as identifying (1) the specific management unit within the bank responsible for evaluating the matters dis- closed and taking corrective action where it is called for, (2) milestones and/or anticipated completion dates for implementing the changes needed and being acted on, and (3) target dates ANNEX C SUMMARY ACCOUNT OF THE DISPOSTION OF ALL REPORTED INTERNAL CONTROL ISSUES for follow-up to ensure corrective actions have been fully implemented or appropriately re- solved and thus can be dropped from the Tracker. NOTE 1 IAD also identified a methodological issue with Management’s pass rate which is still open. 68 ANNEX D DECRIPTION OF THE QUALITY RATING PROCESS USED BY IEG Annex D A Description of the Quality Rating Process Used by IEG in Evaluating the Approach and Methods in Management’s Assessment and the IAD Review of Part I Introduction 1. In designing its evaluation, IEG saw the need to create a standardized framework of questions to be asked of the different segments of work that would constitute management’s assessment and the IAD review: the approach and methods that management would use in its process mapping and verification of the design effectiveness of key controls; the methods it would use for sampling and testing those controls; the clarity and robustness of the results achieved in the testing; and the quality of conclusions that management arrived at in completing its assess- ment; and a parallel application of the review methods, results, and conclusions arrived at by IAD. 2. Standardization was seen as important, for various reasons: there were initially 30 (later 32) different BPMs contained in management’s representation of IDA’s business proc- esses, and it would be important for IEG to apply a uniform set of questions, based on pre- established criteria, so that each module would be evaluated by IEG against a common stan- dard. By creating a standard template of questions, a platform would also be created on which similar future evaluations could rely, and which could provide the basis for comparisons over time. 3. Since management had divided its assessment into two major parts, this gave an addi- tional reason to create a framework to capture issues at two levels: at the level of business transactions (Part I of the review), and then a quite different set of issues to be captured in the entity-level assessment to be conducted in Part II. IEG therefore created two separate (but linked) templates of questions. The first dealt with IDA business processes, the second with the COSO framework. This annex describes the content and application of the first of the tem- plates (The Business Process Template), which IEG has applied in Part I. The COSO Frame- work Template will be used and described in the IEG report to be written on completion of Part II. ANNEX D DECRIPTION OF THE QUALITY RATING PROCESS USED BY IEG The Business Process Template 4. IEG formulated the template using 56 questions, organized around the different seg- ments of the work described above. The questions were formulated following research con- ducted by IEG of questions used in similar reviews in other agencies, including the U.S. GAO, but IEG made its own adaptation of the questions to suit the case of IDA. Table D.1 provides a listing of the main categories of the questions, six for the management assessment and four for the IAD review. Table D.1. Question Categories in the Business Process Template Question Categories Number of Questions FOR MANAGEMENT’S ASSESSMENT 1. Strategic Relevance and Importance 5 2. Mapping the Business Process 11 3. Assessment of Control design 10 4. Testing of Control Compliance 12 5. Linkage to COSO Framework 6 6. Quality of Conclusions 4 SubTotal 48 FOR THE IAD REVIEW 1. Scope of Work 1 2. Criteria and Standards 1 3. Documentation and Evidence 4 4. Quality of Conclusions 2 SubTotal 8 TOTAL 56 The Rating System 5. The template also contains a rating system. This was based on criteria that were built around the degree of certainty with which management’s assessment (and the IAD review) could be said to have shown that the methods used were such as to show clearly (or otherwise) that controls were well mapped, designed, tested and so forth. As shown in Box D.1, the sys- tem uses a four-part rating system, ranging from highly satisfactory (1) to less than satisfactory (4). Its purpose was to reduce qualitative judgments about different processes to quantified in- dicators, which would assist in making summary overall judgments. Box D.1. Elements of the 4-Part Rating System Rating (numerical equivalent) Criteria Highly Satisfactory(HS) (1) High degree of certainty that the assessment/review showed that controls in the process element are well designed, operate effectively, etc. OR: Clear evidence that the Assessment/review showed that there were deficiencies/weakness in the controls process. Satisfactory(S) (2) Reasonable degree of certainty that the Assessment/Validation showed that controls in the process element are well designed, operate effectively, etc. OR: Some evidence that the assessment/review showed that there were deficiencies/weakness in the controls process. 70 ANNEX D DECRIPTION OF THE QUALITY RATING PROCESS USED BY IEG Satisfactory with Qualification ( 3) Uncertainty as to whether the assessment/review showed that controls in the (SQ) process element are well designed, operate effectively, without deficiencies etc. Less than Satisfactory(LS) ( 4) Clear evidence that the assessment/review did NOT show that controls in the process element are well designed, operate effectively, etc.; OR failed to uncover the presence of deficiencies or weaknesses Not Applicable (NA) Not Applicable The Evaluation and Rating Process 6. IEG assembled evaluation panels, consisting of its core team of external consultants, together with a selected number of other experts with significant experience in the Bank (e.g., lending operations, legal, procurement, financial and loan management, safeguards). The pan- els examined the materials presented to IEG by management for each of the Modules, they ap- plied the template to the findings of the different segments of the assessment, and came to a consensus on the rating to be given to the process referred to by each question. Given that management had divided Part I into two parts (IA and IB), the panels used the template twice, applying Questions 1-3 in the session dealing with Part IA, and questions 4-6 for Part IB. Overall Results 7. A summary of the results that were achieved by this process have been given in the main text of the report (table 1) and in the Statistical Appendix (tables SA.1 to SA.5). The sum- maries describe the fact that an overall rating of 2.5 was given for the outcome of Part IA (process mapping and control design) and 2.4 for Part IB (compliance testing). This implied that for both Parts IA and IB, IEG found management’s approach and methods to be generally satisfactory, but with some qualifications. More Detailed Breakdown of Results 8. IEG regarded this tool as adding significant value to its evaluation. Part of this value was in distilling the evaluation into an overall rating. However, possibly even more valuable were the insights that the Template provided at the level of individual modules, where the template questions often led to a searching examination of Management’s processes and re- sults. In this vein, it is instructive to look in more detail at what were the major factors that de- tracted from the overall rating being (2) — fully satisfactory. 9. An overall summary of the distribution of quality ratings—for both Part IA and IB—is shown in Table D.2. It shows that almost two-thirds of the ratings for all categories were at “2”, i.e. were fully satisfactory, while some 26 to 29 percent were rated at “3”, meaning that proc- esses were seen as satisfactory with qualification. IEG emphasizes that this rating is still above the line, but that some factor or process (very often descriptive materials, rather than substan- tive deficiencies) could have been better presented, or improved upon in some way. The table also shows that IEG was quite stringent in its ratings, giving ratings of “1” (Highly Satisfactory) to no more than 2 percent of all processes covered in the questions. Against this, ratings of Less than Satisfactory accounted for 7 to 8 percent of the total. ANNEX D DECRIPTION OF THE QUALITY RATING PROCESS USED BY IEG Table D.2. Distribution of IEG Quality Ratings All Modules Distribution by Rating PART IA (Management) 1 2 3 4 Distribution by No. 15 450 184 51 Distribution by Percentage 2% 64% 26% 7% PART IB (Management) Distribution by No. 4 272 129 35 Distribution by Percentage 1% 62% 29% 8% PART I OVERALL (IAD) 0 88% 12% 0 Some Highlighted Features 10. The template was used as a tool to evaluate Management and IAD’s approach and meth- ods. The template ratings are a comment on whether the results management achieved can be taken to be credible, given the methods that were used. In general, the IEG ratings suggest that the methods were indeed credible and satisfactory, but there were also some qualifications. Most of these were of the nature of descriptions which could have been improved, but others, such as the treatment of risk, were of more operational significance. 11. Process Example—Module 28 SIL Safeguards: To illustrate how the template uncov- ered some useful insights, in the evaluation of the SIL Safeguards module, the evaluation panel had initially rated most elements in the Template at “4” i.e. less than satisfactory. This arose mainly from a lack of clarity as to how management had drawn the process map, and what precise processes had underpinned the clearance of the two key controls. On account of the low rating IEG sought a meeting with management, and asked for an examination of the actual documentation on each step in the key controls, which was contained in the Data Room. Following this meeting, it was clear to IEG that the processes had, in fact, been correctly fol- lowed, but that the (Part IB) test results materials presented to IEG had not been sufficiently explicit. This interchange was a learning experience for both sides, and the ratings were amended, though they reflected the fact that this lack of clarity had occurred. 12. Overall, IEG regards the template to have been a useful addition to its arsenal of evaluative tools. Its questions appeared relevant to the different categories of process exam- ined, and the rating results provide a fair, if rigorous, indication of the quality of management and IAD approaches to Part I of the review. Management’s assessment was rated separately for Part IA and Part IB; IAD was rated for Part I overall (see Table D.2 above). IEG is currently refining its version of the COSO Framework template, which will be used in Part II to evaluate management’s assessment and the IAD review of Part II. 72 Annex E Statistical Appendix Table SA.1. Summary of IEG Ratings of Management’s Part IB Approach and Testing Methods Function Rated Rating Function Rated Rating (By Template Question (By Template Question Numbers) Numbers) Choosing the Sample (4.01) 2.37 Testing Methods (4.02) 2.07 4.01 a 2.19 4.02a 2.07 4.01b 2.59 4.02b 2.00 Testing Results (4.03) 2.68 Quality of Conclusions (6.00) 2.43 4.03a 2.79 6.01a 2.21 4.03b 2.36 6.01b 2.57 4.03c 2.48 6.01c 2.35 4.03d 3.00 6.01d 2.95 4.03e 2.40 Rating System: 1 = highly satisfactory; 2 = satisfactory; 3 = satisfactory with qualification; 4 = less than satisfactory; N/A = not applicable. Table SA.2. Summary of Quality Ratings of Management’s Assessment of the Design Effectiveness of Key Controls (Part IA) Quality Dimensions Average Ratings Overall R1 R2 R3 R4 R5 Mapping the Business Process 2.38 Origin, Method, and Criteria 2.31 2.58 2.00 2.19 2.12 Accuracy and Completeness 2.50 2.31 2.73 2.00 2.42 Identification of Key Controls 2.19 2.38 2.04 2.08 Assessment of Control Design 2.62 Identifying Process Risks 2.81 2.46 4.00 2.31 2.15 Matching Risks with Process Design 2.31 2.35 2.08 2.44 Overall Average Rating 2.45 2.42 2.57 2.20 2.23 Rating Scale: 1 = highly satisfactory; 2 = satisfactory; 3 = satisfactory with qualification; 4 = less than satisfactory. Annex E Statistical Appendix Table SA.3. Summary of Quality Ratings of Management’s Assessment of the Design Effectiveness of Key Controls by Distribution of Ratings (Part IA) Distribution by Rating Mean 1 2 3 4 N Mapping the Business Process 2.38 0 62 38 0 26 Origin, Method, and Criteria 2.31 0 69 31 0 26 Clarity of IDA Operational Objective? 2.58 4 54 23 19 26 Clarity of method and criteria? 2.00 0 100 0 0 26 BPM established under Bank BP or OP? 2.19 12 62 23 4 26 Management sought input in process area? 2.12 0 88 12 0 26 Accuracy and Completeness 2.50 4 46 46 4 26 Process has been clearly titled? 2.31 4 62 35 0 26 Risks to BPM clearly stated? 2.73 4 38 38 19 26 Ownership of process clearly designated? 2.00 4 92 4 0 26 Management sought input in process? 2.42 12 50 23 15 26 Identification of Key Controls 2.19 0 81 19 0 26 Clear definition of key controls? 2.38 0 69 23 8 26 Relevance of mapped BPM controls? 2.04 0 96 4 0 26 Differentiation between controls for financial reporting 2.62 0 92 8 0 25 and other COSO objectives? Assessment of Control Design 2.62 0 38 62 0 26 Identifying Process Risks 2.46 0 23 73 4 26 Clear identification of risks that the control points are designed 2.46 8 38 54 0 26 to alleviate? Risks have been categorized (fin/op/rep) and analyzed? 4.00 0 0 0 100 26 Management documentation relates to the policies and procedures 2.31 0 73 23 4 26 of controls and risks? Management consulted with the most authoritative sources? 2.15 0 85 15 0 26 Matching Risks with Process Design 2.31 0 69 31 0 26 Management adequately matched the design with the risks? 2.35 i. Built in checks and balances 2.81 0 35 50 15 26 ii. Involved specialized staff 2.15 0 85 15 0 26 iii. Involved appropriate operational units and mgmt levels? 2.08 0 92 8 0 26 Design process is known by relevant staff? 2.08 8 81 8 4 26 Mgmt. has shown that controls extend to cover external risks? 2.44 0 56 44 0 25 Rating Scale: 1 = highly satisfactory; 2 = satisfactory; 3 = satisfactory with qualification; 4 = less than satisfactory. Note: Modules 1-3 and Modules 25 & 26 were assessed together; therefore maximum number of observations is 26 rather than 29. 74 Annex E Statistical Appendix Table SA.4. Summary of Quality Ratings of Management’s Assessment of the Control Compliance of Key Controls by Distribution of Ratings (Part IB) Distribution by Rating Mean 1 2 3 4 N Testing of Control Compliance 2.39 0 64 32 4 28 Choosing the Sample to be Tested 2.37 0 67 30 4 27 Clear, well-explained sampling methodology? 2.19 0 85 11 4 27 Criteria give a credible, comprehensive sample? 2.59 0 56 30 15 27 Geared to highest risk processes? 2.00 0 100 0 0 1 Testing Methods 2.07 0 96 0 4 28 Well-explained, explicit testing methodology, followed consistently? 2.07 4 89 4 4 28 Where methods were different, were differences justified? 2.00 0 100 0 0 2 Testing Results 2.68 0 46 39 14 28 Were results robust, credible, unambiguous? 2.79 4 32 46 18 28 Where control weaknesses were revealed, was it clearly stated whether due to compliance or control design? 2.36 0 68 27 5 22 Where control weaknesses, did tests identify origin and did findings lend themselves to seeking remedies among relevant units? 2.48 10 43 38 10 21 Where control weaknesses, was additional work done to determine how widespread, cause, consequences? 3.00 0 22 56 22 18 Where mitigating controls identified, were they well-explained and documented? 2.40 0 80 0 20 15 Were mitigating controls arguments justified? 2.43 0 64 29 7 14 Quality of Conclusions 2.43 0 57 43 0 28 Is there a conclusion that is clear, concrete, and concise? 2.21 0 79 21 0 28 Does the conclusion reflect the findings? 2.57 0 46 50 4 28 Were exceptions adequately identified as exceptions or deficiencies? 2.35 0 65 35 0 23 Were there proposals for remedial action, if relevant? 2.95 0 32 42 26 19 Rating Scale: 1 = highly satisfactory; 2 = satisfactory; 3 = satisfactory with qualification; 4 = less than satisfactory. Note: Modules 1-3 and Modules 25 & 26 were assessed together; also, module 29 was not completed; therefore maximum number of observations is 28. (This table includes modules 30, 31, and 32.) Annex E Statistical Appendix TABLE SA.5. AVERAGE QUALITY RATINGS OF MANAGEMENT’S ASSESSMENT OF THE DESIGN EFFECTIVENESS OF KEY CONTROLS BY BUSINESS FUNCTION Mapping the Business Process Assessing Control Design Origin, Overall Business Function Method & Accuracy & ID of Key ID of Proc- Match Average Overall Criteria Completeness Controls Overall ess Risks Risks Programming & Alloca- tion 2.50 2.00 3.00 2.00 3.00 2.50 3.00 2.57 Lending Products 2.25 2.00 2.00 2.50 2.75 2.75 2.25 2.36 Legal 2.33 2.33 2.33 2.00 2.33 3.00 2.00 2.33 Financial Management 2.00 2.00 2.00 2.00 2.00 2.50 2.00 2.07 Procurement 2.00 2.00 2.00 2.00 2.00 2.67 2.00 2.10 Loan Administration 2.67 2.67 3.00 2.33 2.89 3.00 2.56 2.73 Quality Assurance 3.00 3.00 3.00 2.00 3.00 3.00 2.00 2.71 Safeguards 2.00 2.00 2.00 2.00 2.50 2.50 2.00 2.14 Rating Scale: 1 = Highly Satisfactory; 2 = Satisfactory; 3 = Satisfactory with Qualification; 4 = Less than Satisfactory; N/A - Not Applicable. 76 ANNEX E STATISTICAL APPENDIX Table SA.6. Projects-Related Concentrations (Projects with at least three noncompliances) Country Module No. of Exceptions Non compliance rate (%) Nicaragua Frontloading 1 3 100.0 Gambia 5 5 17.2 Burkina Faso 5 3 10.3 Mongolia 5 4 28.6 Yemen 5 4 28.6 Mali 7 4 11.8 Timor-Leste 7 3 9.1 Honduras 7 3 8.8 Soã Tomé & Principe 9 5 25.0 Djibouti 9 4 20.0 Haiti 10 3 15.8 Vietnam 12 3 30.0 Gambia 12 5 45.5 Mongolia 12 4 40.0 Serbia-Montenegro 12 3 37.5 India 12 3 42.9 Bangladesh 12 3 33.3 Mongolia 14 4 23.5 Bangladesh 14 8 47.1 Honduras 15 3 33.3 China 15 4 57.1 India 15 3 42.9 Mozambique 17 3 14.3 Vietnam 17 3 14.3 Tajikistan 17 3 15.8 Yemen, Republic of 17 5 25.0 Africa 17 3 14.3 Timor-Leste 18 3 27.3 São Tomé & Principe 24 10 71.4 Djibouti 24 5 35.7 Guinea-Bissau 24 3 60.0 Rwanda 25 4 100.0 India 25 3 75.0 Pakistan 25 3 75.0 TOTAL NONCOMPLIANCES 132 77 Annex E Statistical Appendix Table SA.7. Summary of Sample Projects with Three or More Noncompliances Module Number of Noncompliances 3 4 5 OR > PROJECTS 1 Allocation Nicaragua Frontloading X 5 Core SIL Burkina Faso X Yemen X Mongolia X Gambia X 7 Core DPL Mali X Timor Leste X Honduras X 9 Contractual Remedies Soã Tomé & Principe X Djibouti X 10 SIL Legal Regime Haiti X 12 FM SIL Vietnam X Gambia X Mongolia X Serbia-Montenegro X India X Bangladesh X 14 Procurement SIL Mongolia X Bangladesh X 15 Procurement Complaints Honduras X China X India X 17 Loan Management SIL Mozambique X Vietnam X Tajikistan X Yemen X Africa Regional X 18 Loan Management DPL Timor Leste X 24 Loan Management Soã Tomé Principe X Suspensions Djbouti X Guinea Bissau X 25 Loan Management Closing Rwanda X India X Pakistan X Source: IEG calculations from management test results data. Annex E Statistical Appendix Table SA.8. Controls-Related Concentrations (Control Steps with at least three non compliances) No. of Non Non- compliance Module Control Letter Number compliances rate(%) 5 4 C 5 5 33.3 5 4 E 4 26.7 5 5 C 3 23.1 5 6 A 1 3 20.0 5 6 A 2 4 26.7 7 3 B 6 3 42.9 9 1 B (ii)2 3 100.0 12 2 D 5 50.0 12 3 A 1 3 20.0 12 3 A 3 3 33.3 12 4 A 1 8 90.0 12 4 A 7 5 38.5 12 5 A 5 35.7 14 7 A 4 3 23.1 14 8 A 6 54.5 15 2 B 6 60.0 15 2 C 2 3 30.0 15 2 D 2 3 30.0 17 4 A 1 3 37.5 17 4 A 9 6 75.0 17 4 A 10 10 100.0 19 5 A 2 4 40.0 24 1 A 3 75.0 24 1 B 3 75.0 24 2 B 4 100.0 25 1 A 3 25.0 25 1 B 3 25.0 25 2 B 3 25.0 TOTAL NONCOMPLIANCES 117 Source: Management results data Annex E Statistical Appendix Table SA.9. Summary of Controls with Three or More NonCompliant Control Steps Module # and Title Specific Control Number of NonCompliant Control Steps 3 4 5 or > 5 Core SIL Control 4C5 X Control 4E X Control 5C X Control 6A1 X Control 6A2 X 7 Core DPL Control 3B6 X 9 Contractual Remedies Control 1B(ii)2 X 12 FM SIL Control 2D X Control 3A1 X Control 3A3 X Control 4A1 X Control 4A7 X Control 5A X 15 Procurement Complaints Control 2B X Control 2C2 X Control 2D2 X 14 Procurement SIL Control 74 X Control 8A X 17 Loan Management SIL Control 4A1 X Control 4A9 X Control 4A10 X 19 LM Application Review Control 5A2 X 24 LM Suspensions Control 1A X Control IB X Control 2B X 25 LM Loan Closing Control IA X Control IB X Control 2B X Source: IEG calculations from management’s test results Annex E Statistical Appendix Table SA.10. Analysis of the Incidence of Noncompliance by Projects and by Control Steps Summary of Controls with Three or More NonCompliant Control Steps “Vertical Cut” Number of Modules Number of Controls Incidence of Noncompliance Steps 3 4 5 or > 10 28 15 4 9 Number of Noncompliances 117 Percent Of Total Noncompliances 48 Summary of Sample Projects with Three or More NonCompliances “Horizontal Cut” No. of Modules No. of Projects Number of Noncompliant Control Steps 3 4 5 or > 12 34 19 8 7 Number of Noncompliances 132 Percent Of Total Noncompliances 54 Source: IEG calculations from Management test results Table SA.11. Links Identified by Management between Key Controls and the Five COSO Components, shown by Business Function Control Risk Control Monitoring & Information & Module Function No. BPMs Environment Assessment Activities Learning Communication Programming and Lending Products Internal Programming & Allocation 4 1 7 Lending Products 4 6 15 2 1 Fiduciary Services Related to Lending Financial Management 2 13 2 Loan Administration 10 32 Legal 3 22 2 Procurement 3 1 19 Safeguards 2 3 3 1 Quality Assurance QAG Processes 1 7 DISTRIBUTION OF 29 11 118 7 1 COSO LINKS Annex E Statistical Appendix Table SA.12. Distribution of BPMs According to Strategic Relevance and Risk Ranking Risk Categories* 1 2 3 Number Distribution 14 8 7 • IDA, FRM, & Post-Conflict • Corporate Review (ROC/OC) • Project Changes Allocation • Procurement Noncompliance • LOA—Special Commitment • CAS Products • Loan Management—SIL • LOA—Amendment or Extension • SIL—Project Cycle • Loan Management—DPL • LOA—Refund Process • DPL—Project Cycle • LOA—Application Review • LOA—Cancellation Process • Contractual Remedies • LOA—Suspensions • LOA—Closings • SIL—Legal Regime • QAG—QAE and QSA (Standard & Special) • DPL—Legal Regime • Safeguards—OPCQC • FM—SIL • FM—DPL • SIL—Procurement • Procurement Complaints • Safeguards—SIL Average Quality Rating for Business Process Mapping 2.25 2.50 2.50 Average Quality Rating for Testing Methods and Results * STRATEGIC RELEVANCE AND RISK RANKING 1 = Highly relevant, critical: heavy weight in management; major risks; high frequency of occurrence 2 = Relevant, but not critical: average weight in management; some risk; average frequency 3 = Relevant but not critical; moderate weight; moderate or minor risk; infrequent Attachment 1 (1) International Development Association Management Report on Its Review of Internal Controls Part IB Prepared by: Operations Policy and Country Services Vice Presidency Controllers, Strategy and Resource Management Vice Presidency May 29, 2007 (3) CONTENTS Executive Summary ......................................................................................................... (9) Introduction and Background ........................................................................................ (11) Methodology ................................................................................................................... (13) Summary Results of Management’s Assessment .......................................................... (15) Management’s Findings and Recommendations ........................................................... (16) Operating Effectiveness of Key Controls .....................................................................................(16) Deficiencies Identified During Part IB ..........................................................................................(18) Issues Identified and Recommendations Made During Part IA ...............................................(22) Addressing Issues Raised by IAD.................................................................................(22) Addressing IEG Recommendation ...............................................................................(25) Addressing Specific Issues Identified During Part IA by Management, IAD and IEG .....................................................................................(27) Completing Work on the Processing Relating to Debt Reporting, Country Policy and Institutional Assessment and Post-Conflict Performance Indicators ...................(28) Attachment Annexes 1. Management Testing Methodology .......................................................................................(31) 2. Compliance Testing – Sampling Methodology ....................................................................(39) 3. Standard Used in Assessing Deficiencies, Significant Deficiencies and Material Weaknesses..........................................................................................................................(46) 4. OP/BP Status of Updates ........................................................................................................(48) (5) ACRONYMS AAA Analytical and Advisory Activities ACTKD Accounting Department – Knowledge Dissemination Unit ACTRC Accounting Department – Operational Risk and Controls Unit APL Adaptable Program Loan BPs Bank Procedures – a component of the Bank’s Operational Manual CAS Country Assistance Strategy COSO Committee of Sponsoring Organizations – issued an internal control framework CPIA Country Policy and Institutional Assessment CSR Controllers, Strategy and Resource Management Vice Presidency DEC Development Economics Vice Presidency DPL Development Policy Loan ERL Emergency Recovery Loan FA Financing Agreement FIL Financial Intermediary Loan FRM Resource Mobilization Department GCC General Computer Controls IAD Internal Auditing Department ICFR Internal Control over Financial Reporting IEG Independent Evaluation Group IL Investment Lending operations IRIS One of the document retention systems used by the Bank ISG Information Solutions Group ISR Implementation Status and Results report IT Information Technology LAS Loan Administration System LIL Learning and Innovation Loan LOA Loan Department OPCIL OPCS – Investment Lending Unit OPCS Operational Policy and Country Services Vice Presidency OPCQC OPCS - Quality Assurance and Compliance Unit OPs Operational Policies – a component of the Bank’s Operational Manual PAD Project Appraisal Document PCPI Post Conflict Performance Indicators PMT Project Management Team PRSC Poverty Reduction Support Credit QAG Quality Assurance Group QSA Quality at Supervision Assessment SIL Specific Investment Loan SIML Sector Investment and Maintenance Loan SWAp Sector Wide Approach lending operation TA Technical Assistance operation TTL Task Team Leader (7) ATTACHMENT 1 I. EXECUTIVE SUMMARY 1. As reflected in the IDA 14 Replenishment Report1 Management committed to carry out, during the period of IDA 14, an independent comprehensive assessment of its control framework including internal controls over IDA operations and compliance with its charter and policies, and making such assessment available to the public after its disclosure has been approved by IDA’s Executive Directors. 2. In October 2006, Management completed the report on its review and findings relating to Part IA, which examined and identified internal controls that apply to IDA operations at the transaction level and assessed their design effectiveness. The report containing Management’s assessment of the design effectiveness of these controls was included as an Annex to the Independent Evaluation Group’s (IEG) Report, Review of IDA Internal Controls – An Evaluation of Management’s Assessment and the IAD Review, October 18, 2006, AC2006-0099 (the “October 2006 IEG Report”), discussed with the Audit Committee on October 30, 2006. 3. Building on the results and findings of Part IA, during Part IB Management has tested the compliance with 29 business processes and 1152 controls in a sample of documents related to Country Assistance Strategies (CAS), Investment Lending operations (IL) and Development Policy Lending operations (DPL) that were processed through various points in the project cycle during the period under review, i.e. July 1, 2005 to February 28, 2006. As explained in Annex 2, the sampling and testing methodology used by Management did not expand the testing sample – either to see if the frequency of an identified deficiency can be reduced by expansion of a sample size or to verify that effectiveness of a given control in a small sample can be confirmed through a larger sample. The following table summarizes the testing results of the key controls: Table 1 – Summary of Testing Results Controls 3 Activity Total Passed Failed Fiduciary Financial Management 6 4 24 Procurement 10 7 34 Disbursements 13 12 1 Sub-total 29 23 6 Non-fiduciary 86 82 4 Total Controls tested in Part IB 115 105 10 1 See, Report from the Executive Directors of the International Development Association to the Board of Governors, Additions to IDA Resources: Fourteenth Replenishment, Working Together to Achieve the Millennium Development Goals, (approved by the Executive Directors of IDA on March 10, 2005), paragraph 39, under the Disclosure bullet. 2 As reflected in Table 2 of the report, during the performance of the testing steps, Management was not able to test 7 of the 122 controls identified as key during Part IA, thus testing only 115 of the 122 such controls. Three of these key controls were duplicated in other modules, two key controls were deemed to have design issues that precluded testing and two key controls had attributes of a conditional nature which did not occur in the sample selected. 3 Financial management and disbursement controls include fiduciary controls for investment and development policy lending operations. 4 The financial management and procurement controls that management deemed to have failed relate to investment lending operations only. (9) ATTACHMENT 1 4 The impact of the deficiencies and exceptions identified by Management in Part IB is described in detail in paragraphs 25 and 26 of this report. During Part II of this exercise, Management will conduct a “drill down” into the causes and impacts of these exceptions and deficiencies so as to permit an accurate assessment of the operating effectiveness of the key controls where such exceptions and deficiencies were identified and recommend appropriate remedial action. 5. With the completion of Parts IA and IB of Management’s review, Management is of the view that the design and operational effectiveness of identified processes and associated key controls are adequate to ensure compliance with IDA’s polices and procedures that funds are used for the purposes intended, except for: (i) the following significant deficiencies (described in more detail in paragraph 25 and defined in Annex 3)  Timely accessibility of relevant documents  Relevance of regional variances in financial management and procurement guidelines  Not keeping pace with needed updates to OPs and BPs, particularly in the area of investment lending; and (ii) the need to further review the fiduciary controls in the areas of financial management, procurement and safeguards of investment operations by examining the quality aspects supporting the specialists’ inputs. 6. Finally, the testing and evaluation performed under Part IB was aimed at resolving, where possible, the issues and potential deficiencies identified during Part IA by the Internal Auditing Department’s (IAD) review of Management’s Part IA activities, and IEG’s evaluation of Management’s work, as reflected in the October 2006 IEG Report. Management resolved many of these issues. Please refer to the report for further discussion and details. (10) ATTACHMENT 1 II. INTRODUCTION AND BACKGROUND 7. In October 2006, Management completed the report on its review and findings relating to Part IA, which examined and identified internal controls that apply to IDA operations at the transaction level and assessed their design effectiveness. The report containing Management’s assessment of the design effectiveness of these controls, was included as an Annex to the October 2006 IEG Report, discussed with the Audit Committee on October 30, 2006. As set out in the said report, Management has carried out this work by: (a) identifying (i) the key instruments through which IDA carries out its operations (namely, CAS, IL and DPL), and (ii) the policies that govern each such instrument; and (b) mapping business processes and associated controls put in place to operationalize such policies and procedures. 8. Management’s assessment was based on its review of documentation and information relating to the said policies, procedures and related processes, supplemented by extensive consultations with the “sponsors” of the policies, procedures, and processes identified. In addition, to confirm the accuracy of the mapped processes and associated controls, verify how they operate in practice, and help assess the design effectiveness of key controls within these processes, Management conducted extensive walkthroughs with owners/implementers of each such process, that included detailed questions and requests for additional information and clarifications to verify the accuracy and design effectiveness of each such process and associated controls. Building on this work and findings, during Part IB Management assessed the operating effectiveness of the processes and associated controls identified and verified during Part IA. This was done by testing the level of compliance with the said processes and associated controls using a mixture of random and targeted (for steps that did not occur frequently) samples of transactional documents relating to the three main instruments (CAS, IL and DPL) which were subject to the said processes and controls during the period under review, July 1, 2005, to February 28, 2006. 9. This report sets out Management’s assessment relating to operating effectiveness of the transaction-level key controls identified in Part IA as controls designed to ensure compliance with the relevant Articles’ provisions and policies governing IDA's operations. Specifically, the objectives of the compliance testing work performed by Management during Part IB were to: (i) assess the operating effectiveness of the key controls identified during Part IA by testing the level of actual compliance with these controls and their attributes (control steps) as evidenced by relevant transaction-level and other documents provided by the regions and other relevant departments; (ii) resolve issues and potential deficiencies identified by Management’s activities during Part IA, IAD’s review of these activities, and IEG’s evaluation of Management’s work; and (iii) complete the documentation, walkthroughs and testing for three additional processes, namely, Debt Reporting, Country Policy & Institutional Assessment, and Post-Conflict Performance Indicators. Management’s findings relating to each of these three objectives are set out in Part IV of this Report. (11) ATTACHMENT 1 10. It is Management’s understanding that any opinion delivered by IAD and IEG following Part IB may be subject to the outcome of the efficiency and effectiveness assessment that would be conducted in Part II. (12) ATTACHMENT 1 III. METHODOLOGY 11. Management is conducting the assessment of IDA’s internal controls in the context of the COSO5 internal control framework. IDA adopted the COSO framework as its controls methodology in 1995. This framework is widely used by leading financial institutions in the United States and is also seen as a model in many other parts of the world. The COSO framework is an all encompassing process which covers all aspects of internal control of an organization’s operation. It considers not only the evaluation of formal controls, but also informal controls, such as ethics, trust, communication, organization behavior and leadership, and incorporates “top-down” as well as “bottom-up” analysis. Based on the COSO framework, as well as the structure for the annual review of the Internal Controls over Financial Reporting (ICFR) Management assesses the organization’s internal controls, and an independent third party verifies such assessment. 12. As a result of the nature of the commitment made under IDA 14, namely to review compliance with applicable charter provisions and policies in place, Management focused Part I of its assessment on transaction-level controls in order to bring out the crucial details of operational compliance. As noted by IAD and IEG in their respective reports on Part IA, an alternative and a more conventional approach would have been to commence this work with entity-level, rather than transaction-level, controls. Management believes, however, that focusing on transaction-level controls first was called for by the very formulation of the IDA 14 commitment which very much focused on operational or transaction-level compliance. This approach has allowed for a systematic, detailed, and manageable analysis of the transaction-level controls that apply to IDA operations and has facilitated the development of specific and actionable findings that affect the core of IDA’s operations. As explained in paragraph 9 above, as part of this exercise, Management verified the content and design of individual processes and controls mapped by walkthroughs with owners/implementers of each such process. In this regard it is important to note that the financial reporting model, where walkthroughs are usually applied to a transaction sequence from beginning-to-end, required modifications to reflect the substantively and qualitatively different business model that applies to IDA’s operations. Specifically, given that the duration of the “project cycle” (from concept to completion) for the primary instruments tested (CAS, IL, and DPL) lasts between 3-7 years, and the diversity and the wide range of owners/implementers involved in each stage and different aspects of the project cycle, the financial reporting-type “beginning-to-end” walkthroughs was not feasible during the review period. 13. Building on the results and findings of Part IA, which identified 30 business processes (plus 2 sub-processes of Financial Resource Management (FRM) included under Module # 1 – IDA Allocation) and 122 controls and approximately 550 related specific controls attributes that evidence and/or bear upon compliance with the said controls that apply to the three instruments and associated products through which IDA carries out its 5 COSO stands for the Committee of Sponsoring Organizations of the Treadway Commission, which published a report in 1992 titled “Internal Controls – Integrated Framework.” (13) ATTACHMENT 1 operations. During Part IB Management tested the compliance with 296 of these 30 business processes and 115 controls in a sample of documents related to CAS, ILs and DPLs that were processed through various points in the project cycle under the mapped processes during the period under review, i.e. July 1, 2005 to February 28, 2006. Table 2 below reconciles the number of controls and attributes identified in Part IA and those tested in Part IB. For a detailed listing of control and attribute counts by process module please see Annex 1, Attachment 2. Table 2 – Reconciliation of Controls and Attributes Controls Attributes Total Identified in Part IA and through additional documentation performed in Part IB 122 550 Less items not included in testing: Duplicate controls/attributes performed in two or more modules (3) (7) Controls/attributes not tested due to design issues (2) (3) Controls/attributes included in testing 117 540 Controls not tested due to attributes not occurring in sample (2) (10) Attributes, embedded in tested controls, not tested due to the attributes not occurring in sample – (64) Controls tested in Part IB 115 466 14. As noted in paragraph 12 and given the duration of the project cycle that applies to each instrument (particularly ILs which take an average of 7 years from identification to completion), it was not possible to test compliance of a single operation or product such as a CAS, IL or DPL, with each of the processes identified. Instead, compliance with individual processes and associated controls that apply to a particular stage or stages of a project cycle was tested against a sample of operations or products that were identified to have gone through the stage or stages in question during the period under review. As such, the compliance testing performed during Part IB was intended to assess the operating effectiveness of specific controls identified as key to each process. It was not designed or implemented to assess compliance of a given single operation or product with all the applicable controls throughout the “project cycle”. 6 Process # 29 Safeguards – Corporate Risk (QACU) was deemed to be deficient during Part IA and was therefore not testable during Part IB. Management has identified this for further follow-up during Part II. (14) ATTACHMENT 1 15. Recognizing that the key controls embedded in the processes identified represent the end-result of a series of events or “attributes” that bear on the operation of each such control, Management’s compliance testing focused not just on the operation of the key controls but also on evidence of compliance with the specific control attributes. As part of its methodology Management defined a “specific control attribute” as a specific step necessary to achieve the objectives of the key control. While recognizing that some attributes are more critical than others, but having no precedent to guide this assessment, Management elected to subject all identified control attributes to the testing methodology, in order to better substantiate its conclusions7. Management’s methodology is set out in more detail in Annex 1. 16. During Part IB Management also evaluated issues that remained unresolved at the conclusion of the testing for Part IA, including: (i) determination of significance and material impact of any of the issues (actual or potential) identified during Part IA by Management, IAD and/or IEG on IDA’s internal controls; and (ii) identification of appropriate remedial actions, if any, to mitigate related risks in order to fully assess their impact. As explained in the findings section of this report, with respect to some of the issues identified during Part IB, the ultimate conclusion on the operating effectiveness of certain key controls (e.g., in the areas of financial management and procurement reviews) requires additional work to "drill down" into the qualitative aspects associated with these processes. These aspects consist of review of the action plans and recommendations arising from financial management and procurement reviews and their subsequent implementation and resolution by the task team leader or borrower as appropriate. This additional evaluation will be conducted together with the issues accumulated relating to efficiency and effectiveness during the last stage of this evaluation, namely the Efficiency and Effectiveness Assessment under Part II. IV. SUMMARY RESULTS OF MANAGEMENT’S ASSESSMENT 17. With the conclusion of Parts IA and IB of Management’s review, Management is of the view that the design and operational effectiveness of identified processes and associated key controls are adequate to ensure compliance with IDA’s polices and procedures that funds are used for the purposes intended, except for: (i) the following significant deficiencies (described in more detail in paragraph 25)  Timely accessibility of relevant documents  Relevance of regional variances in financial and procurement guidelines  Not keeping pace with needed updates to Operational Policies (Ops) and Bank Procedures (BPs), particularly in the area of investment lending; and 7 As explained in Annex 2, the sampling and testing methodology used by Management did not expand the testing sample – either to see if the frequency of an identified deficiency can be reduced by expansion of a sample size or to verify that effectiveness of a given control in a small sample can be confirmed through a larger sample. (15) ATTACHMENT 1 (ii) the need to further review the fiduciary controls in the areas of financial management, procurement and safeguards of investment operations by examining the quality aspects supporting the specialists’ inputs. 18. The efficiency and effectiveness of the current policy and procedural framework will be carefully and strategically reviewed during Part II of this review. The examination of the quality aspects supporting the specialists’ inputs (identified in paragraph 17(ii) above) during Part II will not review the specialists’ conclusions/ratings but rather will focus on whether the required monitoring, follow-up, implementation and resolution of identified issues were carried out. In addition, Management will review whether documentation exists to support the specialists’ conclusion. Also, Management will document and review the controls of the current Quality of Supervision Assessment (QSA-7) being undertaken by the Quality Assurance Group (QAG) during Part II. Management expects that the results of the overall review including the specific work on effectiveness and efficiencies during Part II, will provide tangible support for a strategic approach to the needed rationalization of the policy and procedural framework that governs IDA’s operations. V. MANAGEMENT’S FINDINGS AND RECOMMENDATIONS A. Operating Effectiveness of Key Controls 19. Management’s compliance testing under Part IB was undertaken to enable Management to assess the operating effectiveness of the internal controls associated with IDA’s operations at the transaction level, where daily decisions are made which have a direct impact on the use of IDA resources. As identified by Management in Table 2 above, during the performance of the testing steps, Management did not test 7 controls identified as key during Part IA: three of these key controls were duplicated in other modules; two key controls were deemed to have design issues that precluded testing; and two key controls had attributes of a conditional nature which did not occur in the sample selected. Please see Annex 1 for a specific example and a listing of the modules that these 7 controls related to. Management therefore tested compliance of 115 key controls identified and it believes that not testing the 4 non-duplicated controls does not impact its conclusion. In addition, due to the sampling methodology applied by Management and the desire to test the same project sample for as many of the control attributes as possible, Management was only able to test 466 of the specific control attributes as 74 attributes did not occur in the samples selected due to a control step, whether conditional or not, not having the preconditions required for the specific test attribute to occur, please see Annex 1 for a specific example and Annex 1, Attachment 1 for details by module. 20. The control attributes typically represent the inputs and outputs generated each time a key control is performed. For example, in the Loan Department (LOA) IL process, key control 1 has been defined as the LOA Finance Officer clearing the Negotiations Package (prior to negotiations). The Negotiations Package includes the following documents (which Management has treated as individual attributes): (i) draft Project Appraisal Document (16) ATTACHMENT 1 (PAD); (ii) draft Financing Agreement (FA); and (iii) Notice of Invitation to Negotiate. The draft PAD and draft FA are documents requiring the review of the LOA Finance Officer and are considered to be key attributes, while the Notice of Invitation to Negotiate does not need the Finance Officer's review and is not considered to be key and therefore not identified as a specific attribute to be reviewed/tested for this key control. Management’s methodology in this respect is set out in more detail in Annex 1. 21. Based on data made available by the regions and other relevant departments responsible for implementation of specific processes (e.g., Development Economics (DEC), Resource Mobilization Department (FRM)) Management was able to locate and review about 93% of all the documents required to test compliance with the relevant processes and associated controls identified during Part IA. The thousands of pages of documentation were organized by the Project Management Team into 63 binders. These binders contain the evidence supporting the test results for the specific control attributes and key controls associated with each process tested. These binders, and Management’s analysis and testing of the documentary evidence, were shared with IAD and IEG. This extensive and rigorous process for accumulating, collating and reviewing these documents and testing for the control attributes, while extremely time and labor intensive, was a very valuable undertaking that has already provided important empirical evidence for the needed rationalization and streamlining of the current processes and associated controls especially in the area of investment lending. 22. The results of compliance testing using these documents and applying the methodology described in Annex 1 indicate that the vast majority of attributes operated effectively and that no exceptions were noted in approximately 93% of the tests performed. However, as noted earlier, the compliance testing performed during Part IB was intended to assess the operating effectiveness of specific individual controls identified as key to each process. It was not designed or implemented to assess compliance of a single given operation or product with all the applicable controls throughout the “project cycle.” Therefore, stating that 93% of the tests passed does not mean that 93% of the projects, processes and products are fully compliant with all policies and procedures that govern such project or product. Instead, Management believes that the compliance testing produced evidence that the vast majority of the specific controls identified are being complied with and are operating as designed. 23. The results have shown, however, several deficiencies or exceptions with respect to operation of specific controls, particularly in the areas of document accessibility, financial management, procurement, safeguards and loan administration. Of the 115 tested key controls mentioned above, Management found 10 (or approximately 9%) of these key controls failed to mitigate the risks associated with the specific control. Management considers that 29 of the 115 key controls are fiduciary controls that ensure funds are used for the purposes intended. Of the 10 failed controls, 6 are fiduciary controls (approximately 21% of fiduciary controls were deemed to have failed by Management). The breakdown of the 29 fiduciary controls (and failures in parenthesis) are in the following activities: Financial management – 6 controls (2 failures); Procurement – 10 controls (3 failures); and Loan disbursement processes – 13 controls (1 failure). The impact of these failures has been recognized as a significant deficiency and discussed in paragraph 25B below and included (17) ATTACHMENT 1 in Management’s deficiency tracker provided to IAD and IEG. For details of the key control failures see Annex 1, Attachment 2 – Control Failures by Process Module. Additional, non-fiduciary deficiencies and exceptions are described in more detail in paragraphs 25 and 26 below. During Part II of this exercise, Management will conduct a “drill down” into the causes and impacts of these exceptions and deficiencies so as to permit an accurate assessment of the operating effectiveness of the key controls where such exceptions and deficiencies were identified and recommend appropriate remedial action. 24. Based on these results, and subject to exceptions and deficiencies noted in paragraphs 25 and 26 compliance testing carried out during Part IB provides substantive evidence that:  The performance-based allocation model is being implemented in a manner that directs scarce IDA resources in support of priority development activities in the poorest eligible member countries.  The complementary use of the three primary instruments for carrying out IDA operations (i.e., CAS, IL and DPL), and evidence relating to the application of the processes and controls that apply to them (from identification to completion), confirm that: o IDA financing is being provided in support of developmental priorities and is focused on matters that appropriately fall within IDA’s mandate; and o consistent with the provisions of IDA’s Articles of Agreement, IDA financing is made available for specific projects as well as other “special circumstances” operations, where appropriate.  The documentation relating to the implementation of both the umbrella and specific (e.g., fiduciary, contractual, safeguards) processes and associated controls that apply to CAS, IL and DPL through all stages of the “project cycle” (from identification to completion) evidenced compliance with the key IDA policies and procedures adopted to ensure that IDA funds are used for the purposes intended.  The documentation relating to implementation of specific processes and associated controls applicable to procurement evidenced compliance with IDA’s procurement policies, as reflected in OP/BP 11.00 and Guidelines: Procurement under IBRD Loans and IDA Credits and Guidelines: Selection and Employment of Consultants by World Bank Borrowers and meet the objective of using IDA resources to finance goods, works and services that were procured with due regard for economy and efficiency, except for some deficiencies and exceptions noted in paragraph 25B. B. Deficiencies Identified During Part IB 25. As a result of Management’s Part IA review certain issues were identified and included in Management’s report. The compliance testing during Part IB shed more light on the issues identified during Part IA and also identified other issues which merit either correction and/or a further drill down during Part II to identify causes and impacts and (18) ATTACHMENT 1 recommend appropriate remedial actions. The following significant deficiencies have been identified by Management at the completion of Part I, and include issues already raised in its Part IA report. For a definition of a deficiency, significant deficiency, or material weakness please see Annex 3. A. Timely accessibility of relevant documents. While after an extensive effort Management was able to obtain approximately 93% of the documentation required to support the processes and controls identified under Part IA, these documents were not easily accessible and Management therefore had difficulties in obtaining such documents in a timely manner. A careful look into the causes of delays with the availability of documents during Part IB identified an issue with respect to consistency of workflow authorization, and accessibility of documents relating to IDA’s operations, both of which merit serious examination and improvement. Given that 93% of the documents requested were ultimately produced, Management believes that the issue is one of documents accessibility, rather than existence or retention. While this indeed constitutes a deficiency that needs to be addressed through a time-bound plan of actions, it does not rise to the level of a material weakness in the internal controls system. Management has already commenced addressing this issue by setting up an expert panel to look at workflow authorization, retention, filing and accessibility of operational documents and come up with recommendations for improvements. These recommendations are expected to be issued by June 30, 2007. The panel is composed of representatives of all the regions, the Information Solutions Group (ISG), Controllers, Strategy and Resource Management (CSR) and Operations Policy and Country Services (OPCS). The panel’s recommendations will be vetted with Senior Management and piloted in one or two key processes before they are mainstreamed to cover all operations. To achieve maximum efficiency and effectiveness, the new workflow authorization and system-based new document retention requirements should be mainstreamed only after the main processes that apply to IDA operations have been rationalized and streamlined following the conclusion and recommendations of Part II of this assessment. B. Variances in regional implementation of institutionally endorsed financial management and procurement guidelines. Management identified that financial management and procurement review processes were not always being performed in accordance with the most recent guidance issued by the appropriate sector board. Specifically, guidelines relating to financial management assessments and supervision reviews and procurement reviews need to be reviewed to ascertain whether these variances are appropriate. To address these issues, the relevant sector boards will: (i) review the documentation requirements necessary to provide evidence of appropriate senior level reviews and clearances, and request the regions to follow the documentation requirements on a consistent basis; (ii) perform a periodic qualitative review of the various supervision processes and output reports to ensure quality of the fiduciary (19) ATTACHMENT 1 work is consistent across IDA’s portfolio and adequate supporting documentation exists for decisions taken during supervision. In particular, the procedures for documenting and communicating the financial management and procurement specialists’ recommended ratings in the Implementation Status and Results (ISR) report should be harmonized across the regions and subject to periodic quality reviews; and (iii) enhance documentation processes of procurement complaints to strengthen effectiveness, and review the centralized complaints database to ensure consistency across the regions in responding in a timely manner to complaints and logging of resolutions. C. Not keeping pace with the needed updates to OPs/BPs, particularly in the area of investment lending. Management has also identified that various OPs/BPs are not keeping pace with the changes needed and/or introduced on the ground. Management, IAD and IEG have flagged this issue in their findings during Part IA and IEG stated that this may be a potential material weakness. As part of Management’s assessment of the effectiveness and efficiency of IDA’s internal control framework, Management intends to look at the current processes underlying policy revision to determine how the policy revision process could be made more efficient so as to facilitate more timely updates of operational policies and Bank procedures. Recommendations for improvement are expected to be issued by December 31, 2007. From the work performed during Part I, Management believes that due to compensatory measures adopted to “fill in” the gaps in the current OPs/BPs, including issuance of Operational Memoranda and interim guidance to staff in areas where gaps have been identified, this does not constitute a material weakness. However, this is only a preliminary conclusion since Management’s ability to fully assess the significance or materiality of the impact of this issue on IDA’s internal control framework will require additional consideration and review during the entity- level review and other activities to be performed during Part II. As part of Management’s review of the OP/BP update process, a table (see Annex 4) has been prepared listing all OPs/BPs that have a direct bearing on IDA’s operations and included in this review, with a summary of update status for each. 26. In addition to the significant deficiencies described above, as part of the detailed review of key controls, process flows (i.e., “walkthroughs”) and compliance testing performed with respect to the individual business processes, Management has identified a number of exceptions and deficiencies in certain areas that merit a closer investigation to assess causes and impact and identify appropriate remedial actions. These include: A. Streamlining of Investment Lending operations. Management’s assessment identified that existing processes and documentary requirements are inefficient and onerous. For example, the compliance tests performed on the modules documenting the pre-supervision activities of SILs (including Legal, financial management, procurement, safeguards and Loan Department activities) – from project concept through Board approval – comprise approximately 20 controls with 95 control attributes. There is a great need to rationalize and streamline the (20) ATTACHMENT 1 existing processes and controls so at to make them more effective and efficient in addressing the key risks. During Part II, Management will look at this issue in a more comprehensive fashion with a view of proposing specific recommendations for such rationalization and streamlining. B. Frequency of corporate reviews of Investment Lending products. The assessment also identified a disparity in the frequency of corporate reviews of IL and DPL operations, with all DPLs being subject to such review but relatively few ILs. To address this issue, on March 25, 2007, OPCS issued a new Guidance Note on Management Review of Investment Operations, intended to put in place appropriate measures to ensure that the level of management review that applies (including criteria for corporate review) is aligned with the level of risks involved in specific investment operations. C. Credit information update and Loan Department clearance processes. Results of Management’s testing identified weaknesses in the LOA’s processes in updating the loan administration system (LAS) on a timely basis and documenting clearances provided. Processes requiring attention include ensuring that: (i) credit master data is created at the time of credit set-up in the LAS, and that the information is consistent with the financing agreement and disbursement letter, especially in the set-up of thresholds for prior review and statement of expenditures. All relevant LOA staff will be reminded to complete all the set-up steps in LAS; and (ii) (a) controls surrounding the LOA Finance Officers’ approvals of notices related to suspensions were not able to be tested as the approvals were not available. Finance Officers will be reminded to file approvals in IRIS. In addition, operations sector Directors will be asked to remind task team leaders (TTLs) that all draft suspension notices should be sent to LOA clearance; the final suspension notice should be sent to LOA for action, and (b) historical audit trails relating to imposing and lifting of suspensions for reasons other than credit repayment by the Borrower (e.g. non-payment related) is not readily available in LAS due to system limitations. The Controller will ask the team responsible for implementing the LAS replacement to include this feature in the new system. This new system (iLAP) is expected to become operational in the third quarter of FY08. CSR management will perform periodic testing of these key controls to ensure they are operating as designed. D. Providing clearances from reviewers. During Management’s walkthrough and testing processes, many instances were identified of required clearances being provided by reviewers to the TTL with conditional or “subject to” comments. Some of the reviewers follow up on whether their comments were taken into consideration, while others stated that it was the responsibility of the TTL to ensure these comments were incorporated in their final document. To address this inconsistency Management will review the need for clearer guidance on the (21) ATTACHMENT 1 division of responsibilities between reviewers and the TTL in ensuring that all relevant comments are reflected in the final documents and/or relevant actions. E. Procedures related to the Safeguards Corporate Risk List. During Management’s walkthrough process the quality assurance and compliance unit responsible for review of safeguards (OPCQC) stated that the screening process for possible inclusion of projects onto the safeguards corporate risk list is based on close consultations between OPCQC and regional staff to determine potential risks and appropriate measures to address them, including potential inclusion on the corporate safeguards risk list. These consultations take place in different ways and on different cycles, depending on the project management system used in each region. Since this was originally designed as a management advisory system rather than as a specific control mechanism, the results of the screening and the subsequent decision to include or remove projects from the list are not recorded in a standardized manner. As a result, management was unable to perform compliance testing on Module # 29 Safeguards – Corporate Risk List during Part IB. Following the findings of Part IA, Management has taken steps to standardize the approach used by OPCQC and the regions in screening and tracking projects on the safeguards corporate risk list and has provided guidance to the regions on this matter. During Part II, Management will revise documentation for module 29 to reflect the new guidance and assess whether sufficient data has been accumulated in order to test its application. F. Debt Reporting Process. Pursuant to OP/BP 14.10, IDA borrowing countries are required to provide IDA with quarterly and annual reports on their external debt. Management’s review verified compliance with the annual reporting requirement, but identified lack of compliance and IDA follow-up relating to quarterly reports. While Management believes that control objectives related to Debt Reporting were achieved despite the issue with quarterly reports, Management will review OP/BP 14.10 with the view to eliminate the current divergence between the OP/BP requirements and the process as it is performed in practice. During Part IB testing Management did not review the process relating to the broader debt sustainability analyses required under IDA 14, including the grant allocation framework based on the level of debt distress. Since these new processes were adopted in July 2006 they fell outside the testing period under review (July 1, 2005 – February, 28, 2006). During Part II Management will document the Process put in place since July 2006 and will test its operating effectiveness provided the process has been applied. 27. In addition to the above, as part of the compliance phase of Part I, Management identified minor issues/exceptions/deficiencies in the processes reviewed and has developed corrective actions to address them. These have been shared with IAD and IEG. Management will update IAD and IEG as to the status of the identified actions at the completion of Part II and will include their status in its final report. C. Issues Identified and Recommendations Made During Part IA (22) ATTACHMENT 1 28. The second objective of the testing and evaluation performed under Part IB was aimed at resolving, where possible, the issues and potential deficiencies identified during Part IA by (i) Management as a result of its own activities, (ii) IAD’s review of these activities, and (iii) IEG’s evaluation of Management’s work, as reflected in the October 2006 IEG Report. As set out below, Management was able to resolve many of these issues. Addressing Issues Raised by IAD 29. The following section describes the key issues identified by IAD as a result of their review of Management’s activities in Part IA together with the recommendations provided by IEG as part of their evaluation of Management’s activities and IAD’s issues from Part IA. 30. The following is a synopsis of the key issues identified by IAD and Management’s response. Issue 1: IDA processes selected (Management’s Methodology) Management Action: The methodology applied by management to rationalize its approach to review compliance with the fiduciary aspects of lending operations in IDA's charter (the work performed in Parts IA and IB) has been described in paragraphs 13 to 24 of Management's Report on its Review of IDA Internal Controls - Part IA, included as Attachment 1 to the October 2006 IEG Report. This methodology has been described in the Revised Work Plan that was shared with IEG, and IAD and discussed with the Audit Committee on July 17, 2006. The initial focus on process and transaction-level controls under the methodology applied makes it difficult to provide an overall definitive conclusion on the effectiveness and efficiency of internal controls over IDA’s operations at this stage. However, Management believes that the staged approach to this exercise was necessary in light of the unprecedented nature of this assessment, the intensity and scope of work required, as well as its primary objective of assessing the existing controls for ensuring that IDA funds are used for the purposes intended. Given the trade-offs, in a resource constrained environment and real time-limitations on what could be done, the approach followed proved to be preferable as the findings, and related action plans adopted to address issues identified, are likely to be the major themes of the entire assessment. Issue 2: Information Technology (IT) controls. Management Action: Management believes that IT controls must be included in its review of IDA's internal control framework. Management has indicated from the beginning of this review that IT controls would be reviewed during Part II. Management will identify all relevant key IT systems impacting the operations with automated compliance controls and document them accordingly. Management believes that few of its current documented processes will be impacted by such review based on the relatively low level of automation associated with most of these processes; however, should review of IT controls under Part II identify the need (23) ATTACHMENT 1 where it will be necessary to re-work any of the existing documentation and compliance testing, Management will do so accordingly and report on its findings during Part II. Management will conduct a form of review of General Computer Controls (GCC) during Part II. Issue 3: Fraud and corruption controls. Management Action: The view that key controls to prevent fraud and corruption should have been specifically identified and assessed by Management takes a narrow view of the control framework needed to ensure that “funds are used for the purposes intended”. Since fraud and corruption are a key cause of misuse of funds, prevention and detection of fraud and corruption have to be embedded and indeed be one of the key aspects of the control framework in place to ensure that funds are used for the purposes intended. At the country level, systemic issues of fraud and corruption and the associated risks are addressed as part of the strategic country dialogue relating to areas of governance, public expenditures and institutional weaknesses, all of which are assessed and reflected as appropriate in the CAS. At the project level, fraud and corruption issues are addressed as part of the review and appraisal processes and are also assessed as part of the fiduciary aspects of project preparation, appraisal, and supervision activities, together with all the fiduciary controls documented as part of the assessment. Management has identified and documented key fiduciary controls which also mitigate the risks associated with fraud and corruption (e.g. LOA’s segregation of duties in review and approval of withdrawal applications, procurement prior reviews and post reviews, country financial assessments, financial management risk ratings, etc.). In addition to work already performed during Part I, Management will use the results of the Bank's fraud and anti-corruption controls program used for the review of the ICFR process, to support its review of the fraud and corruption controls on an IDA entity-wide basis. Issue 4: Outdated Operational Policies (OPs) and Bank Procedures (BPs). Management Action: This point is similar to that raised by IEG (recommendation 2). Updating and streamlining operational policies and procedures governing investment lending operations is the primary focus of the final stage of the modernization agenda which is expected to be completed within the next 18 to 24 months. As part of this process Management has prepared a table (see Annex 4) of all OPs/BPs that have a direct bearing on IDA’s operations and included in this review, with a summary of update status for each. This table has been shared with IAD and IEG. It is expected that the results of this evaluation, including the efficiency and effectiveness review to be carried out under Part II would form and important input into this work. (24) ATTACHMENT 1 Issue 5: Categorization and remediation of deficiencies. Management Action: Management has evaluated the magnitude of any control deficiency (either raised by IAD, IEG or management), together with findings of non- compliance during the operational effectiveness testing and shared its evaluation with IAD and IEG. The key findings are summarized paragraphs 25 and 26 of this report. Issue 6: Document retention and accessibility. Management Action: Management has commenced its review of the issues identified to date by setting up an expert panel to look at document retention, filing and accessibility of operational documents and develop recommendations for improvements. In light of the business units’ ability to provide full documentation during the walkthroughs performed during Part IA, and the high level of documentation obtained as part of the compliance testing in Part IB, the issue appears to be not one of compliance but rather of ensuring the appropriate documentation is easily and readily available to support the various processes and decisions reached. Issue 7: Assessment of entity-level controls. Management Action: The review of entity-level controls will be carried out during Part II. Management will closely consult on the scope of the activities to be included in this review with IAD and IEG by the end of May 2007. It is expected that the review of entity-level controls will include tone at the top, assignment of authority and responsibility, appropriate policies and procedures and various entity-wide programs. Issue 8: Walkthroughs of process documentation. Management Action: Based on discussions with IAD, Management believes that the compliance testing methodology which included testing individual credits through all the controls in a procedure have alleviated IAD's concerns about the design effectiveness by the completion of Part IB. We do not believe this is an issue any longer. Addressing IEG Recommendations 31. The following is a synopsis of the recommendations made by IEG and Management’s response. Recommendation 1: Confirm the validity of the Business Process Mapping cluster. Management Action: In order to confirm that all investment lending operations have the same key controls as Specific Investment Loan (SIL) operations, during Part IB Management included in its compliance testing sample investment loans other than SILs (4 IDA grants, 2 Sector Wide Approaches (SWAp), 2 Adaptable Program Loans (APL), 2 Technical Assistance (TA), 2 Emergency Recovery Loans (ERL) and 1 Sector (25) ATTACHMENT 1 Investment and Maintenance Loans (SIML) operations were added to the test samples). These were reviewed to ensure that the processes and key controls followed for these types of instruments were consistent with those documented for SILs (excluding the special nuances of each product, where applicable). The results of this testing indicated that the majority of controls were the same for all IL products and compliance was consistent across all products. The results have been made available to IAD and IEG as part of the testing. Management’s decision to exclude Analytical and Advisory Activities (AAA) and other Knowledge Products in Part I of this exercise was consistent with its focus and main objective of assessing the internal controls in place for ensuring how borrowers use IDA resources for the purposes intended. Management will review the AAA activities to determine if the key controls for the majority of these activities are consistent and can be easily documented and tested during Part II. Recommendation 2: OP/BP status. Management Action: This point is similar to that raised by IAD (issue 4). Updating and streamlining operational policies and procedures governing investment lending operations is the primary focus of the final stage of the modernization agenda which is expected to be completed within the next 18 to 24 months. As part of this process Management prepared a list of all OPs/BPs that have a direct bearing on IDA’s operations and included in this review, with a summary of update status for each, see Annex 4. This list has been shared with IAD and IEG. As stated above, it is expected that the results of this evaluation, including the efficiency and effectiveness review to be carried out under Part II would be an important input into this work. Recommendation 3: Complete the remaining stages of the IDA Review. Management Action: With the issuance of this report, Management has completed its assessment of the first part of the review of IDA’s internal controls. The testing results for 28 of the 29 modules reviewed were delivered to IAD in separate groups during the period from January 25 through February 9. The last module was delivered to IAD on March 1, 2007. IEG began receiving the modules upon completion of IAD’s review. Management will commence Part II immediately upon issuance of this report. During this part Management will assess whether the overall control framework over IDA’s operations, including corporate governance, entity-level controls and IT controls, is effective and efficient to provide reasonable assurance to Senior Management and the Board that IDA’s operations are carried out in a manner that complies with the provisions of IDA’s charter and internal operational policies. Management will closely consult with IAD and IEG on the methodology and scope of this work by the end of May 2007 so that work can commence and be completed by end-2007. Recommendation 4: Resolve issues and potential deficiencies with IAD. (26) ATTACHMENT 1 Management Action: In completing Part IB, the potential issues identified by Management during its assessment and by IAD’s review have been evaluated to determine their impact on IDA’s internal controls and the remedial actions, if any, that may be required to mitigate risks. Management’s listing of the identified deficiencies and their resolution has been shared with IAD and IEG. Issues of efficiency and effectiveness accumulated from Parts IA and IB will be reviewed and evaluated during Part II. Recommendation 5: Manage the risk framework. Management Action: During the second half of FY07 Management will review the COSO Enterprise Risk Management framework for potential strengthening and or adaptation of the Bank’s (including IDA) existing Integrated Risk Management Framework. This decision is not expected to impact Part II of the review. Recommendation 6: Mainstream internal controls reviews. Management Action: During the second half of FY07 Management will commence discussions with the Audit Committee to consider the value to the Board and Shareholders of adopting a process for periodic or ongoing monitoring and reporting on internal controls in addition to the review of internal control over financial reporting. Addressing Specific Issues Identified During Part IA by Management, IAD and IEG 32. During Part IA Management, IAD and IEG identified specific issues with the documentation and design effectiveness of the identified key controls. Management reviewed all of these issues with IAD and IEG and resolved them as appropriate or in the case of the 41 control design issues indicated in the table below, evaluated the magnitude of the issues and categorized them as deficiencies, significant deficiencies or material weaknesses. Those issues that have been deemed to be issues of efficiency and effectiveness have been carried forward to Part II and will be categorized as part of that process. It should be noted that there is some overlap in the issues identified by the three groups, so the actual total number is less than the simple sums shown in the table below. 33. The following table indicates the preliminary work performed by management in grouping all these issues by type. (27) ATTACHMENT 1 Issues Identified During Part IA By Description Management IAD IEG Total Total 56 59 11 126 Issues resolved and closed (4) (33) (3) (40) Documentation updated - (3) (2) (5) Efficiency & Effectiveness (36) (5) (3) (44) Control design issues for evaluation during Part IB 16 18 3 37 Control Design issues defined as: By-pass existing control 1 2 - 3 Improperly designed control 11 8 3 22 Regional variance 3 - - 3 OP/BP update required 1 8 - 9 16 18 3 37 34. The above table depicts the 126 issues documented by Management, IAD, and IEG during Part IA and is broken out between two levels of categorization:  The first level categorizes issues not by content, but by how they were tagged for resolution: (a) issues resolved and closed required additional information and clarification on the process. The Project Management Team (PMT) followed up with the respective process owners to gather additional information that would satisfy IAD and IEG's inquiries; (b) documentation issues were addressed via updates to the PMT's working papers (process flows and key control sheets); and (c) issues of efficiency and effectiveness would be addressed in Part II; and  The second level categorizes the remaining 37 issues by type of control design flaw as identified in the table. As already discussed, these issues have been reviewed and evaluated by Management as to their impact, individually and in aggregate, on IDA’s objectives. D. Completing Work on the Processing Relating to Debt Reporting, Country Policy and Institutional Assessment and Post-Conflict Performance Indicators 35. The documentation and compliance testing of key controls for the Debt Reporting, Country Policy and Institutional Assessment and Post-Conflict Performance Indicators modules # 30, 31 and 32 respectively, was completed during Part IB. The results of the work performed have been incorporated into the overall Part IB findings and results that have been shared with IAD and IEG. (28) Attachment Annexes (29) ANNEX 1 Annex 1: IDA Internal Control Review Management Testing Methodology 1. Under the methodology outlined below, Management began its compliance testing effort on November 21, 2006, by requesting documentation for the sampled items from the Business Units. The compliance testing was completed on March 1, 2007. Collection of sample documentation 2. For each sample chosen per Management’s Sampling Methodology, which has been shared with IAD and IEG, Management requested documentation from appropriate parties throughout the institution in Washington and Country Offices using standardized documentation lists generated from the Testing Plans (also referred to as Audit Programs). The Testing Plans specify the controls and control attributes to be tested for each process module together with the specific testing methods used. See Attachment 1 for a complete count of controls identified, controls tested, and attributes tested for each process module. An OPCS Investment Lending (OPCIL) Operations Officer collected all documentation held within the regional operations departments while two CSR Business Development Officers (from the ACTKD unit) collected documentation for all other areas, including Loan Management and the other remaining centralized processes. See Attachment 3 for the complete breakdown of documentation collection responsibilities. 3. As documentation was received from the various departments, it was organized by sample and by project or by module, depending upon the process for which it was collected. See Attachment 3 for a complete breakdown. This documentation was reviewed and tagged by the staff that collected the documentation in preparation for testing. Testing 4. Testing of the documentation collected began on January 14, 2007. Testing was performed by a Management auditing team consisting of the following qualified accountants from CSR’s Operational Risk and Control (ACTRC) unit:  Manager  Senior Accounting Officer  Senior Financial Officer/Operational Risk  Financial Officer 5. Following the Testing Plan for each process module, the auditors worked with the staff that collected the documentation to review the documentation and determine whether the requirements of each test attribute for each key control had been met in the performance of the process for the given sample set. The audit team used a Testing Results Matrix (31) ANNEX 1 (“matrix”) for each process module to capture the result of each test performed for each sampled activity. The matrix contains columns corresponding to the Testing Plan tests to be performed, and rows corresponding to the samples chosen for testing. 6. In the course of testing, an auditor may have identified changes required within the Testing Plan. These changes, whether due to redundant steps, lack of clarity or any other reason, were reviewed and cleared with the Manager of ACTRC prior to being put into effect. Any changes made to the Testing Plan were highlighted within the plan for future reference by IAD and IEG reviewers against the original Testing Plans which had been distributed to, and reviewed by IAD and IEG prior to the start of Management’s testing process. 7. For each test that passed, the corresponding cell in the matrix was colored green. For tests that failed initially, the audit team worked with the appropriate Business Unit to determine if additional documentation could be made available or if explanations could be provided to validate the exception. If explanations were provided which were deemed as validating an exception, the explanation and any supporting documentation was filed with the other documentary evidence within the corresponding binder. Any comments useful for reviewers relating to a passed test were footnoted on the matrix. If no documentation could be made available and/or no valid explanation could be provided for the exception, then the test was noted as failing using a red color in the corresponding matrix cell. Where possible, the reasons for failures were also footnoted on the matrix. If a control step, whether conditional or not, did not have the preconditions required for the specific test attribute in the selected sample to occur, it was identified as not applicable “N/A”. In certain instances, the reasons for the step being defined as not applicable were footnoted on the matrix. The following example of this has been extracted from the Investment Lending (IL) project supervision process: Specific Risk Identified Credits are allowed to proceed to signing without signing requirements per IDA policy having been met by the borrower and authorized by appropriate parties in IDA. Specific Control to Mitigate Identified Risk Control 5 – Country Director’s signature of Financing Agreement confirming that the appropriate IDA policies have been complied with. Conditional Control Attribute to be Tested Attribute D – If IDA’s signing requirements have not been met, review evidence to verify that the following 5 steps took place. This conditional step was not tested for any of the 15 IL operations selected in Management’s sample as all of the signing requirements were met within the original timeframe. Therefore these 5 specific control attributes have all been identified as N/A in the Testing Results Matrix. (32) ANNEX 1 Evaluating the Results 8. After all tests were performed and failed tests were identified, the audit team reviewed the reasons for each failed test to determine if the failure represented a minor exception or a deficiency. If the auditor determined that the failure represented a minor exception, the rationale was footnoted on the matrix and the corresponding cell in the matrix was changed to the color pink. If the failure was deemed to represent a deficiency, the color in the matrix remained red. 9. The audit team then created a Summary Sheet for each process module, containing the overall percentage of tests passing for each key control within the process and the evaluation of the exceptions and/or deficiencies identified. The evaluation of exceptions or deficiencies includes a discussion of the reasons for the failure as well as an evaluation of the impact of the specific failure on the objectives of the process. In addition, the Summary Sheet contains the auditor’s overall assessment as to the ability of the testing results to provide Management with reasonable assurance that the objectives of the process are being met, taking into consideration the sum of the reasons for all exceptions and deficiencies found and the specific objectives of the process. 10. Certain processes’ test results did not provide reasonable assurance to Management in the opinion of the concerned auditor. For these process modules, the results of testing and the evaluation of the extent of improvement needed within the process were reviewed with and cleared by the Deputy Controller and Director of Accounting. Distribution to IAD and IEG 11. For each completed process module, IAD and IEG were provided with a documentation package which included the following documents:  Summary Sheet  Testing Results Matrix  Final Audit Program  Process Coversheet with Objective  Process Flow Chart  Process Key Control Sheets 12. The supporting documentary evidence for each process module or sample project was then made available for review by IAD and IEG in office set aside by the Controller. See Attachment 3 for a complete listing of Management’s testing auditor responsibilities and delivery dates for each process module to IAD for their review. IEG will review at their discretion, as part of their work in supporting their Part IB report. (33) ANNEX 1 Control and Attribute Counts by Process Module Controls Attributes Duplicate Not Not # Process Name Identified Controls Tested Tested Identified Duplicate Tested Tested 1 IDA Allocation 4 – – 4 12 – (1) 11 2 IDA Allocation Model1 3 IDA Post-Conflict Allocation1 4 CAS Products 3 – – 3 7 – – 7 5 SIL: Specific Investment Loan 9 – – 9 54 – (6) 48 6 Project Changes 1 – – 1 21 – (9) 12 7 DPL: Development Policy Lending 8 (1) 2 – 7 52 – (13) 39 8 Corporate Review (ROC/OC) 2 – (1) 3 1 4 – (1) 3 3 9 Contractual Remedies 3 – – 3 56 – (22) 34 10 Legal – IL 11 (1) 4 – 10 35 (3) 4 (2) 30 11 Legal – DPL 9 (1) 1 – 8 22 (4) 1 – 18 12 Financial Management – IL 5 – (1) 5 4 27 – (3) 5 24 13 Financial Management – DPL 3 – (1) 6 2 5 – (1) 6 4 14 Procurement – IL 9 – (1) 7 8 46 – (9) 7 37 15 Procurement – Complaints 2 – – 2 12 – (3) 9 16 Procurement – Non-Compliance 2 – – 2 6 – (1) 5 17 LOA – IL 5 – – 5 22 – – 22 18 LOA – DPL 6 – – 6 12 – – 12 19 LOA – Application Review 5 – – 5 32 – (2) 30 20 LOA – SC or Application Problem 1 – – 1 4 – – 4 21 LOA – Amendment/Extension 2 – – 2 7 – – 7 22 LOA – Refund Process 1 – – 1 4 – (1) 3 23 LOA – Cancellation Process 2 – – 2 9 – – 9 24 LOA – Suspensions 6 – – 6 18 – – 18 25 LOA – Closing - Standard 2 – – 2 4 – – 4 26 LOA – Closing - Special 2 – – 2 11 – (2) 9 QAG – Quality at Entry & – – – – 27 Supervision 6 6 17 17 1 These two sub-processes were initially recorded as separate processes. After discussion with CFP management they were deemed to be sub-processes of the IDA allocation process and were consequently integrated into process number 1. We have not revised the numbering scheme. There are 28 processes that have been selected by Management for documentation. 2 One control, and its four associated attributes, was tested as part of the Project Changes (#6) and Contractual Remedies (#9) modules. 3 One control, and its one associated attribute, was identified as a design issue during Part IA and was not tested. 4 One control, and its three associated attributes, was tested as part of the Contractual Remedies (#9) module. 5 One control, and its two associated attributes, was determined not to be a control during the operational effectiveness testing. 6 One control, and its one associated attribute, was not testable as no DPL sample required FM clearance for conditions of tranche disbursement. 7 One control, and its nine associated attributes, was not testable as no evidence of disagreement within the task team was identified that would require the control of escalating disagreements to sector and procurement managers to kick in. (34) ANNEX 1 Continued on following page Controls Attributes Duplicate Not Not # Process Name Identified Controls Tested Tested Identified Duplicate Tested Tested 28 Safeguards – IL 3 – – 3 11 – – 11 30 Debt Reporting 3 – – 3 8 – (1) 7 31 CPIA 4 – – 4 17 – – 17 32 PCPI 3 – – 3 15 – – 15 122 (3) (4) 115 550 (7) (77) 466 Less items not included in testing: Safeguards - Corporate Risk 29 (QACU)1 – – – – – – – – Controls performed in other modules (3) 3 – – (7) 7 – – Controls not tested due to design issue (2) – 2 – (3) – 3 – Controls testable 117 – (2) 115 540 – 74 466 1 Deemed deficient during Part 1A. Not testable during Part 1B, will be followed-up during Part II. (35) ANNEX 1 Control Failures by Process Module Operating # Process Name Tested Effectively Failed Description of Failed Control 1 IDA Allocation 4 4 – 4 CAS Products 3 3 – 5 SIL: Specific Investment Loan 9 9 – 6 Project Changes 1 1 – 7 DPL: Development Policy Lending 7 7 – 8 Corporate Review (ROC/OC) 1 1 – 9 Contractual Remedies 3 3 – 10 Legal – IL 10 10 – 11 Legal – DPL 8 8 – 12 Financial Management – IL 4 2 2 During FY06, the FM Sector Board issued new guidelines for FM practices in Bank-financed investment operations. We could not verify because of lack of documented evidence, and change in Regional practices, that the review and approval of the FM Assessment and appraisal stage PADs and Financing Agreements by the RMFM or appointed delegee occurred. The sample testing, based on date prepared, identified that the majority of FMSRs prepared after November 2005 were not in accordance with the suggested requirements in the FM Guidelines as Regions were transitioning to preparing the FMSRs in accordance with the guidelines. Approx. 40% of the projects reviewed had no documentation evidencing that the risk rating identified by the FM specialist was sent to the TTL for inclusion in the ISR. In one instance we noted the ISR had a different rating from the FMSR - and no explanation was attached. 13 Financial Management – DPL 2 2 – 14 Procurement – IL 8 6 2 Issues in preparation of the Form 384 included: (i) a few months delay in preparing the Form 384 after the contract received date; (ii) the 384 not showing the LAS disbursement categories; (iii) the 384 not corresponding to the no objection letter; and (iv) the contract amount recorded in the 384 was lower than that of the bid documents. Unable to verify that the procurement post reviews were carried out in (36) ANNEX 1 accordance with the timing requirements in the most recent procurement supervision plan or PAD in approx. 40% of our sample, due to lack of documentation provided. In one case we also noted a lack of audit evidence to support the post procurement review and the results from it. 15 Procurement – Complaints 2 1 1 Audit evidence on file was missing to indicate internal review and communication with the borrower in some cases. 16 Procurement – Non-Compliance 2 2 – 17 LOA – IL 5 4 1 Loan master data created at the time of credit set-up in LAS, was not consistent with the financing agreement and/or disbursement letter. The majority of issues related to the set-up of prior review and/or SOE thresholds. 18 LOA – DPL 6 6 – 19 LOA – Application Review 5 5 – 20 LOA – SC or Application Problem 1 1 – 21 LOA – Amendment/Extension 2 2 – 22 LOA – Refund Process 1 1 – 23 LOA – Cancellation Process 2 2 – 24 LOA – Suspensions 6 4 2 Controls surrounding FO approvals of notices related to threat of suspension, suspension, and lifting of suspension were not testable in many cases due to the lack of documentary evidence. Verifiable historical audit trail relating to imposing or lifting of suspensions is not readily available in LAS. 25 LOA – Closing - Standard 2 1 1 Lack of evidence of Finance Officer clearances – documentation not made available/provided for testing. 26 LOA – Closing - Special 2 1 1 Housekeeping of the Credit information in LAS is not always performed in a timely manner. 27 QAG – Quality at Entry & 6 6 – Supervision 28 Safeguards – IL 3 3 – 30 Debt Reporting 3 3 – 31 CPIA 4 4 – 32 PCPI 3 3 – 115 105 10 Less items not included in testing: Safeguards - Corporate Risk 29 (QACU) – – – Controls included in testing 115 105 10 (37) ANNEX 1 Testing Process by Module Binder Document Organized Delivered # Process Name Collection By Auditor to IAD 1 IDA Allocation CSR Module SAO 31-Jan-07 4 CAS Products OPCS Project Mgr 1-Feb-07 5 SIL: Specific Investment Loan OPCS Project Mgr 26-Jan-07 6 Project Changes OPCS Module FO 9-Feb-07 7 DPL: Development Policy Lending OPCS Project Mgr 31-Jan-07 8 Corporate Review (ROC/OC) OPCS Project Mgr 31-Jan-07 9 Contractual Remedies OPCS Module SAO 1-Mar-07 10 Legal – IL CSR Project FO 5-Feb-07 11 Legal – DPL CSR Project FO 5-Feb-07 12 Financial Management – IL OPCS Project Mgr 9-Feb-07 13 Financial Management – DPL OPCS Project Mgr 7-Feb-07 14 Procurement – IL OPCS Project SFO 9-Feb-07 15 Procurement – Complaints CSR Module SFO 9-Feb-07 16 Procurement – Non-Compliance OPCS Module SFO 9-Feb-07 17 LOA – IL CSR Module SAO 2-Feb-07 18 LOA – DPL CSR Module SAO 31-Jan-07 19 LOA – Application Review CSR Module Mgr 26-Jan-07 20 LOA – SC or Application Problem CSR Module SAO 25-Jan-07 21 LOA – Amendment/Extension CSR Module FO 25-Jan-07 22 LOA – Refund Process CSR Module SAO 1-Feb-07 23 LOA – Cancellation Process CSR Module SAO 25-Jan-07 24 LOA – Suspensions CSR Module SAO 31-Jan-07 25 LOA – Closing - Standard CSR Module SAO 30-Jan-07 26 LOA – Closing - Special CSR Module SAO 26-Jan-07 QAG – Quality at Entry & 27 CSR Module FO 25-Jan-07 Supervision 28 Safeguards – IL OPCS Project Mgr 6-Feb-07 29 Safeguards - Corporate Risk (QACU) N/A N/A N/A N/A 30 Debt Reporting CSR Module SAO 1-Feb-07 31 CPIA CSR Module SAO 6-Feb-07 32 PCPI CSR Module SAO 31-Jan-07 Auditor Legend: Mgr - Manager SAO - Senior Accounting Officer SFO - Senior Financial Officer FO - Financial Officer (38) Annex 2 Annex 2: IDA Internal Control Review Compliance Testing – Sampling Methodology OVERVIEW 1. To perform compliance testing on the control activities within all documented process modules for Part I of Management’s review of IDA internal controls, sampling was required for both projects/operations as well as several processes with activities performed outside of the project/operation cycle. This document describes Management’s methodology for all required samples including:  Core Investment Lending (IL) and Development Policy Lending (DPL) operations  Country Assistance Strategy (CAS) products  Other non-lending processes SAMPLE SELECTION OF IL AND DPL OPERATIONS 2. The methodology for selecting projects/operations for IDA Internal Control Review compliance testing is outlined below for the primary sample of Specific Investment Loan (SIL) and DPL products as well as for additional IL types with lower lending volumes than the SIL product. A Core IL Operations Sample 3. Based on the universe identified for IL operations (see the sections on SIL and additional IL operations below), the required number of projects to be selected for testing of Investment Lending operations was determined by Management to be 15 each for pre- supervision and supervision activities. This provided a sample size equivalent to the median between the applicable audit requirements applied by IDA’s external auditor used for the review of internal controls related to financial reporting for Weekly and Monthly frequency of activity. 4. Management randomly selected 10 projects for the SIL product (10 projects for pre- supervision activities and 10 different projects for supervision activities) from the SIL Sample Universe as described below. An additional 5 projects, for both pre-supervision and supervision, of other IL operations were selected randomly from the IL Sample Universe as described below to provide the required sample size of 15 for all IL operations. 5. Selection of the pre-supervision and supervision projects for IL operations was performed on the basis of approval date. Projects approved during FY06 formed the pre- supervision sample universe for each operation. Projects approved prior to FY06 formed the supervision universe for each operation. This enabled Management to ensure as far as possible that the operational control activities being tested would have occurred during the testing timeframe for this exercise – July 1, 2005 to February 28, 2006. (39) ANNEX 2 SIL Operations – Identifying the Universe # Projects Starting Project Universe: Downloaded all projects from the 14,776 Operations Portal on March 8, 2006. Remove all projects where: (i) status is non-Active; (ii) projects are non-IBRD/IDA; (iii) IDA Commitment amount is Zero; (iv) not a Development Policy Loan or Specific Investment Loan that was approved since January 1, 2001; and all DPL, PRSC, Sector Adjustment Loan, Structural Adjustment Loan. (14,369) Projects constituting the SIL sample universe for Management’s assessment 407 SIL sample selected for testing: 20 operations (10 pre-supervision and 10 supervision) Additional IL Operations – Identifying the Universe # Projects Starting Project Universe: Downloaded all Non-SIL investment lending projects from the Operations Portal on November 16, 4,060 2006. Remove all projects where: (i) status is non-Active; (ii) projects are non-IBRD/IDA; (iii) IDA Commitment amount is Zero; (iv) all projects that were approved after FY06 (3,770) Projects constituting the Additional IL sample universe for Management’s assessment 290 The additional IL operations were further broken down by the following types and 5 individual samples were selected from the more numerous types below: # Projects Sample Adaptable Program Loan (APL) 119 2 Technical Assistance (TA) 73 1 Emergency Recovery Loan (ERL) 50 1 Sector Investment and Maintenance Loan (SIML) 23 1 Learning and Innovation Loan (LIL) 17 None Financial Intermediary Loan (FIL) 8 None 290 5 6. Please see Attachment 1 for a listing of the selected SIL and additional IL operations included in Management’s testing process. (40) Annex 2 B. Core DPL Operations Sample 7. Management documented the procedures associated with the current policy regime for DPL operations under the revised OP/BP 8.60, issued in August 2004. Compliance testing for these procedures focused on operations governed by this current policy regime during the timeframe for testing in this exercise, July 1, 2005 through February 28, 2006. However, DPL operations with a Concept Review date of August 9, 2004, or earlier are not required to comply with the new OP/BP 8.60. These operations continued to be governed by the previous policy regime under OD 8.60. 8. Management conducted a review of all 21 DPL operations that met the following criteria: a. Active during the testing timeframe and approved in FY05 but processed under the new policy OP/BP 8.60; or b. Approved during the testing timeframe, including those that also then closed during the testing timeframe. 9. Management used random and targeted sampling in selecting the 7 DPL operations, or 33% of the total universe, for testing purposes. Please see Attachment 1 for a listing of the selected DPL operations included in Management’s testing process. SAMPLE SELECTION OF COUNTRY ASSISTANCE STRATEGY OPERATIONS 10. Management identified 13 IDA country assistance strategy (CAS) products with a Board Date within the testing timeframe for this exercise. Management decided that a sample of 5 CAS products, representing 38% of the total universe, was acceptable for testing. The sample for the CAS process was judgmentally selected by the Regions after OPCS requested that the Regions identify CAS products developed during the testing time frame. Management collected documentation for the 5 CAS products during its Inception-to- Completion walkthroughs, and relied on the documentation collected for these CAS products for its testing. 11. The sample of CASs tested by Management included 2 CASs in ECA, 2 CASs in SAR and one CAS in the AFR region. SAMPLE PROJECTS SELECTED FOR OTHER MODULES 12. Not all modules were able to be fully covered by Management’s sampling methodology used for selecting test samples for the IL, DPL and CAS products. Specifically, the following modules required targeted sampling based on specific sample universes: 1: IDA Allocation 8: Corporate Review 15: Procurement Complaints 16: Procurement Noncompliance 19: LOA Application Review (41) ANNEX 2 20: LOA Special Commitment or Application Problem 21: LOA Amendments 22: LOA Refunds 23: LOA Cancellations 24: LOA Suspensions 25: LOA Closing – Standard Procedures 26: LOA Closing – Special Procedures 27: QAG - Quality at Entry Assessment (EA) and, - Quality at Supervision Assessment (QSA) 30: Debt Reporting 31: Country Policy and Institutional Assessment (CPIA) 32: Post Conflict Performance Indicators (PCPI) 13. These modules are either entirely independent, or have processes that are not part, of the project cycle for IL or DPL products, and occur either once per year or random time cycles depending on their conditionalities. For each of these modules, Management requested documentation for any/all occurrences of the activity using the Core IL/DPL sample within the testing timeframe. 14. If results provided at least 5 samples of each activity, Management tested these samples and considered this an adequate sample for activities that are not deemed to be occurring on a daily basis. If results did not provide 5 samples of each activity, Management considered whether additional targeted/judgmental sampling was required to indicate a key control or specific control attribute was operating adequately. Where additional samples were deemed to be necessary, Management relied upon available system reports or other search methods to identify occurrences of the activity. Such methods, where used, were specifically documented to enable re-performance and validation. 15. The individual sampling methodologies for each of these modules, together with those of the IL, DPL and CAS products have been provided to IAD and IEG by Management. (42) Annex 2 IDA Internal Control Review Compliance Testing – Sampling Methodology Listing of Operations Selected for Core Investment Lending (IL) and Development Policy Lending (DPL) Testing Sample of IL Operations Selected for Testing of Pre-Supervision Controls Type Region Country SIL AFR Mozambique SIL AFR Mozambique APL AFR Africa APL AFR Gambia SIML AFR Burkina Faso SIL EAP Vietnam TA EAP Mongolia ERL EAP Lao PDR SIL ECA Tajikistan SIL ECA Georgia SIL LCR Haiti SIL MNA Yemen SIL SAR Sri Lanka SIL SAR Pakistan (43) ANNEX 2 Sample of IL Operations Selected for Testing of Supervision Controls Type Region Country SIL AFR Rwanda SIL AFR Mozambique SIL AFR Africa APL AFR Ethiopia APL AFR Rwanda SIL EAP Mongolia SIL EAP Lao PDR Serbia and SIL ECA Montenegro Serbia and SIL ECA Montenegro SIL LCR Honduras TA LCR Guyana SIL MNA Yemen SIL SAR India ERL SAR India SIML SAR Bangladesh (44) Annex 2 Sample of DPL Operations Selected for Testing of Controls Type Region Country DPL AFR Mozambique DPL AFR Uganda DPL AFR Mali DPL AFR Cameroon DPL EAP Timor-Leste DPL ECA Armenia DPL LCR Honduras Summary of Sample Operations by Region Listed below are the total number of IL (by type) and DPL operations sampled for each of IDA’s operational regions: Product AFR EAP ECA LCR MNA SAR Total SIL 6 3 4 2 2 3 20 APL 4 – – – – – 4 TA - 1 – 1 – – 2 ERL - 1 – – – 1 2 SIML 1 – – – – 1 2 DPL 4 1 1 1 – – 7 Total 15 6 5 4 2 5 37 (45) Annex 3 Annex 3: IDA Internal Control Review Standard Used in Assessing Deficiencies, Significant Deficiencies and Material Weaknesses 1. The Bank is performing its assessment of internal controls over external financial reporting using existing auditing standards on attestation of internal controls over financial reporting as prescribed by generally accepted auditing standards. In performing its review of compliance with IDA’s charter and applicable internal policies and procedures, Management has used the same concepts as those defined in the Auditing Standard No. 2 (AS2) An Audit of Internal Control Over Financial Reporting Performed in Conjunction with An Audit of Financial Statements, issued by the U.S. Public Company Accounting Oversight Board (PCAOB) as much as possible. 2. Management believes that applying the concepts that have been defined by audit standard setters for assessing internal controls over financial reporting will provide the level of comprehensiveness, rigor and consistency required in its self-assessment of internal controls and compliance with IDA’s charter and applicable internal policies and procedures. 3. During Management’s review items that represent deficiencies and which may or may not require remediation were discovered. A control deficiency exists when the design or operation of a control does not allow management or employees, in the normal course of performing their assigned functions, to prevent or detect noncompliance on a timely basis.  A deficiency in design exists when (a) a control necessary to meet the control objective is missing, or (b) an existing control is not properly designed so that, even if the control operates as designed, the control objective is not always met.  A deficiency in operation exists when a properly designed control does not operate as designed, or when the person performing the control does not possess the necessary authority or qualifications to perform the control effectively. 4. Control deficiencies are classified as one of the following: (i) an internal control deficiency; (ii) a significant deficiency16; or (iii) a material weakness17. The classification of the deficiency is based upon the likelihood of occurrence/noncompliance and/or the significance of noncompliance. 16 AS2 defines a significant deficiency as a control deficiency, or a combination of control deficiencies, that adversely affects the company’s ability to initiate, authorize, record, process, or report external financial data reliably in accordance with generally accepted accounting principles such that there is more than remote likelihood that a misstatement of the company’s annual or interim financial statements that is more than inconsequential will not be prevented or detected. 17 AS2 defines a material weakness as a significant deficiency, or combination of significant deficiencies, that results in more than a remote likelihood that a material misstatement of the annual or interim financial statements will not be prevented or detected. (47) ANNEX 3 5. Conclusions about what constitutes a material weakness over compliance or operations are judgmental, more so than in the case of material weaknesses in financial reporting. Therefore, the definition of material weakness has been adapted from the context of the financial reporting definition, with its reliance on materiality in relation to the financial statements, to one using more judgment as to whether the operations and compliance objectives of internal control are met. To guide financial auditors in making these judgments, AS2 identifies examples of attributes the auditor should consider in evaluating identified internal control deficiencies to determine whether the deficiencies, individually or in combination, are significant deficiencies or material weaknesses. Management, IAD and IEG have agreed that clearly defined measures be established for judging operational materiality. These measures have been used as guides by each of the three groups in determining whether identified internal control deficiencies in compliance constitute significant deficiencies or material weaknesses. Identified deficiencies could be significant deficiencies or material weaknesses where the control deficiencies have attributes that could:  impair the achievement of IDA’s objectives;  violate requirements of IDA’s charters or other contractual agreements;  significantly weaken safeguards against waste, loss, or unauthorized use of funds, property, or assets;  involve conflicts of interest;  involve systemic problems in country assistance, partnerships and project lending; and  require the attention of Senior Management, the Board as well as the awareness of external stakeholders. 6. All deficiencies identified during Management’s assessment have been recorded on a summary deficiency schedule. This deficiency schedule outlines Management’s assessment of the deficiency (type of deficiency), any mitigating controls over the deficiency, the potential financial impact, if any, the impact from a non-financial perspective, and management’s determination of how to address the deficiency, i.e. corrective action (remediation). A control deficiency or combination of control deficiencies that, in management’s judgment, represent significant deficiencies in the design or operation of internal control that could adversely affect the organization’s ability to meet its internal control objectives is a “Significant Deficiency”. A significant deficiency or a combination of significant deficiencies that Management determines to be significant enough to be reported outside IDA shall be considered a “Material Weakness”. 7. Management’s report on Part IB includes its assessment of the overall deficiencies and a determination on the impact these deficiencies have individually and in total on the internal controls over IDA’s compliance with its charter and applicable internal policies and procedures. This report includes: (i) Management’s assessment of IDA’s compliance with its charter and applicable internal policies and procedures; and (ii) a description of any significant deficiencies or material weaknesses identified through its assessment, together with their respective remediation plan. (48) ANNEX 4 Annex 4: IDA Internal Control Review OP/BP Status of Updates VOLUME I: STRATEGIES AND PRODUCTS Country Focus Latest Revision In Pipeline to be Updated  2.11 - Country Assistance Strategies BP June, 2005 Possibly in the next 6 months, if determined that changes needed to reflect the GAC action plan Business Products and Instruments  8.60 - Development Policy Lending OP August, 2004 No BP August, 2004  10.00 - Investment Lending: Identification to Board Presentation OP June, 1994 Yes, a proposal for BP June, 1994 modernization, Annex A-Outline for an Investment Project January, 1994 consolidation and Information Document rationalization of this Annex B-Elements of a Project January, 1994 OP/BP is expected to be Implementation Plan presented to the EDs in the Annex C-Operational Plan Contents January, 1994 next 18 to 24 months to Annex D-Outline for a Staff Appraisal January, 1994 reflect lessons learned Report from experience and Annex D1-Outline for Preparing the "Loan January, 1994 reflect recommendations and Project Summary" of IDA internal controls Annex D2-Standard Disbursement Profiles January, 1994 review Annex E-Outline for a Memorandum and January, 1994 Recommendation of the President Annex F-Sample Notice of Invitation to January, 1994 Negotiate Annex G-Telex of Invitation to Negotiate January, 1994 Annex H-Notice of Status of Negotiations January, 1994 Annex I-Loan/Credit/GEF Grant Cover January, 1994 Sheet Annex J-Conditions of Board Presentation January, 1994 Annex K-Streamlined Procedures for Board January, 1994 Presentation (49) ANNEX 4 Latest Revision In Pipeline to be Updated VOLUME II: PROJECT REQUIREMENTS Safeguard Policies  4.01 - Environmental Assessment OP January, 1999 No Annex A-Definitions January, 1999 Annex B-Content of an Environmental January, 1999 Assessment Report for a Category A Project Annex C-Environmental Management Plan January, 1999 BP January, 1999 No Annex A-Environmental Data Sheet for January, 1999 Projects in the IBRD/IDA Lending Program Annex B-Application of EA to Dam and January, 1999 Reservoir Projects Annex C-Application of EA to Projects January, 1999 Involving Pest Management  4.04 - Natural Habitats OP June, 2001 No Annex A-Definitions June, 2001 BP June, 2001 No  4.09 - Pest Management OP December, 1998 No  4.10 - Indigenous Peoples OP July, 2005 No Annex A-Social Assessment July, 2005 Annex B-Indigenous Peoples Plans July, 2005 Annex C-Indigenous Peoples Planning July, 2005 Framework BP July, 2005 No  4.11 – Physical Cultural Resources OP July, 2006 No BP June, 2006 No  4.12 - Involuntary Resettlement OP December, 2001 No Annex A-Involuntary Resettlement December, 2001 Instruments BP December, 2001 No  4.36 – Forests OP November, 2002 No Annex A-Definitions November, 2002 BP November, 2002 (50) ANNEX 4 Latest Revision In Pipeline to be Updated  4.37 - Safety of Dams OP October, 2001 No BP October, 2001 No Annex A-Dam Safety Reports: Content and October, 2001 Timing Fiduciary  10.02 – Financial Management OP March, 2007 Revised OP/BP 10.02 BP March, 2007 effective March 22, 2007 Annex A-Review of Financial Management March, 2007 Systems Annex B-Sample Telexes: Accounting; March, 2007 Financial Reporting; and Auditing Annex C-Audit Reports Compliance System March, 2007  11.00 – Procurement OP July, 2001 Yes. OP and BP to be BP July, 2001 revised in the next 18 Annex A-The World Bank Procurement October, 2002 months to reflect lessons Function learned and recommendations of the internal controls review.  12.00 – Disbursement OP March, 2007 Revised OP/BP 12.00 BP March, 2007 effective March 22, 2007 Annex A-Valuation of Disbursements and March, 2007 Changes in Exchange Rates  12.20 - Special Accounts OP May, 1994 Has been replaced by Annex A-Required Bank Characteristics January, 1994 revised OP 12.00 on Annex A1-Sample Comfort Letter from January, 1994 March 22, 2007. Commercial Bank Holding Special Account Annex B-Subaccounts and Second- January, 1994 Generation Special Accounts BP May, 1994 Has been replaced by revised BP 12.00 on March 22, 2007.  12.30 - Statements of Expenditure OP September, 1993 Has been replaced by BP August, 1993 revised OP/BP 12.00 on March 22, 2007. (51) ANNEX 4 Latest Revision In Pipeline to be Updated Financial  12.10 - Retroactive Financing OP July, 2002 Has been replaced by an OP Memo on expenditure eligibility for projects in countries without approved country financing parameters on March 27, 2007. Management  13.05 - Project Supervision OP July, 2001 Yes, OP and BP to be BP July, 2001 revised in the next 18 to 24 months to reflect lessons learned and recommendations of the internal controls review Contractual  3.10 - Financial Terms and Conditions of IBRD Loans, IBRD Hedging Products, and IDA Credits OP June, 2003 Annex A-Past Loans of IBRD June, 2003 Annex B-Prepayment of IBRD Loans June, 2003 Annex C-Countries Ranked by Per Capita July 2006 Updated at least 2-3 times Income a year Annex D-IBRD/IDA Countries: Per Capita July 2006 Updated at least 2-3 times Incomes, Lending Eligibility, and a year Repayment Terms BP June, 2003  7.00 - Lending Operations: Choice of Borrower and Contractual Agreements OP February, 2001 No  13.00 - Signing of Legal Documents and Effectiveness of Loans and Credit OP February, 2002 Possibly, if changes needed based on results of IDA internal controls review BP February, 2002 Same as above re OP  13.30 - Closing Dates OP February, 2002 Possibly, if changes needed based on results of IDA internal controls review BP February, 2002 Same as above re OP (52) ANNEX 4 Latest Revision In Pipeline to be Updated  13.40 - Suspension of Disbursements OP February, 1996 Currently under review. BP February, 1996 Revisions expected in the Annex A-Sample Notice to the Borrower January, 1996 next 6-12 months for a Suspension Unrelated to Payment Annex B-Sample Notice to the Executive January, 1996 Directors for a Suspension Unrelated to Payment Annex C-Sample Notice of Cancellation January, 1996 Sent to the Borrower Annex D-Sample Notice of Cancellation January, 1996 Sent to the Executive Directors Annex E-Sample Notice to the Borrower for January, 1996 Lifting a Suspension Annex F-Sample Notice Sent to the January, 1996 Executive Directors for Lifting a Suspension Unrelated to Payment Annex G-Sample Notice to the Borrower January, 1996 When Payment Is 30 Days Overdue Annex H-Sample Notice of Impending January, 1996 Suspension Sent to the Borrower When Payment Is 45 Days Overdue Annex I-Sample Notice of Suspension Sent January, 1996 to the Borrower When Payment Is 60 Days Overdue Annex J-Sample Notice of Suspension Sent January, 1996 to the Executive Directors for a Payment- Related Suspension Annex K-Sample Notice to the Executive January, 1996 Directors for Lifting a Payment- Related Suspension  13.50 – Cancellations OP August, 2002 Possibly, if changes needed based on results of IDA internal controls review BP August, 2002 Same as for OP above (53) ANNEX 4 Latest Revision In Pipeline to be Updated  14.10 - External Debt Reporting and Financial Statements OP October, 1999 No BP October, 1999 No Annex A-Sample Letter on Financial and August, 2006 Economic Data: IBRD Borrowing Country Annex B-Sample Letter on Financial and August, 2006 Economic Data: IDA Borrowing Country Annex B1 - Countries Subject to IDA's August, 2006 Non-Concessional Borrowing Policy Annex C-Sample Letter of Representations January, 1999 regarding a Borrower's/Project Entity's Financial Condition (54) Attachment 2 IAD Review of Management’s Assessment (55) ATTACHMENT 2 (56) ATTACHMENT 2 (57) ATTACHMENT 2 (58) ATTACHMENT 2 (59) ATTACHMENT 2 (60) ATTACHMENT 2 (61) ATTACHMENT 2 (62) ATTACHMENT 2 (63) ATTACHMENT 2 (64) ATTACHMENT 2 (65) ATTACHMENT 2 (66) ATTACHMENT 2 (67) ATTACHMENT 2 (68) ATTACHMENT 2 (69) Attachment 3 (71) ATTACHMENT 3 STATEMENT BY THE EXTERNAL ADVISORY PANEL TO THE INDEPENDENT EVALUATION GROUP (IEG) IN RELATION TO THEIR REVIEW OF IDA CONTROLS 1. Terms of Reference. The Terms of Reference for Panel members referred to IEG’s conduct of an independent evaluation of the self assessment being completed by management (the Controller’s Department) and the subsequent review to be conducted by the Internal Audit Department (IAD). Panel members were to be provided with a selected set of materials as background reading to familiarize themselves with the nature of overall task. As well, once the IEG evaluation study reached the stage at which its findings were incorporated in a draft report. Panel members working individually, were expected to review the draft report and participate in a Panel discussion to be held in Washington, DC. That discussion would involve the IEG evaluation project team and selected members of IEG management. The major purpose for involving the Panel was to have to provide an independent opinion on the IEG evaluation approach and findings directly to the Board of IDA. This it would do in a Letter of Comment. It would be a relatively brief statement, focusing on highlighted issues, but would not be constrained in any way from making observations on any aspects of the overall review it may deem relevant, whether these may pertain to the overall review itself, or to the part that was played by IEG in making its evaluations. 2. Timing and Coverage of the Panel’s Review The Panel conducted its review in the five days from 5-9 March 2007 in Washington, DC. The review covered mainly the almost completed Part I: Processes and controls applicable to the fiduciary aspects of IDA (The Compliance Assessment) which had been divided into two parts—Part IA: Process Mapping and Design Effectiveness; and Part IB: Testing of Controls Operation. The Panel reviewed and discussed the IEG Report on the Completion of Part IA (dated October 16, 2006) and the Preliminary Draft IEG Report on the Completion of Part IB. As well, the Panel provided comments on IEG’s Thoughts on Part II of the IAD Internal Control Assessment. The Panel had access to the draft Management report on its Review of Internal Controls—Part IB. The Panel did not benefit from sight of the IAD Part IB report, since this was not yet ready for release. 3. Working Papers and Documents The Panel was provided with a range of papers and documents for information as well as to enhance its understanding and comprehension of the issues and the approaches, methodologies and techniques used. A full set of these documents as well as working papers with comments and cross references to the former have been given to IEG. (73) ATTACHMENT 3 4. Related Background The Panel’s attention was drawn to the then United States General Accounting Office Report to Congressional Committees (June 2003) entitled “World Bank Group—Important Steps taken on Internal Control but additional assessments should be made”. In that report, at least two relevant observations were made for this review as follows (Page 3): ‘Bank Group management does not include an assertion on internal control over operations and compliance with key provisions of its bank charters and policies, and it has not asked the external auditor to give an opinion on those internal controls.’ And, ‘The Bank Group’s external financial statement audits do not, and are not intended to, provide specific assurance about the internal control over the Bank Group’s operations and whether the funds are spent for their intended purposes. Given the inherent users in the Bank’s activities, additional assurance on these other categories of internal control—operations and compliance—would provide an added level of assurance to the Bank Group and its member countries that funds were used for their ‘intended purposes’. In the IDA14 Replenishment Report approved by the Executive Directors of IDA on March 10, 2005, Bank management ‘committed to carry out an independent comprehensive assessment of its control framework including controls over IDA objectives and compliance with its charter and policies’ (Page 39). The interpretation of this commitment and its impact on the nature and direction of the review have been, and will continue to be, of central importance to a successful outcome. In a report entitled ‘Integrated Risk Management Framework and Implementation’ as requested by Executive Directors during a joint Audit Committee and CODE meeting on May 15, 2002, it was stated that: ‘Development effectiveness is the central management challenge for the Bank. It requires a focus not just on getting the best results from ongoing activities but also on choosing the right activities and scale to engage in-taking appropriate risks’ (Page 1). The report goes on to say that: “The Board faces a wide range of increasingly complex risks. Sound risk management—to meet the dual requirement of development effectiveness and resource mobilization—must be comprehensive. To ensure that opportunities are not missed and the full range of risks is considered, it helps to organize risk management around four focal areas: strategic effectiveness, operational efficiency, stakeholder support, and financial soundness’ (Page 1). (74) ATTACHMENT 3 Finally, ‘But it is also crucial to take into account the interactions among the different types of risks. So these four focal areas have to be treated only as an organizing principle, as part of a whole,’ (Page 1). The Panel also noted the ‘IBRD and IDA FY06 COSO year-end report’ prepared by the Controllers, Strategy and Resource Management Vice Presidency of October 11, 2006 and comments on the Risk and Opportunity Workshops providing the: ‘ability to take the institution’s pulse’ on the control environment and other aspects of risk management and to ensure their findings become an integral part of the IRM effort on risk awareness and measurement (Page 19). This observation has implications that go well beyond financial reporting controls. It also reflects the interrelationship between risk and control and the requirement for cost/benefit assessments. 5. Panel Observations. The Panel was impressed by the professionalism, competence, understanding and commitment of the IEG personnel involved in the evaluation. We also met with the Auditor General and two other senior IAD staff on two occasions as well as with a senior group of management personnel. These were full and frank discussions. We also had an opportunity to meet with the Chair of the Audit Committee. Details of the meetings are included in the Working Papers. The Panel notes that there were different views on the so-called bottom up approach to Part I as opposed to a top down approach starting with entity level controls. In emphasizing, for Part I, the fiduciary aspects of IDA controls, and while keeping within the COSO framework, Management decided to approach its assessment by focusing on the business processes that IDA performs on a daily basis in the execution of its operations. The Panel understands the pragmatic approach taken for Part I while recognizing that final conclusions will be made on the review after the completion of Part II. In fact it may be necessary to vary some of the conclusions of Part I as the result of the work on Part II. The strength of the approach in Part II is the top level strategic focus reflecting decisions made about the application of the overall integrated COSO risk management framework and associated entity-wide controls within the governance arrangements that reflect both ‘tone at the top’ and the authority and accountability which is assigned to the review and any agreed outcomes. The necessary leadership is reinforced by both top down and bottom up approaches with the resultant commitment at all levels of the organization. Any initiatives taken should reflect the culture of the organization, in particular by strong values and ethics. Identified risks and associated controls can be better integrated in a cohesive, consistent and transparent manner which should result in better understanding (75) ATTACHMENT 3 and commitment to the framework throughout the Bank. That is, controls and risk treatments should be coincidental business process and entity levels and mutually reinforcing. The Panel notes the extent and value of work done on Part I which is in the final stages of completion. What it has done is to provide information about business processes and operation that was not readily available in the past. This has led to better understanding of the business and provides a useful basis for future analysis and strategic decision making. Some concerns were expressed, particularly by IAD, about a lack of documentation. The Panel also notes the views that, in most cases, documentation was available but difficult to discover/access. In that respect, it strongly supports the Management initiative on Enterprise Content Management to improve documentation, link directly with work flows and automate. The Panel agrees that documentation is a common problem across both the private and public sectors and is generally being given a higher priority both to facilitate decision-making and to enhance accountability. The Panel supports the use of suitable standards against which accomplishment/performance can sensibly be assessed. This is a discipline that all such reviews should involve and is an essential element of quality assurance. The Panel notes the Management’s comment, that “The audit standards to be used define the degree of rigor in controls assessment and testing”. Management indicated that it used a set of standards based on concepts which are contained in the United States Audit Standard 2 (AS2) criteria. These standards are as directed by the Sarbanes-Oxley legislation (SOX) and were designed to strengthen financial reporting under the COSO framework. The standards agreed by the Management, IAD and IEG in assessing deficiencies, significant deficiencies, and material weaknesses are appended as Attachment 2 to the Approach Paper submitted to the Audit Committee on July 12, 2006. The Panel also notes the differences of view between Management and IAD about the use of concept of “walkthroughs” under the above standard. The IEG has broadly crafted the management approach as meeting the intentions of the standard. However, this is an issue that is still to be resolved between Management and IAD. The Panel supports the requirements for an evidential trail which can be provided in alternative ways, for example by informed expert comment. But what this does raise is the question of the use of such standards as a guide for exercises of this kind or as a prescriptive element of the review. The Panel would be inclined to the notion of a Guide. The Panel also supports the use of the INTOSAI Guidelines for Internal Control Standards for the Public Sector as suggested in the paper ‘Thoughts on Part II of the IDA Internal Control Assessment’ given the nature and constituency of the Bank with both its public and private sector profiles. Another common issue for both the private and public sectors is achieving a suitable balance between conformance and performance imperatives as a central element of good governance. Put simply, the challenge is to provide both adequate assurance and achieve required results. They are not alternatives and sound risk management frameworks assist those in charge of governance to decide on an appropriate balance at specific points in time and over time. In the Panel’s view, this is what makes Part II of the review so important in building on the extensive investment in Part I. The latter has been largely about compliance (76) ATTACHMENT 3 or conformance, the former is now a challenge to focus more on what is to be achieved both efficiently and effectively. The Panel observes that there are different skills required for financial statement, assurance and performance audits. The Auditor General, not surprisingly, has a strong focus on the former and the related controls and risk factors. The Panel accepts that IAD may also have some capacity to undertake reviews of administrative effectiveness. While noting the issue of ‘independence’ in reporting in the relationships with the President and the Board which gave IEG the review role in this exercise, the Panel also notes that Part II may well involve issues of policy effectiveness which are more within the province and scope as well as the competence of IEG. The Panel understands the problems that can arise with apparently overlapping functions and responsibilities and the need for open, as well as close, consultation and cooperation to achieve required outcomes. Indeed, this was reflected in a degree of frustration in the Audit Committee last year which all parties are determined will not be repeated. This is not a matter of achieving agreement at the expense of informed debate on the issues but rather ensuring that respective roles are recognized as well as the contributions made, facts are agreed and differences of opinion are clarified so that informed decisions can be made. This requires mutual trust, respect and cooperation in a genuine ‘partnership’ to achieve the best results for the organization. The above issues are important for the timely achievement of Part II. As noted earlier, Part I is in the final stages of completion with a focus on the testing in Part IB. This work is necessary as part of good quality assurance and has been important element of the learning process. While there remain some matters of detail to be resolved which reflect on the confidence in the statistical sampling methodology and interpretation of a number of the elements of the business processes, the Panel notes IEG’s view that the approach may well have wider benefits for future understanding of the various interrelationships and structures of those processes and their resulting efficiency and effectiveness, including risk assessments and controls. The Panel notes that Management, IAD and IEG will meet to resolve and clarify some issues and concerns in the process. Moreover, there is confidence that the Board will receive considerable assurance that the processes and controls put in place are largely working as intended. Part II will provide an opportunity to make improvements in the existing arrangements but also, hopefully, provide insights which could result in different approaches and achieve better results. In discussions with Management the Panel understood that there will be some changes and enhancements to their Draft Report on Part IB (of February 28, 2007). The Panel was encouraged by the response and a more helpful report should result. While the IEG Report on the completion of Part IB is still work-in-progress, the structure and content are supported. Timing is a factor. As noted earlier, there are issues and queries to be resolved between the main parties—Management, IAD and IEG. Management stressed that Part IB is an ‘interim assessment’. The Auditor General may not give an opinion on Part IB but more likely will opine on whether Management has done sufficient work to justify their conclusions and whether their report is fairly stated. It is probably fair to say that, at this (77) ATTACHMENT 3 date, there is a greater distance to travel for agreement between IAD and Management than between Management and IEG. 6. Overall Conclusions The Panel is impressed by the professional approach and commitment of the three major parties to the review—Management, IAD and IEG. Considerable effort and resources have been dedicated to the task. The Panel is not in a position to conclude on the cost/benefit of what has been done to date. However, it notes the paucity of available information on similar exercises elsewhere in areas where many organizations are dealing with demands for better controls, greater assurance, and better outcomes—whether more cost effective programs or more shareholder value. There has been a greater focus on governance frameworks and on associated organization culture that reflects high level values and an ethical approach with greater transparency and associated accountability. The Panel strongly supports adherence to the COSO Integrated Risk Management framework. This requires considerable ongoing support from the top of the Bank and commitment at all levels of the organization. The Panel also notes the initiatives taken over a number of years in relation to that framework, particularly in the area of controls for financial reporting and the Bank’s financial statements. These are not covered in the reviews basically because there has not been any adverse audit opinion expressed on them, as the Panel understands. Nevertheless, they are an important part of the integrated framework which is increasingly covering both non-financial as well as financial information. In particular, the Panel notes the growing importance of Information Management and Communication controls in the ever increasing technological era and the associated risks which have often added to management complexity as well as facilitating organizational effectiveness. The Panel wishes all parties well in what is now likely to be the more complex part of the reviews. Hopefully, the efforts to date will facilitate the conduct of Part II but timing will continue to be a challenge, particularly if it is considered that the result should be available for Replenishment 15 discussions early in 2008. The issue of timing is also important for any effective conversion to be made between business process controls reviewed in Part I and entity controls in Part II. The nature of Part II is likely to demand considerable time and commitment of many Senior Executives and Board members. A number of clear milestones need to be established and the risks to the timetable managed accordingly. Greater emphasis will need to be placed on cooperation and timely information exchange as well as on key decision points. Issues about priority will need to be addressed at the outset. No doubt Part II will be facilitated by the insights and lessons learned, from Part I. An early sharing of these across the key participants would facilitate a successful outcome. Patrick Barrett Bjarne MØrk-Eidem Vijay Shunglu (78)