DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8D-FFC94685AE00 Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 DECEMBER 8, 2020 REPORT NO. FY21-2-4-2104615 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed w ithout World Bank Group authorization. DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 Distribution Issued by: Anke D'Angelo, Vice President and Auditor General , GIAVP OIAk.t- O' a~ To: David Malpass, President, World Bank Group, EXC ■ REPORT NO. FY21-2-4-2104615 2 DocuSign Envelope ID: 3FD6ED33-467D-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 1. EXECUTIVE SUMMARY 1.1 Background On August 27, 2020, World Bank management announced reports of data irregularities in the Doing Business 2018 report (published in October 2017) and the Doing Business 2020 report (published in October 2019). Development Economics Vice Presidency (DEC) management conducted a review of data changes in the Doing Business (DB) reports from 2016 to 2020 in response to the data irregularities reported by the DB team. The review was completed in December 2020 and the results of the review have been described in the report: “Review of Data Irregularities in Doing Business”. DEC management defines the data irregularities as “data changes that were made outside of the appropriate review process and were not justified by the DB methodology or by any new information provided to the DB team”. For details on the review process followed by management, refer to Appendix 2 DEC Management Review Process Flow. The management review was limited to data changes that occurred after the circulation of data as part of the institutional review process 1 and before the final publication of the corresponding DB reports. The review was carried out by the current DEC management (who has been in place since June 2020, and who did not have managerial responsibility for the production and publication of the DB reports covered in this review), with inputs from the DB team. As requested by the World Bank Group President’s Office (EXC), Group Internal Audit (GIA) conducted an independent verification of the DEC management review. The detailed objectives, scope, and approach are provided in the Terms of Reference in Appendix 3. Separately, at the request of the Board and DEC management, GIA completed an assurance review of “Data integrity in the production process of the Doing Business Report” (FY21-2-2- 2103656). The report was issued on December 8, 2020. 1 After the internal reviews within the DB team and DEC management, the draft DB report including the underlying data is shared with wider WBG staff and management for feedback. This institutional review process is also known as the Bank Wide Review (BWR). REPORT NO. FY21-2-4-2104615 3 DocuSign Envelope ID: 3FD6ED33-467D-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 1.2 Overall Conclusion GIA verified that DEC management systematically reviewed all data changes that occurred after the institutional review process for the DB reports from 2016 to 2020 in accordance with the process described in its report: “Review of Data Irregularities in Doing Business” (version dated December 3, 2020). GIA concluded that the review process as defined by management is sufficiently complete to identify data irregularities. DEC management concluded that these data changes were reasonable except for nine changes, affecting four countries, which were reported by the DB team as data irregularities. The nine confirmed data irregularities are corrected in the Data Management System 2 (DMS) and documented in DEC management’s review report. GIA’s verification identified one data discrepancy (refer to verification objective 1 in section 2) and potential deviations (refer to verification objective 2 in section 2) in the DEC management’s review process. GIA verified that management has addressed the data discrepancy and potential deviations. GIA’s verification did not cover the overall accuracy of the data and DEC management’s technical judgement to determine changes as reasonable. The high-level conclusions of GIA’s verification of DEC management’s review are presented in section 2. The Detailed Verification Results are described in Appendix 1. 2 DMS is the system used by DEC where the DB data is processed, reviewed and approved for final publication. REPORT NO. FY21-2-4-2104615 4 DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 2. Summary of Verification Results GIA verified each of the six verification objectives as outlined in the TOR (refer to Appendix 3). The table below includes GIA's initial observations based on its verification work, DEC management's subsequent actions to address GIA's observations, and GIA's conclusion after verification of management's subsequent actions. The detailed verification steps and results are presented in Appendix 1. Subsequent GIA Verification Objective GIA's Initial Observation Management Action Conclusion 1 DEC management Management identified all data DEC management Verified identified all data changes changes except one in 0816 included the 0816 that were made after that was excluded due to an data change with an DEC's circulation of the oversight (human error) in data explanation in the DEC DB data for the retrieval. management review institutional review report and confirmed process prior to its this change was not a publication data irregularity. 2 DEC management DEC management reviewed DEC management Verified reviewed each of the each of the data changes and updated the changes to identify those confirmed nine changes lacking explanations for the lacking justification reasonable explanation. DEC four changes identified according to clearly management used their by GIA. DEC defined and verifiable technical judgement and the management criteria changes reported as potential confirmed that the four irregularities by the DB team as changes are not data criteria to identify changes irregularities. lacking justification. In addition, GIA identified four changes, two in 0816 and two in 0817, with incomplete explanations as potential deviations. 3 DEC management DEC management reviewed the No subsequent action Verified. reviewed all changes nine changes lacking was required. GIA did not, lacking justification reasonable explanation with however, according to clearly follow-up conversations and assess DEC defined and verifiable emails with the DB team and management's confirmed them as irregularities. technical In addition, five data changes in judgement. 0816 and an inconsistent application of DB methodology for a specific country in DB 17 REPORT NO. FY21-2-4-2104615 5 DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 Subsequent GIA Verification Objective GIA's Initial Observation Management Action Conclusion criteria, and identified were flagged by the DB team in irregularities 3 their emails to DEC management as potential irregularities. DEC management had review ed these changes before GIA's verification and reconfirmed that they are not data irregularities as no specific improprieties were identified. 4 DEC management has Management confirmed and No subsequent action Verified decided and implemented corrected the nine irregularities w as required. corrective actions to in OMS. The corrections were in address the data accordance with management's irregularities in review process. accordance with management's review process 5 The DB data in excel The DB data in excel annexed No subsequent action Verified annexed to the to the management review w as required. management review report matches the data in the report matches the data in excel master files. the DB database 6 The impact of the The recalculated country No subsequent action Verified correction of data on the rankings are accurate based on w as required. country ranking mentioned the corrected data irregularities in the management report by DEC management. is accurate following the existing DB scoring method 4 3 The wording of the objective as stated in GIA's TOR is changed from "DEC management investigated all changes lacking justification according to clearly defined and verifiable criteria, and identified irregularities" to "DEC management reviewed all changes lacking justification according to clearly defined and verifiable criteria, to better reflect management's terminology. Refer to Appendix 3 for the Terms of Reference. 4 The wording of the objective as stated in GIA's TOR is changed from " The impact of the correction of data on the country ranking mentioned in the management report is accurate following the algorithm set by DEC managemenr to " The impact of the correction of data on the country ranking mentioned in the management report is accurate following the ex isting DB scoring method"to better reflect management's terminology. Refer to Appendix 3 for the Terms of Reference. REPORT NO. FY21-2-4-2104615 6 DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 Appendix 1: Detailed Verification Results This section describes for each of the six verification objectives as defined in the Terms of Reference in Appendix 3: (i) the results of the verification exercise; and (ii) the steps taken by GIA to verify each of the six objectives. 1. DEC management identified all data changes that were made after DEC's circulation of the DB data for the Bank-Wide Review prior to its publication DEC management identified all data changes except one related to 0816 that was excluded due to an oversight (human error) in data retrieval. Subsequently, DEC management included the DB 16 change with an explanation in the DEC management review report and confirmed this change was not a data irregularity. GIA matched more than 41 ,000 data points between the draft DB Report (PDF) circulated for the institutional data review process and the published DB Report (PDF) for 0816, 0817, 0818, 0819 and 0820. The PDFs were obtained independently by GIA from the emails circulated for the institutional review process and directly from the internet where the DB reports are published for all the years (0816 to 0820). GIA identified one additional data change as follows: Year Country Topic/Indicator Change• 1 DB 16 Swaziland Getting credit: Private bureau coverage 41.3% to 46.4% (% of adults) *Within the change description, the first value is as reflected in the draft DB Report (PDF) circulated for institutional review process and the second value is as reflected in the published DB Report (PDF) for the respective DB year. 2. DEC management reviewed each of the changes to identify those lacking justification according to clearly defined and verifiable criteria DEC management reviewed each of the changes and confirmed that nine lacked reasonable explanations. DEC management used their technical judgement and the changes reported by the DB team as potential irregularities, to identify changes lacking justification. In addition, GIA identified four further changes, two in 0816 and two in 0817 with incomplete explanations as potential deviations. DEC management subsequently updated the explanations for the four changes to provide further clarifications. DEC management confirmed the four potential deviations were not considered as data irregularities. GIA reviewed the explanations for each of the changes provided either at the time of the change or updated at the time of the management review and compiled a list of those changes with REPORT NO. FY21-2-4-2104615 7 DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 missing or incomplete explanations. GIA verified the completeness and not the technical soundness of the explanations. The four additional changes are as follows: Year Country Topic/Indicator 1 DB17 Brunei Darussalam Dealing with Construction Permits 2 DB17 Malaysia Paying Taxes 3 DB16 Sri Lanka Getting Electricity 4 DB16 Belgium Paying Taxes 3. DEC management reviewed all changes lacking justification according to clearly defined and verifiable criteria, and identified irregularities DEC management reviewed the nine changes lacking reasonable explanation with follow-up conversations and emails with the DB team and identified them as data irregularities. In addition, five changes in DB16 and an inconsistent application of DB methodology for a specific country in DB17 were flagged by the DB team in their emails to DEC management as potential irregularities. DEC management had reviewed these changes based on subsequent follow-up discussions with the DB team before GIA's verification. DEC management reconfirmed that these changes were not data irregularities as no specific improprieties were identified. DEC management's criteria for identifying irregularities are: (i) changes without reasonable explanations; and (ii) changes flagged by the DB team as potential irregularities. For criteria (i), GIA verified the changes with missing or incomplete explanations as stated in verification objective #2 above. For criteria (ii), GIA verified email communication between the DB team and DEC management to identify concerns flagged by the DB team . GIA also interviewed four out of the 10 Topic Leaders to substantiate information in the emails. GIA did not assess DEC management's technical judgement. REPORT NO. FY21-2-4-2104615 8 DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 The five changes in DB 16 are as follows: Year Country Topic/Indicator 1 0B16 Indonesia Getting Electricity 2 0B16 Slovakia Getting Electricity 3 0B16 India (Mumbai) Getting Electricity 4 0B16 Chile Getting Electricity 5 0B16 Mexico Getting Electricity In addition, concerns were raised by DB team in the treatment of India during DB17 regarding the inconsistent application of the DB data collection methodology for the "Dealing with Construction Permits" indicator. These concerns were raised during DEC management's review. Management reconfirmed that they were not data irregularities during GIA's verification. 4. DEC management has decided and implemented corrective actions to address the data irregularities in accordance with management's review process DEC management confirmed and corrected the nine irregularities in OMS. The corrections were made in accordance with management's review process. GIA verified in OMS that the data was corrected by DEC management in accordance with management's review and approval process, as of the date of this verification. The review process includes a three-layer approval in OMS by the topic leader, peer reviewer and the data manager. The nine topics with corrected data irregularities are as follows: Year Country Topic/ Indicator DB20 Azerbaijan Getting Electricity Number of Procedures Time (days) 2 DB20 Azerbaijan Enforcing Contracts Quality of Judicial Process Index 3 DB20 Azerbaijan Trading Across Borders Time to export: border compliance (hours) Time to import: border compliance (hours) Time to export: documentary compliance (hours) Time to import: documentary compliance (hours) Cost to export: documentary compliance (USO) 4 DB20 Saudi Arabia Legal Rights (Getting Credit) Strength of Legal Rights Index 5 DB20 Saudi Arabia Paying Taxes Time (hours) REPORT NO. FY21-2-4-2104615 9 DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 Year Country Topic/Indicator 6 DB20 United Arab Emirates Paying Taxes Time (hours) 7 DB 18-20 China Starting a Business Procedures (Beijing) Procedures (Shanghai) Time in days (Beijing) Time in days (Shanghai) 8 DB 18-20 China Legal Rights (Getting Credit) Strength of Legal Rights Index (both cities) 9 DB 18-20 China Paying Taxes Time in hours (Beijing) Time in hours (Shanghai) 5. The DB data in Excel annexed to the management review report matches the data in the DB database The DB data in excel annexed to the management review report matches the data in the excel master files 5. GIA reconciled the yearly DB data between the five excel files annexed to the DEC management review report and the corresponding data from the master files for DB 16-DB20 downloaded from the DB database as explained below: • GIA compared more than 96,000 data points in the five excel files annexed to the DEC management review report with the Masterfile that was circulated for the institutional review process and the Masterfile that was used for the published DB reports for each of the five years. • All changes identified in this comparison were identified by DEC management and verified by GIA without any discrepancy. • GIA verified that each of the data changes identified by DEC management had a corresponding explanation for the change without any discrepancy. 6. The impact of the correction of data on the country ranking mentioned in the DEC management review report is accurate following the DB scoring method The recalculated rankings for the four countries that were impacted by the irregularities are accurate. 5 GIA matched the DB data in the annexed excel to the contemporaneous master files (downloads of the data from the DB database at the time of DB report circulation for the institutional review and final publication). REPORT NO. FY21-2-4-2104615 10 DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 GIA independently verified the new ranking of the four economies affected by the data irregularities, using the corrected scores as explained below: • GIA obtained the historical score and ranking for all economies on November 17, 2020 from the Data Series as published in the Doing Business website. • GIA verified the new ranking for the four affected countries using the corrected score instead of the historical score, by retaining the data constant for all other countries. • GIA compared the new ranking verified by GIA with the ranking reported by DEC management in its review report. GIA verified the mathematical accuracy of the country rankings applying the existing scoring method followed by DEC management. GIA did not verify the underlying scoring method. The corrected score and ranking of the countries are as follows: DB Year Country Irregular score and ranking Corrected score and ranking 1 0820 Azerbaijan Score: 76.7; Ranking: 34 Score: 78.5; Ranking: 28 2 0820 Saudi Arabia Score: 71.6; Ranking: 62 Score: 70.9; Ranking: 63 3 0820 United Arab Emirates Score: 80.9; Ranking: 16 Score: 80.8; Ranking: 16 4 0818 China Score: 65.3; Ranking: 78 Score: 64.5; Ranking: 85 REPORT NO. FY21-2-4-2104615 11 DocuSign Envelope ID: 3FD6ED33-467D-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 APPENDIX 2: DEC Management Review Process Flow Click HERE for an enlarged PDF version. Source: GIA’s analysis based on meetings and documents provided by DEC management. REPORT NO. FY21-2-4-2104615 12 DocuSig, Envelope ID: 3FD6ED33 4670 4854 8F8D FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing ~ 1 WORLD BANKGROUI: Business Reports from 2016 to 2020 ~ Group lntemol Audit APPENDIX 3: Terms of Reference :::1: A1C71 ll!C-0370-◄ ,7E-37llE-M i50 1C71A6A Oocu5'gn : nvelope 1 ~ WORLD BANK GROUP OFFICE MEMORANDUM DATE: Novermer 6, 2020 TO: David Malpass. President. Wor1d Bank Group, EXC FROM: Anke D'Angelo, Vice President and Auditor General, GIAVP 8v.kt. Q•a~ EXTENSION: 34181 SUBJECT: Tenns of Reference for the Independent Verification of the Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 Please find attached our Terms of Reference (TOR) for the above-mentioned engagement. The engagement wm be conciJcted by If you have any questions regarding this TOR or if we can be of further assistance, please do not hesitate to contact me Attachments REPORT NO. FY21-2-4-2104615 13 DocuSign Envelope ID: 3FD6ED33-467D-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing WORLD BANK GROUP Business Reports from 2016 to 2020 Group Internal Audlt oc S ~ E v pe D. A • 99 •O D-4 E-9 9E-'I! 50 1C "A6A lndepende \lerific.ation ot t he n.agement Rev·ew of Data rregularities in the Doi "(l Business Reports from 2016 to 2820 e nce T:erms of ReJer, I ntroduction On Augus 27, 2020, World Ba management announced data ·rre ulariti es in the Doing Bu · ess 2018 and oj g B siness 2020 reports published ·n October o 20117 sind 2019, res pee ·ve . In response, DEC management conducted a rev·e "'of dat a, i1Tegularities in the Doi g 81.1$lness Re.p()rts from 2016 to 2020. Tile management review was led b a Develo ment Economics Vice Presidency (DEC) Director and oomple ed in October 2020 In th is review, management analyze d d'a1a irregularities an d decided ow ey wit be corrected in the Doi g Bus, ess 2021 repo . As per the req est fmm the Worlld Bank Group Pres·dent's Office (EXC), Group lntemal Aud" (GIA) wil con duct the independen verification o 1he management review co mpleted in October 2020. l !ngagement Objectives The primary -01:ljectives of th is engagement are: (i o re ie e process o lov1ed by management ·n their re ·ew • Of DB data changes and, eri i or otherwise comment on its comp e eness: 0i) verify tha management c-0nd1.1c edits re iew of DB data charnges in accordanoe • h the review process indicated in its report and there ore reached reasonably defons be oon usions: and ("i) oo,ifir · tha he corrections made to the DB data are oonsisten wfth the conclusions of the re ie . Specifically, GIA's eriflca ion \. i confirm whether: 1. DEC management i de i ed all data changes tllat were made a· er DEC's circu ation o the DB da a fo r the Bank-Wi de Revie prior o its publica io11; 2_ DEC management , reviewed each of the changes to identi~• hose lacking justifica ·on according to dearly defined and verifiable ori eria; , 3. DEC management in estigated all changes lacking justification accorcling o clear1 defined and verifiable c • eria. and "dentified irregularities; 4. DEC N a.n. agement has decided and implernen ed corrective actions to address the data irregularities in accordance :with management's review process; 5. The OB data n Excel annexed to the managemen review report matches e data in the DB database; and 6 The ·mpact o the conection of da a ori he co try ra n i119 mentioned in the managen e t repo is accurate fol owing the alg;orithm set by DEC managememt. GIA wil also assess the adequacy of coritro s in the management r ev·e •; process. REPORT NO. FY21-2-4-2104615 14 DocuSign Envelope ID: 3FD6ED33-4670-4B54-8F8O-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing Business Reports from 2016 to 2020 OoeuSga E.cvuope D . AIC7 ' 000-03"O-411E.S TQE~2501C71ABA Independent Venfication of the Management Review of Data /i1f;. , WORLD BANK GROUP Irregularities in the Doing Business Reports from 2016 to 2020 w . . Engagement Scope The scope of verification covers activities management undertook to complete the management review of data irregularities for the DB report from 2016 to 2020. The following will NOT be included in the scope of verification: Adequacy of management's technical judgement to detemrine data irregularities; however the review will seek to confirm that clear and verifiable criteria were defined and applied accordingly; 2 Adequacy of management's actions to correct data Irregularities; however, the review will seek to confirm that corrections were implemented for all irregular data changes deterrrrined under the review process, 3. Overall acruracy of the updated DB data following the management review; 4 Adequacy of the underlying DB methodology to coDect analyze. and aggregate data; and 5 Any potential data irregularities that were introduced in the concerned historical data sets for the 2016 to 2020 DB reports before the Bank-Wide Reviews and after the publlcation of the DB reports Engagement Approach The engagement will involve; Obtaining details on the approach used in the management review to understand the activities involved and assess the adequacy of controls, 2. Obtaining documented evidence from management that the activities described in the approach took place; 3. Conducting data analytics to independently identify data changes between the data used for the Bank-Wide-Review and the published data: and 4 Conducting data analytics to confirm that all corrective changes via the management review are accurately reflected in the Excel files atlached to the management report. Administration and Reporting The engagement commenced on November 4, 2020 with a target date for the draft report by November 30, 2020. The Final Report will be issued soon thereafter. REPORT NO. FY21-2-4-2104615 15 DocuSign Envelope ID: 3FD6ED33-467D-4B54-8F8D-FFC94685AE00 Verification Report of Management Review of Data Irregularities in the Doing WORLD BANK GROUP Business Reports from 2016 to 2020 Group Internal Audlt ! O Ci ASA I depe den Verificmlon o the Management Rev e of Data lmgulaf ·e1 i 1he Doing Bu iness Report from 2016 to 2020 Appendix 1: Report Distribution 1e dra and fina l reports identifying ey issues ~I be addressed to the President andi copied to the fo o Ning stakell o' ders: Unit Title MDOPS ~ anaging Di rector, Opera on£ MDOPS Direc or, s a egy a d Ope a ·ons MDDPP ~ anag1ng Di recto r MDDPP Direc or, sv ategy a d Ope a ·o s MDDPP Special Assistant DECVP Vice Presiden a cl Chie· Eco omisl DECVP Directo r, o den , a.nd Ch1 ice o Vice P1es1 e Economist ECRVP Vice Presiden Key c ent a.nage ent and s aJf REPORT NO. FY21-2-4-2104615 16