Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) Report Number: ICRR0023144 1. Project Data Project ID Project Name P092429 STATE STATS SYST 2 Country Practice Area(Lead) Russian Federation Poverty and Equity L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-48670 25-Aug-2012 9,963,039.08 Bank Approval Date Closing Date (Actual) 28-Jun-2007 31-Dec-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 10,000,000.00 0.00 Revised Commitment 9,963,039.08 0.00 Actual 9,963,039.08 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Clay Wescott Jennifer L. Keller IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement (p. 6) and the Project Appraisal Document (p. 5), the project's objectives were "to assist the government of the Russian Federation to produce reliable, timely, and accurate economic, financial, socio-demographic, and other data for policy formulation and decision-making in line with international good practice and frameworks." The objectives were not revised during implementation. Outcome targets were revised to be more ambitious as additional government financing was provided for the project, and new intermediate indicators were introduced to reflect added activities, but the outcome indicators Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) themselves remained essentially the same (with some adjustments for clarification). A split rating is not required. For purposes of this Review, the objectives are defined as follows: 1. Produce reliable, accurate economic, financial, socio-demographic, and other data in line with international good practice and frameworks 2. Produce timely economic, financial, socio-demographic, and other data in line with international good practice and frameworks 3. Produce data for policy formulation and decision-making b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The PAD (p. 2) noted that, given Russia's large fiscal surplus at appraisal, the project's value lay in its facilitation of Russian access to global knowledge and skills through analytical and advisory support and through bringing together different stakeholders and parts of government and forging consensus on key issues. The project had five components: A. Modernization of economic statistics methodology (appraisal: US$3.35 million, of which US$0.71 million financed by the Bank; actual: US$6.51 million). This component was to promote harmonization of the System of National Accounts (SNA) with international standards, including the introduction of internationally compatible classifications; development of institutional sector accounts; construction of assets and liabilities balance sheets for non-financial assets; improvement of capital accounts data; improvement of constant price estimates by enhancing the Consumer Price Index (CPI), Producer Price Index (PPI), and other price indices and use of double-deflation methods; and development of the methodological basis for the compilation of basic input-output tables built on the newly-introduced classifications and results of a 2006 agricultural census. With additional government financing to the project, new activities were added, including the development of methodologies for constructing tables for calculation processes, accounting for intellectual property in national accounts, business statistics, centralized assessment of macroeconomic indicators, compiling environmental economic accounts, estimating natural resources productivity, compilation of assets accounts for fossil mineral and energy resources, and treatment of enterprise groups. B. Development of modern design and technology for the statistical data collection, processing, and dissemination system (appraisal: US$34.42 million, of which US$6.77 million financed by the Bank; actual US$103.85 million). This component was to support the Federal Service of State Statistics of the Russian Federation's (ROSSTAT's) strategic goal of improving data survey development, data gathering, data analysis, and data dissemination processes through the use of information technology (IT). The component Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) was to support ROSSTAT's implementation of new infrastructure developed during the predecessor State Statistical System (STATSYS) project (US$30 million, 1999-2006) and tested in 15 pilot regions of the Russian Federation through technical assistance on international best practices in institutional development and limited hardware and software investment. This component supported improved performance monitoring of government entities in the social sphere, financial sphere, and real sector, to support a shift to performance-based budgeting. With additional government financing to the project, new activities were added, including the introduction of an integrated approach to organizing and conducting household surveys, upgrading technology for keeping and using statistical registers, developing database software for calculating System of National Accounts (SNA) aggregates, developing various tools and systems for ROSSTAT, improving IT policy-making, integrating computer information infrastructure, developing software for demographic data processing, and developing a concept for using "Big Data" in government statistics. C. Enhancement of social statistics (appraisal: US$2.09 million, of which US$0.43 million financed by the Bank; actual US$4.6 million). This component was to strengthen statistics on population incomes, poverty, and living standards through the modernization of household sample surveys and administrative data collection. It was to support improved cooperation and coordination of regional statistical offices with the agencies responsible for vital registration, upgrade systems of vital registration recording and migration registration, improve methods for the analysis of labor supply in the economy, and further develop labor force and employer surveys. With additional government financing to the project, new activities were added, including upgrading the methodology and technology for processing micro-data, developing methodologies that form the statistical base for calculating indicators of income inequality and poverty, improving the empirical base underlying several statistical practices, supporting the compilation of a wide range of indicators for monitoring the Sustainable Development Goals (SDGs), developing methodologies for estimating household income/expenditure/savings, constructing improved mortality tables, and improving short-term demographic forecasting. D. Human resource development (appraisal: US$6.46 million, of which US$1.31 million financed by the Bank; actual US$6.78 million). This component was to establish a modern and sustainable system of ongoing professional education for statisticians in different data-providing agencies by defining the needed skills mix for modern statistical work, developing appropriate new job descriptions, and designing training and re-training programs. With additional government financing to the project, new activities were added, including the development and implementation of training materials on sample design, statistical data editing, SNA and macroeconomic statistics, price statistics, and modern technologies for the production and dissemination of statistical data. E. Project management (appraisal: US$3.68 million, of which US$0.79 million financed by the Bank; actual US$7.88 million). This component was to support the Bureau of Economic Analysis Foundation (BEA), as the project implementation unit (PIU). Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Total project costs at appraisal were US$50 million, to be financed by a US$10 million Specific Investment Loan from the International Bank for Reconstruction and Development under the umbrella of STATCAP Adaptable Program Lending and a US$40 million contribution from the government. Russia had adopted a Federal Targeted Program (FTP, 2007-2011) for the development of state statistics and a Statistical Master Plan (SMP), meeting the eligibility criteria for STATCAP. Project financing made up 25 percent of the US$200 million FTP budget. During implementation, the planned government contribution more than tripled to US$121 million (an additional US$16 million in 2014, US$40 million in 2015, and US$25 million in 2018), of which US$118.7 million was disbursed. Actual Bank financing was US$9.96 million, leading to an actual total project cost of US$128.6 million. The project was approved on June 28, 2007, and became effective on April 22, 2008. It underwent a mid- term review in February 2011. It closed on December 31, 2021, over nine years after its original closing date of August 25, 2012. The project was restructured five times:  May 17, 2012: Extension of the closing date to February 28, 2014 following slow project start-up due to the effects of the 2008-2009 financial crisis.  January 25, 2014: Extension of the closing date to December 31, 2015; change to components and output/outcome indicators to reflect additional government financing.  May 19, 2015: Extension of the closing date to December 31, 2018; change to components and output/outcome indicators to reflect additional government financing.  February 16, 2018: Extension of the closing date to June 30, 2021; change to components and output/outcome indicators to reflect additional government financing.  September 17, 2020: Extension of the closing date to December 31, 2021, to accommodate delays due to the COVID-19 pandemic. With closure extending more than ten years beyond the project's date of effectiveness, an interim ICR should have been produced and reviewed by IEG at the ten-year mark, but this was not done. In a meeting with IEG on June 13, 2023, the project team explained that, due to the evolving nature of the Bank’s relationship with the government at that time, it was decided to concentrate instead on possibilities for extending the project in order to continue support for the production of high-quality statistical data. 3. Relevance of Objectives Rationale At appraisal, Russia was still in the process of transforming its state statistical system from one serving the needs of a centrally planned economy to one meeting the needs of a more open society and market economy. It was recognized that implementing a broad-based poverty reduction strategy and sustaining job-creating economic growth could not be achieved without a steady supply of trustworthy and readily available social and economic statistics. ROSSTAT and other data-providing federal and regional agencies were experiencing ongoing shortcomings: uneven capacity to supply quality data according to international standards; insufficient coordination of information and statistical resources; an absence of regulations on Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) rights and responsibilities of federal executive bodies in the areas of developing and using statistical resources; and incompatibility of information systems, hampering inter-agency data exchange. Two medium-term federal government programs (1992-1996 and 1997-2002) and a third inter-agency program completed in 2006 produced a virtually new statistical system largely conforming to the basic principles of official statistics as set out by the United Nations Statistical Commission. Significant additional work remained, however, to further adopt international standards and classifications. The government established an SMP and third medium-term FTP, "Development of the State Statistics in Russia, 2007- 2011," that envisioned accelerated investments in infrastructure, upgrading of personnel skills, launching of new surveys and data collection initiatives, and progress toward establishing e-government. The government's emphasis on transparency was indicated by its subscription to the International Monetary Fund (IMF) Special Data Dissemination Standard in January 2005. The project remained relevant to government strategy throughout its lifetime, as it was integrated into the subprogram "Official Statistics" of the State Program "Economic Development and Innovation Economy (2015-2024)." The additional activities incorporated into the project's implementation plan in 2018 reflected new tasks and objectives outlined in the "Government Strategy for Information Society Development for 2017-2030." The objectives were aligned with the Bank's Country Partnership Strategy (CPS) at appraisal (2006-2009), which contained pillars on sustaining rapid growth, improving public sector management and performance, improving the delivery of social services, and enhancing Russia's global performance. The project underpinned each of these objectives, contributing to the generation of the statistical data needed to support, monitor, and evaluate administrative reforms, promote performance management and performance-based budgeting, adopt international accounting standards, and facilitate municipal reform and other key programs. In addition, under the second pillar on improving public sector management and performance, the CPS contained a specific outcome on improving the quality of Russian statistics. The most recent CPS (2012-2016) contained a cross-cutting theme of improving governance and transparency, with goals to increase the transparency of government activities and achieve modern standards of public service delivery based on the needs of citizens and businesses. This project was explicitly situated under that theme. The project's objectives were appropriately framed, given the Bank's previous engagement in the sector. The STASYS project promoted the reform of primary statistical procedures and developed the main elements of a modern information technology network. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) Produce reliable, accurate economic, financial, socio-demographic, and other data in line with international good practice and frameworks. Rationale The theory of change for this objective held that the provision of consultancies and technical services, training of staff, and establishment of adequate policy, regulatory, coordination, management, human resources, and quality assurance frameworks would contribute to the modernization of Russia's methodology for economic statistics. In parallel, the project's support for government procurement of IT equipment and software would contribute to the development of modern design and technology for statistical data collection, processing, and dissemination. These technological and methodological improvements would contribute to enhanced statistical infrastructure and processes in the areas of classifications and standards, questionnaire design, sample selection, data collection and editing, imputation, analysis, and dissemination procedures. With core statistical programs enhanced to meet international standards and national needs in terms of coverage, content, comparability over time, and coherence, the system would be able to produce more reliable and accurate economic, financial, socio-demographic, and other data. These results would bring Russia in line with international good practice and frameworks, increasing Russia's capacity to benchmark progress along a wide range of statistical indicators against international comparators. This theory of change assumes that the IT infrastructure will be sufficient to facilitate modern methods for the collection, production, and dissemination of data; that there will be adequate funds and resources to maintain the infrastructure; and that the government will sustain political support for the production and dissemination of accurate and transparent data. In principle, this theory of change was credible, though the assumption of sustained political support for meeting international statistical standards has not held up. Outputs that apply to all project objectives Research, analysis, methodological, and organizational tools were developed to facilitate transition to the 2008 SNA standards (the latest version of the international statistical standard for national accounts). As part of this transition, other methodologies were developed: for analyzing and accounting enterprise groups, compiling national accounts for natural resources, treatment of expenditures on military equipment as fixed capital formation, treatment of intellectual property products as fixed capital, assessment of the consumption of fixed capital as value-added, and others, all of which were applied for obtaining annual and quarterly series of gross domestic product (GDP) and its components beginning in 2011. The ROSSTAT computing and information system was modernized to support a transition to centralized statistical data processing and systems integration. All main tasks for maintaining and further developing this IT infrastructure have been centralized at the federal level, to promote the optimization of resources. A technological platform was developed and implemented for an integrated approach to organizing and conducting sample surveys of households. Methodological guidelines were developed for conducting thematic surveys of living conditions, poverty, household income and budgets, and the labor force. United Nations standards were introduced for statistical practices on measuring well-being and quality of life, including SDG indicators. In 2010, a Scientific and Methodological Council was established at ROSSTAT, charged with developing strategies for Russian statistics, promoting a wide range of research, soliciting experts' opinions on methodological problems, preparing recommendations for dealing with strategic development issues, and providing ongoing advisory support. In 2011, a Public Statistical Council was established to incorporate input Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) from the expert community and civil society. The ICR (p. 127) states that both Councils have met frequently and effectively supported "greater openness, planning, and implementation of the statistical production process and dissemination of statistical products." Modern computer equipment was provided to all ROSSTAT staff, with ongoing upgrading of hardware and software (most financed by the government rather than by the project). All statisticians and staff were equipped with computers, had access to the internet, and possessed modern software tools and packages for survey design, data collection, data verification, and data analysis. Educational and methodological materials were developed for training ROSSTAT employees. Over 30,000 ROSSTAT staff were trained under the project, exceeding annual targets during the entire project period. ROSSTAT training budgets exceeded project targets each year. Training courses and modules covered, among other topics, the use of modern information technologies for statistical production, quality management in official statistics, social/demographic/health statistics, national accounts, supply and use tables, price and finance statistics, labor/education/science/culture statistics, and seasonal adjustment of social and economic statistics. Changes to surveys and modules for data collection were introduced each year to keep abreast of new economic and social developments and the needs of users. Between 3 and 16 changes to forms or modules were introduced each year of project implementation through 2020. Following the 2019 Federal Statistical Observation Act, 75 changes were introduced in 2021. In addition, ROSSTAT reduced the number of forms in use from 258 in 2008 to 210 in 2021, streamlining and simplifying reporting. In total, these changes met the target of changes having been introduced in 75 percent of all forms, though no information is provided on the nature, scope, or quality of these changes. An indicator measured the percentage of statistical indicators based on electronic economic descriptions (digital templates for statistical data). Baseline: 34 percent in 2018 (the first year this indicator was measured). Target: 60 percent in 2021. Actual: 60 percent in 2021. The preparation of methodological guidelines for compiling satellite accounts in the social sphere, as recommended by the Organization for Economic Cooperation and Development (OECD), did not take place as planned. (Russia's accession to the OECD was put permanently on hold following its illegal annexation of Crimea in 2014.) Intermediate Outcomes An indicator measured the degree of implementation of the 2008 SNA. Baseline: 10 percent in 2015 (the first year this indicator was measured). Target: 93 percent by 2018. Actual: 95.5 percent by 2021. The ICR (pp. 134-135) details the methodology for calculating this indicator, noting that "the higher the value of the indicator, the more adequate and comparable Russian macroeconomic indicators are with statistical data from other countries" (ICR, p. 135). An indicator measured the extent to which statistical questionnaires of international organizations were completed with harmonized indicators. Baseline: 87 percent in 2007. Target: 90 percent by 2012. Actual: 90 percent by 2010, maintained at that level through 2014. In 2013, the indicator was changed to the extent of harmonization of the Russian Federal Plan for Statistical Work with official OECD questionnaires, which Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) monitored the process of completing 70 new questionnaires recommended by OECD. Baseline: 70 percent in 2013. Target: 77 percent by 2018. Actual: 79 percent by 2021. Compliance with the household income and expenditure methodology of the International Conference of Labor Statisticians increased from "partially complies" in 2007 to "fully complies" in 2012, following the introduction of sample surveys on household incomes and participation in social programs. Outcomes Periodic surveys of the level of users' trust in official statistical information were implemented beginning in 2008 (annually through 2011, and then every two years). The ICR (pp. 115-117) presents a detailed explanation of the calculation of the users' trust index. The indicator is the ratio of the overall trust index for the reference period to the value in the base period. Baseline: 1 in 2007. Target: 1.04 by 2012. Actual: 1.03 by 2021. The ICR (p. 19) notes that, despite not reaching targets, these increases represent an important overall achievement "considering the low level of public confidence for any official information produced in Russia." The COVID-19 pandemic disrupted the final scheduled survey collection and update. The link provided in the ICR to access detailed survey results (http://anbico.ru) is no longer active. In addition to a complex composite index, the ICR could have demonstrated progress on improved user trust in a more straightforward manner by presenting data on a set of representative questions directly from the user surveys. Data quality assessments for National Accounts Statistics (per the Data Quality Assessment Framework for National Accounts Statistics as detailed in 2012 IMF Guidelines) were conducted in 2010, 2013, 2015, 2017, and 2019-2020. The assessments measured prerequisites of quality, assurances of integrity, methodological soundness, accuracy and reliability, serviceability to users, and accessibility to users (ICR, pp. 119-121). The scores for data quality increased steadily across the project implementation period, from 3.1 (on a four-point scale) in 2010 to 3.3 in 2019-2020, with the most significant increase (from 3.12 to 3.26) taking place between 2013 and 2015, driven largely by improvements in the sub-scores on methodological soundness and accuracy/reliability (ICR, p. 121). An indicator measured the number of classifications used in the system of state statistics compatible with the equivalent international classifications. Baseline: 7 classifications in 2007. Target: 12 classifications by 2016. Actual: 12 classifications by 2021. Achieving this target allows systematizing and structuring of official statistical information following international standards for classifying socio-economic processes and phenomena. Another indicator measured the number of SDG indicators compiled by official statistical bodies, based on international methodologies, and introduced into the Federal Plan of Statistical Works. Baseline: 90 indicators in 2018. Target: 108 indicators by 2021. Actual: 112 indicators by 2021. A national set of SDG indicators for the Russian Federation was defined in 2020. (These indicators superseded prior analogous indicators on the Millennium Development Goals.) IEG has validated the ICR’s statement (p. 23) that Russia exceeded IMF Special Data Dissemination Standard metadata certification requirements for all categories in 2022. Rating Substantial Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) OBJECTIVE 2 Objective Produce timely economic, financial, socio-demographic, and other data in line with international good practice and frameworks. Rationale The theory of change for this objective largely draws from that described under the first objective, with emphasis on outputs related to digitization of data collection, reporting, and processing; conducting of sample surveys rather than full count surveys; and application of modern technological tools for statistical analysis. The increased efficiencies realized through these modernized practices were expected to contribute to timelier production of economic, financial, socio-demographic, and other data. Relevant outputs were discussed under the first objective. Intermediate Outcomes An indicator measured the share of regional statistical office employees dealing with receiving and processing primary statistical data. Baseline: 100 percent in 2015 (the first year the indicator was measured). Target: 40 percent by 2018. Actual: 35 percent by 2021. This decline in acceptance and processing of paper forms demonstrates improvement in statistical process automation, contributing to timeliness. The availability of data collected through remote methods, which the ICR (p. 133) defines as "the implementation of modern technologies in statistical practice," increased from 10 percent in 2010-2012 to 61 percent in 2013. This indicator was later replaced by the share of official statistical reporting forms that could be submitted in electronic form. Baseline: 80 percent in 2015. Target: 100 percent by 2018. Actual: 100 percent by 2018 and each year thereafter through project closure. Another indicator measured the number of systemic statistical tasks moved to centralized primary statistical data processing (through the ROSSTAT computing center). Baseline: 0 in 2014. Target: 45 tasks by 2018. Actual: 146 tasks by 2021. In principle, centralized data processing both improves the quality of data output and reduces processing time. (This indicator replaced an earlier indicator on the number of statistical tasks forwarded to the United Foundation of Algorithms and Programs.) The number of surveys conducted by ROSSTAT in various economic and social fields done on a census/full count basis (rather than a sample basis) was 54 in 2007, increased to a peak of 61 in 2014, but then declined to 55 in 2017 and 44 in the pandemic years of 2020 and 2021 (no target was set). Over the project period, more than 700 sample surveys were conducted in lieu of full-count surveys. The ICR (p. 21) states that this indicator demonstrates an improvement of methodological and organizational procedures and a "significant reduction of time, costs, and response burden," but a more meaningful indicator would have measured the ratio of full-count to sample surveys rather than the absolute number. Outcomes An indicator measured the share of information provided by ROSSTAT to users according to established time schedules (internationally accepted release schedules and the calendar on the ROSSTAT website). Baseline: 97.8 percent in 2006. Target: 100 percent by 2012. Actual: 100 percent by 2008, maintained at 100 Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) percent throughout the project's lifetime. The ICR (p. 124) notes that it would have been preferable to introduce more differentiated indicators to capture progress since 2008. According to the ICR (p. 23), Russia exceeds IMF Special Data Dissemination Standard timeliness requirements for employment, unemployment, wages/earnings, CPI, and PPI. Rating Substantial OBJECTIVE 3 Objective Produce data for policy formulation and decision-making. Rationale The theory of change for this objective held that the production and dissemination of more relevant, timelier, higher-quality statistical data and analysis, achieved under the first and second objectives, would be available to policy- and decision-makers. Relevant outputs were discussed under the first objective. Intermediate Outcomes and Outcomes The project's results framework did not contain indicators specific to this part of the objective. The ICR presents some indicators on general availability of data that could, in principle, be useful for policy formulation and decision-making, though the indicators do not demonstrate actual use by policy- or decision-makers:  An indicator measured the number of annual visits to the ROSSTAT website. Baseline: 500,000 in 2007. Target: 1 million by 2012. Actual: 3 million by 2021.  In 2016, microdata from household budget surveys and labor force surveys were made available on the ROSSTAT website. Results of new enterprise surveys have been posted every two years since 2012. In addition, the ICR provides information beyond the project indicators on statistical methodologies and data supported by the project that would support policy formulation and decision-making:  A methodology for the anonymization of microdata for presentation to users for analytical purposes was developed and tested.  Real GDP data rebased to 2011 prices have been made available from 2011. In April 2016, ROSSTAT released GDP estimates compiled according to the 2008 SNA.  Aggregate price indices are now compiled for each major good and service item, for all of Russia's regions and for the country as a whole. Detailed data on total annual sales by economic activity, which are used to develop weights for the PPI, are published on the ROSSTAT website. Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Russia's overall score on the World Bank's Statistical Performance Indicators (SPI) improved from 71.59 in 2016 to 78.19 in 2019, placing it in the fourth quintile in 2019. Its strongest dimensions of SPI performance are data used by international organizations (91) and standards and methods (90). Through the time of project closure, the project contributed to improved methodological soundness in the production of Russian statistics through the application of international standards, increased efficiency of data exchange, improved reliability of data sources and of intermediate and output statistics, better reliability of administrative data, and improved data timeliness. Project indicators were, for the most part, of acceptable quality and adequately measured the achievement of objectives; where this was not the case, the ICR presented relevant additional information. The project team reported to IEG in a meeting on June 13, 2023, that the quality of data related to project outputs and outcomes did not deteriorate across the lengthy project period, despite international sanctions and a challenging political environment. IEG finds this assertion to be credible, as the project closed before limitations on release of statistical data emerged following Russia’s invasion of Ukraine in February 2022 (see Section 7). Overall Efficacy Rating Substantial 5. Efficiency The PAD (pp. 13-14) noted that the project was not amenable to a cost-benefit analysis, as national statistical offices have little significant cost recovery. Project economic benefits were to accrue from three sources: improvements in the effectiveness and efficiency of statistical operations and agencies would produce broader coverage of and higher-quality data for the same level of expenditure; better data would enhance the potential for evidence-based decision-making at policy, program, and project levels; and the project would help address the significant costs of missing or inaccurate data. The ICR (pp. 27-29) postulates that economic benefits were realized through a variety of channels: improved efficiency of statistical operations contributed to better- functioning economic sectors by facilitating more coverage and higher-quality data; production of better-quality data enhanced the potential for evidence-based decision making at the policy, program, and project levels; and data collection costs were reduced through improved IT systems and use of electronic information and survey submission for respondents; technical and information base upgrades reduced data processing and storage costs. These arguments in the ICR are reasonable, supporting the hypothesis that a relatively small investment Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) of Bank funds helped leverage a larger country-financed program that enabled strong development outcomes in statistics. In terms of operational efficiency, the ICR (p. 27) cites statements from ROSSTAT, Ministry of Finance, and Ministry of Economic Development staff that project funds were "utilized efficiently on the targeted activities with a focus on value for money and strict spending controls." Along those same lines, annual audits and site visits to ROSSTAT regional offices by the Accounts Chamber found that the value-for-money principle was followed in project implementation. BEA (the project PIU) was simultaneously implementing two other Bank-financed projects, contributing to economies of scale and cost savings under the project management component. Overall, the ICR states that "most of the planned activities were completed as per established schedule" and that project resources were used efficiently, with the project completed without cost overruns "by its closing date of December 31, 2021" (ICR, p. 27). Project extensions incorporated significant additional activities made possible by increased government co-financing. The ICR (p. 32) describes some early delays caused by issues with the Ministry of Finance's approval of annual project budgets and implementation/procurement plans and by personnel turnover in participating government entities. The ICR also describes delays in procurement resulting from an insufficient number of candidates in the Russian statistical consultancy market to meet the Bank's procurement rules, an issue that was never "fully resolved" (ICR, pp. 26-27), though the project team later added that challenges were addressed through close communication and coordination without incurring additional costs. Overall, implementation inefficiencies were minor. The project extensions reflected the evolution of the Bank's portfolio during the project period and the important role the project played in maintaining the Bank's engagement in Russia and its ongoing knowledge work during a politically challenging time, especially its work on economic monitoring and poverty. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's objectives were highly relevant to Russia's country context, government strategy, and Bank strategy. All objectives were substantially achieved. Project efficiency was substantial. Overall, these ratings are Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) indicative of only minor shortcomings in the project's preparation, implementation, and achievement, resulting in an Outcome rating of Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The project's institutional, technological, and training outcomes were, in principle, sustainable. Almost all project activities and results had been institutionalized and incorporated into ROSSTAT's regular work. According to the ICR, the government has supported recurrent and maintenance costs for equipment, and both hardware and software upgrades are adequately covered by budget allocations. A new ROSSTAT strategy (2024) was prepared before project completion reflecting the project's results in several areas and was approved by the government. The ICR presents further details on this strategy. However, the sustainability of some of the project's results may be at risk due to changes in what statistics are disclosed and when some statistics that were previously available are no longer disclosed (e.g., foreign trade statistics, including those related to imports, exports, and trade within the Eurasian Economic Union). Beginning in May 2022, the Ministry of Finance stopped publishing information on federal budget expenditures and sources of financing for its deficit. ROSSTAT continues to publish some major social and economic indicators, but the periodicity of the data release has changed from daily to weekly or monthly. 8. Assessment of Bank Performance a. Quality-at-Entry The project's objectives, theory of change, and results chain were clear at the time of appraisal. Project preparation benefited from a well-developed Statistical Master Plan, prepared with the Bank's assistance, that provided a long-term vision and development framework. A comprehensive Project Operations Manual, implementation plan, and procurement plan were in place at the outset. The PAD (p. 8) noted that key lessons had been learned from the experience of the predecessor STASYS project: (i) the need for close supervision and coordination of equipment delivery, software testing and installation, and civil works for computing centers, and (ii) the likelihood that software contracts will require relatively lengthy periods. Risk assessment at appraisal acknowledged the prevalence of corruption, specifying a series of mitigation measures: setting procurement thresholds and frequency of the Bank's prior review at levels that would allow an appropriate level of control; enhancing disclosure and transparency of project-related information; setting up an appropriate complaints handling mechanism; enhancing internal controls and limiting cash payments from project funds; relying on regular audits by the Accounting Chamber of the Russian Federation and internal audit units of the Ministry of Finance; and intensive Bank supervision (PAD, p. 11). The project's overall risk rating was moderate, with moderate risks to the achievement of outcomes stemming from the commitment of the government to support statistical work and ROSSTAT's capacity to sustain the development of the statistical system. Mitigation actions included training officials and other data users in the use of statistics, improving data quality to promote the value of statistics, and revising the legal to enhance ROSSTAT's central role. The only risk assessed as high was the possibility Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) of a monopoly for procurement of software, to be mitigated by increasing the number of vendors; it is unclear how the Bank was to implement this mitigation measure, and this risk eventually materialized, presenting procurement challenges and causing delays throughout the project's lifetime (see Section 5). The ICR (p. 36) notes that a formal Decision Review was not held, a departure from standard practice for investment project financing preparation; the ICR states that “to some extent,” the Decision Review was substituted by the findings and recommendations from the STASYS project. The project team later clarified, however, that a decision meeting was held on February 15, 2006, and the team provided documentation to that effect. Quality-at-Entry Rating Satisfactory b. Quality of supervision The project team remained engaged with the government in a challenging political environment. Although Additional Financing was considered for the project, the government's approach was to continue implementation and expansion through its own increased co-financing. This allowed the Bank to maintain technical involvement in an important reform area and, as the project team noted in its conversation with IEG, maintain overall engagement and dialogue with the client. A formal mid-term review was not held, as the client considered it optional because it was not part of the Loan Agreement, but a mid-term report was completed. An interim ICR (required for a project that has not closed after ten years) was suggested by the project team and was drafted but was not completed or formally submitted (ICR, p. 37; see Section 2e). Supervision was consistent, drawing on technical expertise from the Bank's Development Data Group, Poverty Global Practice, and operational knowledge of the project team (located both in Russia and in Washington). Close supervision in the project's early years helped overcome start-up challenges. Performance reporting was adequate. The fiduciary team joined most supervision missions and exercised close supervision of BEA's compliance with financial management and procurement requirements. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) a. M&E Design Regular project monitoring used a set of Data Quality Assessment Framework indicators developed by the IMF. Other indicators and monitoring data were drawn from IMF Reports on the Observance of Standards and Codes, IMF multi-sector statistical mission reports, surveys of users of statistical products, and independent assessments of statistical products against the master plan. A project evaluation was to be carried out by an independent agency selected by BEA (PAD, p. 10). An Interagency Coordination Committee, similar to that used for the STASYS project, was to ensure checks and balances on implementation decisions. The PAD (pp. 24-25) specified a clear results framework with a logical results chain and adequate outcome and intermediate outcome indicators. The indicators were, for the most part specific, measurable, achievable, relevant, and time-bound (SMART). Not all baselines and targets were available at approval, but they were incorporated when available in a timely manner. Arrangements and institutional responsibility for results monitoring and reporting, centered at ROSSTAT, were clearly specified. The ICR (p. 33) notes that the indicators were aligned with the FTP for the development of state statistics in Russia and measured progress in the broad context of reforming the state statistical service; as such, the indicators did not measure the specific contribution of the project to observed outcomes. Key elements of statistical system strengthening supported by the project -- statistical infrastructure, IT modernization, staff knowledge, improving the quality of specific statistical products -- could have been tracked with more refined indicators. b. M&E Implementation ROSSTAT, with support from BEA, carried out regular monitoring and comprehensive annual reporting on project implementation as planned. Outcome indicators were slightly adjusted in 2018 to increase clarity, but their basic definition remained unchanged, and targets were adjusted upward to take into account additional time and financing made available to the project. Intermediate indicators were adjusted and added in accordance with new activities introduced to the project. User satisfaction surveys were incorporated into the results framework and were conducted on a regular basis in keeping with international best practice. The ICR's Annex 5 provides an extensive discussion of the evolution of the project's results indicators. The ICR does not state whether the planned project evaluation was conducted; the project team confirmed that an independent third party surveyed users annually, and that a final assessment was completed on all M&E indicators and ROSSTAT outputs on the accuracy, timeliness, and quality of data. c. M&E Utilization Information on project monitoring and results was reported regularly to the Bank, Ministry of Finance, Ministry of Economic Development, and Accounts Chamber. The results of these assessments were used to analyze intermediate results, adjust and add activities, and update indicators. M&E Quality Rating Modest Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) 10. Other Issues a. Safeguards The project was classified as environmental assessment category C. No safeguard policies were triggered. b. Fiduciary Compliance The project's financial management (FM) risk was assessed as moderate at appraisal (PAD, p. 14). Corruption was recognized as a "major issue for doing business in the country" (PAD, p. 14), and a series of mitigation measures was specified (see Section 8a). A Country Financial Accountability Assessment (CFAA) in 2001 had concluded that gradual progress in public financial management had been made over the preceding decade but that Russia's financial management and procurement capacity and infrastructure were not yet at a level of performance that would allow the Bank to rely on those systems. The CFAA therefore recommended that fiduciary functions (disbursement, procurement, accounting, and reporting) continue to be outsourced to specialized agencies, in this case BEA. Audits were to be conducted by private-sector audit firms. The Bank's financial management team regularly reviewed and rated project FM as satisfactory. All quarterly unaudited interim reports were submitted on time and were acceptable to the Bank. No major control or accountability issues were identified during annual external audits. All had clean opinions. The Bank carried out regular procurement reviews as part of supervision missions and rated procurement satisfactory throughout the implementation cycle, with staffing and planning found to be sufficient. Post- reviews were regularly conducted on a sample basis (ICR, p. 35). Some procurement delays were encountered due to an insufficient number of candidates for consultancies (ICR, pp. 26-27). The ICR (pp. 32, 33) notes that BEA was investigated by the World Bank Integrity Vice Presidency (INT) but that no violations were found. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) Failure to follow standard Bank Performance Satisfactory Moderately Satisfactory procedures for interim ICR submission. Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR (pp. 41-42) presented several insightful lessons and recommendations, with some re-stated here: Embedding institutionalization of activities during a complex statistical reform project is an essential condition for sustainability. In this case, ROSSTAT agreed to formally embed all project-developed methodologies and procedures into internal executive orders; consultants' contributions, after piloting, were accepted as part of regular survey operations; and the project-supported staff training program was coordinated with ROSSTAT's methodology and IT product development plans. Large, complex, interconnected activities under statistical development projects require close monitoring and pre-planning for unforeseen delays. In this case, delays in the delivery of equipment, network installation, and approvals for remodeling of a server room produced lags between the piloting and implementation of new statistical applications. In environments where political, technological, and other contexts are likely to experience change, a multiphase programmatic approach may be preferable to a single operation. In this case, a different instrument would have allowed for different phases of the project to be reviewed in a timely manner and assessed on their own merits. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is clear and reasonably well written, with an abundance of data and meticulous attention to outcome and intermediate outcome indicators. The quality of the analysis and lessons is satisfactory. In several instances, the ICR compensated for the lack of adequate data in the project's results framework by presenting additional relevant information. However, there are important shortcomings. The ICR is extremely long (160 pages, with annexes) and repetitive. The presentation of results data is highly challenging to follow, with the project's many intermediate indicators presented in different orders in the main text, datasheet, and results Annex 5. Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STATE STATS SYST 2 (P092429) a. Quality of ICR Rating Substantial Page 18 of 18