ICRR 12861 Report Number : ICRR12861 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/25/2008 PROJ ID : P050487 Appraisal Actual Project Name : Development Of State Project Costs (US$M): US$M ): 38.5 54.8 Statistical System Project Country : Russian Federation Loan/ US$M): Loan /Credit (US$M): 30.0 30.0 Sector Board : EP Cofinancing (US$M ): US$M): 8.5 24.8 Sector (s): Central government administration (100%) Theme (s): Economic statistics modeling and forecasting (29% - P) Infrastructure services for private sector development (29% - P) Macroeconomic management (14% - S) Analysis of economic growth (14% - S) Legal institutions for a market economy (14% - S) L/C Number : L4468 Board Approval Date : 05/13/1999 Partners involved : Closing Date : 06/30/2004 12/31/2006 Evaluator : Panel Reviewer : Group Manager : Group : Dusan Vujovic Gita Gopal Ali Khadr IEGCR 2. Project Objectives and Components: a. Objectives: The development objectives of the Russia Development of the State Statistical System Project (STASYS) were: (1) an improved macroeconomic management by providing a required information base; and (2) an improved availability of data needed for a market economy . This was to be achieved by focusing on the following priorities in the development of the state statistical system : (i) eliminate data gaps in the macroeconomic framework used for economic policy and management; (ii) capture new and dynamic economic and social activities; (iii) meet the needs of analysis of policy makers and the business community; (iv) introduce international standards and methodologies; (v) increase the efficiency and productivity of statistical production through improved organization, data collection and processing methods and equipment; and (vi) improve the quality of analysis and the monitoring of economic development along with the modernization of decision-making methods for economic policy . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The project included 10 components focused on three main areas: developing institutional capacity of GOSCOMSTAT (1-4), strengthening other collection agencies (5-9), and improving project management (10): (1) Improvement of organization, management strategy and perspective planning; (2) Fostering knowledge of standards and methodology; (3) Strengthening of statistical infrastructure; (4) Reorganization of data collection, processing, and dissemination . (5) Improvement of Public Finance Statistics and the Compilation of General Government Sector Accounts in the SNA; (6) Introduction and Compilation of Financial Statistics; (7) Improvement of Coverage and Quality of BoP Statistics; (8) Improvement of the Capabilities of the Government in the Utilization of Statistical Information for Economic Monitoring, Forecast, and Policy Making; (9) Improvement of Vital Statistics Processing . (10) Establishment of a Directorate under the existing PIU (the Economic Analysis Bureau) with the purpose of Program management to support the development of the state statistical system . During project implementation the Bank and the Government agreed on a number changes to the original components of the project. Albeit nominally extensive, these changes were fully consistent with the project objectives spelled out in the legal documents and did not necessitate amendments to the loan agreement . The changes were as follows: some GFS activities (planned under a separate project ) were subsumed under the Component 5. Component 6 was dropped due to lack of interest of the Central Bank of Russia (CBR). Component 9 was substantially reduced due to lack of engagement by the Registry Department and was replaced by TA for the implementation of the All -Russia Population Census in 2003. Component 3 was substantially scaled back and focused to new methodologies for data compilation . Key activities and performance indicators on replacing censuses with sample surveys were reorganized and consolidated under Component 4 (i.e. subcomponents 4.1 (on economic statistics) and 4.2 (on households). An all-Russia product classifier by economic activities substituted the development of a classification unit Component 8. A new Component (11) was added to separate the work on the Federal Customs Service database from the rest of the work on the Balance of Payments statistics . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Total project cost has increased from US$ 38.56 M at appraisal to US$54.79 mainly due to significant scaling up of components 1,2 and 4. Bank financing remained unchanged at US $ 30.0 M while the borrower's share increased by US$16.3 M, from US$8.50 M to US$24.79 M, to secure borrowers 80 percent contribution in IFI-supported projects starting December 24, 2004. This was administratively implemented by a significant reduction in the share of Bank financing for specific disbursement categories in the loan agreement . As indicated above and documented in the Annex 1 of the ICR, the brunt of additional financing (US$17 M) went to further replication of technical solutions utilized in the information system under component 4. The closing date was extended twice . First from June 30, 2004 to April 30, 2005 to allow for completion of original project activities including the agricultural census methodology, the new classification of products by economic activity and inputs into the new chart of accounts in public administration . The second extension of the closing date (till December 31, 2006) was requested by the borrower along with the extension of project activities and substantially increased counterpart financing mandated by new legislation . 3. Relevance of Objectives & Design: The project objectives (improved coverage, quality and availability of statistical data provided in a timely manner ) were highly relevant for the Russian Federation and remain highly relevant to this day . The project design was well suited for the objectives and chosen components appeared broadly relevant in addressing the capacity needs of the Goscomstat and other collection agencies, despite numerous changes to the structure of components to eliminate overlaps and to channel the main trust of the project design onto specific improvements . The key objectives . This project was prepared in the aftermath of the Russian financial crisis of 1998 which exacerbated the political and economic turmoil triggered by the profound decline of the Russian economy during the start of transition . A reformist and technocratic government of the time clearly recognized the need to overhaul the statistical system and improve the quality of statistical information as a key to better macroeconomic management and implementation of institutional reforms. The transition posed major challenges to the statistical system due to the emergence of a large and dynamic private sector, expansion of modern services, and the changing role of the state in economic management . With significant assistance from development partners GOSCOMSTAT moved early to comply with the 1993 System of National Accounts (SNA), collect data on prices and systematically produce the Consumer Price Index (CPI) and other price indexes. The Bank supported many data efforts in that broader context (including initiatives to strengthen the estimation of national accounts, improve the compilation of the Balance of Payments, money and banking, government finance statistics, introduce new types of social statistics, conduct input -output analysis, household and labor force surveys, develop a set of poverty indicators, etc .). In 1997, the authorities approved a Federal Earmarked Program for the Development of the Statistical System of the Russian Federation for 1997-2000. Despite these improvements, key weaknesses in the statistical system persisted, however, due to excessive reliance on census data collection methods, continued use of old Soviet -type accounting standards in enterprises and banks, and inadequate compliance with statistical reporting requirements . This adversely affected the timely provision of quality macroeconomic data needed to inform macroeconomic policy considerations and investor decisions . Goskomstat and the Bank realized that a deep overhaul of the statistical system is needed to provide quality data for sound macroeconomic management and to inform private investment decisions . This necessitated fundamental changes in data collection and processing practices and the introduction of modern estimation and analytical techniques. It also required a wider scope of reform including : (a) all government agencies involved in data collection and processing, looking both at their individual capacity building and improved coordination among them; (b) adequate legal basis for statistical services; (c) better management capacity for the statistical system as a whole; and (d) added emphasis on improving data dissemination and building relationships with end -users. 4. Achievement of Objectives (Efficacy): (1) Provide a required information base for improved macroeconomic management - substantial achievement ; (2) an improved availability of data needed for a market economy - substantial achievement . With a range of new and improved statistical data products, internationally comparability of data, better dissemination and the substantial strengthening of economic modeling and forecasting activities using the new data, both development objectives were substantially achieved . Looking at individual priorities the project had substantial achievement in almost all priority areas as it helped : (i) eliminate data gaps, duplications and overlaps in all important areas for macroeconomic policy and management (except debt statistics due to some outstanding legal issues ) as evidenced through new statistical products, improved methodology and marked reduction in duplicate publications; (ii) capture new and dynamic economic and social activities (especially in labor force, entrepreneurial activity, agriculture, etc.) as well as make advances in constructing a sub -regional development index, measuring innovative capacity and competitiveness; (iii) provide critical improvements in meeting the needs for analysis for policy makers and the business community; (iv) observe international standards and methodologies in national accounts, government financial statistics, and balance of payments statistics, to name the most important areas; (v) increase the efficiency and productivity of statistical production in RF by implementing two key projects : The first was aimed at converting from the Soviet all -census data collection framework to an integrated census -sample survey framework which is more efficient and suitable for a market economy - with moderate progress (i.e. progress was "commendable though less than anticipated at project appraisal "). The second was to modernize data collection, organization, processing and dissemination systems in RosStat were substantial progress has been achieved; and (vi) improve the quality of analysis, monitoring and forecasting economic development by building a new (demand driven) macroeconomic forecasting model . The model has been used to measure the impact of tax exemptions on public revenues, of oil trade on Russian oil prices, etc . 5. Efficiency (not applicable to DPLs): The efficiency of the project itself was affected by implementation delays and multiple extensions of the closing date. On the positive side, during implementation the project generated a much larger share of confinancing and interest from the borrower supporting the finalization of the Statistical Master Plan, new Federally Targeted Program and the follow-on STASYS 2 project. This also leads the way to a new model of lending with smaller financial resources from the Bank leveraging a much larger borrower -financed program. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Despite some implementation delays, this project consistently and substantially achieved both its overall development objectives (required information for improved macroeconomic management and improved data availability for a market economy) and most of the specific project goals identified as priority activities . Realignment and changes in the project components, including significant scaling up of components 1,2 and 4, the introduction of component 11, and elimination (component 6) or significant scaling down (component 9) of components with weak ownership in the host institutions or alternative sources of free financing, did not affect the broadly defined development objectives. Combined with strong client ownership and continued relevance of project objectives, confirmed by a follow-on project and substantial confinancing from the borrower, as well as its contribution to the development of a new model of lending based on smaller Bank loans leveraging much larger borrower -financed programs. a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The new Law on statistics approved by the Duma after STASYS project closure (November 2007) formally verifies new data collection and processing methods used in the RosStat, as well as new statistical products developed with the help of this project. a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: Bank responded in a timely manner and very quickly prepared a product which addressed the immediate needs of the borrower. The Bank was well aware of the complex political economy and multiple agency coordination issues behind this project (as clearly noted in the lessons learned ). A conscious decision was made to use the window of opportunity and go with a quick project preparation and approval by the Board, rather than wait for all the political issues to be resolved and institutional arrangements to be in place . This approach is quite understandable as some of the key political economy and institutional issues could only be addressed and resolved in real time, during the ratification process and in project implementation . The only missing element was proper ex ante time allowance for unforeseeable delays and project risks (as identified in the lessons learned ). In addition to its immediate DO's for the Russian Federation, this project also paved the way to the development of a standardized statistical development project which became the core of the StatCap program, on the one hand, and led to the development of a knowledge -intensive leveraged Bank lending instrument, on the other . at -Entry :Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: Despite some initial implementation problems and slower implementation pace, this project earned a full and true ownership by the authorities both in RosStat, the Bureau of Economic Analysis and in all core economic ministries which have by now grown accustomed to timely provision of quality data and economic analysis and forecasting which would have been impossible without unified and quality data sources . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: Although the STASYS project preceded the current effort to rely on quantifiable project outcome indicators, it utilized the Results Framework methodology . The PAD contained a number of process-oriented performance indicators which made initial reporting on the “results� difficult. In mid 2005 the team refocused the indicators on key priorities that emerged during implementation : improvement of the data collection and processing systems, upgrading of staff skills, methodological improvements to the national accounts data, integration of statistical and administrative data, and passage of the Law on Statistics . The follow-on project (STASYS2) builds on this and more effectively shepherds project outcomes . IMF's SDDS report cards due later this year might be a useful monitoring indicator. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: ICR lists seven valuable lessons learned in the STASYS project which are fully applicable to all complex, IT intensive multi-agency projects, and beyond . Lessons range from multi-agency coordination problem to accommodating non-technical considerations, providing legal basis and external agency as the PIU . We would like to emphasize the importance of realistic project preparation and implementation with special attention given to adequate time allowances for unforeseeable delays and project risks which turned out to be a major stumbling block in ratification and project implementation . We would add an additional lesson learned from this project -- the value of continuously engaging the borrower in an evolving and expanding project design leading to enhanced ownership and greater leveraging of Bank lending resources for development results . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: Well written, well documented and informed ICR . a.Quality of ICR Rating : Satisfactory