43167 On the Access and Use by World Bank Staff of the ESW Knowledge Base Jaime B. Quizon Oliver Rajakaruna On the Access and Use by World Bank Staff of the ESW Knowledge Base Jaime B. Quizon Oliver Rajakaruna WBI Evaluation Studies No. EG07-131 The World Bank Institute The World Bank Washington D.C. June 2007 Acknowledgments The World Bank Institute Evaluation Group (WBIEG) prepared this report under the direction of Richard Tobin. Jaime B. Quizon, who served as task team leader for the evaluation, is the principal author. The authors thank Patrick Grasso (IEGWB), John Pollner (OPCIL), and Edward A. Strudwicke (IDU) who reviewed the study and offered valuable suggestions for its improvement. The report also benefited from review by Jan Weetjens (HRSLB). The authors thank Humberto S. Diaz for his assistance with formatting and graphics. WBIEG evaluates learning by staff of the World Bank and activities of the World Bank Institute (WBI). The findings, interpretations, and conclusions expressed in WBI Evaluation Studies are entirely those of the authors and do not necessarily represent the views of the World Bank Group, including WBI. WBI Evaluation Studies are available at http://www.worldbank.org/wbi/evaluation Suggested citation: Quizon, Jaime B., and Oliver Rajakaruna 2007. On the Access and Use by World Bank Staff of the ESW Knowledge Base. Report No. EG07-131 Washington, DC: World Bank Institute. Vice President, World Bank Institute Frannie Léautier Manager, Institute Evaluation Group Richard Tobin Task Team Leader Jaime B. Quizon ii Contents EXECUTIVE SUMMARY.......................................................................................................V 1. INTRODUCTION........................................................................................................... 1 2. OBJECTIVES................................................................................................................ 2 3. METHODOLOGY.......................................................................................................... 3 4. RESULTS...................................................................................................................... 5 The availability of ESW reports in the ImageBank.................................................... 5 On the capture and dissemination of knowledge from ESW reports.......................... 7 On the use of knowledge from ESW reports .............................................................. 9 5. IMPROVING THE OPERATIONAL IMPACTS OF ESW................................................. 14 APPENDIXES Appendix A: Interviews with TTLs.............................................................................. 19 Appendix B: Web-based surveys of Bank operations staff.......................................... 21 Appendix C: Survey A.................................................................................................. 24 Appendix D: Survey B.................................................................................................. 29 Appendix E: Guideline questions for interviews with TTLs........................................ 32 Appendix F: Procedural Guidance to TTLs on ESW ................................................... 34 iii Acronyms and abbreviations BW Business Warehouse ESW Economic and Sector Work IDU Internal Documents Unit IRIS Integrated Records and Information Service KLB Knowledge and Learning Board OPCS Operations Policy and Country Services QAG Quality Assurance Group SAP Systems Analysis and Product SLC Staff learning coordinator TTL Task team leader WBIEG World Bank Institute Evaluation Group iv EXECUTIVE SUMMARY As part of the Knowledge and Learning Board's (KLB) mandate for knowledge management, it aims to make knowledge embedded in economic and sector work (ESW) readily accessible to the Bank's operational teams. To promote staff learning from ESW and its use in Bank operations, an understanding is needed of how operational teams use the knowledge captured in ESW. To what extent does ESW contribute to staff knowledge and learning? What are the effective ways for capturing the knowledge from ESW and making it available for staff learning and use? And how does this ESW knowledge relate to business processes and outputs, such as other ESW? This study: (1) investigates how, and the degree to which, ESW knowledge is accessed and used within the Bank, including: (a) what constrains and facilitates access and use of this knowledge base within the Bank, and, (b) how this knowledge base contributes to the Bank's business objectives; (2) explores how knowledge from a typical ESW is captured, stored, and shared among Bank staff; and (3) identifies and recommends ways for promoting staff access, use, and learning from completed ESW. This study uses three approaches: surveys of Bank staff, a desk review, and semistructured interviews with ESW task team leaders and some of their managers, country directors, and staff learning coordinators. The results indicate that: · Operations staffs' awareness of ESW products related to their work and the usefulness of these products in their work are "moderate" (i.e., average staff self-ratings of 3.2 on an ascending 5-point scale). · Informal networks within the Bank are as important, if not more so, than structured events (e.g., workshops, seminars, brown bag lunches) for raising awareness of ESW reports relevant to their operational work · Disseminating new ESW knowledge to Bank staff is often opportunistic rather than something planned beforehand. A dissemination strategy is not a usual component of the concept note of a typical ESW. However, most task team leaders of ESW desire opportunities for disseminating their findings widely to relevant Bank staff. · Seventy-nine percent of the Bank's operations staff actively searched for ESW for use in their work in the past six months. Eighteen percent of them, using various channels, were not successful in locating any of the ESW reports they sought or their summaries. Difficulties with accessing ESW reports are a key constraint to the wide use by Bank staff of the ESW v knowledge base in their operational work. Less than 30 percent of completed ESW products are available in ImageBank, the required depository for all ESW products. · Approximately two-thirds of respondents noted that they had used some knowledge from ESW in their work in the past six months. The most frequently cited uses were in project-related operations work and in other ESW, research, or applied activities. Few respondents reported uses in policy-specific contexts or in client or staff training. To improve the operational impacts of ESW reports, staff suggested that the Bank and task team leaders: (a) Produce and disseminate short and targeted summaries. These summaries should be mandatory for all ESW. Use videoconferences more widely for dissemination. (b) Build processes for continuous dissemination of key ESW findings to staff and clients. (c) Identify strategies and good practices for disseminating the results of ESW to staff. (d) Organize internal access to all ESW, including concept notes and the status of reports in progress, in a single, user-friendly webpage. Simplify ESW information storage and retrieval systems. (e) Include an action plan, budget, and staff resources for disseminating the results of ESW to operations staff as part of any ESW proposal or concept note. (f) Use the ESW process as an "action learning" tool that, similar to the process for developing country assistance strategies, (a) engages staff and clients from different regions and sectors, and (b) produces an "action matrix" for follow-up activities. (g) Build and distribute "how to" toolkits or guidelines for disseminating ESW products internally for maximum business impact. (h) Offer incentives to operational staff to read and use ESW reports, perhaps by providing staff with time to read ESW reports by instituting a unit-based SAP code for "research" where staff can charge their time as needed. (i) Establish KLB awards or recognition for "best knowledge management practices," including for dissemination of "new" knowledge to staff. (j) Recognize and reward knowledge sharing and dissemination as much as knowledge creation and application (i.e., technical and operational achievements) in annual reviews of performance and in the development of results agreements. vi 1. INTRODUCTION 1.1 Economic and sector work (ESW) advances the knowledge base of the development community because it produces knowledge for clients.1 ESW is valuable and useful also to Bank staff not only to the staff engaged in the ESW but also to certain staff and units within the Bank for whom this knowledge is relevant. How (and how often) do Bank operational staff access the knowledge embodied in ESW to improve the effectiveness and quality of service to Bank clients? 1.2 As part of the Knowledge and Learning Board's (KLB) mandate for knowledge management, it aims to make knowledge embedded in ESW readily accessible to the Bank's operational teams. To promote staff learning from ESW and its use in Bank operations, an understanding is needed of how operational teams use the knowledge captured in ESW. For example, to what extent does ESW contribute to staff knowledge and learning? What are some "best practices" for capturing the knowledge from ESW and making these available for staff learning and use? What are the effective ways for capturing the knowledge from ESW and making it available for staff learning and use? And how does this ESW knowledge relate to business processes and outputs, such as other ESW? 1.3 At an annual cost of about US$127 million in FY05, ESW is a major component of the World Bank's operational services to its client members. As a business product, ESW is subject to regular reviews from the Bank's Quality Assurance Group (QAG). The Independent Evaluation Group also frequently comments on the quality of ESW in its country and sector reviews. These assessments improve the value of future ESW from a business perspective. This study complements, rather than duplicates, such reviews. 1The Bank has two categories of ESW: reports and policy notes. This study considers only the ESW classified as reports. To qualify as ESW, an activity must: (a) involve original analytic effort; (b) be undertaken with the intent of influencing an external client's policies and programs, and, (c) be "owned" by a specific Bank unit, i.e., the output must represent the views of the Bank (rather than be attributed to individual authors ­ be they Bank staff or consultants). ESW is the most well-known knowledge products of the Bank. In terms of their annual budgets, ESW is also the largest funded knowledge product within the Bank. 1 2. OBJECTIVES 2.1 A key goal of this study is to identify the processes whereby operational teams derive value from ESW. The study aims to understand how the knowledge base associated with ESW might be exploited and enhanced for staff use and learning and in support of the Bank's business objectives. The specific objectives are to: · Investigate how and the degree to which this knowledge is accessed and used within the Bank, including (a) how this knowledge contributes to the Bank's business objectives and (b) what constrains and facilitates access to and use of the ESW knowledge base within the Bank; · Explore how knowledge from a typical ESW is captured, stored, and shared among Bank staff; and, · Identify and recommend ways to promote staff access to, use of, and learning from ESW. 2.2 Table 1 lists the main questions under each objective for which this study sought answers, including the key information sources for these answers. Table 1: Evaluation objectives, questions, and information sources Evaluation Objectives and Questions Information Source A. Investigate how and the degree to which ESW knowledge is accessed and used A1. Web-based within the Bank, including how this contributes to the Bank's business survey objectives A2. Semistructured · To what degree are Bank staff aware of the ESW that is directly related to interviews their operational work? How do Bank staff typically become aware of ESW A3. Follow-on related to their work? survey · How accessible to Bank staff is the knowledge from ESW? To what extent is the current "system" for accessing this knowledge reliable and user- friendly? · To what degree have Bank staff accessed and used ESW for operational work (e.g., lending, supervision, technical assistance, strategic planning, etc)? How and to what degree have Bank units used the existing ESW knowledge base in staff learning? B. Explore how knowledge from a typical ESW is captured, stored, and shared B1. Semistructured among Bank staff interviews · How is the knowledge from completed ESW captured, stored, and shared? B2. Follow-on survey C. Identify and recommend ways to promote staff (and potentially client) access, C1. Semistructured use, and learning from completed ESW. interviews · How do existing systems, policies, and incentives encourage or discourage C2. Follow-on sharing and use of knowledge from ESW? What can be done to encourage survey more knowledge sharing and wider use of ESW knowledge within the Bank? 2 3. METHODOLOGY 3.1 This study used three complementary research methods. The first was a desk study that considered the availability of ESW in ImageBank, the official repository of formal Bank reports and the reasons why some ESW is not included in ImageBank. 3.2 The second approach involved semistructured interviews with the task team leaders (TTLs) of selected ESW and some of their managers, relevant country directors, and staff learning coordinators (SLCs). This approach was intended to get a picture of the supply side of ESW. WBIEG randomly selected one ESW product from FY03-05 from each sectoral and regional vice presidential unit.2 Interviews focused on how the "new" knowledge generated by the ESW was captured, verified, stored, and shared (including the underlying policies and incentives for doing so). The interviews allowed WBIEG to: (a) understand better the dissemination of ESW knowledge that occurs after the ESW product has fulfilled its main business-related function, and (b) identify and describe some useful practices that can support staff learning. 3.3 The third approach used staff surveys. The surveys were intended to understand the demand side of ESW. WBIEG conducted two surveys, the first of which was a general web-based survey of randomly selected operations managers and higher-level (GE+ level) operations staff about their access to and awareness and use of ESW in their current work. WBIEG used the surveys to gauge the overall availability and usefulness of ESW. The second survey was a follow-up on the earlier interviews. The TTLs, their managers, and country directors were asked to identify Bank operations staff or units for whom the knowledge from the ESW would be relevant and useful. WBIEG surveyed a sample of the Bank staff identified in this manner. This follow-up survey of the potential staff users of a specific ESW addressed: (a) Their awareness of the ESW; (b) The accessibility and their knowledge of the key findings; the usefulness and their actual use of the knowledge from (or findings of) the ESW in their operational work (e.g., lending, supervision, and strategic planning activities); (c) Their thoughts on whether and how existing systems, policies, and practices encourage or discourage sharing and use of "new" knowledge from ESW; and, 2According to the Bank's Business Warehouse (BW), as of September 2006 (when this study began), there were 2,095 ESW products in FY03-05. BW updates this figure regularly, and it reported that there were 2,154 ESW products in FY03-05 as of March 2007. 3 (d) Their suggestions for possibly broadening staff use and learning of ESW in support of the Bank's business. 3.4 Appendices A and B describe the interviews and the surveys, including the sampling framework, sample sizes and distributions, survey and interview methods, response rates, and the questionnaires used. 4 4. RESULTS The availability of ESW reports in the ImageBank 4.1 The World Bank's Internal Documents Unit (IDU, or more formally, the Library and Archives of Development) has the mandate to collect and make available Bank reports, including ESW products. While certain documents, such as specific Board reports, are systematically sent to the IDU, TTLs are supposed to submit all formal reports, such as ESW products, to IDU's ImageBank.3 If TTLs do not submit their reports, the ImageBank is unlikely to have them. 4.2 By design, ImageBank is the central "warehouse" of Bank reports, including ESW, which relevant Bank managers have authorized for internal access and use.4 ImageBank provides easy access to ESW documents, so it is the Bank's principal repository of ESW knowledge and is an essential part of the Bank's management of knowledge.5 4.3 What percentage of completed ESW is available in ImageBank? What are the main reasons for some ESW products not being available in ImageBank? What can the Bank do to get more ESW reports in ImageBank? 4.4 To answer these questions, WBIEG retrieved a list of all ESW delivered by the regions and the networks in FY03-05 from Business Warehouse (BW). Table 2 shows the breakdown of these ESW reports by region, network, and fiscal year. 3TTLs are expected to submit all Bank reports, especially those funded through a P0 number, to the IDU by completing a web-based submission form accompanied by an electronic copy of the document or by providing the IDU with a paper version. For ESW, it is worth distinguishing between "grey cover" and "non-grey cover" reports. "Grey-cover" reports are typically presented to the Bank's Board of Directors and almost mandatorily deposited in ImageBank. This is not the case with "non-grey cover" ESW reports, and it is with these reports where most of the non-reporting problems arise. 4The Bank's Integrated Records and Information Service (IRIS) is a depository for business records which include `in progress' ESW that must be deposited in ImageBank upon completion. 5ESW knowledge can be captured and stored in other forms and media. For example, some ESW reports are available in their complete or summary forms from related regional or sectoral websites. Also, summaries of the key ESW findings may be available in various formats (e.g., brochures, research briefs, newsletters, diskettes, etc.). 5 Table 2: ESW products by region and network, FY03-05 Supplements ESW reports Supplements and Fiscal year delivered delivered reports delivered (#) FY03-05 Total 146 1,949 2,095 FY03 Total 57 653 710 Regions 49 540 589 Networks 8 113 121 FY04 Total 46 666 712 Regions 43 546 589 Networks 3 120 123 FY05 Total 43 630 673 Regions 36 507 543 Networks 7 123 130 Source: Business Warehouse. Notes: BW uses the following definitions: Supplements Delivered (#) = count of ESW associated or an offshoot ("child") product of a main or bigger ("parent") ESW product; Total Delivered (#) = count of delivered ESW products excluding supplements. The table excludes 59 ESW reports produced by LEGVP and OPCVP. 4.5 WBIEG randomly sampled 50 ESW "reports delivered" from each of the three fiscal years to determine whether these reports are available in ImageBank. Using the P0 number and the "detailed search" option, WBIEG found that 26 percent (39 of 150) of the reports delivered were available.6 To determine why a large percentage of ESW products were not available in ImageBank, WBIEG contacted the relevant TTLs of the 75 ESW products (from FY04-05) that were not in ImageBank. These TTLs were asked to indicate the main reason for this unavailability (table 3).7 The responses suggest issues to be addressed to improve TTLs' delivery of ESW reports to ImageBank. Table 3: Reasons why ESW products are not in ImageBank Number of TTLs Should not be there because: In process 10 No permission from the relevant government to release study 9 Not an ESW (conferences, WBI activity). 4 Should be there, but not submitted because: Did not know Bank procedure or TTL responsibilities 26 ESW available but on another website(s) 10 Published in combination with another ESW 3 No response from TTL to WBIEG query 4 Not enough information to contact TTL8 9 Total ESW not in the ImageBank 75 6 Operations Policy and Country Services conducted a similar assessment of the availability of ESW in ImageBank. It found that approximately 27 percent of FY05-06 ESW products were available in ImageBank. 7WBIEG did not track ESW products completed in FY03 because of their age and the resources available for the evaluation. 8This means that no contact information for the TTL listed in BW was available from any of the publicly available data sources in the Bank. This suggests that the TTL is no longer in the Bank's service and that the Bank had no publicly available contact information for him/her. 6 4.6 Several of the TTLs who were informed that their ESW products were not in ImageBank then complied with their obligation to do so.9 It is therefore possible to increase compliance with the submission requirement by routinely reminding staff of its existence.10 Similarly, specific incentives for complying TTLs (e.g., KLB awards for ESW use in operational work) or simpler recognitions (e.g., invitations to submit or "thank you" notes) could raise submissions of ESW reports to ImageBank. 4.7 Finally, it should be noted that unless IDU is informed of the P0 number of an ESW, the status of the ESW will not be reflected in IDU's "meta data." "Meta data" track a knowledge stream that begins in SAP, is recorded and reported in BW, and is supposed to be deposited as a finished product in ImageBank. Thus, for example, for tracking purposes, the TTL of an ESW that is eventually combined with another major report must file a "meta data" statement with or without an attached ESW document. Additionally, a knowledge product for sale to a government (that does not want the analysis published) must file its P0 number with IDU, even though the filing may not have a document accompanying the "meta data" statement. A difficulty with tracking the status of ESW in ImageBank is the failure of some TTLs to file P0 numbers with the IDU.11 On the capture and dissemination of knowledge from ESW reports 4.8 Results from WBIEG's interviews support the earlier finding on the general unavailability of ESW in ImageBank. Of the 12 ESW reports randomly selected for discussion with their TTLs, five reports were not available in ImageBank at the time of the interview. Of the remaining seven, five were available in relevant regional, country, or network websites. Also, three of the 12 TTLs did not know of their responsibility to deposit their completed ESW in ImageBank. Some staff were also confused about the role of ImageBank, a depository of formal reports, versus that of the Integrated Records and Information Service (IRIS), a depository for project files, general correspondence, business records, etc., including "in progress" ESW.12 The differences in functions of these Bank-wide information storage and retrieval systems should be reiterated to Bank staff. In addition, as discussed later, the system for information storage and retrieval should be greatly simplified. 4.9 Most ESW reports, such as country economic memorandum and public expenditure reviews, supported by the Bank's region and some reports, such as poverty or sectoral assessments, supported by the networks, follow common formats or guidelines. These "standard" reports typically use earlier or similar ESW products as 9 It is likely that the most of the "no responses" (n=4) are in the general category "should be there, but not submitted." 10This positive response was likely due to WBIEG's use of telephone calls (followed by emails) to remind ESW TTLs of the IDU's filing requirement. The success of this approach lies in contrast with IDU's own experience in FY06. At that time, IDU sent email requests to TTLs of all ESW reports which had reached the "delivery to client" stage. The response IDU received was close to zero and, for this reason, IDU discontinued this effort. 11IDU needs to have the P0 number of an ESW in order to catalog the material correctly and, in most cases, IDU receives this information. The few cases in which P0 numbers are unavailable relate mostly to World Bank publications, i.e., material published by the Bank with an ISBN. 12The same confusion is suggested also in some respondents' answers to open-ended questions in the surveys. 7 references, and this is one area of use of the Bank's ESW knowledge base.13 The majority of ESW products are country focused 10 of the 12 ESW discussed in interviews were specific to a country. Country-focused ESW is typically disseminated within a region and among its staff for whom the ESW is relevant. 4.10 Due to common topical interests, regionally based ESW reports are more likely than their sector-based counterparts to reflect collaboration with other organizations, such as the International Monetary Fund and the Inter-American Development Bank. In this sense, ESW reports are useful staff tools for bridging research and learning partnerships while also leveraging resources with others outside the Bank. These ESW- based relationships also facilitate the exchange of past knowledge, including from the Bank's ESW knowledge base, among partner institutions. 4.11 Sector-based ESW often applies new frameworks or analytical approaches to alternative subsectors or countries. Such ESW is also more likely than regionally based ESW to result in publications, including journal articles, briefs, toolkits, and handbooks. Because of the nature of sector-based ESW products, they are more widely disseminated among Bank staff in different regions than is regionally based ESW. 4.12 Dissemination of an ESW report within the Bank starts with a concept note that informs Bank staff of the study's objectives and timeframe. At various later stages of the ESW exercise, peer reviews and presentations or discussions of draft materials are standard practices that enhance visibility within the Bank. The ESW process is, in itself, a learning experience for Bank staff, where new knowledge is created and disseminated, albeit only within a limited circle of participants and peer reviewers during this process. Broader dissemination occasionally happens outside this standard ESW process when other staff, particularly reviewers, use the "new" ESW knowledge in other Bank activities. 4.13 The interviews suggest that TTLs desire wide dissemination of their ESW products. Nonetheless, since the main audiences of these products are clients, the nature and extent of Bank-wide dissemination vary. Although relevant knowledge from a few ESW products is used in staff training, particularly during sector weeks, the dissemination of new ESW knowledge to Bank staff is often an afterthought, particularly in the case of "standard" reports. In fact, staff dissemination of ESW knowledge is often opportunistic rather than something planned beforehand. A dissemination strategy, which includes the allocation of resources for dissemination, is typically not a component of concept notes for ESW (although perhaps it should be). TTLs noted that lack of resources, particularly of staff time, was their key constraint to disseminating "new" ESW knowledge to Bank operations staff in learning activities such as staff training events and sector weeks.14 13Seven of the 12 ESW products discussed in WBIEG's interviews were "standard" ESW reports. Some of these reports, such as public expenditure reviews, are repeated every few years, so the issues and findings in these reports could be repeated. Not everything in an ESW product is new knowledge. 14This is best expressed by the comment of a TTL who said that "... even if the knowledge could be shared among the staff (as was the case with his ESW), the `reproduction time' to fulfill the needs of those who 8 4.14 For the typical TTL, the submission to and approval by clients and Bank management of the ESW product is the key objective.15 A few TTLs produce and circulate summaries of one or more of their key ESW findings in various media, such as discussion briefs, research notes, newsletters, brown bag lunches, and compacts disks that target clients and Bank staff. Nonetheless, TTLs noted that the impacts of these dissemination efforts on the operational work of Bank staff (outside the original country/regional focus of the ESW) remained uncertain. Among the TTLs interviewed, nearly all mentioned their need for guidance with disseminating ESW knowledge to Bank staff, particularly assistance with "best practices" or appropriate roadmaps for achieving operational impacts from their ESW. On the use of knowledge from ESW reports 4.15 To understand the demand side of ESW, WBIEG surveyed a random sample of over 700 GE+ staff on their awareness and use of ESW reports. Another survey of 58 Bank staff focused on awareness and use of a specific ESW product identified in the interviews with the TTLs. In the discussion that follows, these are referred to as survey A (the large survey) and survey B (the small survey), respectively.16 4.16 How do Bank staff rate their own awareness and use of ESW reports and their knowledge of key messages from these studies? Table 4 summarizes some results from the two surveys. The results from the larger sample reveal that Bank staff rated themselves as having, on average, moderate awareness of ESW related to their work (average rating of 3.2 on a 5-point scale, where 1 = "no awareness," 3 = "moderate awareness," and 5 = "full awareness"). Likewise, they rated their knowledge of the key messages from these ESW products and the usefulness of these products in their operational work as moderate (average ratings of 3.0 and 3.2, respectively on similar 5- point scales).17 These results suggest that Bank staff recognize that there is room for improving the impacts of ESW on their knowledge and operational work. Any such improvements should begin with raising awareness of ESW products. 4.17 Survey B found high staff self-ratings for awareness of a specific ESW related to their work. Seventy-nine percent of respondents mentioned that they were aware of the specific ESW about which they were queried. In addition, as shown in table 4, survey B respondents rated their knowledge and the usefulness of this ESW slightly higher than their survey A counterparts (mean scores of 3.7 and 3.5 respectively on the same 5-point scale).18 These higher ratings reflect the selective nature of the survey B are requesting `know how' of the created knowledge is a problem." Of course, however, if TTLs are not aware of the submission requirement, then any amount of time will not improve submission rates. 15As one TTL noted, "... the ESW process is largely output, rather than impact, driven." 16Appendix B describes these surveys in detail. 17There are no statistically significant differences in responses to these questions by region versus network, by grade level of staff, or by location (resident mission versus headquarters) of operational staff. Because of small sample sizes, it is difficult to compare survey responses across regions and/or across sectors. However, differences in responses by grade categories of staff, by years of work in the Bank (grouped data), and by location of operational staff are possible to report, and these are discussed, where appropriate and relevant, in various parts of this report. Additional detailed analysis using these latter variables can be provided on request. 18These differences between surveys are statistically significant. T-values for the test of differences of the means from the two surveys are t = 3.31 for "knowledge" and t = 1.14 for "usefulness." Chi-square tests 9 sample. These are still modest ratings, however, and they suggest that there is room to improve on the knowledge and operational impacts of ESW products on Bank staff. Table 4: Respondents' self-ratings on various ESW concerns 1 2 3 4 5 Weighted average None Low Moderate High Full Total rating Survey A % % % % % N % A. Awareness of work-related ESW reports 4 17 42 32 5 340 100 3.2 B. Knowledge of key messages in ESW 5 23 43 26 3 339 100 3.0 C. Usefulness of ESW reports 5 18 33 41 3 337 100 3.2 Survey B D. Knowledge of key messages in a specific ESW 4 13 25 25 33 23 100 3.7 E. Usefulness of a specific ESW 9 14 23 32 22 22 100 3.5 4.18 Table 5 lists the sources of staff awareness of ESW reports, arranged according to their importance. These results are from the two surveys.19 The results suggest that informal networks within the Bank are as important, if not more so, than structured arrangements (e.g., workshops, seminars, brown bag lunches) for becoming aware of ESW reports relevant to operational work. In addition, staff awareness of relevant ESW comes from active initiatives by staff themselves as much as from routine dissemination efforts (e.g., e-mails, posters, flyers, brochures, newsletters, research briefs, etc).20 Help desks are not useful in raising staff awareness of available ESW, at least in the opinions of those interviewed. 4.19 Respondents in survey A were asked to estimate the number of times they had searched ESW reports for knowledge or new ideas in the past six months. Of the 339 respondents to this question, 21 percent had never searched. Of those who did search, they had searched an average of 7.8 times over the past six months. Some 18 percent (49 of 267) reported no success in their search for any ESW report.21 Again, this finding, together with the general unavailability of completed ESW in Image Bank noted earlier, confirms that difficulties with accessing ESW reports are a key constraint to the use of this knowledge base by Bank staff in operational work.22 show that the distributions of these "knowledge" and "usefulness" variables are significantly different between surveys A and B. 19Both surveys asked respondents to check as many possible sources of ESW reports as listed in table 5. 20How Bank staff become aware of new ESW depends, in part, on their job grade. For instance, whereas GE-GF staff cite "active online search" more than their GG+ counterparts (i.e., 31 percent versus 18 percent, respectively from survey A), GG+ staff cite "Bank colleagues via word of mouth, email, or interoffice mail" more often than GE-GF staff as a source of their awareness of work-related ESW. A chi- square test shows a significant statistical difference between GE-GF staff vs. GG+ staff distributions of their source of ESW awareness (from survey A). 21Of these 49 respondents, 15 of them cited specific instances in which they failed: "download from Bank intranet website" (eight citations), "download from IRIS" and "request from member(s) of ESW originating unit" (five citations each), "download from Bank's external website" (four citations), and "download from ImageBank" and "request or borrow from a colleague" (two citations each). 22It is possible that difficulties arise because some TTLs may not know how to conduct a proper search for ESW products, but WBIEG's surveys were not designed to investigate the level of "search skills" among Bank staff. 10 Table 5: Sources of staff awareness of work-related ESW Survey A N % Bank colleagues via word of mouth, e-mail, or interoffice mail 257 26 Active online search 183 19 E-mails, posters, flyers, brochures, or newsletters from the ESW originating unit(s) 168 17 Workshops, seminars, BBLs, or other activities that relate to my work or interests 145 15 Bank's intranet announcement or mention 123 12 Research briefs from the ESW originating units 73 7 Visit (in person) to Bank/Fund libraries 19 2 External clients or other non-Bank persons 14 1 Help desks 8 1 Total number of respondents = 331 Total 990 100 Survey B N % Bank colleagues via word of mouth, e-mail, or interoffice mail 26 66 Workshop, seminar, BBL, or other activities that relate to my work or interests 6 15 Active online search 2 5 E-mails, posters, flyers, brochures, or newsletters from the ESW originating unit(s) 2 5 Research brief from the ESW originating unit(s) 1 3 Bank's intranet announcement or mention 1 3 External client or other non-Bank people 1 3 Total number of respondents = 18 Total 39 100 4.20 Clearly, addressing the related access and awareness problems is overdue. While the Bank has been in the knowledge management business for a decade and used to be cited often as a leader in this field,23 many Bank staff are unaware of, or have difficulty accessing, ESW knowledge. The Bank needs to find solutions to these issues, for otherwise, it is failing in knowledge management with its own staff. 4.21 The 218 respondents who reported some successes in their search had a cumulative success rate of 88 percent with locating either the full ESW reports they wanted or their summaries in various forms.24 These respondents used different, often multiple, channels to locate the ESW reports they sought (table 6). As shown, "download from the Bank intranet webpage of the unit that produced or sponsored the ESW study" was cited the most number of times (i.e., by 114 of 218 respondents). 23For five consecutive years ending in 2004, the Bank Group was named one of the world's top 20 Most Admired Knowledge Enterprises by Teleos, an independent knowledge management and intellectual capital research company. 24Not all ESW products are available as short summaries, and WBIEG has no estimates of the number of ESW summaries (and their form) that responsible Bank units publish and disseminate to staff. 11 Table 6: Access channels to ESW in the past six months, actual versus preferred Actual Preferred Channels used # % # % A. Download from Bank web page of the unit that produced 114 30 107 37 or sponsored the ESW study B. Download from ImageBank 93 25 74 25 C. Request a hard or electronic copy of the ESW report 52 14 51 17 from the member(s) of its originating unit D. Request or borrow from a colleague 47 12 9 3 E. Download from IRIS 27 7 21 7 F. Download from Bank external web 24 6 18 6 G. Request or borrow from a client 12 3 0 0 H. Request or borrow from a Bank library 2 1 8 3 I. Other channels 6 2 6 2 Sum of cited channels / preferred sources 377 100 294 100 4.22 When asked about their preferred source of ESW reports, the same respondents mentioned the above-mentioned channels in the same order, except for "request or borrow from a colleague," which ranked fifth among respondents' preferences. Chi- square tests show no significant differences between the response distributions for "actual" versus "preferred," or between the "actual" (and "preferred") responses of headquarters-based versus resident mission-based staff. However, one noteworthy statistical difference, albeit not shown in table 6, is between "actual" versus "preferred" responses for resident mission-based staff who appear to prefer "download from Bank webpage" and to dislike "request or borrow from a colleague." Overall, these results indicate a preference for self-managed (i.e., less dependent on others), and thereby quicker, ESW access channels. 4.23 A majority (65 percent) preferred reading hard copy reports to reading electronic copies (35 percent); reading well-written summaries (66 percent) to reading the full reports (27 percent); and studying the reports themselves (52 percent) rather than attending workshops, seminars, or brown bag lunches on these reports (40 percent).25 For ESW summaries, the same respondents rated them as slightly above "moderately effective" (i.e., mean score of 3.4 on a 5-point scale where 1 = "not effective," 3 = "moderately effective," and 5 = "fully effective") and slightly above "moderately useful" (i.e., mean score of 3.4 on a similar 5-point scale). These findings are shown in table 7. Survey B results show slightly higher mean scores on effectiveness and usefulness (i.e., t = 2.8 and t = 2.0, respectively). 4.24 Survey A asked respondents whether they had used knowledge from an ESW product in their work in the past six months. Of the 320 people who responded to this question, 67 percent noted that they had used some knowledge from ESW in their work in the past six months. Table 8 shows the specific areas where these respondents, from their write-in survey comments, reported using knowledge from ESW. As shown, the most frequently cited uses were in project-related operations work (30 percent), and in 25For the corresponding three questions in survey A to which these results pertain, the number of survey respondents varied from 271 to 297. 12 other ESW, research, or applied activities (20 percent). Only a few respondents reported uses in policy-specific contexts (9 percent) and in client or staff training (2 percent).26 Table 7: Respondents' ratings of the effectiveness and usefulness of ESW summaries 1 2 3 4 5 6 Moderately Fully Weighted Not effective effective or effective or average or useful Low useful High useful Total rating Survey A % % % % % N % A. Effectiveness of short summaries 2 13 40 36 9 298 100 3.4 B. Usefulness of short summaries 2 14 39 35 10 300 100 3.4 Survey B C. Effectiveness of short summaries 0 7 23 53 17 30 100 3.8 D. Usefulness of short summaries 0 7 33 43 17 30 100 3.7 4.25 Compared with the survey A findings, the results from the smaller survey indicate slightly different responses to the open-ended question about how ESW products are used. This again is due to the composition of the respondents. As noted in table 8, the results from survey B show considerable use of specific ESW products in project-related work. These results confirm what was gathered from the interviews with TTLs. Ten of the 12 TTLs noted that their reports had direct and immediate operational applications. Table 8: Uses of knowledge from ESW Survey A Survey B N % N % Project work, of which 112 30 16 45 Identification, preparation, design, and development (of projects) 87 14 Implementation and supervision (of projects) 25 2 Program work, of which 44 12 3 8 Program development 29 2 Program implementation 15 1 ESW, research and applied work, knowledge management 75 20 7 19 Technical assistance and capacity building 62 17 3 8 Policy preparation, design and implementation 33 9 4 11 Country assistance strategy and strategy planning 27 7 2 6 Training 8 2 Others 11 3 1 3 Total 372 100 36 100 26These is no significant difference between headquarters-based versus mission-based staff in their uses of ESW. 13 5. IMPROVING THE OPERATIONAL IMPACTS OF ESW 5.1 This study was designed to uncover issues and problems of "access and use by World Bank staff of the ESW knowledge" and did not intend to devise solutions to its potential constraints. Instead, WBIEG's approach is to present suggestions and constraints as important issues to be addressed in the future. An illustration of this approach is sampled in table 8 where "Uses of Knowledge from ESW" are summary responses to an open-ended survey question (A-10b). Similar open-ended questions (A- 8, A-9, and A-11) asked staff respondents to identify the main constraints to their access and use of the knowledge from ESW products in their operational work. Their responses to these survey A questions (including relevant comments from the interviews) are summarized below. 5.2 Approximately 27.4 percent of survey A respondents noted that there were no major hindrances to access and use of ESW in their Bank work.27 5.3 Among those who identified constraints to access and use, the main ones included: (i) Shortage of staff time to read (and to disseminate) ESW. As one respondent put it, ESW reports can be "... too lengthy, complex and technical. Also there are simply too many ESW [reports]. on the same issues. The sheer volume of available reports is overwhelming." Another respondent noted difficulties with identifying a priori the "... range of issues covered" in an ESW report. Another mentioned the "lack of meaningful ESW summaries" that are user-friendly (i.e., available in other relevant languages). Also, "... there are no incentives or rewards to disseminating ESW knowledge to Bank staff" (27.4 percent of survey A respondents who identified constraints to access and use mentioned this concern). (ii) Absence of convenient "search and download" facilities. Staff cited problems with: unhelpful web/ImageBank/IRIS "search" facilities; confusing coding issues including "incongruence of ESW titles with their contents" and "lack of clarity of topics from ESW titles"; slow downloads and difficulties with getting assistance, particularly for staff in country offices; lack of centralized information on and easy links to the Bank's ESW knowledge database; and the absence of systematic announcements of new ESW products and their main results (14.3 percent of survey A respondents who identified constraints to access and use mentioned this concern). 27It is difficult to quantify open-ended responses. Nonetheless, we provide some estimates of the frequency of certain responses to open-ended questions in this and later paragraphs. 14 (iii) Lack of relevance and the poor quality of ESW. Though not related to the issue of access to ESW, respondents mentioned the unrelatedness of ESW to their project work and the uncertain importance of these reports (e.g., "too technical and unable to operationalize findings"; "too academic to be transferable in specific contexts," "lack of solutions," "of operational guidelines," and "of updates or new information") as reasons for their less common use (9.5 percent of survey A respondents who identified constraints to access and use mentioned this concern). 5.4 The surveys and interviews also asked Bank staff for suggestions for improving the operational impacts of ESW reports. The suggestions address the three main constraints to enhanced use of knowledge in ESW that were noted above. Several of these represent existing practices, but they are implemented sporadically or only in a few Bank units. · Identify strategies and good practices for disseminating (or communicating) ESW results to staff as is the case, for instance, with reports on ESW from QAG but with a focus on impacts. Provide examples of how knowledge from ESW products has been used in operations work28 (21.9 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). · Organize internal access to all ESW, including of concept notes and of reports in progress, in a single, user-friendly webpage.29 This webpage would have useful links that are organized by region, country, sector, and topic and have search capabilities based on key words. The page could also provide a weblink (or telephone link) to a central "help desk" preferably patterned after the Bank's global "help desk" (i.e., 32121). (17.0 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). · Use the ESW process as an "action-learning" tool that, similar to the process for country assistance strategies (a) engages staff and clients from different regions and sectors early in the process, and (b) produces an "action matrix" for follow-up activities. (11.6 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). · Include an action plan and budget for disseminating ESW results to operational staff including those in resident missions as part of any ESW proposal or concept note. Incorporate "dissemination to Bank staff" as a criterion for QAG ratings of ESW. (10.8 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). 28Included here might be lessons from (or useful examples of) knowledge dissemination to staff of key messages from Bank products like the World Development Report, evaluation reports, etc. 29This is already done on a limited basis by some units. It could be expanded, with ESW put more comprehensively on one site. However, the risk is that if this effort is not done with full coverage, it might lose credibility quickly. 15 · Build processes for continuous dissemination of key ESW findings to staff and clients as these become available (e.g., circulate a list of all new ESW and their status periodically, including how these are being used in Bank operations). (7.8 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). · Build and distribute toolkits or guidelines for disseminating various types of ESW (sectoral versus regional) internally for maximum impact (as part of building a knowledge management culture within the Bank). Remind TTLs to deposit their ESW products in ImageBank. Perhaps electronic management approval controls might be considered so that when a manager approves a final ESW, s/he needs to do so electronically, and the ESW report is simultaneously sent to ImageBank. (6.6 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). · Offer incentives to operational staff to read and use ESW reports, e.g., provide staff with a fixed amount of time to read ESW products by instituting a unit-based SAP code for "research" where staff can charge their time as needed. (5.9 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). · Produce and disseminate short and targeted summaries, preferably electronic, and other outputs (e.g., videos, workshops, newsletters, brown bag lunches). Make summaries mandatory for all ESW. Use videoconferences for dissemination. (5.9 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). · Recognize and reward knowledge sharing and dissemination as much as knowledge creation and application (i.e., technical and operational achievements) in annual reviews of performance and in the development of results agreements. (2.9 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). · Establish awards or recognition for "best knowledge management practices." The dissemination of ESW for operational use may be a special award criterion, if not an independent award (e.g., KLB award for excellence and innovation in disseminating ESW reports for operational use). KLB awards could also be patterned after the Bank's "Spot Awards" to include some monetary awards. (1.6 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). · Finally, OPCS should revise its several procedural guidelines on ESW (Appendix F) to ensure consistency and to ensure that these guidelines make clear authors' obligation to submit ESW products to the ImageBank. OPCS guidance now indicates that these products should be filed in IRIS.30 The fact that there are various ESW repositories currently, including regional and 30In general, Bank staff are not sure of the differences between IRIS, ImageBank, and the intranet for recordkeeping. This needs to be clarified. 16 sectoral websites, constitutes a challenge to staff. The system for ESW information storage and retrieval should be greatly simplified. (1.6 percent of survey A respondents who offered suggestions for improving ESW impacts mentioned this item). 17 18 APPENDIX A: INTERVIEWS WITH TTLS Sample selection In selecting the TTLs for interviews, WBIEG first sought a list of all ESW reports completed in FY04-05 from Business Warehouse (BW). This search identified 666 ESW products completed in FY04 (regions = 546 and networks = 120) and 630 in FY05 (regions = 507 and networks = 123).31 For each ESW product, BW lists project number (or P0 number), a brief title assigned by the SAP information system, the name of the TTL, the region, the network, and some other ESW particulars. From the list of 1,296 ESW products, WBIEG excluded all policy notes. WBIEG then randomly selected 12 from the remaining list, one each for the six regions and six networks.32 Table A.1 shows the ESW products from FY04-05 ESW from which the random sample was chosen. Table A.1: Distribution of completed ESW, by fiscal year, region, and network Combined Delivered Total total Total cost (million supplements (N) delivered (N) delivered (N) US$) FY04 total 46 666 712 107,782 Regions 43 546 589 92,540 AFR 23 152 175 22,698 EAP 12 67 79 14,904 ECA 4 121 125 20,813 LCR 1 80 81 12,030 MNA 2 62 64 9,173 SAR 1 64 65 12,922 Networks 3 120 123 15,242 ESD 0 11 11 1,311 FSE 1 52 53 7,433 INF 2 9 11 799 HDN 0 20 20 2,604 PRM 0 4 4 245 PSD 0 24 24 2,850 (Table A 1 continues on next page) 31These reports were delivered at cumulative costs of US $107.8 million and $127.4 million, respectively. The costs include deliveries of 46 and 43 ESW supplements for the same two years, respectively. Therefore, the two years' combined total is 1,296 ESW products and 89 supplements with a cumulative cost of $235.2 million. 32Prior to July 1, 2004, outputs types of (a) consultations and country dialogue and (b) conferences or workshop were ESW products. Thus, while selecting ESW products to be sampled, WBIEG found a few such titles such as a "workshop" or a "training program" offered to clients and staff. WBIEG did not include any of these in its sample used to determine which TTLs would be interviewed. 19 (Table A 1 continued) Combined Delivered Total total supplements (N) delivered (N) delivered (N) Total cost (US$) FY05 total 43 630 673 127,398 Regions 36 507 543 106,824 AFR 15 130 145 35,256 EAP 11 66 77 13,277 ECA 0 98 98 17,004 LCR 3 71 74 14,956 MNA 5 64 69 12,913 SAR 2 78 80 13,418 Networks 7 123 130 20,574 ESD 1 21 22 2,584 FSE 1 36 37 6,188 INF 3 21 24 3,132 HDN 1 17 18 3,152 PRM 1 7 8 3,171 PSD 0 21 21 2,347 FY04 and FY 05 totals 89 1,296 1,385 235,180 Regions only 79 1,053 1,132 199,364 Networks only 10 243 253 35,816 The interviews WBIEG conducted face-to-face interviews lasting about 45 minutes with the TTLs of the selected ESW products. WBIEG recorded the interviews only when the interviewee granted permission. A set of guideline questions steered these informal interviews. A sample list of these guideline questions is in Appendix E. 20 APPENDIX B: WEB-BASED SURVEYS OF BANK OPERATIONS STAFF Sample Selection for Survey A For the web-based survey A, WBIEG initially considered all active regular Bank staff in the GE-GH levels working in the regions, networks, and other units for whom ESW would be relevant (e.g., WBI, Development Economics, etc.). WBIEG then deleted staff in these units whose job titles related to general services, administration, communications, procurement (other than specialists), cofinancing, facilities, and resource management. From this shortened list, totaling 3,798 staff members, WBIEG randomly drew a 20 percent sample. Table B.1: Distribution of survey population and sample response rates Survey population Responded sample Total Total Response PMU GE-GG GH+ sampled GE-GG GH+ responses rates Regions 432 111 543 200 44 244 45% AFR 122 29 151 59 10 69 46% EAP 72 17 89 35 5 40 45% ECA 75 26 101 38 14 52 51% LCR 69 13 82 27 6 33 40% MNA 34 14 48 18 5 23 48% SAR 60 12 72 23 4 27 38% Networks 103 37 140 52 13 65 46% ESD 23 8 31 9 1 10 32% FSE 17 2 19 9 1 10 53% HDN 13 8 21 4 4 8 38% INF 34 10 44 20 4 24 55% PRM 12 6 18 8 3 11 61% PSD 4 3 7 2 2 29% Operations 58 19 77 26 5 31 40% OPCS 11 5 16 1 1 6% DEC 23 9 32 12 3 15 47% WBI 24 5 29 13 2 15 52% Total 593 167 760 278 62 340 45% Survey population by location Survey respondents by location HQ 493 206 42% Resident Missions 267 134 50% Total 760 340 45% Survey B Headquarters 35 17 49% Resident missions 23 13 57% Total 58 30 52% 21 Table B.1 shows the distribution of the sample by the unit and staff grade.33 Of this sample, 493 were based in Washington while the remainder (or 267) were scattered across 82 resident missions (see table B.2). The sample includes 208 job titles. The sample included 32 managers or country managers and 21 sector managers. The web-based survey consisted of 14 questions of which five were open-ended questions with subparts, and three gathered information on grade level, years of service, and location. The questionnaire took less than 15 minutes to complete. An initial request to participants was sent via e-mail, followed by a reminder e-mail after a week. A second reminder was sent to those who had not submitted responses after three weeks from the initial survey launch date. Several telephone calls were also made to remind people to complete the surveys. A final e-mail reminder was sent five weeks after the initial launch date. As shown in table B.1, 340 respondents (45 percent of the sample) completed and returned the questionnaire. Another 90 staff in the sample viewed the questionnaire but declined to complete and submit it. Of these 90 Bank staff, 19 respondents sent e-mails explaining that their work had no relation to the use of the ESW knowledge base.34 Table B.1 shows that Bank staff in grades GE-GG were more likely to respond to the survey (47 percent response rate) than those in grades H+ (37 percent response rate). Survey response rates also differ by location of the staff (e.g., for survey A, response rates were 42 percent versus 50 percent for staff in HQ vs. staff in resident missions, respectively). While these differences in response rates could influence the overall ratings found, section 4 reports tests of statistical differences for various ratings across various "types" of survey respondents. Finally, WBIEG compared the survey responses of the first 50 respondents with those of the last 50 to determine if delayed responses (and perhaps the nonresponses) were on account of a lack of knowledge of ESW. The results showed no statistically significant differences in the responses of these two groups. Sample selection for the follow-up survey B that pertains to specific ESW WBIEG requested the TTLs of 12 randomly selected ESW to suggest names of Bank staff or units (including from outside of their own unit) for whom their ESW products would be relevant. WBIEG then sampled five Bank staff names per interviewed TTL from all the staff names that WBIEG could assemble for a particular ESW. The sample population for this follow-up survey consisted of 58 names.35 This sample 33The GE-GG break point in this table is different from the one in footnote 19. This is because it is HR policy to reveal staff grades only within certain broad bands, particularly when the information for use in situations (e.g., staff surveys) where "confidential" information might be compromised. The results reported in footnote 19 are based on actual reports on job grade by survey respondents. 34If the 19 respondents who noted that ESW are not relevant in their line of work are excluded, the aggregate response rate climbs to 47.5 percent. For all survey questions, WBIEG placed responses from this group in the "not applicable" category. 35WBIEG was two names short of the target of having 60 staff in the sample. This was either because some TTLs were reluctant to share this information or because the primary author was not the TTL and claimed not have this information. 22 covered 41 units both in Washington and at resident missions. Table B.2 shows the distribution of these follow-up respondents by their staff grade and location. The follow-up survey consisted of questions that, while similar to those in the first survey, related directly to staff's awareness, access, and use in his/her operational work of a particular ESW from the interviews. As with the first survey, the follow-up survey required about 15 minutes to complete. WBIEG also tried various attempts to increase response rates with reminder e-mails and telephone calls to staff over a period of four weeks. Table B.2: Characteristics of survey respondents Survey A Survey B Staff Grade Number % Number % GE 37 11 1 3 GF 83 25 2 6 GG 144 43 15 47 GH 68 20 12 38 GI+ 2 1 2 6 Total 334 100 32 100 Years of service Less than 5 years 130 39 7 23 5 up to less than 10 years 89 27 6 19 10 or more years 115 34 18 58 Total 334 100 31 100 Respondents' location: Headquarters 201 62 17 57 Resident missions 122 38 13 43 Total 323 100 30 100 Location: Headquarters 493 65 35 60 Resident missions 267 35 23 40 Total 760 100 58 100 23 APPENDIX C: SURVEY A Survey on the Bank Staffs' Access to and Use of Economic and Sector Work (ESW) Reports This questionnaire seeks information on Bank staffs' access to and use of the knowledge from completed ESW reports (i.e., reports that have been released to their targeted clients). The purpose of this survey is to assess to what extent operational teams currently use the knowledge gathered through ESW in their work, and to identify ways and means for improvement. All survey responses will be treated confidentially and reported only in aggregate form. 1. Each year, the Bank publishes over 700 ESW in different thematic and sectoral areas to provide policy assistance to clients. How would you rate yourself on the following areas: None Low Moderate High Full Areas 1 2 3 4 5 A. Your awareness of ESW reports that are related to your operational work or general interests B. Your knowledge of the key messages in these ESW reports C. The overall usefulness of these ESW reports to your operational work 2. Your awareness of ESW reports that are related to your work or interest usually comes from (select all that apply): Active online search Emails, posters, flyers, brochures, or newsletters from the ESW originating units Research briefs from the ESW originating units Bank colleagues via word of mouth, email, or interoffice mail Workshops, seminars, BBLs, or other activities that relate to my work or interests Bank's intranet announcement or mention Help desks Visit (in person) to Bank/Fund libraries External clients or other non-Bank persons Other channels, please specify: _______________________ 24 3a. In the past six months, have you searched ESW reports for knowledge or new ideas? Yes No 3b. If yes, roughly (how many) ___ times 4. In the past six months, were you successful in finding the ESW reports you sought? Not successful in obtaining any of them Yes, I was successful in locating (how many) ____ different ESW reports or their short summaries out of (how many) ____ that I sought. Among them, (how many) _____ were the original full reports, and (how many) ____ were short summaries in various forms. 5a. If you successfully found ESW reports that you needed in your work in the past six months, which of the following channels did you use and how long, on average, did it take you to obtain the ESW report(s) you sought? Select/answer all that apply. Channel How long did it take you? (A) Requested a [hard or electronic] copy of the ESW report from [the member(s) of) its originating unit ___ Days ___ Hours (B) Downloaded from Bank intranet [web page] of the unit that produced or sponsored the ESW study ___ Days ___ Hours ( C) Downloaded from Bank external web ___ Days ___ Hours (D) Downloaded from ImageBank ___ Days ___ Hours (E) Downloaded from IRIS ___ Days ___ Hours (F) Requested or borrowed from a Bank library ___ Days ___ Hours (G) Requested or borrowed from a colleague ___ Days ___ Hours (H) Requested or borrowed from a client ___ Days ___ Hours (I) Other channel, please specify: ____________________________ ___ Days ___ Hours 5b. Which of the above channels (as referenced above) do you prefer to obtain ESW reports? (A) (B) (C) (D) (E) (F) (G) (H) (I) Other: .............. Other: ............. Other: .............. 25 6. Considering your time and learning style, what do you do to get the knowledge from ESW? Choose one or both for each of the following pairs: (a) read hardcopies versus (b) read (online) electronic copies (a) read the full reports (long (b) read short summaries (e.g., discussion briefs, version) versus "best practice" booklets, research/technical notes) (a) study the reports myself (b) attend workshops/seminars/BBLs that introduce versus these reports 7. Some of the important "new" knowledge and ideas from ESW reports are summarized in various short forms (e.g., discussion briefs, "best practice" handbooks, research/technical notes, or toolkits/guideline booklets) that are disseminated to Bank staff in hardcopy or made available online. 7a. How would you rate the overall effectiveness of these summaries in disseminating key ESW messages? Low Moderate High No Not effective at all effectiveness Fully effective opinion 1 2 3 4 5 7b. How would you rate the usefulness of these (short) summaries to your own work? Low Moderate High No Not useful at all usefulness Totally useful opinion 1 2 3 4 5 8. What barriers hinder your access to the knowledge from ESW reports? How can these be reduced? _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ 9. How can the Bank facilitate staff access to its growing ESW knowledge base? _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ 10a. In the past six months, have you used knowledge from ESW reports in your operational work? Yes No 26 10b. If yes, how or in what specific operational activities (i.e., project/program development, supervision, technical assistance, strategy planning, capacity building or strengthening, policy/program implementation, and others) did you use this knowledge? Please specify. _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ 10c. Did you receive any feedback with regard to the use of ESW in the operation(s) you mention above? Yes No 10d. If yes, from whom (colleagues, peer reviewers, manager, Network anchor, operational adviser, other donors, clients, others) did you receive this feedback? _____________________________________________________________________________ _____________________________________________________________________________ ____________________________________________________________________________ 11a. What are the main constraints you experienced for using the knowledge from existing ESW in your operational work? _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ 11b. What can the Bank do to encourage or facilitate staff use of its ESW knowledge base in the Bank's operational work? _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ 12. Where are you located? World Bank Headquarters Country Office: Please specify: _____________________ 13. In all, how long have you worked for the World Bank (including time as consultant/temp)? Less than a year 1 up to less than 2 years 2 up to less than 5 years 5 up to less than 10 years 10 or more years 27 14. What is your current grade level? GE GF GG GH Other (specify): ____________________________ Thank you. Submission of the completed survey is highly appreciated! 28 APPENDIX D: SURVEY B Survey on the Bank Staffs' Access to and Use of Economic and Sector Work (ESW) Reports The purpose of this survey is to help the Knowledge and Learning Board improve the capture, storage, accessibility, sharing, and use of ESW-generated knowledge for staff learning and operational work. This questionnaire seeks information on your awareness of, access to, and use of a specific ESW report: (NAME of the ESW report released on month/year). All survey responses will be treated confidentially and reported only in aggregate form. 1a. Are you aware of this ESW report? ? Yes No 1b. If yes, how would you rate yourself on the following areas: Areas None Low Moderate High Full 1 2 3 4 5 D. Your knowledge of the key messages in this ESW report E. The overall usefulness of this ESW in your operational work 2. How did you become aware of this ESW report? (select all that apply): Active online search Emails, posters, flyers, brochures, or newsletters from the ESW originating unit(s) Research brief from the ESW originating unit(s) Bank colleagues via word of mouth, email, or interoffice mail Workshop, seminar, BBL, or other activities that relate to my work Bank's intranet announcement or mention Help desk Visit (in person) to Bank/Fund library External client or other non-Bank people Other channels, please specify: _______________________ 3a. Have you used knowledge from this ESW report in your operational work? Yes No 3b. If yes, how or in what specific operational activities (i.e., project/program development, supervision, technical assistance, strategy planning, capacity building or strengthening, policy/program implementation, and others)? Please specify. _______________________________________________________________________________ _______________________________________________________________________________ 29 3c. What would be your estimate of the demand for this ESW by the other Bank staff in general? Please explain. _______________________________________________________________________________ _______________________________________________________________________________ ______________________________________________________________________________ 4. Were there other sources of knowledge (including past ESW) that may have conveyed information equivalently useful to your operational work as that provided by this ESW? Yes No 4b. If yes, please list below, those other sources of knowledge that may have provided equivalent information as that from this ESW? Please elaborate your answer, as necessary. _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ The following questions refer to all ESW in general: 5. Some of the important "new" knowledge and ideas from ESW reports are summarized in various short forms (e.g., discussion briefs, "best practice" handbooks, research/technical notes, or toolkits/guideline booklets) that are disseminated to Bank staff in hardcopy or made available online. 5a. How would you rate the overall effectiveness of these summaries in disseminating key ESW messages? Low Moderate High No Not effective at all effectiveness Fully effective Opinion 1 2 3 4 5 5b. How would you rate the usefulness of these (short) summaries to your own work? Low Moderate High No Not useful at all usefulness Totally useful opinion 1 2 3 4 5 6. What barriers hinder your access to the knowledge from ESW reports? How can these be reduced? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 30 7. How can the Bank facilitate staff access to its growing ESW knowledge base? ______________________________________________________________________________ ______________________________________________________________________________ 8. How do existing systems, policies, and incentives encourage or discourage sharing and use of knowledge from ESW? ______________________________________________________________________________ ______________________________________________________________________________ 9a. What can be done to encourage more sharing of ESW knowledge within the Bank? _____________________________________________________________________________ _____________________________________________________________________________ 9b. What can the Bank do to encourage or facilitate staff use of its ESW knowledge base in the Bank's operational work? _____________________________________________________________________________ _____________________________________________________________________________ 10. What can you recommend as best practices for promoting staff access, use, and learning from completed ESW? _____________________________________________________________________________ _____________________________________________________________________________ 11. Where are you located? World Bank Headquarters Country Office: Please specify: -----------------_ 12. In all, how long have you worked for the World Bank (including time as consultant/temp)? 2 up to less than 5 years 5 up to less than 10 years 10 or more years 13. What is your current grade level? GF GG GH Other (specify): --------------- Thank you. Submission of the completed survey is highly appreciated! 31 APPENDIX E: GUIDELINE QUESTIONS FOR INTERVIEWS WITH TTLS Each year, the WB completes about 700 ESW on various topics. These annual studies amount to a very large collection of ESW knowledge over time. 1. First, we would like to ask you about the interactivity/knowledge sharing between the past and present ESW. · When preparing your ESW, how and to what degree did you actively search past ESW related to the topic of your own ESW? · How and to what degree did you use the existing ESW knowledge base in your own ESW? · In your experience, are there other ESW that cover the same key issues and/or findings of the ESW study you have completed? Please elaborate. · Who were the intended audience of your ESW? To what degree was Bank staff part of this intended audience? Please elaborate. 2. We would like to ask you about the dissemination to Bank staff (within or outside your unit) of your ESW. · How did you make other Bank staff aware of your ESW, its "new" knowledge (or key messages) from your ESW? · Is your ESW available in ImageBank? If no, why not? · What factors may have hindered the dissemination of the "new" knowledge from your ESW? What factors may have facilitated the dissemination of this "new" knowledge? 3. We would also like to know how the knowledge from your ESW has been captured and stored. · Has the knowledge from your ESW been captured and disseminated in summary form (e.g., web-based, diskette, or hard copies of research briefs, unit newsletters, "best practice" brochures, and so on)? · If yes, in what forms? In your view, which of these forms have been the most effective in disseminating the "new" knowledge from your ESW to Bank staff? · Are the key findings from your ESW captured and stored in some other form or media that Bank staff can reference for their use? 32 · Do you think there are things that the Bank can do to improve the capture, storage, and availability/sharing of ESW knowledge among Bank staff? If so, what? 4. We would like to ask you about the use by Bank staff (within or outside your unit) of your ESW. We would like to know also of any outcomes and impacts from this use by Bank staff of your ESW. · How would you characterize the demand for the knowledge in your ESW? Were there a large number of requests with in the Bank? What purpose or potential uses were behind these requests? · Can you think of how Bank staff has used the knowledge from your ESW? What outcomes have resulted from these? Please describe. · What sort of evidence "on the ground" have you seen from these outcomes or results? Please describe. · How and to what degree have Bank units (including your own) used the existing knowledge you're your ESW in staff learning? · How do existing systems (policies, practices, and incentives) encourage or discourage sharing and use of knowledge from ESW? · What would you say are the three main barriers preventing relevant Bank staff from using the knowledge from your ESW? · What would you say are the three main facilitators encouraging Bank staff to use the knowledge from your ESW? 5. Do you think there are things that the Bank can do to improve the use of the ESW knowledge base by Bank staff in their operational work? If so, what? 6. Could you please list 5 Bank operational staff (at least 3 from outside of your unit) for whom the results of your ESW would be relevant for use in their operational work? Name Unit Extension a. __________________ _______ _________ b. __________________ _______ _________ c. __________________ _______ _________ d. __________________ _______ _________ e. __________________ _______ _________ 33 APPENDIX F: PROCEDURAL GUIDANCE TO TTLS ON ESW The Operations Policy and Country Services (OPCS) has published procedures to guide TTLs through the ESW process. These ESW process and guidelines have been evaluated and revised respectively over time.36 The guidelines, which include several definitions, SAP steps, quality concerns, and best practices, are provided conveniently in the OPCS's intranet site.37 This website, however, contains at least two links to separate web pages that each contains instructions or steps for following the ESW process. The first webpage can be found under the "Non- Lending" tab in the OPCS webpage, and the second under the "Operations" portal in the same OPCS homepage. Although they contain similar instructions, these two webpages are not exactly the same, particularly with regard to disclosure and publication. These are reproduced below (sections A and B) for reference. Section C below contains OPCS guidelines for preparing an ESW concept note. A. ESW GUIDELINES FROM THE "NON-LENDING" LINK IN THE OPCS WEBPAGE Identification Create the Activity Initiation Summary (AIS) in SAP. The designated managers (typically the Country Director and Sector Manager) approve the activity and the VPU Release Authority releases the funds. At this time, the actual AIS sign-off date is automatically populated in SAP. Prepare Concept Note. In general, a Concept Note covers: (i) the context, relevance, and timeliness of the task; (ii) the content, objectives and scope of the task; participatory processes to be followed during preparation of the product; (iii) the expected impact of the work; (iv) team composition and resources (Bank, partners, client, including trust funds); (v) the dissemination/follow-up plan, and (vi) the timetable for the activity. The length and level of details for an ESW concept note depends on the activity's scope and complexity. Concept Note Review Peer Review/Quality Enhancement. It is recommended that the ESW activity be subject to peer review/quality enhancement. Guidance on the selection of peer reviewers or the quality enhancement process should be provided by Regional/Network ESW Guidelines. Prepare and Circulate Concept Note Review Package. For smaller tasks, a virtual review meeting may be more appropriate. 36Examples include: "Fixing ESW: Whereare we?" (CODE2000-76, July 11, 2000); "Fixing ESW: Phase II-Challenges and Next Steps" (SEC-M2000-0431, June 28, 2001); the 2002 "Report of the Advisory Panel on Assessing Economic and Sector Work"; and "Improving the Bank's Analytic and Advisory Services ­ Progress Report" (2004). 37This is accessed by typing "OPCS" in the URL bar. 34 Prepare and Circulate Minutes of the Concept Note Review Meeting. Guidelines should include the recommended distribution list for concept notes, minutes of review meetings, draft output, and other relevant material. Record the Activity Implementation Start/Concept Note Date. In the Activity Update Summary (AUS) in SAP, enter the actual milestone Activity Implementation Start/Concept Note i.e., the date when the team initiates the actual development of the product or the date when the concept note is reviewed/approved. Draft Output Review (a) Prepare Draft Review Package and circulate for comments to all relevant staff/managers, including peer reviewers. Based upon comments received, and if needed, organize a meeting to discuss suitability and readiness of draft output for discussion with client. (b) Record the Decision Meeting Date. In AUS, update the actual milestone Decision Meeting date, i.e., the date the draft product is reviewed (review of the objectives, scope, likely impact on clients, new issues that may have emerged since the concept stage, etc.). (c) Prepare and Circulate Decision Meeting Minutes. Delivery to Client (d) Send Written Draft Output to the Client. All ESW outputs should be filed in IRIS. (e) Record the Delivery to Client Date. In AUS, update the actual milestone Delivery to Client, i.e., the date when the results, findings and conclusions of a task are substantially communicated to the client. This is the stage when a task is considered as delivered for counting Bank outputs and used for corporate monitoring. (f) Based on clients' comments, revise the document. The TL should consult with the client about the appropriateness to share the document with the Board, and subsequently to make it available to the public. Identify any confidential/sensitive information and make adjustments to the output as appropriate. Final Delivery/Dissemination Disseminate Output to Relevant Stakeholders. Dissemination is an important part of most ESW tasks. The TL is required to file in IRIS all the dissemination material, such as list of participants, presentations, agenda, talking points, minutes of the meeting, etc. 35 Record the Final delivery Date. In AUS, update the actual milestone Final Delivery, i.e., the date of the finalization of the output, taking into account client comments, and including its related dissemination activities. Disclosure and Publication Approval to Distribute to Board and InfoShop. The Country Director, as the official typically designated by the RVP, clears the document and is responsible for the decision regarding distribution to the Board. [For regional and global ESW, the Chief Economist and the Sector Board Chair, respectively, are typically designated to play this role]. When the output is approved for distribution to the Board (if applicable), the TL prepares a memo from the CD to SECBO, authorizing printing and distribution to the Board. The CD indicates that the document may be made available to the public in the Board Submission form. If the CD decides not to disclose the ESW, the cover of the report carries a standard caveat indicating that its release is restricted. For more information on Board Submission and related forms, go to the SECBO website. External Publication with Disclaimer/Copyrights. The CD [or the official designated by the RVP/NVP] clears the document and is responsible for the decision regarding publication. Use the standard disclaimer and copyright statement for papers approved by RVPs or Heads of other Bank Units: Standard Disclaimer: "This volume is a product of the staff of the International Bank for Reconstruction and Development / The World Bank. The findings, interpretations, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries." Copyright Statement: "The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. (a) For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, www.copyright.com. 36 (b) All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org." Evaluate the Impact of the Task Create the Activity Completions Summary (ACS) in SAP. ACS procedures require the TL to submit an ACS in SAP for all ESW tasks irrespective of their cost. The ACS should be completed within six months of the delivery to client date as long as all tasks associated with the activity are complete. If however all work associated with the finalization of the task have not yet been completed (e.g., dissemination; distribution to the Board, if applicable), a delay in ACS submission may be justifiable. The designated managers (typically the Country Director and Sector Manager) approve the ACS. At this time, the actual ACS date is automatically populated in SAP. B. ESW STEPS FROM THE OPCS "OPERATIONS" PORTAL Identification (i) Activity Initiation Summary (AIS) released (ii) Prepare Concept Note Concept Note Review (i) Select Peer Reviewers (ii) Prepare and Circulate Concept Note Review Package (iii) Prepare and Circulate Minutes of the Concept Note Review Meeting (iv) Record/Update Activity Implementation Start/Concept Note Date Missions and Trust Funds (i) Prepare Terms of Reference for Identification /Preparation Mission (ii) Prepare Back to Office Report for Identification/Preparation Mission (iii) Request for Trust Funds (if applicable) Output Preparation and Review (i) Prepare Written Draft (ii) Select Peer Reviewers 37 (iii) Prepare and Circulate Draft Review Package (iv) Prepare and Circulate Decision Meeting Minutes (v) Record/Update the Decision Meeting Date Delivery to Client (i) Send Output to the Client (ii) Record/Update the Delivery to Client Date Final Delivery/Dissemination (iii) Based on Clients' Comments, Revise the Document (if applicable) (iv) Disseminate Output to Relevant Stakeholders (v) Record/Update the Final Delivery Date Disclosure and Publication ESW Requiring Distribution to Executive Directors for Information (if applicable) (i) Obtain Internal Documents Unit (IDU) Report Number/Cover page (ii) Authorize and Send ESW to SECBO (iii) Send Output to the Printshop (if applicable) ESW not Requiring Distribution to Executive Directors (iv) Obtain Internal Documents Unit (IDU) Report Number/Cover page (v) Send Output to Internal Documents Unit (IDU) (vi) Send Output to the Printshop (if applicable) Evaluate the Impact of the Task (i) Complete the Activity C C. ESW CONCEPT NOTE FRAMEWORK ESW Concept Note Framework Note: The length and level of detail for a ESW concept note depend on the activity's scope and complexity. To ensure your activity is properly coded as ESW, please refer to the ESW/TA Decision Tree. 38 1. Context/Relevance/Audience: o client/audience of this activity o policy/program decision that will be supported by this activity o appropriateness and relevance of the activity in the context of CAS (if applicable) o links to past and on-going activities o value added of this activity 2. Objectives/Result Indicators: o development objective(s) of the activity o result indicators (i.e., interim outcomes) under the given objective(s) 3. Scope/Methodology: o summary of issues to be covered o methodology & rationale why this is the right instrument to achieve the stated objective(s) 4. Consultations with Client and Other Stakeholders: o client ownership and commitment (e.g., access to information, resources, input, dissemination) o scope of complementary work on the topic by development partners, and relationship to proposed ESW activity o contributions from other donors (if applicable) o plan for dissemination and consensus building 5. Resources: o staffing (including consultants) o peer reviewers o management/oversight o estimated costs (BB and TF) 39 6. Timetable: Milestones Date Activity implementation start/concept note [1] Field Work Draft Decision Meeting [2).](if applicable) Delivery to client [3] Final delivery (i.e., dissemination, if applicable) [4] Self-Evaluation (Activity Completion Summary) [5] Note: The TTL should enter in SAP the relevant milestones for (i) Activity implementation start/concept note; (ii) Decision meeting; (iii) Delivery to client; (iv) Final delivery, and; (v) ACS. For more information on how to update milestones in SAP, refer to the x32121 website. [1] Activity Implementation Start/Concept Note is the date when task teams initiate the actual development of the product or the date when the concept note is reviewed/approved. []]Decision Meeting is the date the draft product is reviewed (review of the objectives, scope, likely impact on clients, new issues that may have emerged since the concept stage, etc.). [3]Delivery to client date is the date when the output is delivered to the client i.e., when the results, findings and conclusions of a task are substantially communicated to the client or when the work toward the activity's objective(s) ends. This is the stage when a task is considered as delivered for counting outputs. [4]Final delivery date is date of the finalization of the output, taking into account client comments, and including its related dissemination activities (if applicable). [5]Activity Completion Summary is a self-evaluation mechanism that ties ESW task objectives to interim outcomes. ACS is required for all ESW tasks irrespective of the activity cost. ACS is completed and submitted for approval in SAP and it consists of the following two parts: (i) rating each result indicator selected in AIS/AUS that specifies the interim outcome of a development objective (DO); and (ii) summarizing major conclusions/ recommendations resulting from the task. The ACS should be completed within six months of the delivery to client date as long as all tasks associated with the activity are complete If however all tasks associated with the finalization of an activity have not yet been completed (e.g., dissemination; if applicable), a delay in ACS submission may be justifiable. 40