Impact Evaluation of Introduction to Bank Operations Chaoying Liu WBI Evaluation Studies No. EG04-76 The World Bank Institute The World Bank Washington, D.C. September, 2003 ACKNOWLEDGEMENTS This evaluation report was prepared for the Learning Board under the overall guidance of Marlaine Lockheed, Manager, Evaluation Group. This report was prepared by Chaoying Liu. A number of people made various contributions to this evaluation study and the report, including: Jaime Quizon, Heidi Zia, Crisitina Ling Chard, Violaine Le Rouzic, Yinglang Liu, Wema Jackson Kategile, Hui Xiao, Elina Manjieva (WBIEG) and Hoveida Nobakht, the IBO course task manager (OPCS). Assistance in document preparation was provided by Humberto Diaz and Tanya Loftus (WBIEG). This report also benefited greatly from expert review comments provided by Sukai Prom-Jackson (HDN) and Richard Westin (ECA). Finally the author wishes to extend thanks to all participants who provided valuable information to this study as well as shared their experience with us. The report was discussed at a meeting with the OPCS SLC chaired by Phyllis Pomerantz on October 16, 2003. WBI Evaluation Studies are produced by the Institute Evaluation Group (IEG) to report evaluation results for staff, client, and joint learning events. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at: World Bank Institute Evaluation Group Publications and Briefs Vice President, World Bank Institute Ms. Frannie Léautier Chief Learning Officer Ms. Phyllis Pomerantz Manager, Institute Evaluation Group Ms. Marlaine Lockheed Task Team Leader Ms. Chaoying Liu ii ACRONYMS ACS Administrative and Client Support ANCOVA Analysis of Covariance ASDT American Society for Training & Development CAS Country Assistance Strategy CDF Comprehensive Development Framework ECA Europe and Central Asia Region FY02 Fiscal Year 2002 FY03 Fiscal Year 2003 GA-D Staff in grades A through D GE/F Staff in grade E or F GG/H Staff in grade G or H HDN Human Development Network HQ Headquarters HR Human Resources IBO Introduction to Bank Operations IEG Institute Evaluation Group Non-IBO Non IBO course participants OED Operations Evaluation Department OPCS Operations Policy and Country Services PAD Project Appraisal Document PCA Principal Component Analysis PCD Project Concept Document STC Short-Term Consultant STT Short-Term Temporary VPU Vice-Presidency Unit WB World Bank WBI World Bank Institute WBIEG World Bank Institute Evaluation Group iii iv TABLE OF CONTENTS ACKNOWLEDGEMENTS ................................................................................................ ii ACRONYMS..................................................................................................................... iii TABLE OF CONTENTS.....................................................................................................v EXECUTIVE SUMMARY .............................................................................................. vii 1. THE IBO COURSE.................................................................................................... 1 2. IMPACT EVALUATION METHODOLOGY .......................................................... 3 EVALUATION FRAMEWORK........................................................................................... 3 EVALUATION SCOPE AND QUESTIONS............................................................................ 4 EVALUATION DESIGN .................................................................................................... 4 DATA COLLECTION ....................................................................................................... 5 DATAANALAYSIS ......................................................................................................... 5 3. RESULTS ................................................................................................................... 7 EVALUATION PARTICIPATION....................................................................................... 7 Baseline.......................................................................................7 Follow up completion and drop-out........................................................................ 8 Comparison group (non-IBO)................................................................................. 9 QUALITYOFTHE IBO COURSE...................................................................................... 9 IMPACT OF THE IBO COURSE...................................................................................... 10 Knowledge retention............................................................................................. 10 Comparison with non-IBO participants ................................................................ 11 Information awareness .......................................................................................... 12 Utilization of knowledge acquired from the course.............................................. 12 Comparison with non-IBO participants ................................................................ 14 Participants' perceived impact.............................................................................. 15 FACTORS INFLUENCING KNOWLEDGE UTILIZATION..................................................... 16 Participants' perceptions about influencing factors.............................................. 16 The influence of external factors on the knowledge utilization............................ 16 ANALYSIS OF FACTORS INFLUENCING THE COURSE IMPACTC...................................... 17 Analytic model...................................................................................................... 17 Multiple Regression Analyses .............................................................................. 18 v 4. PARTICIPANTS' COMMENTS AND SUGGESTIONS ....................................... 21 Target audience..................................................................................................... 21 Course content....................................................................................................... 21 Other suggestions .................................................................................................. 21 5. DISCUSSION, CONCLUSIONS AND RECOMMENDATIONS.......................... 23 REFERENCES ................................................................................................................. 27 ANNEXES ........................................................................................................................ 29 ANNEX A: IMPACT EVALUATION OF IBO ­ BASELINE SURVEY ITEMS..................... 31 ANNEX B: IMPACT EVALUATION FOLLOW-UP QUESTIONNAIRE (IBO GROUP) ......... 33 ANNEX C: IMPACT EVALUATION QUESTIONNAIRE (NON-IBO GROUP) .................... 37 ANNEX D: KNOWLEDGE TEST................................................................................... 39 ANNEX E: PRINCIPAL COMPONENT ANALYSIS FOR SCALES FOR GENERAL KNOWLEDGE AND OPERATION KNOWLEDGE ................................................................. 43 ANNEX F: SUMMARY OF DEPENDENT, INDEPENDENT VARIABLES AND SCALE COMPOSITION........................................................................................... 44 ANNEX G: RESULTS OF MULTIPLE REGRESSION ANALYSES (STEP 1).......................... 46 ANNEX H: RESULTS OF MULTIPLE REGRESSION ANALYSES (STEP 2).......................... 47 vi EXECUTIVE SUMMARY The Introduction to Bank Operations course (IBO) is the first unit of the new curriculum on Bank Operations and Portfolio Management developed by Operations Policy and Country Services (OPCS). The IBO course covers most of the key elements of the new curriculum at the basic level through a series of classroom sessions during a 5-day period. The main objectives of the IBO course are: · to help participants (relatively new staff) to familiarize themselves with the World Bank (WB), its core business, policy framework, operations principles, key instruments, and working environment; and · to update and enhance participants' knowledge and skills in conducting business in a professional, integrated and collaborative manner. The primary audience of the course is staff who have joined the Bank recently (within 6 to 36 months), and those who work directly in Operations or plan to do so in the near future. The aim of the impact evaluation was to determine short and intermediate term effects of the course. Specifically, the impact evaluation examined the following questions: · Did IBO increase participants' Operations knowledge? · Did participants utilize the knowledge acquired from the course? · What factors influenced participants' knowledge utilization? · How can the course be improved? The impact evaluation was designed as a prospective follow-up study. Participants were recruited from four selected offerings in FY03 (October, November and December 2002; and January 2003). A group of staff who did not participate in the IBO course was identified through the HR data set to serve as a comparison group. Baseline data for impact evaluation were collected from the IBO participants using a paper-and-pencil questionnaire administrated at the end of course. Three-month follow-up was carried out via a Web-based survey questionnaire. The questionnaire measured participants' course knowledge, their perception of overall course impact and knowledge utilisation as well as factors enabling or disabling their knowledge utilization. The comparison group was surveyed using a similar questionnaire and knowledge test at the same time as the follow up. Overall, the IBO course successfully achieved its objectives. The Operations training helped Bank staff acquire knowledge and skills related to Operations, thereby enhancing staff performance. The evidence of IBO's impact includes: vii · The IBO course received a high rating for all aspects of the course quality. The rating compares favourably with WB staff FY02 formal training courses and American Society for Training & Development's benchmarks (ASTD) on two key criteria ­ increase in knowledge and skill acquired, and applicable of the course to the job (Mean 4.6 & 4.3 vs. 4.2 for WB staff FY02 formal training and 4.2 for ASTD). · The IBO course significantly improved participants' Operations knowledge and the improvement was largely sustained during the follow-up period. Participants' level of knowledge as measured at follow-up was significantly higher than their pre-course knowledge level and higher than that of the comparison group (non-IBO) at the follow-up. · The IBO course increased participants' awareness of helpful information sources available within the Bank. · IBO Participants significantly increased their use of Operations knowledge after completing the course. The comparison made between the IBO and non-IBO groups confirmed that the IBO participants' utilization of Operations-related knowledge was significantly higher than that of non-IBO staff. · Additionally, IBO participants perceived that course impact and subsequent benefits were at moderate to high levels. The study revealed that the IBO impact- knowledge utilization - was influenced by a set of factors. As reported by the participants, self-confidence, job relevance and institutional policy support were the top three factors enhancing their knowledge utilization. As estimated by multiple regression analyses, among the four groups of factors (training, trainees, internal and external factors), external enabling factors had the largest effect of any of the variables on knowledge utilization, standing out well ahead of the influence of other factors that have been shown to be important on their own. This finding suggests that the IBO course can successfully increase participants' understanding about Bank Operations and motivate them to achieve better performance. However, knowledge improvement did not necessarily lead to subsequent utilization. Rather, external factors, such as job relevance and institutional support played a critical role in determining the level of utilization. This result is consistent with most of the WBEG evaluation findings and suggests a need for policy attention. In conjunction with the results of the Level 2 evaluation, the current evaluation would recommend that: 1) the course should be offered Bank­wide and continuously in its present form. 2) the Bank may explore some possible policy measures to ensure the priority participation of Bank staff who need it most. 3) the course may be further improved by: · increasing attention on Operations-related topics and issues; viii · designing more cases and examples reflecting the real life experiences of task managers; and · providing more time for case studies and networking during the course. ix x 1. THE IBO COURSE 1.1 This report presents an impact evaluation of a staff learning program entitled "Introduction to Bank Operations" (IBO). To set the scene, this chapter presents a brief review of the background and design of the course. Chapter 2 provides an overview of the current evaluation study, including its objectives, scope and key evaluation questions as well as the framework and evaluation strategies. Chapter 3 presents the major findings of the evaluation, under the following headings: · Evaluation participation · Quality of the IBO course · Impact of the IBO course · Factors influencing knowledge utilization · The analysis of factors influencing course impact Chapter 4 summarizes the participants' comments and suggestions to improve the training course. The final chapter (Chapter 5) offers discussion, conclusions and recommen- dations. 1.2 The Operations Policy and Country Services Vice Presidency (OPCS) is responsible for developing a new curriculum on Bank Operations. The curriculum will be an upgraded version of the first "Integrated Curriculum on Bank Operations and Portfolio Management" that was developed in response to the needs identified in the Wappenhans report for enhancing staff knowledge and skills in conducting the Bank's business. The new curriculum under development by OPCS reflects the new challenges of the Bank as it targets poverty reduction and sustainable development in a more integrated manner across sectors, in close partnership with clients and development partners. 1.3 The ultimate goal of the new curriculum is to contribute to high quality Bank Operations. This goal will be achieved through a series of training activities coordinated or designed within the new curriculum. IBO is the first unit of the new curriculum. The objectives of the IBO course are: · to help participants (relatively new staff) to familiarize themselves with the World Bank, its core business, policy framework, operation principles, key instruments and working environment; and · to update and enhance participants' knowledge and skills in conducting business in a professional, integrated and collaborative manner. 1 1.4 IBO is a 5-day classroom course covering most of the key elements of the new curriculum at a basic level, as summarized below: · Bank Evolution · Adjustment Lending · Corporate Perspective · Operational Policies · Sector Perspective · Legal Aspects · Country Perspective · Procurement · Results Orientation · Financial Management · Economic and Sector Work · Putting it All Together · Investment Lending · Designing a CAS 1.5 The course is designed for a high level of interactive learning, with a limited use of lecture presentations, and is delivered by experienced Bank staff and consultants. Learning objectives for each session have been clearly set out by the course organizer and activities are facilitated by a series of lectures, group activities and case studies. 1.6 The primary audience for the course is staff who recently joined the Bank (within 6 to 36 months) and those who work directly in Operations or plan to do so in the near future. It is also open to staff who have been with the Bank for a longer time, but who are relatively new to the Operations areas. While regular staff at all levels are welcome at the IBO course, ideally the ratio of non-ACS to ACS staff is maintained at around 4:1, giving priority to ACS staff with direct responsibilities in operational teams. 1.7 A formative evaluation was conducted for the IBO pilot course in FY02 by IEG. The evaluation showed that the pilot course received high ratings from participants in terms of the course contents, design and delivery. A test instrument (Multiple Choice Questions) was developed for Level 2 evaluation during the pilot course. 2 2. IMPACT EVALUATION METHODOLOGY 2.1 As requested by the Learning Board, an in-depth evaluation was carried out in FY03 to determine the impact of the IBO course. The design of this evaluation is discussed in the following sections. EVALUATION FRAMEWORK 2.2 A framework was developed to aid the design of the evaluation. As illustrated in Figure 1, in order to contribute to high quality work and product, an individual needs to work effectively and efficiently. Such desirable behaviour is contingent on the individual's · knowledge and skills relevant to do the tasks; · positive attitude and motivation to practice; and · opportunity and resources available to practice the behavior. Figure 1: IBO framework IBO course outcome chain Job relevance Knowledge/ Other training Skills Support, $, time Relevant External factors Appropriate Quality Utilization Motivation/ IBO Attitude Behavior/ Performance Products Internal factors Information/ Resource Individual profile Past experience 2.3 Although the IBO course acknowledges the importance of all three of these aspects in determining an individual's performance, its primary intention is to raise participants' awareness and provide basic Bank Operations knowledge. It also aims to influence participants' perceptions and motivation to engage in further learning, and to provide participants with useful information resources. By helping individuals build up the 3 fundamental knowledge and skills for Operations, the IBO course is expected to contribute to high quality Bank Operations. 2.4 The evaluation framework also posits a range of factors that may influence the achievement of the course outcomes. For instance, if the course were poorly designed, if it were irrelevant to the target individuals, or if it were designed inappropriately in terms of the level and method of delivery, the learning effect would not be expected to happen. Likewise, if an individual were prevented from using knowledge and skills acquired from the course, the course's intended impact would not be achieved. EVALUATION SCOPE AND QUESTIONS 2.5 The framework provides help in determining both the scope and specific questions for the present evaluation. As the purpose of IBO is to give an introduction, its content was designed largely to provide knowledge/information and to raise awareness. It therefore differs from most training courses that focus onbehavioral change. Although the ultimate goal of the course is to enhance staff performance, such behavioural outcomes and impact of behavioral changes would be very hard to observe or detect within the timeframe of the current evaluation. Acknowledging the unique characteristics and the evaluability of the program, the present impact evaluation was designed, in conjunction with Level 2 evaluation, to examine the short and intermediate course impact on course participants. Specifically, the evaluation examined the following questions: · Did IBO increase participants' Operations knowledge? · Did participants utilize the knowledge acquired from the course? · What factors influenced participants' knowledge utilization? · How can the course be improved? EVALUATION DESIGN 2.6 The impact evaluation used a prospective follow-up design that compared: · Baseline (end of course) measures with follow-up measures for the IBO course participants. · IBO course participants (IBO) with non-IBO staff onseveral measures. 2.7 All IBO participants from four FY03 offerings in (October, November and December 2002; and January 2003) who completed the course and its Level 2 evaluation were eligible for the impact evaluation study. These constituted the IBO participants. A group of staff who did not participate in the IBO course was identified through the HR data set , according to a set of demographic characteristics matched with the course 1 2 1The HR data set contains unit records for all Bank staff members (with open and term appointment) listed on November 30, 2002. It was created as an aid for the impact evaluation of staff learning programs. To ensure confidentiality, the CODE_ID-to-person correspondence is maintained by HR and is not available to IEG. 4 participants. The matching process was undertaken using the "Filter" function in the Microsoft Excel program. To ensure a sufficiently high response rate, the study extracted two or three non-IBO staff to match each IBO participant. In the case this procedure identified more than three similar individuals from the HR data set, only three finalists were randomly selected and contacted by the study. These staff constituted the non-IBO comparison group. DATA COLLECTION 2.8 The IBO course participants were surveyed at the end of the course, using a paper-and-pencil questionnaire, together with standard Level 1 and knowledge tests (Level 2 evaluation). The survey collected baseline information on participants' current level of course-related knowledge utilization, their motivation to utilize and their perceptions of factors influencing knowledge utilization. 2.9 A follow-up was carried out three months after the course completion. A web-based survey questionnaire was used via Lotus Notes to collect information from the course participants. The questionnaire measures participants' perceptions and practice in the following aspects: · Overall course impact; · Knowledge utilization; · Factors influencing knowledge utilization; and · Current Operations knowledge level. 2.10 The comparison group (non-IBO group) was surveyed using a similar questionnaire at the same time as the follow up. The survey asked their familiarity with and their utilization of Operations-related knowledge in their work3. Additionally, they were invited to complete the same knowledge test as IBO participants. The baseline questionnaire and follow-up survey questionnaires for IBO participants and for the comparison group (non-IBO) are attached in Annexes A, B and C. DATA ANALAYSIS 2.11 The evaluation employed paired t-tests to determine the significance of differences in outcome measures between baseline (end of course) and follow-up within the IBO group (treatment group). One limitation of the study design is that study participants were not 2The study followed the predetermined sequence (station-VPUs-grade level-length of service) to precede the matching process, even though there was no perfect matching for any pairs due to the limitation of the HR data set. For example, there was a high proportion of STC/T staff in the IBO group, whereas there was almost no STC/T staff list in the HR provided data set. 3 The question asked for the non-IBO staff was: The IBO course is designed to help new staff familiarize themselves with knowledge and information related to the following topics. To what degree did you usethe knowledge and information related to the following topics over the past three months? (1=Low, 5=High) 5 randomly sampled to represent all IBO participants and the matching process could not fully equate the treatment group (IBO) and no treatment group (non-IBO) in all important variables (observable and not observed). Keeping this limitation in mind, the study employed Analysis of Covariance (ANCOVA) to detect the difference between the IBO and non-IBO groups in key outcome measures, controlling for demographic variables (station, VPUs, grade and length of service) (Wildt & Ahtola 1978). This controlling strategy was based on an assumption that the participants' demographical characteristics may be related to their current knowledge and skills in Operations. That is, staff working for the Bank longer may have better knowledge than those who are new to the Bank. In addition, another possible confounding factor, having other concurrent Operations training, was considered in the analysis. It was believed that controlled in this way, the net impact of the IBO course on the outcome measures could be more accurately estimated. The statistical criterion for accepting an observed difference was set at the p<.10 level of significance. 6 3. RESULTS EVALUATION PARTICIPATION Baseline 3.1 The evaluation was conducted between October 2002 and May 2003. Implementation involved collecting baseline data at the end of each course for three Headquarter (HQ) IBO offerings and one field offering. 3.2 Table 1 presents the characteristics of the IBO participants at baseline (end of course). In the HQ offerings (October, November and December 2002), the proportions of Washington-based staff and HQ staff working with regional VPUs were significantly higher than staff from field and other VPUs. More staff at grade level GE+ than ACS staff attended the course. A smaller share of participants in the December offering worked in Anchor or other VPUs and more participants in the January field offering were at grade levels of GA-GD. There was no significant difference between the offerings with regard to participants' length of service with the Bank. Table 1: Characteristics of participants at the baseline Oct Nov Dec Jan Total Stat. Variables (N=31) (N=28) (N=31) (N=30) (N=120) test2 Stationa Field 16% 32% 29% 90% 42% 45*** HQ 84% 68% 71% 7% 58% VPUs Regional 65% 70% 90% 70% 73% 6.0 Anchor & other 35% 30% 10% 30% 27% Grade levela GA-D 19% 7% 23% 40% 23% 14.9 GE/F 45% 45% 32% 20% 36% GG/H 7% 18% 23% 13% 15% STC/T 10% 14% 16% 17% 14% Other 19% 14% 3% 3% 10% Average of Bank Service 3.2 2.4 3.3 2.8 2.9 t=0.5 Note: (a) Removed missing value cases; *** p<.01 7 Follow up completion and drop-out 3.3 Follow up was carried out three months after each offering. Several email and telephone reminders were used to enhance the response rate for the follow-up. Table 2 shows the number and percentage of participants at baseline and follow-up. As can be seen, the overall drop-out rate4 (proportion not completing the follow-up) for the study was approximately 34 percent (ranging from 29 percent to 50 percent). The biggest drop-out occurred for the January offering. This offering was given in the field and following up with this group using reminder strategies appeared particularly ineffective, perhaps due to the distance and tight time frame. Table 2: Participation and completion Offering Baseline Follow up Response rate Oct 02 31 22 71% Nov 02 28 20 71% Dec 02 31 22 71% Jan 03 30 15 50% Total 120 79 66% 3.4 To help understand the reason for continuation or drop-out during the evaluation, comparisons were made between returned participants and drop-outs. Table 3 shows that there are no significant differences between the two groups (Columns A and B) with respect to selected participant characteristics and their tested knowledge (end of course test). Overall, both groups were likely equivalent and drop-out should not critically threaten the validity of study design. Table 3: Characteristics of returned participants, drop-outs and returned non-IBO5 (A) (B) (C) Stat. test Stat. test Variables Returned Drop-outs Non-IBO (N=79) (N=41) (N=59) (A) vs (B) (A) vs. (C) Station Field 37% 51% 48% 2=4.6 2=1.6 HQ 64% 46% 52% VPUs Regional 76% 68% 64% 2=0.8 2=2.1 Anchors & others 24% 32% 36% Grade level GA-D 19% 29% 32% 2=10.0 2=15.1*** GE/F 38% 32% 46% GG/H 14% 17% 19% STC/T 18% 7% 3% Other 11% 7% Average years of Bank Service 3.0 2.9 5.4 t=.17 t=3.59*** Average of pre-test score 16.8 14.3 16.3a t=.18 t=.11 Average of post-test score 22.9 21.4 t=1.6 n.a Note: (a) It was assumed that Non-IBO staff's knowledge level at follow up should be equivalent to the IBO participants' pre-course knowledge score. *** p<.01 4Drop-out refers to those IBO participants who filled in the baseline survey but did not responded to the follow-up survey. 5As the non-IBO contact provided by HR did not match exactly the list that the study requested in terms of number of cases, it was impossible for the study to identify who were the non-IBO no­returns and to make a similar comparison analysis with the returned non-IBO respondents. 8 Comparison group (non-IBO) 3.5 Over 300 non-IBO staff, identified as a comparison group, were invited to fill out the survey questionnaire via email during the same period as the IBO participants' follow up. Telephone and email reminders were also employed to enhance the returns from the comparison group. Ultimately, 59 (20 percent) non-IBO staff completed the survey questionnaire and knowledge test. Characteristics of non-IBO staff are also summarized in Table 3 (Column C). 3.6 The initial one-to-one match on the four key variables (station, VPUs, grade, length of service) was not retained in the return sample. A number of reasons may be responsible for this result. First, the initial matching was not perfect for each individual. Second, the match between the study list and the HR list, as a whole was not perfect. This was due at least in part to the fact that some of identified staff on the study list had left the Bank during the study period. Third, the impact of drop-out from the IBO participants was substantial (some IBO participants dropped out, but their non IBO matches responded to the survey). Finally, the non-IBO response rate was very low, which tended to exaggerate group differences. Nevertheless, as shown in the last column of Table 3, the knowledge level of the non-IBO group at the time of follow up was similar to the pre-course knowledge level of the IBO group (t=.11). This finding means that the non-IBO group (at the follow-up point in time) was just as knowledgeable as the IBO group just before the latter had started their IBO training. This condition allows the study to use these non-IBO staff as the comparison group. QUALITY OF THE IBO COURSE 3.7 The quality of the course was assessed by the standard Level 1 evaluation administered at the end of course. Table 4 presents the means of individual items pertaining to three aspects of the course. It compares with WB FY02 formal staff training courses and ASTD benchmarks on two key criteria: (a) increase in knowledge and skill acquired and (b) applicable of the course to the job (Mean 4.6 and 4.3 vs. 4.2 for WB formal staff learning and 4.2 for ASTD). The overall high average score for all aspects of course quality as well as the high percentage of respondents who rated the course as "four" or "five" indicate that the course quality was high. 9 Table 4: Participants' assessment of course quality (N=120) Course aspects Mean Percent r ating 4/5 Quality (1=Low 5=High) Content 4.4 93% Structure 4.3 87% Materials 4.3 85% Overall quality 4.5 93% Usefulness (1=Low 5=High) Usefulness 4.4 86% Relevance 4.6 94% Increasing knowledge 4.6 92% Applicable 4.3 82% Quantity (1=Unbalanced 3=Balanced) Rating 3 Sufficient theoretical content 2.6 67% Sufficient practical content 2.6 60% Sufficiency for presentation 2.7 67% Sufficiency for participation 2.7 74% Pace of the course 2.4 52% IMPACT OF THE IBO COURSE 3.8 The impact of IBO was examined by looking at four key indicators: · Knowledge retention; · Information awareness; · Utilization of knowledge acquired from the course; and · Participants' perceived impact. Knowledge retention 3.9 A 36-item knowledge questionnaire covering the course content was developed for the IBO Level 2 evaluation (Rouzic 2003). Ideally, the same questionnaire would be administered to measure knowledge retention at the follow up. However this was not considered feasible as administration of a long test could lead some participants to ignore the entire survey, resulting in a low response rate. Likewise, an extensive knowledge test could also discourage non-IBO staff from participating in the evaluation study. Based on these considerations, a shorter 12-item test was constructed to assess participants' knowledge at the three-month follow-up, using the remaining items in the Level 2 test item pool. For the purpose of comparisons, the test score obtained from the follow-up was converted to a score comparable to pre and end of course tests (Level 2). Statistical reliability test shows that the internalreliability (alpha) for the 12-item knowledge test was .80. Annex D presents details regarding the methodology followed for constructing the knowledge test and final items in the test. 3.10 A profile of the change in participants' knowledge of Operations for all IBO offerings is shown in Figure 2. IBO participants demonstrate large and significant improvements in their understanding of Operations knowledge from before to just after 10 attending the IBO course, suggesting that IBO had an immediate impact on participants' knowledge. Moreover, their knowledge level measured at the three-month follow-up was still higher than on the pre-course test. Figure 2: IBO participants' knowledge profile at three time points 80 70 70.8 66.5 69.7 64.6 63.7 64.0 60.5 60 58.8 53.0 56.6 52.7 50 50.7 46.6 answers 43.1 40 38.6 30 correct of 20 %10 0 Oct Nov Dec Jan All Pre-test IBO Offerings Post-test Follow-up 3.11 Furthermore, Table 5 presents the results of statistical t-tests with matched end-of-course and follow-up scores. For all offerings, other than December participants, the knowledge scores declined at the three-month follow up. However, this observed decline was not significant for most offerings and the overall comparison shows that participants' follow-up score was still significantly higher than their pre-course knowledge score. This comparison suggests that the impact of IBO on knowledge enhancement was sustained over time. Table 5: Mean course knowledge scores at post course and at 3-month follow up Pre-test Post-test 3 months Pre vs post Post vs. 3 mth Pre vs. 3 mth Mean SD Mean SD Mean SD t-test t-test t-test Oct (n=20) 18.3 5.0 23.3 5.1 21.2 6.8 -4.7*** 1.4 -1.7 Nov (n=17) 15.5 5.3 25.4 4.9 22.9 6.7 -5.4*** 1.8** -3.7*** Dec (n=22) 19.1 6.1 23.9 4.2 25.1 5.1 -4.4*** -0.8 -3.0*** Jan (n=15) 12.7 7.2 18.7 4.2 17.4 6.3 -3.9*** 1.3 -2.9*** All (n=74) 16.76 6.6 23.0 5.1 22.0 6.7 -8.9*** 1.5 -5.6** *** p<.01, ** p<.05 Comparison with non-IBO participants 3.12 Further analysis of knowledge retention was carried out by comparing knowledge levels between the IBO participants and non-IBO participants at follow-up. The knowledge levels for both groups are shown in Figure 3. It can be seen clearly from the comparison, that IBO participants' knowledge level is nearly 15 percent higher than that of the non-IBO group. Analysis of Covariance (ANCOVA) was employed to test the 11 significance of the knowledge scores between the groups, controlling for four demographic variables (station, VPUs, grade and length of service). The test result confirms that this observed 15 percent difference is significant (F=15.5, p<.01), reinforcing the conclusion that participants increased their knowledge as a result of attending the IBO course. Figure 3: Comparison of knowledge level between IBO and non-IBO at 3 month follow-up 70 60 50 answer 40 61.0 30 correct 46.2 of 20 %10 0 non-IBO IBO Information awareness 3.13 One of the objectives of the course is to raise participants' awareness of helpful information or sources available within the Bank. A number of commonly used information sources such as manuals, templates and web-sites relating to the Operations were introduced at the course. At follow-up, both IBO participants and non-IBO staff were surveyed about their awareness and the level of use of these sources. Although the survey showed no difference between the groups in their level of use of these resources, IBO participants clearly showed a better awareness of information sources than non-IBO staff. Table 6 presents an example showing a difference between the groups in awareness of some information sources. Table 6: Awareness of information sources (percent of respondents) Never heard of IBO (n=79) Non-IBO (n=59) the following sources Red Book 1.3% 18.6% CDF 1.3% 13.6% Evaluation Manual 2.5% 13.6% PAD &PCD 1.3% 6.8% OPCS website 1.3% 10.2% OED website 1.3% 10.2% OPCS help desk/line 2.5% 11.9% Utilization of knowledge acquired from the course 3.14 The IBO course provides knowledge and information that are believed most applicable to the participants. The follow-up survey revealed that more than 90 percent of participants who returned the survey report having an opportunity to use the knowledge 12 and skills acquired from the course. The average level of utilization was 3.4 on a 5-point rating scale, with 5 representing the highest level of utilization. As illustrated in Figure 4, among the users, approximately 40 percent used at a high level (4-5) and over 12 percent reported a very high level of utilization. Figure 4: Level of use of course content (n=73) 5 12.3 5=high) 4 34.2 (1=low, 3 39.7 use 2 8.2 of 1 2.7 Level 0 10 20 30 40 50 Percentage of Users at Each Level 3.15 Of 14 IBO sessions, 11 deal with specific Operations topics. To aid and simplify the analysis of participants' knowledge utilization of these topics, the study employed a Principal Component Analysis (PCA), which derived two indices from participant responses: General Bank operations knowledge (General knowledge 6 ) and Specific operations knowledge (Operations knowledge 7 ). The statistical result of the PCA regarding these scales is provided in Annex E. 3.16 As illustrated in Figure 5, in comparison with their baseline level (end of course), IBO participants increased their level of utilization for both General and Operations knowledge by over 10 percentage points each, and the increase was significant (General t=3.43, p<.01; Operations t=3.88, p<.01). 6General knowledge includes Corporate perspective, Sector perspective, Country perspective, Result orientation and Economic Sector Work. 7Operations knowledge contains Investment lending, Adjustment lending, Operational policies, Legal aspects, Procurement and Financial management. 13 Figure 5: Comparision of level of knowledge utilization between baseline and follow up 100 90 % 80 67.0 70 63.4 56.7 60 52.3 utilization 50 of 40 30 Level 20 10 0 General Operations Knowledge area Baseline 3 months Comparison with non-IBO participants 3.17 Similarly, a comparison was made between the IBO participants and non-IBO staff for their reported level of knowledge utilization3. As shown in Figure 6, in comparison with non-IBO staff, IBO participants reported a higher level of use of both General and Operations knowledge during the follow-up period. A statistical test (ANCOVA) confirms that the observed difference between the two groups is significant (F=3.06, p<.05 for General knowledge) and (F=2.05, p<.10 for Operations knowledge). As both IBO participants and non-IBO staff were exposed to similar environments during the follow-up period, the difference between the two groups in their knowledge utilization is an evidence of the IBO course's impact on participants. Figure 6: Comparision of level of knowledge utilization between IBO and non-IBO at follow up 100 Le 90 vel of 80 util 70 65.7 63.3 iza 57.2 60 56.7 tio 50 n % 40 30 20 10 0 General Operations Knowledge area Non-IBO IBO 14 Participants' perceived impact 3.18 At the end of course, the IBO participants were asked to indicate the extent to which the course had influenced their perceptions and motivation. Table 7 below shows the item means and the percentage of participants' rating of four or five for each item. Overall, the ratings suggest that the course not only boosted participants' knowledge, but also enhanced their motivation and self-confidence in carrying out their daily work. The most positive example is that 90 percent of respondents were motivated to pursue further learning. Table 7: Participants' rating short-term impact of the course "To what extent did the course enhance your:" Percent rating N Mean (1=not at all, 5=very much) 4/ 5 a. Self-confidence 119 3.9 72% b. Organization awareness 119 4.3 86% c. Commitment to organization goals 116 4.2 80% d. Motivation to achieve high standards 120 4.1 78% e. Commitment to work effectively as an individual 118 4.1 79% f. Commitment to work cooperatively as a team player 117 4.3 83% g. Motivation to engage in further learning 119 4.6 90% h. Rational and strategic thinking 119 4.1 76% 3.19 At follow-up, the survey included four questions that assess participants' perception of the impact of the course and subsequent benefits. On a five-point rating scale (from very low to very high), participants gave an average rating of 3.5 on the questions asked (Figure 7), suggesting that participants perceived that the course's impact and subsequent benefits were at a moderate to high level. Figure 7: Participants perceived course impact 3.6 Overall impact 3.5 Enhanced performance Beneficial to unit 3.5 Beneficial to clients 3.4 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Mean score (1-5, low-high) 15 FACTORS INFLUENCING KNOWLEDGE UTILIZATION Participants' perceptions about influencing factors 3.20 Like most behavioral changes, knowledge utilization needs to be facilitated by positive environmental factors. At follow-up, participants were asked to rate the extent to which the group of factors listed in Table 8 helped them use the knowledge gained from the IBO course. As reported by the participants, confidence, job relevance and institutional policy support were the top three enabling factors. On the other hand, t-tests conducted to compare mean ratings at baseline (end of course survey) and follow up with matched data show that ratings on all factors decreased significantly. This means that after three months although respondents feel at least a modest level of environmental support, they felt less support than they anticipated at the end of the course, especially for the aspect of available resources (time and people), manager support and colleague support. Table 8: Participants' perceived influence on knowledge utilization Degree of influence Follow up (1=very low, 5=very high) N End courseMean Mean Statistical t-test a. Self-confidence 68 3.9 3.5 2.3** b. Job relevance 75 4.1 3.5 4.5*** c. Institutional policies 74 3.9 3.4 3.4*** d. Available resources 71 3.8 2.8 5.7*** e. Manager encouragement 73 4.1 3.1 5.5*** f. Colleague encouragement 74 3.8 2.9 5.6*** *** p<.01, ** p<.05 The influence of external factors on the knowledge utilization 3.21 A further analysis was carried out to estimate how the above factors affected knowledge utilization. Using a Principal Component Analysis (PCA), an index was created containing five external factors listed in Table 8 (see details about the scale in Annex F). Based on the median index score, the IBO group was split into two groups: low support and high support. A comparison was then made between the level of knowledge utilization and the knowledge score of the two groups. As demonstrated in Figure 8, the IBO participants who perceived higher external support showed a significantly higher level of knowledge utilization than those who felt they received lower support (t=2.44, p<.05 for General knowledge; t=2.21, p<.05 for Operations knowledge). In fact, the utilization level of the low-support group was nearly comparable to those who did not attend the course (non-IBO). Importantly, the difference in utilization between the low and high support groups was not due to the fact that the low-support IBO participants did not learn from the course, (their post course knowledge score was essentially the same as the high support IBO participants and higher than the non-IBO), nor due to the demographic differences of the individuals. This analysis implies that the different level of external support appeared to strongly influence the IBO participants' utilization practice. The relationship between these two aspects will be examined further in the next section. 16 Figure 8: Comparison of knowledge utilization, test score between low support IBO, high support IBO and non IBO at 3-month follow up 90 80 answer 72 70 61 68 65 64 58 60 57 57 (%) 50 46 use/correct 40 of 30 Level 20 10 General Operations Post test score Low support Knowledge area High support Non IBO ANALYSIS OF FACTORS INFLUENCING THE COURSE IMPACTC 3.22 In the preceding sections, the key evaluation question "did IBO have an impact on participants?" was addressed by analyzing four impact indicators. As anticipated, the result was positive: the IBO course did enhance participants' understanding of Bank Operations and influenced their utilization behavior. This section will turn attention to the next question of the evaluation: "Under what conditions was the course impact achieved?" Analytic model 3.23 An analytic model was constructed to address this research question (Figure 9). Essentially, this model is the simplified version of Figure 1. It posits that the impact of the IBO course is a function of four groups of variables, namely: training, trainees, internal and external enabling/disabling factors. This diagram is used to illustrate the possible causal links between the program components and the final outcome. Knowledge utilization can be treated as a dependent variable, whereas the training, trainees, enabling/disabling factors are the independent variables. The hypotheses implied in the model can be spelled out as follows: · A high quality and useful course motivates participants to use the knowledge acquired (training factors); · The level of utilization is influenced by individual characteristics (trainee factors); · Utilization practice is also influenced by an individual's capacity (knowledge, skills), motivation and attitudes/perceptions (internal factors); and · All of the above effects are enhanced or constrained by a positive or negative environment (external factors). 17 Detailed information on how these factors were defined, measured and used in the study is summarized in Annex F. Figure 9: Analytic model -the relationship between the impact and influencing factors External factors Training factors Trainee factor Trainee factors Knowledge Utilization Internal factors External factors Multiple Regression Analyses 3.24 Several multiple regression analyses were conducted to test the hypotheses implied in the analytic model and to determine the extent to which these variables impacted knowledge utilization (General and Operations combined). 3.25 In the first analysis, all independent variables (training, trainee, internal and external factors) were included in the regression model to estimate the magnitude of effect of a single group factor on knowledge utilization (dependent variable). 3.26 Annex G summarizes the coefficients of the regression analysis, showing that only three variables had significant effects on knowledge utilization, namely: perceived course usefulness, self-confidence and external support factors. That is, if an individual perceived the course was more useful, or when he/she was more confident about his/her ability to use the knowledge, or when he/she received more support from the external environment, his/her utilization was likely to be higher. Neither demographic variables appeared to affect knowledge utilization, nor did the participants' knowledge levels (at end of the course and follow-up). 3.27 The regression analysis was computed by introducing only the three variables shown to significantly affect knowledge utilization in the fist step of regression. These variables were introduced into the regression model by blocks according to a predetermined causal sequence, as summarized in the box below. The advantage of introducing the independent variables by block is that this creates an opportunity for the evaluation to not only look at the relationship between dependent and independent variables, but also to investigate the possible interactions among the independent variables. Model A: utilization=f(perceived course usefulness) Model B: utilization=f(course usefulness, self-confidence) Model C: utilization=f(course usefulness, self-confidence, external factors) 18 3.28 The result of the second set of regression analyses, with interpretation, is found in Annex H. Overall, both Models A and B presented patterns very similar to the first regression analysis. A notable change was seen in Model C. When all these variables were included in one equation (Model C), the effect of external support factors appear to have much stronger effect on knowledge utilization. 3.29 In sum, the regression analyses performed for the knowledge utilization impact variable largely support the study's key hypothesis concerning the influence of a group of factors and the result is consistent with the previous analysis of knowledge utilization with respect to different levels of external support. Among the four groups of factors examined (training, trainees, internal and external factors), external enabling/support factors had the largest effect of any variables on knowledge utilization, standing out well ahead of the influence of other factors that have been shown to be important on their own. The IBO course successfully enhanced participants' knowledge of Bank's Operations and motivated participants to achieve a better performance. However, knowledge improvement was insufficient in leading to subsequent utilization behavior. Rather, external factors such as job relevance and institutional support appear to play the strongest role in determining the level of knowledge utilization. The results also indicate that external factors may influence knowledge utilization directly, and may also affect utilization by synergying the effect of an individual's perception (perceived course usefulness and self-confidence). 3.30 The variables that significantly affect the IBO impact, based on the findings from the multiple regression analyses, are illustrated in Figure 10. This model, which incorporates the findings from other analyses, enhances our understanding of the role of staff learning intervention and the conditions under which an intervention program produces desired outcomes. Figure 10: Factors influencing the course impact ­ knowledge utilization Course usefulness 1.94* 3.66*** External Knowledge Utilization Confidence 3.23** factors *** p<.01, **p<.05, * p<.10 19 20 4. PARTICIPANTS' COMMENTS AND SUGGESTIONS 4.1 Participants made a wide range of comments and suggestions for improving the course. Overall, the issues that most concerned by participants were similar to those expressed in the evaluation results of the pilot course in FY02. Target audience 4.2 The most common issue cited by the participants across all four offerings was that the course did not work too well for the mixed group (experienced staff and newcomers, ACS and professional staff). Here are some quotes pertaining to this topic: · Do not match people who work a long time at the Bank with those who just started. · The level (professional at least) of participants should be if not equal, at least alike. Putting together a 1-year experience ACS together with 7 years experience PhD person looks a bit strange.... · Some were more advanced and found many sessions too basic. Others were struggling to understand. 4.3 Participants suggested that the IBO course should be offered to staff with similar experience in the World Bank, and should try to be more selective of the Bank experience and learning of participants. OPCS should consider designing a separate training course on Bank's Operation for ACS staff. Course content 4.4 Although most participants found the course content was comprehensive and practical, some felt that there was a lack of in-depth discussion and critical analysis of the most important messages and key points, such as good or bad past experiences and challenges. Participants suggested that the course should elaborate some underlying principles of Bank policies and instruments, and discuss Operations-related issues such as the Bank's relations with client countries, donors and the Bank's budgeting process. Participants also suggested that the course could enhance learning by selecting more "real world" cases and examples from task managers and sector managers. Other suggestions 4.5 Other suggestions to improve the course design and delivery included: 21 1. Diversifying learning methods; using more interactive teaching and learning methods; 2. Providing exercises that are based on the course content and allowing more time for feedback from instructors; and avoiding the "guess" exercises; 3. Reducing the use of video conferences to a minimum (a comment particularly from the January attendees); 4. Allocating more time to discuss details of the most important issues, to complete exercises, including CAS, and have Questions and Answers; and 5. Providing more time for breaks, lunches and networking. 22 5. DISCUSSION, CONCLUSIONS AND RECOMMENDATIONS 5.1 This report describes an impact evaluation of a staff learning program -- Introduction to Bank Operations (IBO). Basic Bank Operations training like IBO can be an important step in helping Bank staff, especially newly joined staff, acquire essential operation-related knowledge and skills, thereby enhancing their performance. Based on the results of the data analyses, several conclusions regarding the evaluation questions can be drawn: 5.2 Overall, the IBO course successfully achieved its objectives. It was evident that participants increased their knowledge of Operations after attending the course, and such knowledge improvement was retained over time. When they returned to their work after the course, participants increased their utilization of the course knowledge. Additionally, there was evidence from the survey that the majority of IBO participants perceived that they were influenced and motivated by the training course to work effectively as individuals and as team players. All these findings indicate that the course successfully achieved its objectives in helping staff work effectively. 5.3 The examination of the determinants of knowledge utilization revealed that participants' knowledge utilization was significantly influenced by their perception of the course's usefulness, their self-confidence and external environment. This finding supports the notion of most behavioral theories and the study hypothesis that behavioral change is a function of internal and external factors (Ajzen & Fishbein 1980). Motivation, positive perceptions and knowledge are necessary for an individual to take action. However, without the supportive environment, the intention of taking action may not be transferred into action or actions may not achieve the desired outcome. Equally, a training course like the IBO can be effective in enhancing participants' knowledge and skills, but transferring knowledge still requires a supportive environment. 5.4 The findings about the importance of external factors in training may have broad implications. First, the finding of influence of external factors is not new in training literature and in the IEG practice. It has been well-acknowledged that the formula "Business Results=Learning Experience X Working Environment is a fundamental concept in training programs. Business results occur when skills taught in a training program are applied on the job, yielding improved performance (Robinson, D. & Robinson, J 1989). With respect to the IEG practice, many external factors studied in this evaluation were not specific to the IBO course. In an impact evaluation study of knowledge utilization of formal staff training programs, Prom-Jackson, et al. (2002) examined a range of affecting factors. The evaluation revealed that both course quality and environmental factors significantly affect the participants' knowledge utilization. More critically, the study 23 pointed out that for Bank Operations, the working environment plays a more significant role in the use of knowledge and skills acquired from training. Given that the factors examined in this study are also applicable to other staff training programs, any recommendations for subsequent intervention pertaining to knowledge utilization should be applicable to IBO course as well as to other staff learning programs. 5.5 Second, training and external factors, unlike trainee factors, are amenable to policy intervention. Because these factors have powerful effects on behavioral changes, if appropriate policy and measures can be introduced into the course design and implementation, the course should result in great benefits to the individual and consequently to Bank operations. For instance, such benefits might derive from requiring managers to recommend/nominate staff who they consider need Operations knowledge to attend the course or selecting potentially beneficial participants through a screening process. 5.6 In conjunction with the results of the formal Level 2 evaluation, at a collective level the evaluation would recommend that the IBO course should be offered Bank-wide and continuously. For the future of the IBO operation, the findings outlined in this evaluation recommend that: 1) The course should be offered continuously in its present form 5.7 As the IBO course is designed to provide basic and updated knowledge and information about Bank Operations, its content is relevant to the majority of Bank staff. So far, the course has operated just over 12 months and only a small proportion of Bank staff have received the training. Perhaps the main benefits of the course to Banks Operations are yet to be realized. It is important to increase its coverage thereby accelerating the course benefits to Bank Operations. Although some participants suggested having a separate Operations course for ACS and for professional staff, both the Level 2 and the impact evaluations did not differentiate the course effectiveness on these two groups. Therefore, with some improvement, the course should be offered continuously in its present form. 2) The course should ensure the priority participation of staff who need it most 5.8 Despite the fact that most participants can learn from the course, the likelihood of knowledge utilization is higher among those who have greater need (job relevance, etc). The Bank may explore some possible policy measures to ensure the priority participation of these staff, such as employing course criteria and some Level 1 questions to screen prospective attendees during the registration process, focusing on relevance, usefulness, applicability and expectation. Through this process, staff highly relevant to this course will be selected for the priority seats, while those staff expecting high Operations-focused content can be referred to advanced or other specific Operations courses. As claimed by many training experts such as Brinkerhoff (1991), this approach can help training achieve true high-impact learning. Additionally, this top-down approach may be complemented with abottom-up process -- directly asking manager to nominate/recommend their staff to attend the course, which could maximally ensure that the right people are on the training seats. 24 3) The course may be further improved by 5.9 Increasing attention on Operations-related topics and issues while still concentrating on Bank Operations. The focus of the course should continue to be on assisting Bank staff to raise awareness and to understand Bank Operations across a range of areas. It would be useful to discuss Operations-related issues such as the Bank's relationships with client countries, other international agencies and donors, as well as the Bank's budgeting process. 5.10 Enhance learning by including more cases and examples reflecting real life experiences of task mangers. As the majority of the IBO course attendees are less experienced or low to middle levels of Bank staff, inclusion of more cases reflecting the real life experiences of low levels of management such as task managers would be more directly relevant, useful and helpful with learning. 5.11 Providing more time for case studies and networking during the course. As the course contains substantial information and practical issues, participants, particularly some new staff, may need more time to understand, digest and apply their exercises or case study. Allowing participants to have more time to think and to practice (e.g., with case studies) can be expected to enhance their learning. Likewise, more opportunities for participants to informally network during the course should also enhance learning. 25 26 REFERENCES Ajzen, I. & Fishbein, M. (1980). Understanding attitudes and predicting social behavior. Englewood Cliffs, NJ: Prentice-Hall. Albert, M. & Ahtola, O. (1978). Analysis of covariance. Quantitative Applications in the Social Science series #12. Thousand Oaks, CA: Sage Publications. Brinkerhoff, R. (1991). Achieving Results from Training: How to evaluate Human Resource Development to strengthen Programs and Increase Impact. Jossey-Bass Publishers, San Francisco (Calif.). Prom-Jackson, S., Malone T., Palmisano M.& Kategile W (2002). The Impact of FY01 Formal Training ­ Turning Knowledge and Reflection into Action in The World Bank. WBI Evaluation Studies, No. EG03-62, The World Bank Institute, Washington, DC. Robinson, D. & Robinson, J (1989). Training for Impact. Jossey-Bass Publishers, San Francisco (Calif.). Rouzic, V. (2003) Learning from the Introduction to Bank Operations Course, FY03. WBI Evaluation Studies, No. EG03-69, The World Bank Institute, Washington, DC. 27 28 ANNEXES 29 30 ANNEX A: IMPACT EVALUATION OF IBO ­ BASELINE SURVEY ITEMS (COLLECTED VIA LEVEL1 FORM) 1. To what extent was the content of each of the following sessions directly applicable to your work? Not at Always N/A all 2 3 4 a. Bank Evolution ? ? ? ? ? ? b. Corporate Perspective ? ? ? ? ? ? c. Sector Perspective ? ? ? ? ? ? d. Country Perspective: CDF & CAS ? ? ? ? ? ? e. Results Orientation ? ? ? ? ? ? f. Economic and Sector Work ? ? ? ? ? ? g. Investment Lending ? ? ? ? ? ? h. Adjustment Lending ? ? ? ? ? ? i. Operational Policies ? ? ? ? ? ? j. Legal Aspects ? ? ? ? ? ? k. Procurement ? ? ? ? ? ? l. Financial Management ? ? ? ? ? ? m. Developing a CAS ? ? ? ? ? ? 2. To what extent did the course enhance the following? Not at Very No 2 3 4 all much opinion a. Your organizational awareness ? ? ? ? ? ? b. Your commitment to organizational goals ? ? ? ? ? ? c. Your self-confidence ? ? ? ? ? ? d. Your motivation to achieve high standards ? ? ? ? ? ? e. Your commitment to work effectively as an ? ? ? ? ? ? individual f. Your commitment to work cooperatively as a ? ? ? ? ? ? team player g. Your rational and strategic thinking ? ? ? ? ? ? h. Your motivation to continue learning ? ? ? ? ? ? 31 3. In your current working environment to what extent do you expect the following factors to enable or help your application of the course content? Not at Very No 2 3 4 all much opinion a. Self-confidence ? ? ? ? ? ? b. Job relevance ? ? ? ? ? ? c. Institutional policies and rules ? ? ? ? ? ? d. Available resources ($, time, people) ? ? ? ? ? ? e. Encouragement from manager/supervisor ? ? ? ? ? ? f. Encouragement from colleagues ? ? ? ? ? ? For the next question, please give one answer to assess your level of use of the course's knowledge and/or skills and one answer for the related information resources (e.g., manuals, help lines, web sites). Your level of use 4. BEFORE the course, what was your Your level of use of of information resources level of use of knowledge/skills the knowledge/skills and of the information resources related to each session below? low high low high 1 2 3 4 5 1 2 3 4 5 a. Bank Evolution ? ? ? ? ? ? ? ? ? ? b. Corporate Perspective ? ? ? ? ? ? ? ? ? ? c. Sector Perspective ? ? ? ? ? ? ? ? ? ? d. Country Perspective: CDF & CAS ? ? ? ? ? ? ? ? ? ? e. Results Orientation ? ? ? ? ? ? ? ? ? ? f. Economic and Sector Work ? ? ? ? ? ? ? ? ? ? g. Investment Lending ? ? ? ? ? ? ? ? ? ? h. Adjustment Lending ? ? ? ? ? ? ? ? ? ? i. Operational Policies ? ? ? ? ? ? ? ? ? ? j. Legal Aspects ? ? ? ? ? ? ? ? ? ? k. Procurement ? ? ? ? ? ? ? ? ? ? l. Financial Management ? ? ? ? ? ? ? ? ? ? m. Developing a CAS ? ? ? ? ? ? ? ? ? ? 32 ANNEX B: IMPACT EVALUATION FOLLOW-UP QUESTIONNAIRE (IBO GROUP) Thank you for participating in the impact evaluation of "Introduction to Bank Operations" (IBO) course and completing this questionnaire. The questionnaire contains two parts. It should take you about 20 minutes to complete. Part I asks your opinions about the overall course impact Part II asks about your utilization of knowledge and skills gained from the course In order to help us link this to your previous completed forms for this course, would you please indicate the code that you used at the course ____/____/____ (i.e. the first letter of the village/town/city where you were born; the first letter of your mother's first name and the month of birth of your mother (in full) part i Overall course impact 1. When you recall the IBO course that you attended in Oct 2002, how would you rate the degree to which the course: Not 2 3 4 Always N/A at all a. was relevant to your work? ? ? ? ? ? ? b. addressed your learning needs ? ? ? ? ? ? ? c. increased your knowledge and skills ? ? ? ? ? ? ? d. motivated you to pursue further Bank ? ? ? ? ? ? training? e. provided you with clear direction on ? ? ? ? ? ? information resources? f. impacted your overall performance? ? ? ? ? ? ? 2. To what extent have you changed the way you carry out your day to day tasks as a result of attending this course? Not at all 2 3 4 Very much N/A ? ? ? ? ? ? 3. As a result of attending this course, to what extent has Not 2 3 4 Always N/A at all a. your organizational awareness positively ? ? ? ? ? ? impacted your overall performance? b. your commitment to organizational goals ? ? ? ? ? ? positively impacted your overall performance? c. your self-confidence positively impacted your ? ? ? ? ? ? overall performance? d. your motivation to achieve high standards ? ? ? ? ? ? positively impacted your overall performance? 33 e. your commitment to work effectively as an individual positively impacted your overall ? ? ? ? ? ? performance? f. your commitment to work cooperatively as a team player positively impacted your overall ? ? ? ? ? ? performance? g. your rational and strategic thinking positively ? ? ? ? ? ? impacted your overall performance? part II Utilization of knowledge, skills and information 4. Since completing the course, did you have opportunities to use the course contents (knowledge, skills and information)? (If you answer no, go to question #6.) ? Yes ? No Since completing the course, to what extent Very Very 2 3 4 N/A Low High a. have you usedthe course contents (knowledge ? ? ? ? ? ? and skills)? b. has your use of course contents enhanced your ? ? ? ? ? ? performance and productivity? c. has the change in your performance and ? ? ? ? ? ? productivity been beneficial to your unit output? d. has the change in your performance and productivity been beneficial to your clients or ? ? ? ? ? ? stakeholders? 5. Since completing the course, to what degree have you used the knowledge, skills and information provided in the following sessions? Very Very 2 3 4 N/A Low High a. Bank Evolution ? ? ? ? ? ? b. Corporate Perspective ? ? ? ? ? ? c. Sector Perspective ? ? ? ? ? ? d. Country Perspective ? ? ? ? ? ? e. Results Orientation ? ? ? ? ? ? f. Economic and Sector Work ? ? ? ? ? ? g. Investment Lending ? ? ? ? ? ? h. Adjustment Lending ? ? ? ? ? ? i. Operational Policies ? ? ? ? ? ? j. Legal Aspects ? ? ? ? ? ? k. Procurement ? ? ? ? ? ? l. Financial Management ? ? ? ? ? ? m. Developing a CAS ? ? ? ? ? ? 34 6. Since completing the course, how many times have you sought information from the following resources? (N=Never heard of) 0 1-3 4-6 7-9 10+ N a. Course note book ? ? ? ? ? ? b. Bank's Operational Manuals ? ? ? ? ? ? c. "Green book" Guideline ? ? ? ? ? ? d. "Red book" Guideline ? ? ? ? ? ? e. CDF-Aligning Culture and Staff Behaviors ? ? ? ? ? ? f. Evaluation Manual ? ? ? ? ? ? g. PCD and PAD guidelines ? ? ? ? ? ? h. OPCS websites ? ? ? ? ? ? i. OED websites ? ? ? ? ? ? j. OPCS help desk via email/phone ? ? ? ? ? ? 7. Since completing the course, have you attended any of the following suggested operation related training courses?(Check all that apply. If you answer no, go to question #8.) ? Yes ? No Safeguards Financial Management Procurement Lending Instrument Monitoring and Evaluation Project Planning and Preparation Project Supervision Project Cycle Management Trust Fund Learning Accreditation Program Other (please specify) 8. To what extent have the following aspects/factors enabled or helped you to apply the course contents in your job? Very Very 2 3 4 N/A Low High a. Self-confidence ? ? ? ? ? ? b. Job relevance ? ? ? ? ? ? c. Institutional policies and rules ? ? ? ? ? ? d. Available resources ($, time, people) ? ? ? ? ? ? e. Encouragement from manger/supervisors ? ? ? ? ? ? f. Encouragement from colleagues ? ? ? ? ? ? g. Other (please specify) ? ? ? ? ? ? 35 9 To what extent have the following aspects/factors constrained you from applying the course contents in your job? Very Very 2 3 4 N/A Low High a. Self-confidence ? ? ? ? ? ? b. Job relevance ? ? ? ? ? ? c. Institutional policies and rules ? ? ? ? ? ? d. Available resources ($, time, people) ? ? ? ? ? ? e. Discouragement from manger/supervisors ? ? ? ? ? ? f. Discouragement from colleagues ? ? ? ? ? ? g. Other (please specify) ? ? ? ? ? ? n. Please comment on your experience in applying the knowledge and skills you have gained from course. 36 ANNEX C: IMPACT EVALUATION QUESTIONNAIRE (NON-IBO GROUP) Thank you for participating in the impact evaluation of "Introduction to Bank Operations" (IBO) course and completing this questionnaire. YOUR AWARENESS AND FA MILIARITY WITH THE COURSE TOPICS 1. The IBO course is designed to help new staff familiarize with knowledge and information relating to the following topics. To what extent is the content of each of the topic directly applicable to your work? Not at All 2 3 4 Very Much N/A a. Bank Evolution ? ? ? ? ? ? b. Corporate Perspective ? ? ? ? ? ? c. Sector Perspective ? ? ? ? ? ? d. Country Perspective ? ? ? ? ? ? e. Results Orientation ? ? ? ? ? ? f. Economic and Sector Work ? ? ? ? ? ? g. Investment Lending ? ? ? ? ? ? h. Adjustment Lending ? ? ? ? ? ? i. Operational Policies ? ? ? ? ? ? j. Legal Aspects ? ? ? ? ? ? k. Procurement ? ? ? ? ? ? l. Financial Management ? ? ? ? ? ? m. Developing a CAS ? ? ? ? ? ? 2. During the past three months, to what degree have you used the knowledge and information related to the following topics? Very Very 2 3 4 N/A low High a. Bank Evolution ? ? ? ? ? ? b. Corporate Perspective ? ? ? ? ? ? c. Sector Perspective ? ? ? ? ? ? d. Country Perspective ? ? ? ? ? ? e. Results Orientation ? ? ? ? ? ? f. Economic and Sector Work ? ? ? ? ? ? g. Investment Lending ? ? ? ? ? ? h. Adjustment Lending ? ? ? ? ? ? i. Operational Policies ? ? ? ? ? ? j. Legal Aspects ? ? ? ? ? ? 37 k. Procurement ? ? ? ? ? ? l. Financial Management ? ? ? ? ? ? m. Developing a CAS ? ? ? ? ? ? 3. During the past three months, how many times have you sought information from the following resources? (N=Never heard of) 0 1-3 4-6 7-9 10+ N a. Bank's Operational Manuals ? ? ? ? ? ? b. "Green book" Guideline ? ? ? ? ? ? c. "Red book" Guideline ? ? ? ? ? ? d. CDF-Aligning Culture and Staff Behaviors ? ? ? ? ? ? e. Evaluation Manual ? ? ? ? ? ? f. PCD and PAD guidelines ? ? ? ? ? ? g. OPCS websites ? ? ? ? ? ? h. OED websites ? ? ? ? ? ? i. OPCS help desk via email/phone ? ? ? ? ? ? 4. During the past three months, have you attended any of the following suggested operation related training courses? (Check all that apply. If you answer no, go to Part II.) ? Yes ? No Safeguards Financial Management Procurement Lending Instrument Monitoring and Evaluation Project Planning and Preparation Project Supervision Project Cycle Management Trust Fund Learning Accreditation Program Other (please specify) 38 ANNEX D: KNOWLEDGE TEST Methodology of developing the knowledge test 1) Used Specification Matrix (see below) provided by the Level 2 team to calculate "weight" for each teaching module, based on 36 items - the final number of items included in the Level 2 knowledge test; 2) Used the above calculated weight to estimate the number of items needed for the shorter knowledge test, based on 22 items ­ the final number of items remaining in the item pool (unused); 3) Removed some items that are marked as "Low important". A final 12 items (paired) were derived from this procedure; and 4) Two random procedures were then followed to finally select 12 paired items from the item pool (22 paired items) and 12 single items from the paired selected in the previous procedure. Specification Matrix ­ Items in Knowledge Quiz No. of pairs Know- Appli- needed Final No of Import- ledge cation No. of Q pairs Weight for Based on 22 items Content Area ance 60% 40% needed each items Pair 1 Bank Evolution L 1 0 1 0.03 0.6 2 Corporate M 2 0 2 1 Perspective 0.06 1.2 3 IFC and the Bank M 1 1 2 0.06 1.2 1 4 Country H 2 2 4 1 Perspective 0.11 2.4 5 Sector Perspective L 1 0 1 0.03 0.6 6 Results H 3 1 4 1 Orientation 0.11 2.4 7 ESW M 2 1 3 0.08 1.8 1 8 Investment M 3 0 3 1 Lending 0.08 1.8 9 Adjustment M 2 2 4 1 Lending 0.11 2.4 10 Financial Products L 1 0 1 0.03 0.6 11 Operational Policy M 1 1 2 0.06 1.2 1 12 Legal Aspects of L 2 0 2 1 Bank 0.06 1.2 13 Financial M 1 1 2 1 Management 0.06 1.2 14 Procurement M 1 1 2 0.06 1.2 1 15 Develop a CAS H 2 2 4 0.11 2.4 1 Total 25 12 37 0.13 22 12 39 KNOWLEDGE TEST The following questions are a collection of fact, issues and concepts you may wonder or need know in order to perform your duties. The goal of these questions is not to test you, rather assess the coverage ofthe course contents and the course effectiveness. There is only 1 correct answer for each question. 1. Which of the following was not a consequence of the establishment of IDA in 1960? Expansion of resources Greater involvement with private sector projects Lending into new sectors Lending into new clients I don't know 2. What is the key mission of the World Bank Group? Reconstruction and Development Attaining the Millennium Development Goals (MDGs) Enabling client countries' access to affordable development loans Reducing poverty with passion and professionalism I don't know 3. Which of the following is not key to a good CAS? Less than 30 percent of commitments through adjustment lending Comprehensive diagnosis Selective interventions Clear and verifiable scenario triggers I don't know 4. Project assessments and reviews by the Operation's Evaluation Department focus on: Development impact after the project's completion Quality at entry of recently designed projects Adequate evaluation systems in place at the time of project appraisal Feasibility of a project at the time of project appraisal I don't know 5. Which statement regarding the Quality Assurance Group (QAG)'s evaluation of ESW product quality is false? It covers both formal and informal ESW products, including the output itself and the process of preparation QAG's assessment standards are well-defined, and emphasize the same points each year More expensive ESW tasks have a greater chance of being reviewed than less expensive tasks The QAG review is an iterative process in which QAG panel solicits the views of the task team, country manager, and the client I don't know 6. Which of the documents prepared for investment loans listed below has the status of an international treaty? PCD (Project Concept Document) PAD (Project Appraisal Document) 40 LA (Loan Agreement) ICR (Implementation Completion Report) I don't know 7. Which of the following is not an operational policy requirement for adjustment lending? Satisfactory macroeconomic framework Policy conditionality Procurement requirements Use of resources for intended purposes (except for negative list) I don't know 8. What is the main purpose of policy harmonization efforts? To ensure that all international financial institutions benefit from the lessons of Bank development experience To ensure that the Bank benefits from the lessons of other agencies' development experience, and vice versa To minimize transaction costs and improve development effectiveness in client countries To minimize transaction costs for the many international financial institutions involved in supporting development I don't know 9. The Bank enters into loan agreement with: The Government of the country concerned The country itself The people represented by NGOs All development partners engaged in lending to a given country I don't know 10. A country program manager would be advised to assess follow up on a recently performed Country Financial Accountability Assessment (CFAA) through which of the following reporting practice? Common financial and other reports on sector performance Periodic financial monitoring reports (FMRs) and annual accounts The country's normal reporting arrangements on public finances None of the above I don't know 11. Which of the following is most likely to restrict the effectiveness of a Country Procurement Assessment Review (CPAR)? Scarcity of procurement specialists in the country who know Bank procedures Limited budget to develop a CPAR every year Lack of Bank and country follow up on the action plan proposed by the CPAR Lack of a procurement specialist in the Bank's country program team I don't know 12. Which of the statements below describes more accurately the correct sequence between the PRSP and the Bank's CAS? The Bank's CAS should follow and build upon the country's own PRSP The Bank's CAS and the country's PRSP are produced in parallel in close consultation The country's own PRSP should follow and build upon the Bank's CAS 41 There is no single correct sequence for the CAS and the PRSP I don't know 42 ANNEX E: PRINCIPAL COMPONENT ANALYSIS FOR SCALES FOR GENERAL KNOWLEDGE AND OPERATION KNOWLEDGE (Rotated Component Matrix) Factor loading for General and Operation knowledge, percent of variance explained, reliability statistics (Cornbach's alpha) and item means (N=114) Factors Loading Variance Reliability Item means General Knowledge 35.5 .83 Corporate Perspective 0.69 2.45 Sector Perspective 0.81 2.89 Country Perspective 0.73 2.73 Result Orientation 0.74 2.96 Economic & Sector Work 0.75 2.74 Operation Knowledge 29.9 .87 Investment Lending 0.79 2.74 Adjustment Lending 0.62 2.31 Operational Policies 0.78 2.82 Legal Aspects 0.76 2.35 Procurement 0.87 2.69 Financial Management 0.85 2.28 43 ANNEX F: SUMMARY OF DEPENDENT, INDEPENDENT VARIABLES AND SCALE COMPOSITION Dependent variable Knowledge utilization Knowledge utilization is a continuous variable combining the utilization of General and Operations knowledge score, measured at the three-month follow-up survey. The higher the score, the higher the level of knowledge utilization. Independent variables Training factors Training factors concerned in this analysis are the quality and usefulness of the IBO course, as perceived by the participants. Such information was colleted through the end of course Level 1 evaluation. As analyzed earlier (section 3.6), there were 11 single items measuring aspects of quality and usefulness, using a five point scale: 1=poor to 5=good. To simplify the analysis, items were grouped into two indices: course quality and course usefulness, which were derived from a Principal Component Analysis (see table below). The index score is the sum of the component item scores. The higher the score, the higher the perceived course quality and usefulness. Trainee factors Trainee factors are represented by four key participant demographic variables collected from the Level 1 evaluation: country station, VPUs, grade level and length of Bank service. Internal factors Internal enabling factors help participants to utilize knowledge and skills learned from the IBO course. In this study, internal enabling factors are: · Knowledge level (post and follow-up knowledge test scores); · Self-confidence (1=low to 5=high, measured at follow up); and · Motivation. The motivation factor was derived from a Principal Component Analysis based on a group of perception variables collected at the end of course evaluation (see table below). A high score corresponds a high degree of motivation. 44 External enabling/support factors Factors that may affect an individual's application of knowledge, but that are mostly beyond the individual's control, are considered to be external support factors. Based on a group of variables measured at the follow up (listed in Table 8), an index of external enabling/support factor was created using a Principal Component Analysis. The result of the PCA is included in Table below. A higher score corresponds to more positive external support. Conversely, lower scores suggest a relative lack of support -- or the presence of hindering external factors. Independent Variables and Factor Scale Composition Loading on 1st factor Reliability Scale Item PCAa () Training factor · Quality Content (1=Low, 5=High) .81 .79 (Low=poor, High=good) Structure .74 Materials .70 Overall quality .79 · Usefulness Usefulness (1=Low, 5=High) .83 .78 (Low=less, High=more) Relevance .79 Increasing knowledge .79 Applicable .73 Trainee factor Length of Bank service (Continuous) Duty station (1=Field; 2=Headquarter) Network (1=Region, 2=Anchor and other) Grade level (Continuous) Internal factors · Motivation Organization awareness (1=Low, 5=High) .58 .91 (Low=low motivation Commitment to organization goals .76 High=high motivation) Motivation to achieve high standards .89 Commitment to work effectively as an .87 individual .88 Commitment to work cooperatively as a .73 team play .73 Motivation to engaging further learning Rational and strategic thinking · Knowledge score Post course score (Continuous) 3-month follow up (Continuous) · Self-confidence Self-confidence (1=Low, 5=High) External factors Job relevance (1=Low, 5=High) 72 .86 (Low=low support High=high Institutional policies .82 support) Available resources .76 Manager encouragement .81 Colleague encouragement .85 a. PCA -- Principal Component Analysis 45 ANNEX G: RESULTS OF MULTIPLE REGRESSION ANALYSES (STEP 1) Effects of training, trainees, internal and external factors on knowledge utilization (N=68) Independent variables Dependent variable: knowledge utilization Beta t Beta t Beta t Beta t Training Quality 0.17 1.31 Usefulness 0.31 2.40** Trainees Station -0.121 -0.91 VPUs -0.05 -0.37 Grade level 0.10 0.73 Length of Bank service -0.06 -0.47 Internal Motivation 0.18 1.37 factors Self-confidence 0.36 2.85*** Post-test score 0.02 0.19 Follow-up score 0.07 0.53 External Support factors 0.54 5.23*** R2=0.08 R2=0.02 R2=0.21 R2=0.29 ***p<.01; **p<.05; *p<.10 46 ANNEX H: RESULTS OF MULTIPLE REGRESSION ANALYSES (STEP 2) Effects of usefulness, self-confidence and external factors on knowledge utilization (N=68) Independent variables Dependent variable: knowledge utilization Model A Model B Model C Beta t Beta t Beta t Training Usefulness 0.33 2.69** 0.23 1.94* 0.15 1.34 Internal Self-confidence 0.38 3.23*** 0.11 0.81 External Support factors 0.48 3.66*** R2=0.11 R2=0.24 R2=0.38 ***p<.01; **p<.05; *p<.10 · Model A shows that when participants perceived the course to be more useful, they were more likely to utilize the course content; · Model B shows that when the internal factor of self-confidence is introduced together with training factors (usefulness) into the model, the effect of usefulness was decreased slightly by the effect of self-confidence, but still remains significant. A notable improvement observed in model B is that the R Square doubled from model A. This suggests that when participants perceived the course to more useful and when they felt more confident about their abilities, they were more likely to utilize the course knowledge in their work; and · Model C shows another substantial improvement in R Square as the result of introduction of external support factors. The regression generated estimated joint (linear) relationship between training factors, internal factors and external factors which explains nearly 40 percent of the variabilityin the data. The effects of both course usefulness and self-confidence dropped from 30 to 70 percent ( from 0.23 to 0.15; and from 0.38 to 0.11) in model C, becoming insignificant. On the other hand, external factors were found to have a strongly significant effect on the level of knowledge utilization, capturing much of the predictive power of the other two independent variables. 47