31291 Evaluation of the World Bank Institute (WBI) Macroeconomic Management Program Marie-Helen Adrien Jaime B. Quizon (Task Manager) WBI Evaluation Studies No. EG04-79 The World Bank Institute The World Bank Washington, D.C. October 2003 ACKNOWLEDGMENT This report was prepared for the World Bank Institute (WBI) under the overall guidance of Marlaine E. Lockheed, Manager, World Bank Institute Evaluation Group (WBIEG). This report was prepared by Marie-Helene Adrian of Universalia and Jaime B. Quizón (WBIEG) supervised this study. The main findings of this report were presented for comment and discussed with members of the WBI Macroeconomic and Policy Assessment Program on August 5, 2003. This report also benefited greatly from comments by peer reviewers: William Eckert and Cristina Ling Chard (WBIEG) and Hippolyte Fofack (WBIPR). Humberto Diaz (WBIEG) provided expert assistance with document production. WBI Evaluation Studies are produced by the Institute Evaluation Group (IEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at: http://info.worldbank.org/etools/WBIEG/publications/index.cfm?pg=Home&Intro=yes Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group Ms. Marlaine Lockheed Task Team Leader Mr. Jaime B. Quizón ii ACRONYMS ADB African Development Bank AfDB African Bank of Development BCEAO Banque centrale des États de l'Afrique de l'Ouest BEAC Banque des États de l'Afrique centrale CN Canadian National DL Distance Learning GDLN Global Development Learning Network GDP Gross Domestic Product IDB Inter-American Development Bank IEG Institute Evaluation Group IMF International Monetary Fund MM Macroeconomics Management MPAS Macroeconomics and Policy Assessment Skills NGO Non-Governmental Organization PRSP Poverty Reduction Strategy Paper TM Task Manager WB World Bank WBI World Bank Institute WBIEG World Bank Institute Evaluation Group iii iv TABLE OF CONTENTS Acknowledgment ii Acronyms iii Table of Contents v 1. Introduction 1 Background 1 Objectives of the Evaluation 2 Key Evaluation Questions 2 Methodology 2 Data Collection 2 Questionnaire 3 Interviews 3 Data Sources 4 Limitations 4 Review Team 6 2. Courses Evaluated & Return Rates 7 Courses Reviewed 7 Return-Rates 8 3. Effectiveness and effects of WBI MM activities and workshops on participants 11 Introduction 11 Presentation of Findings 11 Effects of MM Courses on Training and Teaching Activities 24 4. Effectiveness of MM Workshop Teaching Methodologies and Learning Activities 29 Introduction 29 Presentation of Findings 29 5. Efficiency of Management of MM Learning Activities 35 Introduction 35 Presentation of Findings 35 6. Conclusions and Lessons Learned 39 v ANNEXES 43 Annex A: List of Findings 45 Annex B: Interview Protocols 49 Annex C: Sample Questionnaire 53 Annex D: People Interviewed 63 Annex E : Survey Results 66 vi 1. INTRODUCTION BACKGROUND 1.1 From 1998 to 2002, the World Bank Institute (WBI) organized the delivery of five training courses to selected participants in Francophone and Anglophone countries of Sub- Saharan Africa), as part of its Macroeconomic Management Program (MM). The MM Program is aimed at fostering the knowledge of key policy makers in transition and developing countries on key macroeconomic issues. Macroeconomic Management is viewed as a key component of development policy. As new and important challenges emerge in the areas of monetary, fiscal, and exchange rate policies in both developing and transition economies, it is important to understand the important developments that have occurred in the last two decades with regards to the analytical tools that have been used to address these issues. 1.2 More specifically, the objectives of the MM program are to: · Expand the capacity of policymakers, academics, Bank staff, staff from other international organizations (such as the IMF and the African Development Bank), and other development practitioners; to analyze current issues in macroeconomic management. · Help design appropriate policy responses to the increasing flows of capital across national borders--flows whose scale and speed make the task of macroeconomic management more complex. · Build relevant analytical and practical skills in the area of policy assessment in government institutions, multilateral organizations, universities, and other research- oriented institutions. 1.3 Over the past few years, WBI conducted a few level 1 and 2 evaluations of MM courses to assess the participants' degree of satisfaction with respect to the workshop (level 1) and the degree of learning that occurred during the workshop (level 2). Recognizing that level 1 and 2 evaluations are limited in scope, WBI has been eager to undertake a level 3 evaluation to better understand long-term behavioral effects on individual learners.1 As a result, the Evaluation Unit of WBI contracted Universalia Management Group in March 2003 to perform a level 3 evaluation of the MM Program. A level 3 evaluation generally seeks to assess effects and results of workshops and results of the program on the participants and their respective organizations. 1According to Donald Kirkpatrick, who identified the four levels of evaluations, a level 3-evaluation measures long- term behavioral impact on individual learners. WBI conducted a level 3-evaluation for one of its Program training courses that was held in Sao Paolo, Brazil, in January of 2001. For more information on Kirkpatrick's 4 levels of evaluations, see: http://coe.sdsu.edu/eet/Articles/k4levels/index.htm 1 OBJECTIVES OF THE EVALUATION 1.4 The objective of this report is to present WBI with an evaluation of the five courses MM delivered in Sub-Saharan Africa from 1998-2002. Specifically, WBI and Universalia agreed that the evaluation would focus on the following objectives: · Study effects and results of the workshops on building individual skills of the participants, the ability of the participants to apply the workshop to the workplace, illustrations of potential results that could be attributable to the workshop. In addition, contextual factors affecting the viability of the results will be presented. · Evaluate the management of the program, including: the design of the workshop, the targeting of participants, the selection process, the follow-up to the workshop, the evaluation of the results; return on investments and value for money of the various delivery methodologies. · Look at lessons learned for WBI that could be integrated into future workshops activities with the aim of increasing the effectiveness and efficiency of the programming. Key Evaluation Questions 1.5 At the proposal stage, the evaluation team developed an evaluation matrix that summarized the main issues and questions to be addressed. There were originally five key evaluation questions to be answered. However, as two of the questions relating to the effectiveness of teaching methodologies and WBI learning activities were complementary, the evaluation team amalgamated the two for clarity of reporting. The report therefore addresses the following four-key evaluation questions: 1) How effective and what are the effects of WBI MM activities and workshops; 2) How effective are the WBI MM workshop teaching methodologies and learning activities; 3) How efficient is the management of the MM learning activities; 4) What are the lessons learned from WBI MM courses and workshops METHODOLOGY Data Collection 1.6 To assess the effects of its MM activities review, Universalia has collected primary data using a methodological mix of qualitative and quantitative data collection techniques. Qualitative data collection methods included document review, in-person interviews and phone-interviews. With regard to the quantitative data-collection technique, Universalia designed and conducted a survey of participants of the MM workshops. The following provides a description of those methods. 2 Questionnaire 1.7 A questionnaire survey (see Annex III) was carried out to survey MM workshop participants. The questionnaire was sent to all participants (with current contact information) in each of the five MM courses. This questionnaire included questions on knowledge of the Courses and their mandates, relevance of activities, evidence of results, as well as the operation and management ofthe courses themselves. 1.8 Universalia administered the questionnaire through a Web-based approach. Participants completed the questionnaire through a website link posted on the "survey monkey" website.2 For participants who could not access the Internet, Universalia sent the questionnaire by fax. WB and BCEAO field offices assisted Universalia in following-up with participants who have left their employment since the training occurred. Our data entry specialists and statistical analysts then ensured that the results were processed, interpreted, and fed back into the evaluation process in a timely manner. Interviews 1.9 Interviews provide a wealth of information that allows evaluators to explore, clarify, and validate ideas. In addition to in-person interviews conducted at the WB headquarters in Washington, D.C., with course managers, Universalia conducted telephone interviews with three groups of stakeholders, namely: · Supervisors of MM Course Participants; · MM Course Moderators and; · Representatives of local Partner Organizations that assisted WBI in the facilitation of the courses. 1.10 During the questionnaire stage, Universalia asked WBI MM course participants to provide, on a voluntary basis, the name of their manager/supervisor who approved their participation to the WBI training course. The purpose of collecting data from the supervisors was to determine the impact the course had on the behavioral changes of learners at the individual and organizational levels. 1.11 Course moderators were also interviewed to determine effective methods of course facilitation and delivery, among other things. Representatives from local Partner Organizations, on the other hand, were canvassed as to their relationships with WBI and how this relationship affected the delivery of the course. 1.12 Universalia has employed an ongoing data analysis to feed results into the evaluation process throughout its duration. As new sources of data were received, analyzed, interpreted, triangulated, and fed back into the process for stakeholder clarification, a chain of evidence was 2For an illustration, pleaseconsult: www.surveymonkey.com 3 created on which final evaluation findings are based. The evaluation team also developed a log of lessons learned that can be shared with WBI. Data Sources 1.13 The main sources of data for this study were the individuals involved in all aspects of the five MM Program training courses. These individuals represented four different groups: the course participants themselves, the managers/supervisors of the course participants, the moderators of the courses, and finally, representatives of local partner organizations who provided logistical and technical support to the operation of the courses. Most data was collected from questionnaire responses from MM Program participants who had attended one of the five courses. 1.14 Contact information for WBI MM course participants, course Moderators and local Partner Organizations was either found on the WBI MM Program website3 or obtained through WB offices in Washington, D.C., as well as WB and BCEAO field offices. Limitations 1.15 The evaluation team encountered a few constraints in the course of this evaluation. The first constraint has to do with the mandate itself, that is to say, conducting a level-3 evaluation. Indeed, conducting a level-3 evaluation and assessing the effects and/or impact of a course in the workplace requires that at least the following elements be in place: · A definition of what specific behavior/knowledge must be modified by the course and the expected potential changes in the professional environment, as a result of having attended the course · A baseline of participant's level before the course on each behavior the workshop is attempting to modify, for the purpose of engaging in a comparison. 1.16 None of the five MM courses evaluated had identified the expected transformational changes that should occur in the workplace as a result of the course, nor had a baseline been established. 1.17 The source of data was another element of limitation. Typically, a level 3 evaluation attempts to obtain data from one or more of the following sources: a) the participant; b) the participant's supervisor(s); c) a sample of the participant's peers; d) a sample of the participant's subordinate(s). Enlarging the sources of data to include these respondents increases the reliability of the observations made with respect to behavior modifications. In the present mandate, no list (contacts, address, names) of either peers, subordinates, or supervisors existed and the evaluation team had to rely on the goodwill of the workshop's participants to submit the names and 3http://www.worldbank.org/wbi/macroeconomics/activities_recent.htm#gdln), click on specific course for the detailed information. 4 coordinates of their supervisor and allow the team to contact him/her. As a result, the sample of subordinates interviewed is skewed to those names provided by consenting participants. 1.18 In addition, data collection presented several challenges to ensure that return-rate would be acceptable. Among obstacles faced in contacting WBI course stakeholders, it is worth noting that: · Several of the email addresses provided by the WBI to the evaluation team proved to be invalid. Out of 200 participants, we had 132 email addresses. However, 24 of these emails were incorrect or no longer in use. The evaluation team then attempted to contact directly the participants with stale email addresses. In most cases, contact was not established. However, a few were contacted and provided their fax numbers where they could be sent the questionnaire. · It was difficult to reach participants for courses that took place in 2000 and 2001. Many former participants were no longer accessible at their workplace while others were reluctant to evaluate a course they didn't remember well. Moreover, several participants that attended the WBI face-to-face training course in Abidjan in March of 2000 had new positions, which made it difficult for the evaluation team to find supervisors that had worked with them at the time of the course. Former supervisors were also reluctant to assess the performance of employee they had not seen for a while. · It was difficult to contact staff from the African Bank of Development (AfDB) who had been involved in the organization of the face-to-face course in Abidjan in March 2000. This can be explained by the recent relocation of the AfDB Headquarters from Abidjan to Tunis (Tunisia) due to political instability in Ivory Coast. At the time of the evaluation, AfDB staffs were not yet installed in Tunis and phone connections with the new headquarters were problematic. As a result, the evaluation team could not interview representatives of the partner organization for the Abidjan face-to-face course. · The evaluation team also encountered difficulty in reaching course participants of the WBI face-to-face training course that was held in Dakar in July of 2002. The list of participants initially provided to the evaluation team did not have proper e-mail addresses and phone numbers. After several inquiries, the evaluation team was able to secure another list from the BCEAO field office in Senegal, but this slowed the process and limited in part the response-rate for this course. 1.19 Despite the above limitations, the evaluation team nevertheless succeeded in obtaining good return-rates as will be explained in the following section. 1.20 In addition, the evaluation team was not able to obtain financial figures detailing overall costs of the five training course (two face-to-face and three GDLN), despite a request made to WBI course managers. These figures would have included overall cost of expenses (excluding staff time) for all five courses evaluated. The lack of information on this matter limited the extent of the evaluation of the efficiency topic. 5 1.21 Finally, it should be noted that here was no control group as part of this evaluation. In experimental research or evaluation, a control group does not receive any treatment or intervention and is therefore used as a benchmark to assess whether behavioral changes happened among the evaluation's respondents. The absence of a control group made it difficult to assess whether or not there was a cause-effect relationship between training beneficiaries and their self-reported data. REVIEW TEAM 1.22 The evaluation team consisted of three Canadian consultants from Universalia Management Group: · Dr. Marie-Helene Adrien, Team Leader · Suzanne Kirouac, Evaluation Specialist · Simon Thibault, Evaluation Specialist 1.23 The team was responsible for the following activities: · Development of data collection tools (survey, interview protocols, etc.); · Data collection and analysis; · Reporting; · On-going coordination with WBI. 6 2. COURSES EVALUATED & RETURN RATES COURSES REVIEWED 2.1 The following table provides a description of the MM courses delivered in Sub-Saharan Africa which constitute the focus of this study: two face-to-face courses and three GDLN courses. The number of participants is based on the lists provided on WBI's website as well as on the information provided by the WB headquarters and field offices (including from BCEAO field offices). Figure 1 Location, date and audiences of face-to-face and GDLN courses in Francophone and Anglophone Sub-Sahara Africa COURSE LOCATION DATE NUMBER OF PARTICIPANTS' COUNTRY PARTICIPANTS 2 Face-to-Face Courses Macroeconomics Dakar, July 15-25, 37 Bénin (3), Burkina Faso (3), Côte d'Ivoire (4), Management for Financial Sénégal 2002 Guinée Bissau (1), Mali (2), Niger (2), Sénégal Stability and Poverty (3), Togo (2), Cameroun (4), Centre Afrique (2), Reduction Congo (4), Gabon (3), Guinée Équatoriale (2), Tchad (2) La Gestion Abidjan, Côte March 13- 34 participants Angola (1 ), Bénin (3), Burkina Faso (2), Macroéconomique: Débats d'Ivoire 23, 2000 (45 with WB Burundi (1), Cameroun (6 ), Comores (1), Côte Actuels et Méthodes staff) d'Ivoire (5), Djibouti (1), Gabon (1), Guinée (1), Quantitatives Guinée Bissau (1), Madagascar (2), Niger (3), Centre Afrique (3), Congo (1 ), Rwanda (1), Sao Tome & Principe (1), Sénégal (3), Tchad (3), Togo (5 ) 3 GDLN Courses Gestion macroéconomique Sub-Sahara April 3-May 59 Burkina Faso (16), Senegal (24), Ivory Coast et stratégies de réduction de Africa 29, 2002 (19) la pauvreté en Afrique subsaharienne Macroeconomic Anglophone May 4-June 23 Ghana (5), Uganda (8), Tanzania (10) Management and Poverty Africa 29, 2001 Reduction Strategies in sub-Saharan Africa La Gestion Francophone October 5- 47 Benin (25), Denmark (1), France, (1), Ivory Macroéconomique en Africa December Coast (15), Senegal (4), France (1) Afrique subsaharienne: 5, 2000 Analyse et Enjeux Actuels 7 RETURN-RATES 2.2 Out of the 108 participants in the five face-to-face and GDLN courses4 who received the questionnaire, a total of eighty one (81) individuals responded on-line or by fax. This means that 70% of respondents who received the questionnaire filled out and returned it. This 70% return- rate is sufficient for reliability of the data and excellent considering the limitations described previously5. The numeric breakdown is as follows: · Macroeconomics Management for Financial Stability and Poverty Reduction Strategies (Dakar, July 15-25, 2002) ­ 8 responses · La Gestion Macroéconomique: Débats Actuels et Méthodes Quantitatives (Abidjan, March 13-23, 2000) ­ 11 responses · Gestion Macroéconomique et Stratégies de Réduction de la Pauvreté en Afrique subsaharienne (GDLN, Sub-Sahara Africa, April 3 ­ May 29, 2002) ­ 30 responses · La Gestion Macroéconomique en Afrique subsaharienne: Analyse et Enjeux Actuels (GDLN; Francophone Africa, October 5 ­ December 5, 2000) ­ 22 responses · Macroeconomic Management and Poverty Reduction Strategies in sub-Saharan Africa (GDLN, Anglophone Africa, May 4 ­ June 19, 2001) ­ 10 responses Return Rates E A 12% 10% Macroeconomics Management for Financial Stability and B A Poverty Reduction Strategies (Dakar, July 15-25, 2002) 14% B La Gestion Macroéconomique: Débats Actuels et Méthodes Quantitatives (Abidjan, March 13-23) D Gestion macroéconomique et stratégies de réduction de la 27% C pauvreté en Afrique subsaharienne (GDLN, Sub-Sahara Africa, April 3-May 29, 2002) D La Gestion Macroéconomique en Afrique subsaharienne (GDLN, Francophone Africa, October 5 - December 5, 2000) C E Macroeconomic Management and Poverty Reduction 37% Strategies (GDLN, Anglophone Africa, May 4 - June 19, 2001) 4This total does not include the 11 WB staff that attended the face-to-face course in Abidjan in March 2000. 5See limitations in securing correct addresses from participants. 8 2.3 In addition to the 81 responses to the questionnaires, the evaluation team interviewed the following people, either in-person or by phone: · The course manager, Mr. Agénor; · 17 Supervisors of MM Course Participants; · 11 Course Moderators; · 9 Representatives from local WBI partner organizations A list of people interviewed is provided in Annex D. 9 10 3. EFFECTIVENESS AND EFFECTS OF WBI MM ACTIVITIES AND WORKSHOPS ON PARTICIPANTS INTRODUCTION 3.1 One of the main goals of this evaluation was to highlight the extent to which the course targeted the right audience and the relative influence the five MM courses had upon the participants in terms of behavioral changes occurring after the training at the individual and organizational levels. Data presented in this section was drawn from questionnaire responses and interviews. Data provides indication on the extent to which WBI training activities enhanced participants' awareness, capacities and learning opportunities. It also details whether courses participants were able to apply their MM learning in their professional environment, while discussing factors facilitating or preventing participants' use of knowledge and skills acquired from WBI courses. PRESENTATIONOF FINDINGS 3.2 The following graphs highlight the responses from course participants who completed the questionnaire, either by fax or on-line. For clarity and legibility purposes, the evaluation team only selected a sample of the graphs that it felt provided relevant responses to the key evaluation questions. The complete series of graphs, which reflect results to all questions posed in the questionnaire, can be found in Annex E. 3.3 In the absence of specifically pre-determined criteria of effectiveness and effects of the MM course, the evaluation team and WBI managers developed the following ones: Indicators ofeffects and effectiveness used for the purpose of this study Illustrative indicators of Effectiveness6 Targeting the right audience Acquisition of new learning at the end of the workshop Participant's satisfaction with the content of the workshop Quality material delivered Effective delivery by moderators (Indicators continues on the next page.) 6The majority of these indicators were tested at the level 2 evaluation, in the post-workshop evaluation form. 11 (Indicators continued.) Illustrative indicators of effects7 Promotion or job enrichment further to the workshop Retention of learning and use of learning in everyday professional situations Regular use of the course material further to the workshop in everyday professional situations Contribution to the development, final approval, or rejection of economic policies in the country Undertaking of policy research on a theme studied during the workshop Sharing with colleagues results of research on themes studies during the workshop Increased ability to solve problems Contribution to publication or research Increased national, regional networking for professional purpose by former participants Finding 1: Overall, the level 1 evaluations conducted for the five MM courses under review reveal a high level of satisfaction with the training courses. 3.4 Although the mandate of this review was not to survey the level of satisfaction of participants with respect to the course, the evaluation team reviewed the level 1 evaluation for the five MM courses under review. In general, results were very positive. Participants were particularly pleased with the relevance of the activity, the overall usefulness of the training and the extent to which they had acquired new information. They were more critical, however, about the focus of the activity on what they specifically needed to learn. Finding 2: WBI MM Courses reviewed were effective in reaching targeted audiences. To which sector does your organization belong? Which of the following activites best describes your profesion? 5% 7% 5% 22% 35% 32% 58% 36% Public Sector (national) Public Sector (international) Teaching & Training Analysis and research Private Sector Non-governmental sector Public policy development Other 3.5 One of the main objectives of the MPAS program is to enhance analytical and practical skills of government policymakers, academics, international organization staff (including WB) as well as development practitioners, to analyze current issues in macroeconomic management and 7These indicators were developed with WBI managers. 12 policy assessment since they work primarily for government institutions (ministries), multilateral organizations, central banks, universities, and other research-oriented institutions. WBI deliberately sought to attract these group audiences since they are actors who can have an impact upon economic policies in their given country8. 3.6 Questionnaire results suggest that the MM courses organizers were successful in targeting those audiences, especially from the public sector. The majority of respondents indicated that they are employed by the state (national) public sector (58%) of their respective countries or by the international public sector (32%), comprised of multilateral organizations. These two groups represent higher proportions of participants, for they are more closely related or exposed to policy development in their country. The remaining respondents indicated that they are from either the private sector (5%) or from the non-governmental (NGO) sector (5%), two sectors less directly exposed to policy development. In terms of professional activities, 36% and 35% of respondents described their occupation as comprising of public policy development or analysis of research, while 7% are involved with teaching and training activities. Twenty two percent (22%) of respondents selected the option "other." Finding 3: The vast majority of respondents have maintained their positions of employment with the same organization. The majority of those that did change positions, report that they did so as a result of knowledge gained during the course. Since you attended the course, have you worked for the same If you answered "no" to the last question, do you agree that the organization? course encouraged and/or helped you find another position? 3% 9% 14% 29% 7% 88% 50% Yes No, I began working for another organization shortly after the course ended Completely disagree Somewhat disagree Rather agree No, I recently started working for another organization Completely agree Do not know 3.7 A key objective of any given training program is that course participants are able to disseminate learned knowledge upon their return to their organization. New skills and knowledge gained, however, can create new professional opportunities, especially in the context of a developing country. It is not unusual for trainees to approach (or be approached by) other organizations for more interesting or lucrative employment opportunities upon completion of 8It may be argued that certain course participants, particularly those involved in teaching, training, and research (academics and researchers) do not necessarily have much of a direct impact upon economic policies. However, it is important to remember that these individuals are often the means by which such information is disseminated, as academics are often the vehicles by which policies are passed down to students, who will in turn be future policy makers. Additionally, the course participants are not always the most senior people, and thus they may not necessarily have direct influence upon economic policy development. 13 their training courses. Hence, employee retention may become a key issue in affecting the transfer of knowledge to the original organization. 3.8 This was not the case with participants of the five MM courses assessed. Survey results indicate that an overwhelming majority (88%) of trainees maintained their positions of employment with the same organization, which they worked for prior to the courses. Most of the knowledge gained by course participants was therefore kept within the organization in question. 3.9 Nine percent (9%) of respondents, however, indicated that they had begun working for another organization. Interestingly, just under two-thirds (64%) of these individuals credited the course with assisting them in finding another position, while 36% indicated that the courses had little or no impact upon their employment switch. However, it should be qualified that the respondents did not indicate to which type of organization they switched to. In the event of a change of position, then, participants therefore viewed MM courses as having had an impact on their ability to acquire other employment opportunities following completion of the course. Finding 4: The majority of respondents indicated that they used and retained knowledge gained through MM courses in their work, a view generally supported by supervisors of participants interviewed. I used concepts and techniques learned in the course of my work I did not retain the concepts and principles discussed in the course, because they had little relevance with my work 4% 8% 4% 3% 6% 11% 26% 29% 58% 51% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know 3.10 The above results suggest that MM courses were successful in introducing concepts and techniques that have been subsequently utilized in the workplace by course participants. Seventy-seven percent (77%) of respondents stated that they have used the concepts and techniques taught during the courses in their current work. This high figure suggests that participants perceived the courses' content as appropriate to their learning needs. 3.11 The fact that only 10% of respondents to the questionnaire indicated that they did not retain the concepts and principles taught during the courses as they had little relevance to their work further reinforces the above observations. Most respondents (87%) indeed suggested the contrary and hinted that they retained the course's content as it was relevant to their work and learning imperatives. 3.12 Supervisors interviewed during the course of the evaluation generally shared the view that their employee had been able to retain and use knowledge gained at the workshop in their 14 working environment. Many stressed, for example, that the training course provided their employee with the latest developments on macroeconomic and/or poverty reduction issues. MM course were viewed as a good opportunity to update their employees' knowledge on macroeconomics and policy issues relevant to their work. 3.13 A review of result broken-down by courses generally showed consistent results from respondents that attended the 2000 and 2002 courses. Finding 5: A majority of respondents indicated that they had used the course materials on some or several occasions following the conclusion of the courses. I used course materials on some occasions I used course materials on several occasions 1% 16% 17% 22% 20% 12% 21% 42% 49% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know 3.14 Use of course material by trainees is a first-step indication of the utilization and transfers of knowledge to the workplace. Seventy one percent (71%) of participants indicated that they used the course materials on some occasions, while 59% used the course materials on several occasions. These results further reinforce the view that MM courses were relevant to the participants' needs as most used the material after the training session. 3.15 Comments gathered from supervisors generally supported the above assertions. Several supervisors referred to the usefulness of the course materials. Some, for example, specified that their employee had shared course materials with fellow colleagues through formal (presentation, meetings, etc.) and informal settings. On the other hand, other supervisors emphasized the importance of providing user-friendly materials to facilitate broad use in the workplace. A few course moderators also stressed the importance of distributing course materials to participants a few days in advance to ensure better preparation for the training. 3.16 A review (primarily through observation of data and not through a T-Test) of results broken-down by courses revealed no significant differences between the perceptions of respondents that attended 2000 and 2002 training courses, except for the May 3-29 GDLN course, where half of participants stated that they had not used the course material on several occasions. 15 Finding 6: Knowledge gained from training courses is mainly being utilized by those with influence at the organizational and national levels. In your professional activities, at which level do you exert influence? 4% 36% 52% 8% At the level of my organization At the local level At the national level At the global level 3.17 The above results show that knowledge gained from MM courses was likely to be utilized at the level of the organization and, to a lesser extent, at the national level. A majority of participants indicated exerting influence mainly at the organizational level (52%), while over a third indicated that they had influence at the national level (36%). 3.18 At first sight, this information may appear disappointing, as one of the goals of the MM courses is to promote macroeconomic management policy on the state level as a means of helping to design appropriate policy responses. If we deconstruct this finding further, however, the information is not as disappointing as initially suggested. In fact, MM face-to-face courses usually target audiences with higher positions of responsibilities such as senior government officials, policy makers from Central Banks and experienced economists from international or private institutions. Such participants, especially in the public sector, generally exert influence at the national level due to the nature of their responsibilities. GDLN course audiences, on the other hand, consist of mostly mid-level professionals in the public sector (central banks and finance ministries) and junior trainers in academia (universities, research and training institutes). Most of these people do not have a significant role at the decision-making level of their organization, although they can also be involved in the elaboration of national policies or researches. The fact that a little more than the majority of questionnaires respondents and over a third of them said that they exerted influence at the organizational and national is consistent with the number of courses evaluated (2 face-to-face and 3 GDLN). 3.19 It is also important to note, as suggested above, that mid-professionals in Ministries and Central Banks can have an impact in the development of macroeconomic policies or at the national level. For example, if a lower ranking individual from an Economic Ministry of a given country attended a WBI course and subsequently reported to her/his supervisor as to what was learned, the individual may perceive that his influence will remain at the Ministry level. However, if the supervisor were to employ the knowledge with the Minister or another important 16 decision-maker, then the participant would have had a wider impact without necessarily knowing it. Moreover, as one moderator emphasized, mid-level professionals are often the ones involved in the development of policies due to their technical skills and greater availability, thus contributing to shaping policies at the state level. Finding 7: Since attending the courses, a small portion of training participants credited the course in having a major influence in their ability to implement economic policies discussed in the course. Since you attended the course, have you implemented any If you answered yes to the last question, what difference did the course economic policies related to those discussed in the course? make in your ability to implement any economic policies? 11% 11% 44% 56% 36% 42% No difference A small difference A moderate difference Yes No A major difference A complete difference N/A 3.20 Implementation of policy derived from the concepts and knowledge gained from the five MM program courses may be considered an important benchmark of MM course success. Results drawn from questionnaire responses show mixed results. Although close to half of the respondents (44%) said that they have implemented economic policies related to those discussed in the course, only 36% of them credited the course to having a major difference with the ability to implement new economic policies, while 42% indicated that the course had made a moderate difference. 3.21 Comments from supervisors, on the other hand, were generally positive. Many stressed that the course facilitated the involvement of course participants in the development of existing or new policies initiatives. Although not all supervisors could elaborate extensively on the matter, several of them provided clear-cut example of how the course directly benefited participants in supporting policy developments. In Burkina Faso, for example, a supervisor noted that his employee got involved in the elaboration of a regional Poverty Reduction Strategy Paper (PRSP), which is based on experience gained with their national PRSPs, after his participation in the MM training. In Niger, a supervisor said that his employee had played a key role in raising the awareness of his Secretariat on the importance of developing a log frame to monitor their country's poverty reduction strategy. In Uganda, a supervisor from the Ministry of Finance stated that his employee used the knowledge from his MM course in the area of fiscal discipline and for the reduction of fiscal deficit. Although a smaller portion of supervisors could not elaborate on whether the MM course did encourage the development of economic policies related to those explored in the training session, most of them nevertheless emphasized the relevance and usefulness of the course to their employees. 17 3.22 A review of results broken- down by courses revealed no "...he (his employee) has helped to refine existing policies like the significant differences between the promotional policy and health insurance policy." perceptions of respondents that A participant's supervisor attended 2000 and 2002 training courses. Finding 8: Since their course attendance, the majority of respondents believed that they have not had an influence on the approval or rejection of economic policies related to the courses. For the minority who stated that they had influence, however, most credited taking the courses with their ability to do so. Since you attended the course, have you had influence on the If you answered yes to the last question, what difference did the course approval or the rejection of any economic policies related to those make in your ability to have an influence on the approval or the rejection of discussed in the course? any economic policies? 15% 13% 6% 6% 37% 38% 85% No difference A small difference A moderate difference Yes No A major difference A complete difference N/A 3.23 The above results suggest that course participants have not had a great influence on the approval or rejection of economic policies relating to those discussed in the course. A large majority (85%) indicated their lack of influence. However, of those respondents who did have influence (15%), 75% of them credited the courses with having a moderate to major difference in their ability to do so. 3.24 Decision-making power over the approval or rejection of economic policies requires significant influence, as one exerted by a minister or ministerial advisers. Although it is a stated MM program objective to seek participation of high-level government officials at face-to-face courses, participants seldom have such a level of influence. High-level decision-makers are usually too busy to avail themselves for a long training period. Moreover, this finding should be considered in light of the fact that the selection of participants does not only target individuals who can play a decision-making on policies. 3.25 Furthermore, it should also be kept in mind that although mid-level professionals from government offices or central Banks do not have the authority to approve or reject economic policies, they nevertheless remain key in the elaboration of these. Despite these nuances, WBI and its partner organizations should nonetheless pay further attention to the selection process of its participants to ensure higher attendance of high-level policy makers. 18 3.26 Perceptions of respondents that attended 2000 and 2002 training courses did not vary significantly, thereby indicating consistency of results over time. Finding 9: Since attending the courses, the majority of course participants have not carried out economic policy research related to research discussed in the courses. Since you participated in the course, have you carried out economic policy development research related to the research discussed in the course? 33% 67% Yes No 3.27 Two-thirds (67%) of course participants indicate that they had not carried out research relating to economic policy development related to the research discussed in the course, while one-third (33%) stated that they had done so. This finding is consistent with the fact that 35% of respondents identified themselves as working primarily on research and analysis activities (see finding 1). 3.28 An analysis of results broken-down by courses revealed no major differences between the 2000 and 2002 courses' results, except for the July 15-25 2002 face-to-face course, where a majority of respondents (albeit small) said that they carried out economic policy research related to those in the course. 19 Finding 10: Respondents indicated that they have not had the opportunity to pass along course information on economic policies. They nevertheless pointed out that they shared course materials and discussed concepts and techniques dealt with in the course with colleagues. Since you attended the course, have you had the opportunity to If you answered yes to the last question, what difference did the course pass on information concerning any economic policies related to make in your ability to pass on information about any economic policies? those discussed in the course? 15% 13% 6% 6% 37% 38% 85% No difference A small difference A moderate difference Yes No A major difference A complete difference N/A I shared course materials with some colleagues I discussed with colleagues about concepts and techniques dealt with in the course 3% 5% 15% 11% 35% 9% 38% 35% 49% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know 3.29 Although respondents discussed with colleagues the concepts and techniques reviewed in the course and shared course materials, they nevertheless stressed that they have not had the opportunity to pass on information concerning any economic policies related to those discussed in the course. At first, these results may appear contradictory. However, it also suggests that participants probably briefly and informally exchanged anecdotes on their training experience and not in a systematic and thorough manner, which would have facilitated the passing of information on economic policies related to those of the course. 3.30 Comments gathered from supervisors of participants reinforced this impression. When supervisors did mention that their employee passed on information on the course or shared course materials with colleagues, their tone and responses suggested that it was generally done in an ad hoc manner. Formalized feedback procedures, for example at BCEAO where employees are requested to provide feedback on their training through a formal presentation to colleagues 20 did not seem to be a common practice among other organizations, although it did happen in a few instances. As one respondent noted, systematic and effective information sharing is still not part of the organizational culture in the African public sector in Africa. 3.31 The apparent contradiction may be viewed in this light. It can also be simply explained by the fact that although a majority of respondents did talk about the course content and share materials, they perhaps did not get the opportunity to pass on information on economic policies related to those discussed in the course due to the lack of formal and effective feedback processes. Finding 11: After attending the courses, the majority of respondents indicated that that they feel more capable of providing advice with respect to economic policies. Since you attended the course, to what extent do you feel ready to Since you attended the course, have you given advice about the provide advice about economic policies? development, rejection or adoption of economic policies? 2% 3% 12% 20% 37% 50% 25% I feel as ready as before the course 51% I feel a bit more prepared than before the course I feel much more prepared then before the course I cannot say N/A Yes No N/A 3.32 The MM training program seeks, among its key objectives, to expand the capacity of targeted participants to analyze current issues in macroeconomic management and to build relevant practical and analytical skills in the area of policy assessment. The above results suggest that course participants feel more ready (and therefore more confident) to provide advice on economic policies. Seventy-five percent (75%) of respondents indicated an increased ability to provide advice on economic policies following the conclusion of the course(s) than they would have been able to prior to participation. These results are in line with comments made by supervisors, several of whom stressed the increased capacity or confidence of their colleague to be involved in policy development work. 3.33 On the other hand, although 75% of respondents feel an increased capacity to provide advice on economic policies, only 37% have actually done so. It would appear that there is some disconnect between the result and that of Finding 10, in which only 15% of respondents indicated that they had passed along information concerning economic policies related to the course. This discrepancy may have resulted due to confusion in terms of what was meant by "pass along information" (Finding 10), and also due to cultural norms on disseminating information to others. 3.34 It should finally be noted that a review of results broken-down by courses revealed some differences between the perceptions of respondents that attended 2000 training courses and those 21 that attended the 2002 training courses. For example, a greater percentage of respondents that attended 2000 training courses had given advice about the development, rejection or adoption of economic policies following their course participation. Finding 12: At the individual level, respondents indicated that MM courses enhanced their professional skills, provided them with new ideas and increased ability to solve problems which they may face in their position. The course helped me improve my professional skills The course provided me with several new ideas on the way to deal with my work 6% 1% 7% 7% 3% 9% 33% 42% 44% 48% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know The course enabled me to find concrete solutions I faced in my position 4% 13% 18% 17% 48% Completely disagree Somewhat disagree Rather agree Completely agree Do not know 3.35 On the individual participant level, survey results indicate that the professional skills of participants (86%) were enhanced due to their involvement in MM courses. A similarly large majority (81%) indicated that the courses provided them with several new ideas and methodologies with which to approach their work. Finally, 66% of respondents stated that participation in the courses enabled them to determine and find concrete solutions to problems they faced in their position. These very positive results indicate that the MM courses provide a sound analytical framework that participants could use in their work (see p. 8 in Annex E). Such observations were generally congruent with comments from supervisors, although the picture was not as overly optimistic, some supervisors being more cautious or reserved in their assessment. 22 Results broken-down by courses revealed no significant differences between the perceptions of respondents that attended 2000 and 2002 training courses Finding 13: Overall, respondents indicated that participation in the courses had a positive impact on their work. The course had a positive impact on my work 3% 4% 7% 35% 51% Completely disagree Somewhat disagree Rather agree Completely agree Do not know 3.36 Almost all respondents (86%) felt that their participation in the course subsequently had a positive impact on their work. This result reinforces the overall impression conveyed by participants that MM courses have had a positive contribution on their learning which they could apply to their work. Supervisors, by and large, concurred with the participants' views about the training program's positive impact on the employee's work. Although some supervisors could not provide illustrations of concrete results, several of them did notice, among other things, improvements or an increased performance of their employee in performing work-related tasks. In Senegal, for example, a supervisor noted his employee's increased capacity to perform banking activity analyses. In Ivory Coast, another supervisor stressed the "new vision" of his colleague, who could make better links with regards to the different aspects of the budgetary policy. In Burkina Faso, a supervisor emphasized an increase in his employee's understanding with regard to the interplay between macroeconomic issues and the country's poverty reduction strategy. Again, in Burkina Faso, another supervisor observed an increased understanding of key economic concepts as well as key issues related to GDP on the part of his employee. He also noticed a tendency with his employee to taking proactive initiatives and working more autonomously, thus illustrating greater confidence. 3.37 The review of results broken-down by courses revealed no significant differences between the perceptions of respondents that attended 2000 and 2002 training courses Finding 14: Supervisors identified a set of key factors facilitating or limiting use of knowledge and skills acquired from MM course in the workplace. 3.38 When asked about factors facilitating or limiting the use of knowledge and skills acquired from training in the workplace, supervisors provided several interesting responses. Among these, a key set of ideas stood out as important factors in facilitating or limiting the transfer of knowledge from MM course. This included: 23 · The learning ability of the course participant to absorb new knowledge and skills and use them; · The motivation and willingness of the course participant to learn and implement new ideas; · The availability and user-friendly format of course materials for participants' colleagues (use of CD-Rom, vignettes, etc.); · Easy access to Internet resources to download course material or information, as well as to participate in organized online-chat debates; · The creation of formalized feedback processes that would be supervised to ensure proper transfer or knowledge (not in an ad-hoc and informal way as is currently the case in most organizations); · Openness and support from supervisors towards new concepts and ideas, and a willingness to support their employees in implementation of that knowledge (trial and error) and; · An organizational culture that facilitates the acceptance and implementation of new concepts and ideas. 3.39 In the latter case, it was interesting to note that organizational culture open to change and innovation appears to be present in newly created government bodies, such as cells and secretariats established to follow-up on the implementation of poverty reduction strategies. Yet, it also appears that resistance to change is still a reality limiting the inception of new knowledge and ideas in other government settings. EFFECTSOF MM COURSES ON TRAINING AND TEACHING ACTIVITIES 3.40 A number of course participants were university professors or others involved in similar training roles. The following sub-section focuses on the training and teaching activities which these individuals undertook following the conclusion of the course(s), and the effects that the course had upon these activities. 24 Finding 15: Almost one-third of course participants have been involved in the subsequent training of others following the conclusion of the seminars. Since you attended the course, have you trained people concerning the themes studied in the course? 29% 71% Yes No 3.41 Dissemination of knowledge on macroeconomic management policies and procedures to individuals who are likely to have an impact, direct or indirect, on the economic policies of the state is among the key objectives of the MM program. Of those attending the courses, a certain percentage (7%) classified themselves as being in the "teaching and training" profession (see finding 1). This does not mean, however, that individuals who classified themselves in other professions are necessarily excluded from providing training. In this sense, training may take place within the organization or within an academic institution. In either case, information provided from the courses is being exposed to a wider audience. 3.42 The above results reveal that since the conclusion of the courses, 29% of all participants have been involved in training others in MM principles and themes. Although a large majority (71%) was not involved with training activities, this is consistent with the fact that a majority of participants are not at all involved with training activities. Finding 16: A majority of participants indicated that they are better prepared to deliver training in macroeconomic management principles following the courses rather than prior to them. Since you attended this course, to what extent do you feel better prepared to teach these people? 5% 24% 28% 19% 24% I feel as ready as I did before the course I feel a bit more prepared than before the course I feel much better prepared than before the course I cannot say N/A 25 3.43 If the field of "N/A" (not applicable) were to be removed from this question, the results would indicate that a larger majority of respondents felt either a bit more or much more prepared to teach aspects of MM than they would have been prior to their participation in the courses. This result suggests that in addition to providing participants with appropriate frameworks and policies on macroeconomic management, the MM training program increased the ability of trainers to subsequently share their knowledge with others more effectively. Finding 17: One-half of respondents indicated that they had utilized parts of the courses into their training, teaching, and research activities. Since you attended the course, have you incorporated parts of the course into your training, teaching or research activities? 4% 2% 16% 14% 34% 30% Yes - I was a resource person in other sessions of the course Yes - I organized a similar course Yes - I incorporated the course contents into my training and teaching activities Yes - I used the course in my research activities No N/A 3.44 Forty-four (44%) of respondents indicated that they had utilized elements of the MM courses when conducting research and teaching/training activities. Interestingly, 4% of course participants stated that they were a resource person in other sessions of the course, while 2% organized a similar course upon completion of the course in question. Conversely, 34% of those surveyed indicated that they had not incorporated parts of the course into their own training, teaching or research activities. Finding 18: Of the individuals subsequently taught by participants who had completed the courses, the majority were university students. Whom did you teach since you attended the course? 10% 6% 22% 4% 52% 4% 2% Undergraduate Students Graduate students Economists Professionals Other No one so far N/A 26 3.45 Although the majority of respondents indicated that they had not, as yet, had the opportunity to provide training or instruction to those who had not been involved in the courses, for those individuals who did provide training, the recipients were mainly university students at the graduate (22%) and undergraduate (6%) levels. Economists and Professionals (each at 4%) constituted the remainder of the teaching audience. 3.46 As was noted by course stakeholders during interviews, exposing individuals such as university students to macroeconomic management principles at a relatively early stage in their careers is also an important component of the effects chain of the MM program. Information must not only be disseminated to already established policy-makers or researchers, but also to those individuals who are likely to hold similar positions in the future. Finding 19: The majority of respondents have never used the WB internet resources to assist them in the preparation of subsequent training activities. Have you used the Internet resources of the WB MPAS Program as an information source to prepare for your training activities? 20% 24% 56% Yes No N/A 3.47 For those individuals who are involved in disseminating the knowledge gained during their participation in the courses to others, just under one quarter (24%) indicated that they have accessed the MM website as an information resource to assist in the preparation and implementation of their training activities. Conversely, 56% indicated that they have never used the MM internet resource in the same capacity. 3.48 One of the main impediments to the use of the internet in many countries in Africa is the lack of stable connections, either resulting from poor service providers or the unreliability of electricity. Another issue, which sometimes comes to the forefront, is the prohibitive cost of internet use. All these factors impact upon the use of the MM website as an information resource. Academics interviewed by telephone remarked that they are much more comfortable with hard copies of the data available during the courses, and furthermore that the internet is not an effective tool to assist in their work, due to unreliability. These observations were corroborated by moderators and partner organizations who emphasized that internet connections were still limited in the public sector in Africa, though the situation is improving in some countries like Senegal due to cheaper connection fees. Poor and costly access to the Internet therefore weakens the use of internet-based follow-up activities with course participants. 27 28 4. EFFECTIVENESS OF MM WORKSHOP TEACHING METHODOLOGIES AND LEARNING ACTIVITIES INTRODUCTION 4.1 The courses under review were delivered through face-to-face delivery and through GDLN methodology. Data presented in this section provides indication on the effectiveness of teaching methodologies of MM training workshops. It discussed, among other things, which delivery mechanism lends itself to better learning as well as what could be done to enhance the effectiveness of teaching methodologies.9 The section also describes what features of WBI learning activities are related to greater effectiveness of course effects, as measured by participants and other course stakeholders. PRESENTATIONOF FINDINGS Finding 20: The overwhelming majority of participants who responded to the questionnaire found the design of MM program courses to be appropriate. Other course stakeholders provided mixed reactions however. Course design is appropriate 2% 5% 25% 68% Completely disagree Somewhat disagree Rather agree Completely agree Do not know 4.2 The design, or conception, of a training program is an important component in determining the likely effects of training activities. If the format is not appropriate, training is doomed to failure even if one has the best trainers. If the design is appropriate, on the other hand, training activities are likely to the originally set. 9 It should be noted that, although some general judgment could be made by the evaluation team, comparisons between the effectiveness of the two methodologies (GDLN vs face-to-face) were difficult to make given the small sample of courses reviewed and the fact that participants could only rate the course that they participated in, either face-to-face or GDLN. 29 4.3 Level 1 evaluations conducted by WBI as part of each course10 indicate that In the case of the MM program, participants were overwhelmingly pleased with the design of the course. Ninety-three percent (93%) of course participants indicated that the design of their particular course(s) was appropriate (to varying degrees) to their learning needs. These results would appear to indicate that the methodologies and designs of the seminars have been well researched prior to implementation, and moreover, that the subject matter is appropriate for participants. 4.4 MM moderators and representatives of WBI partner organizations did not provide such a clear-cut assessment. While some of them felt that the design of the MM course was adequate, several other provided criticisms, especially with regards to the design of the GDLN courses. It was stressed, among other things, that GDLN courses were too theoretical and did not provide enough attention to practical matters and topics related to participants concerns. Respondents also emphasized that interactions among participants were not stimulated enough, although it was acknowledged that such interactions are not easy to generate in a distance education setting. Greater use of local moderators through locally supervised debates and discussion was perceived as a way to address that problem. 4.5 It should be noted that changes were made to the design and delivery of GDLN courses over time if one compares the design of the first and last GDLN course under review. Presentations in the last GDLN, for example, were said to be shorter to ensure greater interaction among participants. On the other hand, MM moderators and representatives were generally pleased with the design of face-to-face courses held in Abidjan and Dakar, although some suggested that partner organizations could play a bigger role in the design of the course. Finding 21: Although participants indicated that both in-person and GDLN were effective methods for disseminating knowledge, the majority indicated a preference for in-person training as opposed to distance-learning (GDLN). In the same spirit, moderators viewed in- person training as a superior teaching methodology in comparison to distance learning. I believe classroom training is preferable to long-distance training I believe long-distance training is preferable to classroom training to transmit knowledge adequately to participants to transmit knowledge adequately to participants 10% 15% 20% 20% 7% 5% 31% 20% 38% 34% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know 4.6 In analyzing this specific data from participants, the evaluation team assumed that respondents could compare the two teaching methodologies based on their previous experience. 10Level 1 evaluations for the 5 courses under review were made available to the evaluation team as part of the document review. 30 Of the two methods of training delivery within the MM program, the majority of respondents indicated their preference for classroom sessions as opposed to GDLN video conferencing. Sixty five percent (65%) preferred classroom training as opposed to only 27% for the GDLN methodology, while an identical 20% on both sides completely disagreed with the use of classroom and GDLN as opposed to the other. This data should nevertheless be taken with some caution as it is not possible to assess the extent to which respondents to this question had in fact been exposed to both methodologies. 4.7 Courses moderators also indicated their preference for in-person training, which they generally viewed as a superior teaching methodology to ensure proper learning. When asked about the advantages and disadvantages of both methods, they usually stressed that the key advantage for organizing GDLN course was economical. If compared to face-to-face courses, they viewed video conferencing technology as a comparatively cost-effective alternative to reach audiences in several countries without asking participants and moderators to leave. Some moderators also liked the GDLN delivery medium as it incorporates a diversity of experiences into the discussion. 4.8 Yet, most respondents were clearly of the opinion that face-to-face teaching methodology lends itself to better learning than GDLN. They felt that the quality of knowledge transfer through GDLN was not as good. Video conferencing, it was suggested, does not allow much flexibility. Due to time constraints and distance, the message must be simplified. In addition, distance-learning does not facilitate practical exercises with direct interactions with teachers. Personal contacts among participant and with teachers in informal settings, was thus viewed as key to stimulate exchanges and learning. The delivery format of GDLN courses does not facilitate such personal contacts. Finding 22: A majority of respondents indicated that they had visited the website of the MM Program. However, only a minority indicated that they had used the website as a reference to keep up with developments in the MM field. I visited the website of the MPAS Program I used the website of the MPAS Program as a reference to keep up to date with recent developments in the field 4% 4% 10% 25% 23% 34% 12% 35% 17% 36% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know 4.9 This finding is congruent with the general attitude of participants towards the internet and the current limitations of employing web technology in Africa. Fifty-nine percent (59%) of 31 respondents visited the website of the MPAS program. Only 45% of respondents have visited the website to keep up-to-date with recent developments in the field. 4.10 It would be simplistic to interpret these figures simply as a result of technology or infrastructure issues which make it difficult to access the internet or the website in Africa, but this is certainly a contributing factor to the low numbers. Conversations with moderators and partners revealed that the unreliability of internet service as well as poor computer facilities, limited the ability to access MM program information, especially for participants from the public sector. As one respondent stressed, computer and internet access, far from being available to all, is often limited to high-level bureaucrats and their secretaries. 4.11 These results illustrate the challenges faced by course managers in ensuring that course participants can continue to have access to cutting-edge information on macroeconomic management after their participation in the training. Finding 23: The MM training program has so far failed to provide effective support mechanisms that would facilitate contact with participants and course organizers in order to ensure continuous learning. I contacted training course participants to discuss work-related The course provided me with good opportunities for contacting and issues we had dealt with during the course networking with other professionals in my field 11% 1% 1% 13% 26% 43% 23% 33% 27% 22% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know I contacted course organizers to discuss certain aspects of the course contents 4% 11% 14% 52% 19% Completely disagree Somewhat disagree Rather agree Completely agree Do not know 4.12 The MM program seeks to bring together like-minded professionals from different countries to discuss issues in macroeconomic management. At the conclusion of MM courses, 32 participants are encouraged to continue the dialogue amongst themselves to further training and learning in MM policies. The reality is that participants are essentially left to themselves. 4.13 As such, about two-thirds of participants (65%) indicated that they had little contact with other participants following the conclusion of the courses. In the same vein, the majority of participants (71%) indicated that they did not contact the course organizers to discuss the contents of the courses. The majority of participants (53%) also indicated that the courses did not result in networking opportunities with other professionals in the field. 4.14 These results reveal the weakness (or absence) of follow-up mechanisms that should be put in place to ensure continuous learning. Although it was acknowledged that follow-up activities can be costly to organize and difficult to manage, moderators and members of partners organizations nevertheless proposed a few alternatives to help workshop participants consolidate their learning. It was stressed, for example, that more attention should be paid to ensure that formal feedback mechanisms are put in place within the participants' organizations. Organizing follow-up meetings was also viewed as a way to facilitate exchange among course participants after the course (internet chat rooms do not seem to be an effective way to facilitate exchange as discussed above). 4.15 Many also emphasized the greater role that local moderators could play to coordinate the different initiatives or mechanisms to facilitate the transfer of knowledge from the course to the workplace. Supervision of follow-up activities, in short, was viewed as crucial. Inthe absence of formalized processes, many hinted, participants are left to fend for themselves and one cannot hope to assist them in cementing the knowledge gained during the course. 4.16 WBI provides support through its website which course participants may access to clarify lessons learned as well as to dialogue on MM issues. However, as we shall observe in Finding 27, this model of support is not always effective given the limitations of internet access and infrastructure problems within Africa. For their part, the Partner organizations are not providing resources. Given the lack of funds and infrastructure available to them, it is unlikely that additional formalized support processes could be put in place without WBI assistance. Finding 24: Moderators provided a few ideas as what could be done to enhance the effectiveness of teaching methodologies. 4.17 When asked about ideas to enhance the effectiveness of teaching methodologies, moderators interviewed provided a set of suggestions. Among important ones, it was recommended to: · Ensure that presentations and exercises are addressing participants' concerns (i.e. their national or regional macroeconomic realities); · Privilege case studies as an effective way to stimulate learning; · Focus on practical matters rather than theoretical ones; · Perform a better screening of participants to ensure evenness among participants' backgrounds and knowledge; 33 · Work with practical tools and instruments to solve problems 4.18 Moderators also provided more specific suggestions to enhance the effectiveness of GDLN courses. This included: · Making short and concise presentations (content should be simplified); · Increasing the number of local meetings where the local moderator can debrief the course content with local participants, thereby ensuring that concepts and notions are assimilated; · Allowing more time for discussions and debates at the local level; · Providing participants with Internet access and an email account. 34 5. EFFICIENCY OF MANAGEMENT OF MM LEARNING ACTIVITIES INTRODUCTION 5.1 Data presented in this section provides indication on the efficiency of the management of MM activities. It discussed, among other things, the nature of the partnerships between WBI and its partner organizations. The section also discusses factors that may impede the use (and thereby sustainability) of knowledge and capacity built, as well as the cost-effective nature of the training courses. PRESENTATIONOF FINDINGS Finding 25: The management of the courses was generally felt to be adequate by course stakeholders. Course management is adequate 3% 5% 28% 64% Completely disagree Somewhat disagree Rather agree Completely agree Do not know 5.2 MM program courses are jointly managed by the WBI and local partner organizations. For the organization of the three GDLN courses reviewed, WBI partnered with distance learning and research centers in the countries targeted by the evaluation. For the two face-to-face courses assessed, WBI received the support of the AfDB, as well as the BCEAO and the BEAC. 5.3 Partner organizations generally manage the entire logistical operation of GDLN or face- to-face courses. Responsibilities include providing a meeting space and the facilities, selecting candidates and course moderators and ensuring the lodging and transportations of participants (if face-to-face). WBI's responsibilities are to design the curriculum of the course and select course facilitators (teachers). 35 5.4 Survey results indicate that 92% of respondents found that the management of their course was adequate. Representatives of partner organizations also found the management of the course to be adequate. Their comments generally stressed the good relationship that they have established with WBI and their desire to pursue such collaboration. 5.5 On the whole, partner organizations were seen to play more of the role of an executing agency. In the selection of participants, for example, they simply selected those that reflected the profile decided by WBI, depending on the nature of the course (GDLN vs face-to-face). Although representatives of partner organizations appreciated the clear sharing of role and responsibilities, some hinted that it would have been interesting for partner organization to have a bigger involvement, for instance, in the definition of the course curriculum. Finding 26: Respondents generally indicated that the benefits of participating in the courses were more significant than the cost of participation. I believe the cost benefit ratio of the program courses is adequate 2% 15% 22% 61% Completely disagree Somewhat disagree Rather agree Completely agree Do not know 5.6 One of the issues raised by the WBI for this evaluation was to determine if the costs of participating in the courses outweighed the potential benefits to course participants, and by extension, their organizations and institutions. Supervisors, partners, and moderators have commented that the cost of registration and attendance for selected participants is quite high. 5.7 Having said this, a large majority of respondents (83%) indicated that the benefits of taking the course were more significant than the costs. In fact, only 2% of participants felt that the cost of the course did not justify the benefits. Comments from partners and moderators were also consistent with this view, although many said that high fees could certainly act as a deterrent and limit participation. This is especially true for research institutions and NGOs, which generally cannot afford to participate in these training sessions unless invited by the WBI or another organization. 5.8 Comments gathered from supervisors were also consistent with the above observations. Considering the resources invested, they generally felt that the course was beneficial for their organization and would recommend their staff to participate in other similar courses. It should be noted, however, that many supervisors did not remember (or were not really aware) of the 36 registration costs, which was usually paid by another department. It should also be specified that several participants were invited by WBI, which paid for all expenses. Finding 27: Lack of proper computer equipment and internet access are viewed as obstacles to the proper use of course tools and methods. This may in turn impede on the use of the knowledge and capacity built, thereby raising the issue of training sustainability. Identify the obstacles you encountered while trying to use the methods and tools of the course 17% 2% 26% 5% 2% 5% 17% 26% Lack of necessary computer equipment Lack of/little access to Internet Lack of support from the course supervisors/colleagues Lack of support from subordinate staff Lack of time Other obstacles I have not met any obstacles N/A 5.9 The above graph illustrates a common theme with regards to the use of the internet derived from other findings. Fifty two percent (52%) of respondents indicated that the single largest obstacle to the proper use of course tools and methods was a lack of computer equipment (hardware and software) and a lack of, or little, internet access. 5.10 The lack of proper equipment and internet access obviously undermines the capacity of former participants to use course material and follow latest developments in the MM field. Hence, if there are no meeting opportunities where participants can exchange information with colleagues to stay up-to-date, knowledge and skills acquired (capacity built) at the training course slowly erodes thus undermining the sustainability of the entire process. 37 38 6. CONCLUSIONS AND LESSONS LEARNED 6.1 The results of this evaluation, in the context of its methodological limitations, nevertheless suggest that the MM courses are appreciated by participants, are sound in their content and respond to participant's needs. Moreover, they indicate that participants, given their own scope of authority and responsibilities in their work environment, are able to transfer or use the knowledge acquired. 6.2 This is positive and overall conclusions of the evaluation suggest that the MM course has been as effective as it could be given the program model that was followed. At this stage, the evaluation team would like to take a forward-looking approach and explore ways in which WBI could build upon the results of this evaluation to increase its effectiveness and effects. At issue from the perspective of the evaluation team are the following points: Targeting of participants 6.3 There is a balance to strike between selecting participants and asking the borrowing country to select the participants. Selection of participants is key to ensuring that transfer of knowledge will occur. On the one hand, it is important that the local institutions propose candidates that they feel will benefit the most from the course. On the other hand, in order to maximize the effectiveness of the course WBI and the local institutions could review together the following questions prior to finalizing the selection: · Are the participants likely to benefit from the course given their present roles and responsibilities? · Do participants have adequate credibility and authority in the organization to initiate change as they attempt to transfer their new learning in the workplace? · How likely is it that the participants will remain within the organization within the next 3-5 years after the workshop to lead and promote the desired changes? · Will the participants have access, upon return, to sufficient materials and resources to implement the changes? · Is the acceptance of some participants dictated primarily by the concern of breaking even in the overhead costs? · Have the selected participants demonstrated in the past an ability to share knowledge with their peers? 39 · Are the supervisors of the targeted participants supportive of the workshop initiative? Leverage for change 6.4 Among some of the observations made by the respondents is the extent to which it is fair to expect changes in the workplace, as a result of the course, if only one or two individuals are selected from an organization. The transferability of newly acquired knowledge could be enhanced through the following activities: · Selecting more than one participant per organization with a view of providing internal support to the participants as they attempt to use or to transfer their new knowledge. · Increasing the awareness of the supervisors of the workshop's participants with respect to the workshop content. The evaluation team has been involved in several level 3 evaluations one of which in particular conducted with the Canadian National (CN) in Canada in which the single most important criteria for increasing transfer of new learning on the job turned out to be the support from the supervisor11. · Selecting participants from organizations that have demonstrated openness to and a support for change and to the application of new learning in the workplace. Developing a stronger methodology for conducting level 3 evaluations 6.5 As the WBI engages in more systematic level 3 evaluations of its courses, more holistic methods for conducting evaluations at this level should be implemented. We have noted some of the limitations of this evaluation in the methodology section. It is worth expanding and suggesting that as the Evaluation Unit of the WBI plans its course evaluation it should ensure that: · When level 3 evaluations are planned, the set of behavioral changes and potential effects in the workplace are identified. This will allow collecting data through observations or interviews on these specific expected changes. · At the planning stage, potential respondents for the level- 3 evaluation are properly identified and a database of names is established, including, for each participant name of supervisors, peers, and subordinates. · Criteria for effects, effectiveness and impacts are reviewed and discussed within WBI to ensure that the evaluations assess the proper dimension of effects and impacts. 11Evaluation of CN Supervisory Training for first level-supervisors, conducted by Universalia 1993-195. Over 20,000 first-level supervisors of CN undertook this three-day course. The review indicated that involving supervisors prior to the workshop and increasing their awareness about the course was highly effective in ensuring transfer of knowledge on the job. CN is a key player in the North American rail industry. 40 · A baseline is established for each respondent prior to the course to determine where their current level of exhibiting the desired behavioral changes is at. This baseline can be produced through self-reporting or through interviews or surveys of the supervisors /subordinates. The same group of respondents interviewed prior to the course should be interviewed after the course. 6.6 Level 3 evaluations are costly and a great deal of planning and investment. Given the existing resources of WBI an adequate balance should be reached between investing for course development, delivery and evaluation. Enlarging the scope of coverage versus deepening the scope of ongoing support: maximizing the return of capacity development resources 6.7 The WBI model of delivery is characterized by two elements: it targets a wide range of participants and increases its targeted audience. On the other hand, it does not provide strong additional support to former participants upon completion of the workshop. Such a model has its strengths and weaknesses but review of effects occurs more often when some resources are taken to deepen the level of support pre and post workshop. Follow-up mechanisms to ensure efficient knowledge transfer 6.8 Closely linked to the issue of breadth and depth is the need to establish more systematic follow-up mechanisms to ensure greater effects. Respondents have noted the limited interaction with the course moderators or amongst participants after the workshop and the need to create a more enabling environment for transferring knowledge and for increased effects. Presently some mechanisms are in place, a Web site, a list of participants and, in other courses given by the WBI, a Newsletter and an electronic chat group where issues and questions can be shared. The experience of the evaluation team in creating favorable conditions for a transfer of learning suggests that as additional aid is provided to participants their chances of applying the knowledge in the workplace increases.12There is no shortage of ideas for follow-up but, rather, limited resources that must be allocated in the most strategic fashion. 6.9 On a different, yet related matter, WBI could contemplate the use of a tracer study to ensure close follow-up of courses graduates. A tracer-study is a good way of establishing the course graduates' location, employment status and to understand the extent to which the graduate may have benefited from the course. It is nevertheless costly and time consuming. Other issues to consider for WBI 6.10 A final set of questions related to the competitiveness of WBI in delivering training courses in Africa should also be contemplated. In particular, WBI should consider: 12Universalia has acquired significant knowledge on the matter through its consulting work for the IDB 41 · Who are WBI's competitors in the organisation and delivery of similar training courses in Anglophone and Francophone Africa? · Whether MM training participants, if given the choice and money, would have attended WBI training course or other training courses organized by competitors? · Are there any geographical differences that are worth considering for WBI when it organizes training seminars (for example, between West and East Africa, etc.)? 42 ANNEXES 43 44 ANNEX A: LIST OF FINDINGS Finding 1: Overall, the level 1 evaluations conducted for the five MM courses under review reveal a high level of satisfaction with the training courses. Finding 2: WBI MM Courses reviewed were effective in reaching targeted audiences. Finding 3: The vast majority of respondents have maintained their positions of employment with the same organization. The majority of those that did change positions, report that they did so as a result of knowledge gained during the course. Finding 4: The majority of respondents indicated that they used and retained knowledge gained through MM courses in their work, a view generally supported by supervisors of participants interviewed. Finding 5: A majority of respondents indicated that they had used the course materials on some or several occasions following the conclusion of the courses. Finding 6: Knowledge gained from training courses is mainly being utilized by those with influence at the organizational and national levels. Finding 7: Since attending the courses, a small portion of training participants credited the course in having a major influence in their ability to implement economic policies discussed in the course. Finding 8: Since their course attendance, the majority of respondents believed that they have not had an influence on the approval or rejection of economic policies related to the courses. For the minority who stated that they had influence, however, most credited taking the courses with their ability to do so. Finding 9: Since attending the courses, the majority of course participants have not carried out economic policy research related to research discussed in the courses. Finding 10:Respondents indicated that they have not had the opportunity to pass along course information on economic policies. They nevertheless pointed out that they shared course materials and discussed concepts and techniques dealt with in the course with colleagues. Finding 11:After attending the courses, the majority of respondents indicated that that they feel more capable of providing advice with respect to economic policies. Finding 12:At the individual level, respondents indicated that MM courses enhanced their professional skills, provided them with new ideas and increased ability to solve problems which they may face in their position. 45 Finding 13:Overall, respondents indicated that participation in the courses had a positive impact on their work. Finding 14:Supervisors identified a set of key factors facilitating or limiting use of knowledge and skills acquired from MM course in the workplace. Finding 15:Almost one-third of course participants have been involved in the subsequent training of others following the conclusion of the seminars. Finding 16:A majority of participants indicated that they are better prepared to deliver training in macroeconomic management principles following the courses rather than prior to them. Finding 17:One-half of respondents indicated that they had utilized parts of the courses into their training, teaching, and research activities. Finding 18:Of the individuals subsequently taught by participants who had completed the courses, the majority were university students. Finding 19:The majority of respondents have never used the WB internet resources to assist them in the preparation of subsequent training activities. Finding 20:The overwhelming majority of participants who responded to the questionnaire found the design of MM program courses to be appropriate. Other course stakeholders provided mixed reactions however. Finding 21:Although participants indicated that both in-person and GDLN were effective methods for disseminating knowledge, the majority indicated a preference for in- person training as opposed to distance-learning (GDLN). In the same spirit, moderators viewed in-person training as a superior teaching methodology in comparison to distance learning. Finding 22:A majority of respondents indicated that they had visited the website of the MM Program. However, only a minority indicated that they had used the website as a reference to keep up with developments in the MM field. Finding 23:The MM training program has so far failed to provide effective support mechanisms that would facilitate contact with participants and course organizers in order to ensure continuous learning. Finding 24:Moderators provided a few ideas as what could be done to enhance the effectiveness of teaching methodologies. Finding 25:The management of the courses was generally felt to be adequate by course stakeholders. 46 Finding 26:Respondents generally indicated that the benefits of participating in the courses were more significant than the cost of participation. Finding 27:Lack of proper computer equipment and internet access are viewed as obstacles to the proper use of course tools and methods. This may in turn impede on the use of the knowledge and capacity built, thereby raising the issue of training sustainability. 47 48 ANNEX B: INTERVIEW PROTOCOLS (1) Interview Protocol for Moderators Introduction · Purpose: This interview is to determine the extent to which the course has effected the behavior and performance of the participants in their workplace. Please describe the specific course you moderated? Which course? Dates? · Duration · Confidentiality Course 1) How many participants were in the course you moderated? What do you feel is the optimum number of participants? 2) From your perception, did participants feel that the course was beneficial? In what specific way? 3) From your experience as a moderator, are there other mechanisms which would help to facilitate the transfer of knowledge from course to workplace? What are they? 4) On an individual level, are there other methods which would assist participants in cementing knowledge gained during the course? 5) On a group level, are there other methods which would assist participants in cementing knowledge gained during the course? (e.g. regional networks, associations, etc.)? 6) Are there any follow-up mechanisms in place to assist in post-course learning? What are they, and how effective do you feel they are? Delivery Methods 7) What are the main advantages of face to face vs. GDLN? Disadvantages? 8) Do you believe that learning and knowledge transfer is better achieved by face to face instruction or by GDLN? Does it make a difference? What are the strengths and weaknesses of each? 9) What are the costing comparisons between the face to face and GDLN methods? Other Comments · Do you have any other general comments or recommendations about the MM course? 49 (2) Interview Protocol for Partners Introduction · Purpose · Duration · Confidentiality Relationship with World Bank Institute 1) What is the role of your organization in the development and operation of the course (specify the course) in collaboration of the WBI? What is the relationship? (e.g. did your organization provide funding? Space? Equipment? Other financial resources? Etc.) 2) How would you characterize the relationship of your organization with WBI in the development of the course (e.g. What were the strengths, what were the limitations how could the relationship be improved)? 3) What is your view of the current approaches/strategy (e.g. the courses) towards learning? Do you believe they are effective in ensuring effective transfer of learning in the organization, once the participant comes back on the job? Why? 4) What suggestions do you have for WBI to increase the effects of the course in the workplace? (i.e. more ongoing coaching, more follow-up, etc.) 5) Are there any additional steps your organization could take to further increase the effects of the course on the job? Follow-Up to Course 6) Are there any follow-up methods you feel would facilitate continued transfer of learning? Other Comments · Do you have any other general comments or recommendations about the MM course? 50 (3) Interview Protocol for Managers/ Supervisors Introduction · Purpose: This interview is to determine the extent to which the course has effected the behavior and performance of the participant. We would like to discuss more specifically the effects of the course XXX (give title) delivered on XXX (give date), on Mr. Or Ms. XXX's performance on the job (give name). · Duration · Confidentiality Your role as a supervisor 1) How long have you been the supervisor of (give name)? 2) What is the exact working relationship between you and (name)? Impact of Course 3) Has the course (be specific) affected the performance of the participant in his/her daily activities? Ø How?Examples? Forexample... Ø Hastheparticipantformulatedanynewpoliciesfortheorganizationordepartment as a result of the course? Ø Has the course helped your staff to better deal with MM issues your country? How? Ø Additionally,hasthecoursehadanyeffectonthefollowingareas(tobedetermined from questionnaire)? 4) From your view, what factors have assisted the transfer of knowledge gained by the course to application in the workplace? What factors have limited the transfer? 5) On an organizational level, what can be done to assist the transition to new policies or procedures the course has imparted? 6) Would you recommend further MM training courses for your staff? Follow-Up to Course 7) Has the individual within your department participated in any follow-up activities related to the course? What kinds of activities? 8) Are there any follow-up methods you feel would help facilitate continued transfer of learning? Examples? 51 9) Considering the resources invested, do you feel that overall the course was beneficial to your organization? 10) Considering your financial resources, is it more cost effective for the course to be face to face or via GDLN? 11) In the future, would you recommend your staff participate in other MM courses? Other Comments · Do you have any other general comments or recommendations about the MM course? 52 ANNEX C: SAMPLE QUESTIONNAIRE 53 54 55 56 57 58 59 60 61 62 ANNEX D: PEOPLE INTERVIEWED Face to face Abidjan : 13-23 mars 2000 Moderators M. Christian Dossou-Yovo, Enseignant, Université Nancy, Consultant, Banque Mondiale, France M. Karim El Aynaoui Économiste, Banque mondiale Partners Not reachable. Supervisors M. Malla Ari, Coordinateur du Secrétariat permanent DSRP, Niger M. Allam-Ndoul, Directeur national de la BEAC pour la République Centrafricaine Face to face Dakar : 15-25 juillet 2002 Moderators M. Nazaire Ndefo, Cameroun Partners M. F. Aboutou, BCEAO, Sénégal M. Antonin Dossou, BCEAO, Directeur de la Recherche, Sénégal Supervisors M. Hamado Sawadogo, Chargé de programmes, Ministère de l'Économie et des Finances, Burkina Faso M. Thierno Seydou Niane, Coordinateur de la cellule du suivi du PRSP (Programme de lutte contre la pauvreté), Sénégal M. Célestin Zalle, Directeur général, BCEAO, Burkina Faso GDLN : 5 octobre-5 décembre 2000 Moderators M. Jean-François Brun, Maître de conférences, CERDI, France M. Gérard Chambas, Chargé de Recherches CNRS-CERDI, France M. Jean-Louis Combes, professeur, CERDI, France M. Patrick Doger, CERDI, France Partners Olivier Jammes, Banque mondiale (anciennement du CERDI) 63 Supervisors M. Charlemagne d'Almeida, Chef du service de crédit (BCEAO), Bénin M. Constant Djogo, Directeur de l'agence principale, BCEAO, Bénin M. Adjovi Epiphane, Responsable de l'Axe de modélisation et coordonnateur adjoint du MIMAP (Cellule d'analyse des politiques économiques sous tutelle du Ministère du Plan), Bénin M. Kouassi Kouamé, Directeur général du budget et des finances, Ministère de l'Économie et des Finances), Côte d'Ivoire M. Aka Loko Michel, Sous-directeur, BCEAO, Côte d'Ivoire GDLN : 3 avril-29 mai 2000 Moderators Dr. Toussaint Houeninvo, Cellule d'analyse de Politique économique, Cotonou, Bénin M. Mama Ouattara, Modérateur, Unité de Formation et de Recherches en Sciences économiques, Côte d'Ivoire M. Idrissa Mohamed Ouédraogo, Enseignant, Université de Ouagadougou, Burkina Faso Partners M. Daniel Dubois, Directeur Centre de formation continue (CEFOC), Burkina Faso M. Jacques T. Edjrokinto, Directeur/ Manager, Centre d'Éducation à Distance du Bénin, Bénin M. Fidel Kabouré, Responsable commercial, CEFOC, Burkina Faso M. Yao Kouassi, Directeur Centre d'Éducation à Distance, Côte d'Ivoire M. Paul Ouattara, Responsable de la Formation, CEFOC, Burkina Faso Supervisors M. Siricondy Diallo, Chef de l'Inspection générale d'État, Sénégal M. N'guessan Konan Raphael, Directeur Général du Budget, Côte d'Ivoire M. Joachim Ouedraogo, Directeur de la Surveillance multilatérale, Union économique et monétaire ouest africaine, Burkina Faso M. Malick Tall Yade, Directeur de Cabinet, Ministère de la Jeunesse, Sénégal GDLN anglophone : 4 mai- 29 juin 2001 Moderators Dr. Nichodemus Rudaheranwa, Senior Research Fellow, Economic Policy Research Centre (EPRC), Uganda Partners Mr. John C. Oloa, Uganda Management Institute (UMI), Uganda 64 Supervisors Mr. Richard M. Ssendege, Head of the Research Department (Uganda Parliament, Uganda Dr. Ddumba Ssenpamu, Director of the Institute of Economics, Makerere University, Uganda Mr. Hudu Siita, Director, Project Implementation Monitoring Unit., Ministry of Finance, Uganda Totals People Interviewed Moderators 11 Partners 9 Supervisors 17 65 ANNEX E : SURVEY RESULTS Which course did you attend? Which of the following activites best describes your profesion? 14% 7% 27% 22% 10% 35% 12% 37% La Gestion Macroeconomique en Afrique subsaharienne 36% Macroecomomic Management and Poverty Reduction Strategies Gestion Macroeconomique et Strategies de Reduction de la Pauvreté en Afrique Macroecomomic Management for Financial Stability and Poverty Reduction Teaching & Training Analysis and research La Gestion Macroeconomique: Debats actuels et des methodes quantitatives Public policy development Other Since you attended the course, have you worked for the same If you answered "no" to the last question, do you agree that the organization? course encouraged and/or helped you find another position? 3% 9% 14% 29% 7% 88% 50% Yes No, I began working for another organization shortly after the course ended Completely disagree Somewhat disagree Rather agree No, I recently started working for another organization Completely agree Do not know To which sector does your organization belong? In your professional activities, at which level do you exert influence? 5% 4% 5% 36% 32% 52% 58% 8% Public Sector (national) Public Sector (international) At the level of my organization At the local level Private Sector Non-governmental sector At the national level At the global level 66 I used concepts and techniques learned in the course of my work I used course materials on some occasions 4% 8% 1% 16% 22% 11% 26% 12% 51% 49% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know I used course materials on several occasions I shared course materials with some colleagues 3% 17% 20% 15% 9% 38% 21% 42% 35% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know I discussed with colleagues about concepts and techniques dealt I visited the website of the MPAS Program with in the course 5% 4% 11% 25% 23% 35% 12% 49% 36% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know 67 I used the website of the MPAS Program as a reference to keep up I contacted training course participants to discuss work-related to date with recent developments in the field issues we had dealt with during the course 4% 10% 11% 1% 34% 43% 23% 35% 17% 22% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know I contacted course organizers to discuss certain aspects of the I did not retain the concepts and principles discussed in the course, course contents because they had little relevance with my work 4% 4% 3% 11% 6% 14% 52% 29% 58% 19% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know The course helped me improve my professional skills The course enabled me to find concrete solutions I faced in my position 6% 1% 7% 4% 13% 18% 17% 42% 44% 48% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know 68 The course provides a sound analytical framework The framework is useful to me in my work 1% 3% 6% 4% 42% 39% 57% 48% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know The course provided me with several new ideas on the way to deal The course provided me with good opportunities for contacting and with my work networking with other professionals in my field 7% 3% 1% 9% 13% 26% 33% 33% 48% 27% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know The course had a positive impact on my work Since you participated in the course, have you carried out economic policy development research related to the research 3% 4% discussed in the course? 7% 33% 35% 67% 51% Completely disagree Somewhat disagree Rather agree Yes No Completely agree Do not know 69 If you answered yes to the last question, what difference did the course Since you attended the course, have you influenced the contents of make in preparing you for developing any economic policies? any economic policies similar to the ones discussed in the course? 10% 3% 3% 39% 46% 38% 61% No difference A small difference A moderate difference A major difference A complete difference N/A Yes No If you answered yes to the last question, what difference did the course Since you attended the course, have you had influence on the make in your ability to influence the contents of any economic policies? approval or the rejection of any economic policies related to those discussed in the course? 6% 3% 6% 9% 15% 27% 49% 85% No difference A small difference A moderate difference A major difference A complete difference N/A Yes No If you answered yes to the last question, what difference did the course Since you attended the course, have you had the opportunity to make in your ability to have an influence on the approval or the rejection of make the final decision regarding any economic policies related to any economic policies? the themes discussed in the course? 13% 6% 6% 44% 37% 56% 38% No difference A small difference A moderate difference A major difference A complete difference N/A Yes No 70 If you answered yes to the last question, what difference did the course Since you attended the course, have you had the opportunity to make in your ability to make the final decision concerning any economic pass on information concerning any economic policies related to policies? those discussed in the course? 11% 11% 15% 36% 42% 85% No difference A small difference A moderate difference A major difference A complete difference N/A Yes No If you answered yes to the last question, what difference did the course Since you attended the course, have you implemented any make in your ability to pass on information about any economic policies? economic policies related to those discussed in the course? 13% 6% 6% 44% 37% 56% 38% No difference A small difference A moderate difference A major difference A complete difference N/A Yes No If you answered yes to the last question, what difference did the course Since you attended the course, have you trained people make in your ability to implement any economic policies? concerning the themes studied in the course? 11% 11% 29% 36% 42% 71% No difference A small difference A moderate difference A major difference A complete difference N/A Yes No 71 Whom did you teach since you attended the course? Since you attended this course, to what extent do you feel better prepared to teach these people? 10% 6% 5% 24% 22% 28% 19% 4% 24% 52% 4% 2% I feel as ready as I did before the course Undergraduate Students Graduate students I feel a bit more prepared than before the course Economists Professionals I feel much better prepared than before the course Other No one so far I cannot say N/A N/A Since you attended the course, have you incorporated parts of the Have you used the Internet resources of the WB MPAS Program course into your training, teaching or research activities? as an information source to prepare for your training activities? 4% 2% 20% 24% 16% 14% 34% 30% Yes - I was a resource person in other sessions of the course Yes - I organized a similar course Yes - I incorporated the course contents into my training and teaching activities 56% Yes - I used the course in my research activities No Yes No N/A N/A Since you attended the course, have you given advice about the Since you attended the course, to what extent do you feel ready to development, rejection or adoption of economic policies? provide advice about economic policies? 12% 2% 3% 20% 37% 50% 25% 51% I feel as ready as before the course I feel a bit more prepared than before the course I feel much more prepared then before the course I cannot say Yes No N/A N/A 72 Since you attended the course, which of the following situations Identify the obstacles you encountered while trying to use the best describes your level of authority in final decision making methods and tools of the course relative to your country's economic policies? 0% 17% 19% 28% 2% 26% 5% 7% 2% 10% 5% 19% 17% 26% 17% I made the final decision Lack of necessary computer equipment I gave direct recommendations to policy makers Lack of/little access to Internet There was only one level between the policy maker and myself Lack of support from the course supervisors/colleagues There were two levels between the policy maker and myself Lack of support from subordinate staff Lack of time There were two or more levels between the policy maker and myself Other obstacles I cannot say I have not met any obstacles N/A N/A I believe classroom training (in person) is an adequate teaching I believe long-distance teaching using the video-conference method providing for proper learning network is an appropriate teaching method providing proper 3% 2% 6%learning 2% 10% 10% 19% 43% 42% 63% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know I believe classroom training is preferable to long-distance training I believe long-distance training is preferable to classroom training to transmit knowledge adequately to participants to transmit knowledge adequately to participants 10% 20% 15% 20% 7% 5% 31% 20% 38% 34% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know 73 Course design is appropriate Course management is adequate 2% 5% 3% 5% 25% 28% 68% 64% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know I believe the cost benefit ratio of the program courses is adequate The WB works jointly with partner organization in managing its training courses 2% 2% 15% 36% 34% 22% 61% 28% Completely disagree Somewhat disagree Rather agree Completely disagree Somewhat disagree Rather agree Completely agree Do not know Completely agree Do not know 74