43165
Evaluation of the New
Managers' Leadership
Program (NMLP) -- Phase II
Heidi S. Zia
Joshua Seth Wimpey
Evaluation of the New
Managers' Leadership
Program (NMLP) -- Phase II
Heidi S. Zia
Joshua Seth Wimpey
WBI Evaluation Studies
No. EG08-134
The World Bank Institute
The World Bank
Washington, D.C.
November 2007
Acknowledgments
The World Bank Institute Evaluation Group (WBIEG) prepared this report under the
direction of Richard Tobin. Heidi Zia is the principal author.
The authors thank Antonieta Romero-Follette who carefully analyzed the
qualitative data from NMLP participants and their supervisors. The authors also thank
Vanessa Andris and Cathy Royal who diligently and successfully conducted interviews
with supervisors of NMLP participants and Seth Beckerman for editing this evaluation
report. Thanks also to Marlaine Lockheed and Nidhi Khattri for reading and commenting
on earlier drafts.
WBIEG evaluates learning by staff of the World Bank and activities of the World
Bank Institute (WBI). The findings, interpretations, and conclusions expressed in WBI
Evaluation Studies are entirely those of the authors and do not necessarily represent the
views of the World Bank Group, including WBI.
WBI Evaluation Studies are available at http://www.worldbank.org/wbi/evaluation
Suggested citation: Zia, Heidi S. and Joshua Seth Wimpey. 2007. Evaluation of New
Managers' Leadership Program (NMLP) � Phase II. Report No. EG08-134. Washington,
DC: World Bank Institute.
Acting Vice President, World Bank Institute Rakesh Nangia
Acting Manager, Institute Evaluation Group Nidhi Khattri
Task Team Leader Heidi S. Zia
ii
Contents
ACRONYMS AND ABBREVIATIONS..................................................................................... v
EXECUTIVE SUMMARY .....................................................................................................vii
Approach and methodology .........................................................................................vii
Key findings ................................................................................................................viii
Modules 2 and 3 are rated more positively than Module 1 by NMLP
participants............................................................................................................viii
Results of pre- and post self-evaluations by participants are favorable................viii
Supervisors' assessments indicate improvements in some managerial skills......... ix
Staff Survey results shows no difference between trained and untrained
managerial cohorts................................................................................................... x
Recommendations .......................................................................................................... x
1. INTRODUCTION .............................................................................................................. 1
Course description.......................................................................................................... 1
Objectives, approach, and methodology of evaluation .................................................. 2
Evaluation objectives............................................................................................... 2
Approach and methodology..................................................................................... 3
Evaluation instruments............................................................................................. 4
Evaluation questions................................................................................................ 5
Target population and response rates....................................................................... 5
2. EARLY EVALUATION RESULTS: MANAGERS' VIEW ...................................................... 7
Module 1: Crafting your management role -- Level 1 findings.................................... 7
Module 2: Achieving business results -- Level 1 findings.......................................... 10
Module 3: Leading change -- Level 1 findings........................................................... 12
Comparison across Modules 1, 2, and 3....................................................................... 14
Summary quantitative findings.............................................................................. 14
Summary of qualitative responses ......................................................................... 16
iii
3. IMPACT EVALUATION RESULTS: SELF-ASSESSMENTS BY PARTICIPANTS BEFORE
AND AFTER NMLP ......................................................................................................... 17
Did the participants use their new knowledge and skills?............................................ 17
Did the participants note a change in their own performance? .................................... 18
Participant self-assessments of performance, before and after NMLP.................. 18
Extent to which NMLP made a difference ............................................................ 21
What are the facilitators and barriers for managers to apply new skills?..................... 22
How can the program be improved? ............................................................................ 24
Comments specific to the modules ........................................................................ 25
Recommendations to improve NMLP ................................................................... 25
4. IMPACT EVALUATION RESULTS: THIRD-PARTY ASSESSMENTS FROM SUPERVISORS
AND STAFF SURVEYS .................................................................................................... 27
Supervisors' assessments before and after NMLP....................................................... 27
NMLP staff survey findings......................................................................................... 30
5. CONCLUSIONS.............................................................................................................. 32
APPENDICES...................................................................................................................... 33
Appendix 1: Summary of phase I evaluation methodology, findings, and results... 35
Appendix 2A: End of course level 1 questionnaire � module 1.................................. 37
Appendix 2B: End of course level 1 questionnaire � module 2.................................. 40
Appendix 2C: End of course level 1 questionnaire � module 3.................................. 42
Appendix 3A: New managers' leadership program impact evaluation
questionnaire � pre-NMLP questionnaire ............................................ 44
Appendix 3B: New managers' leadership program impact evaluation post-NMLP
questionnaire ........................................................................................ 45
Appendix 4: Response rates for NMLP pre- and post self-assessments .................. 49
Appendix 5A: Pre-NMLP interviews with supervisors of cohorts 9 and 10............... 50
Appendix 5B: Post-NMLP interviews with supervisors of cohorts 9 and 10
alumni................................................................................................... 51
iv
ACRONYMS AND ABBREVIATIONS
HRSLO Human Resources Services Leadership and Organizational
Effectiveness Group
IBRD International Bank for Reconstruction and Development
IEG Independent Evaluation Group
IFC International Finance Corporation
KLB Knowledge and Learning Board
MIGA Multilateral Investment Guarantee Agency
NMLP New Managers' Leadership Program
WBG World Bank Group
WBIEG World Bank Institute Evaluation Group
v
vi
EXECUTIVE SUMMARY
In July 2002, the World Bank's Human Resources Services Leadership and
Organizational Effectiveness Group (HRSLO) launched a three-module program for
newly appointed and recruited managers. This "New Managers' Leadership Program"
(NMLP) responded to Bank-wide feedback from senior Bank managers, human resources
managers, and other Bank professionals about the need for such a program. The NMLP
aims to:
� Clarify the institution's expectations of managers and strengthen managers'
capabilities to deliver results through effective core relationship skills;
� Increase managers' capabilities to create the results they want in ways that
support work-life balance; and,
� Build managers' knowledge and skills in leading change at their unit level and
provide opportunities to apply their new knowledge to work situations as they
continue to build the cohort as a learning and support community.
The NMLP's three modules are presented over a seven- to eight-month period,
with several weeks separating each module. Each new manager completes all three
modules, generally with the same managerial cohort. As of the date of this evaluation, 10
managerial cohorts have completed the NMLP.
APPROACH AND METHODOLOGY
This is the second evaluation of the New Managers' Leadership Program
(NMLP). The first evaluation, Evaluation of New Managers' Leadership Program (Zia,
Hanson, and Gunnarson 2004), covered Cohorts 3 and 4 and was presented to the
Learning Board in November 2004. After discussing the evaluation, the Knowledge and
Learning Board (KLB) requested that World Bank Institute Evaluation Group (WBIEG)
carry out a second evaluation of the NMLP for subsequent cohorts. Both evaluations
address the same evaluation questions. The first evaluation is referred to as Phase I and
the second as Phase II. The Phase II evaluation focuses on Cohorts 8, 9, and 10 and uses
four evaluation approaches, three of which permit estimates of the impact of the NMLP:
� A level one evaluation for each of the three modules, answered by participants.
� A knowledge, behavior, and skills self-assessment tool completed by
participants one week before and approximately twelve weeks after training.
vii
� Assessments from supervisors of participating managers before and after the
training.
� Comparisons between trained and untrained new managers with respect to
ratings given to them by their own staff on the "my managers" questions in the
2005 Bank Group Staff Survey.
KEY FINDINGS
Modules 2 and 3 are rated more positively than Module 1 by NMLP participants
This evaluation compares Modules 1, 2, and 3 across eight level 1 indicators:
(1) overall quality of training, (2) extent that the training fulfilled participants' learning
needs, (3) extent that the training achieved stated objectives, (4) extent that the training
was useful and applicable to participants on their jobs, (5) effectiveness in helping
participants learn from other managers, (6) quality and relevance of the training content,
(7) quality and relevance of materials used, and (8) quality and relevance of course
delivery method.
As in the case of the Phase I evaluation, Modules 2 and 3 were rated higher than
Module 1. For six of the eight selected indicators: "overall quality of training,"
"fulfilling learning needs," "achieving stated objectives," "quality and relevance of
content," "quality and relevance of materials used," and "quality and relevance of course
delivery method," the ratings for Module 1 were significantly lower than those for
Modules 2 and 3.
Module 3 was rated highest in "quality and relevance of materials" and Module 2
in "effectiveness of this module in helping you learn from other managers." There was
no statistical difference among the three modules for ratings of "usefulness and job
applicability" for which all three modules received high average ratings of 4.1 and above.
Module 2 ranked first for pacing of topics, Module 3 second and Module 1 last.
Ten percent of the respondents rated the pacing of Module 2 as "too slow," 74 percent
rated the pacing as "adequate," and sixteen percent rated it as "too fast."
Although Module 1 was revised after Phase I recommendations, participants'
ratings of this revised module remain similar to those before the revision.
Results of pre- and post self-evaluations by participants are favorable
To measure program impact, participants were asked to rate the extent to which
they are aware of or use 14 "preferred" management knowledge, skills and behaviors
addressed by the NMLP. The results were extremely favorable. After completing
NMLP, participants' self-ratings were higher for 9 of the 14 areas, as compared with their
self-ratings before attending NMLP. Statistically, the improvements (differences in
means) were significant at a .05 (or better) significance level.
viii
The program was especially successful in three areas rated lowest in the pre-
NMLP survey. All three areas benefited from a more than 30 percent increase in ratings
of 4 or 5 after program completion. The three areas of exceptional improvement were
"understand the impact you have on others" (43 percentage point increase in high
ratings), "practice creating a clear vision of what you want to accomplish as a leader" (40
percentage points increase in high ratings) and "understand the institution's expectations
of you as manager" (37 percentage point increase in higher ratings.)
In addition to a pre- and post self-assessment of management skills and behaviors,
participants were asked to rate the extent to which they believed NMLP made a
difference in improving their skills in target areas pre-identified by NMLP as "key target
areas of management improvement." The NMLP had the most impact, according to the
participants, in two areas - "communicating tough issues with my staff" and "improving
teamwork in my unit" - where almost one-half (49 and 45 percent, respectively) reported
that the program made "a lot of difference" in improving their skills in these areas. The
program had the least effect on "managing stress when making tough choices" and
"encouraging my staff to attend learning events," where almost one-fifth (17 and 19
percent, respectively) indicated that NMLP made "no difference."
To identify and measure potential factors that facilitate or hinder the new
managers' ability to practice NMLP management skills, participants were asked about the
extent to which a list of pre-identified facilitators and barriers could potentially influence
their ability to use the new management skills. By far the most important facilitating
factor was "personal commitment" to applying the new management skills. Participants
identified a "heavy volume of day-to-day work" as the greatest obstacle to put in practice
new knowledge after the NMLP.
Supervisors' assessments indicate improvements in some managerial skills
Eighteen of 24 of interviewed supervisors indicated that they had noticed changes
in the managerial skills and behaviors of their NMLP staff four months after the program.
They reported that the participants made less frequent requests for advice, exhibited
stronger leadership, utilized more effective problem-solving mechanisms, and that their
team management had improved. Supervisors also identified several general outcomes as
well as a few specific management skills that they expected each participant to achieve
from the NMLP. In interviews conducted after managers completed the program,
supervisors reported that NMLP participants had, on average, achieved 60 percent of the
predetermined outcomes and 46 percent of predetermined management skills to a "great
or considerable extent."
Fifteen of 24 interviewed supervisors responded "yes" to the following broad
question: "Has the NMLP contributed to the performance of your staff in their new
managerial position?" Overall, they indicated that NMLP is (a) worth the sacrifice of
having the manager away for three weeks, and (b) important for new managers because it
helps them "settle in their position" and exposes them to a "network of colleagues" that
they would have otherwise not known.
ix
Staff Survey results shows no difference between trained and untrained managerial
cohorts.
Comparisons between NMLP cohorts 7 and 8, which had completed the program
before the 2005 Bank Group Staff Survey, and cohorts 9 and 10, which completed the
program later, revealed only one difference in staff ratings of their managers. The only
Staff Survey item that showed a statistically significant improvement in ratings was "My
manager provides me with timely feedback to enhance my performance." Staff ratings of
their managers on other four questions about "my manager" showed no differences
between staff of managers in cohorts 7 and 8, and those of managers in cohorts 9 and 10.
RECOMMENDATIONS
Four months after the course, participants and their supervisors offered
recommendations on means to improve the NMLP's usefulness and impact. The authors
agree with and endorse these recommendations.
Improve the quality and reduce the duration of Module 1. Module 1 was
rated lower than Modules 2 and 3 in six out of eight dimensions. Fourteen respondents
suggested that the module be changed significantly in design, content, and delivery.
Several participants noted that the session was "too slow," "boring," "too generic," and
"a superficial recycling of other Bank courses." Suggestions to make module 1 more
effective included cutting its duration, and adding more coaching, people management
exercises, and resource management skills.
Align NMLP with the day-to-day management issues of the Bank and tailor
the program to Bank needs: NMLP participants and their supervisors highlighted the
need for aligning NMLP to the Bank's work. To increase alignment, NMLP participants
made two suggestions: (a) allocate more time to "real world/actual" Bank/IFC case
studies that cover HR management, change management, and resource management; and
(b) develop action plans that enable participants to practice and apply NMLP skills.
Design refresher courses: Participants noted that refresher courses would
"provide a systematic process for course follow-up." Some suggested combining a
refresher course with one-on-one coaching, and others said that follow-up courses would
help the new managers to assess their own progress and address the skills that take longer
to put into day-to-day practice. The opinion of one supervisor summarizes this
recommendation: "The current NMLP provides a good base, but it needs continuation
and further deepening." A follow-up course would also keep the support network in
operation for the new managers.
Ensure timely course delivery to new incoming managers: Supervising
managers would like the program to be available early in the manager's career.
"Consider timing the program's delivery to take place within the first six months after a
new manager is assigned to his/her new position," suggested one supervisor.
Strengthen NMLP in three specific areas: Supervising managers also indicated
that NMLP could be even more effective than it has been to date if it would emphasize
x
the three topics that continue to be challenging issues in the Bank environment --
cultural diversity, conflict management (including delivering hard news), and resource
management. Improving resource management was also recommended in Phase I of this
evaluation.
xi
xii
1. INTRODUCTION
1.1 This is the second evaluation of the New Managers' Leadership Program
(NMLP), a program sponsored by the Human Resources Services Leadership and
Organization unit (HRSLO). The first evaluation, Evaluation of New Managers'
Leadership Program (Zia, Hanson, and Gunnarson 2004), was presented to the Learning
Board in November 2004. After discussing the evaluation, the Learning Board requested
that WBIEG carry out a second evaluation of the NMLP. The two evaluations address
the same questions at different points in time; hence the first evaluation is referred to as
Phase I and the second as Phase II.1
1.2 This chapter describes the New Manager's Leadership Program, lists the
evaluation objectives, presents the evaluation questions, and describes the evaluation
methodology used to assess program impact.
COURSE DESCRIPTION
1.3 The NMLP, which is based on feedback from senior Bank Group management,
human resource managers, coaches, and other Bank professionals, was developed to
support newly appointed and recruited managers with the aim to:2
� Clarify the institution's expectations of managers and strengthen managers'
capabilities to deliver results through effective core relationship skills;
� Increase managers' capabilities to create the results they want in ways that
support life balance; and
� Build knowledge and skills in leading change at their unit level and provide
opportunities to apply their new knowledge to work situations as they continue
to build the cohort as learning and support community.
1.4 The NMLP covers these elements through three sequential four- to five-day
modules over a seven- to eight-month period. Each group of new managers completes all
1. Appendix 1 provides a summary of the evaluation methodology, findings, and recommendations for
Phase 1.
2. Source: http://intranet.worldbank.org/WBSITE/INTRANET/UNITS/HR/LOE
1
three modules with the same group, a managerial cohort.3 As of the date of this
evaluation, 10 cohorts have completed the NMLP. The three course modules are:4
� Module 1 � Crafting Your Management Role provides insights on the skill of
relationship management by exploring the Bank's work environment,
participants' influencing skills, interpersonal communication skills, and cultural
competencies. This module was modified in early 2005 as a result of WBIEG's
recommendations in November 2004.5
� Module 2 � Achieving Business Results through Personal Mastery focuses
on building self-awareness and understanding as a foundation for personal
management and managing others. Participants are expected to strengthen their
ability to face new managerial challenges by gaining clarity about personal
vision, aligning their personal values with professional demands, building
interpersonal competence and confidence, and accepting responsibility for
outcomes.
� Module 3 � Leading Change mobilizes managers to lead change in the
complex World Bank Group. Participants are expected to apply the concepts
presented in this module to their issues. Those issues are related to leading
strategically in a decentralized environment, both within the Bank Group's
leadership team and in their units.
OBJECTIVES, APPROACH, AND METHODOLOGY OF EVALUATION
Evaluation objectives
1.5 The overall objectives and research questions of this Phase II evaluation remain
the same as those for Phase I: to determine whether NMLP has achieved its mandate "to
facilitate new managers' success by providing them with foundational knowledge and
skills as steps toward becoming world class managers." The evaluation seeks to
understand the usefulness of NMLP from the perspective of the participants as well as
their supervisors. To achieve these objectives, this evaluation:
� Assessed NMLP's success in reaching its goals;
� Assessed the program's usefulness and gauged the extent to which managers
applied the knowledge and skills gained from the program to their day-to-day
work;
3. In a few instances participants had to transfer to a future cohort to complete subsequent modules (due
to unplanned business or time conflicts).
4. Source: http://intranet.worldbank.org/WBSITE/INTRANET/UNITS/HR/LOE.
5. The initial objectives of Module 1 were to craft the management role by aligning strategies with the
Bank's mission and achieving results through effective resource management. The Phase I evaluation
report (Zia, Hanson, and Gunnarson 2004) recommended that "[HRSLO] revisit Module 1 to make
considerable changes or to drop it." The NMLP team decided to make changes that were reflected in the
program offered to cohorts 8, 9, and 10.
2
� Assessed the improvement of Module 1; and
� Provided recommendations for continuous improvement.
Approach and methodology
1.6 The Phase II evaluation focuses on cohorts 8, 9, and 10.6 Building on the Phase I
results and methodology, the Phase II uses an approach that includes four components:7
� Component 1 assesses the perceived quality of each of the three modules and
identifies participants' perceptions of each module immediately after its
completion with a level 1 evaluation for all three cohorts.
� Component 2 assesses impact based on a comparison of posttraining self-
assessments from participants in cohorts 9 and 10 and a pretraining self-
assessment of those who participated in cohorts 9 and 10.8
� Component 3 gathers assessments from supervisors of participating managers.
Supervisors of participants in cohorts 9 and 10 were interviewed twice -- one
month before the training to determine areas of desired improvement and four
months after the training to discuss the extent of changes in the participants'
managerial skills.9
� Component 4 assesses impact based on the "my managers" questions in the
Bank's 2005 staff survey comparing the ratings between NMLP cohorts 7 and 8,
which had completed the program before the 2005 Bank Group Staff Survey,
and cohorts 9 and 10, which completed the program later.10
6. Cohort 8 began on February 14, 2005, cohort 9 began on October 17, 2005, and cohort 10 began on
January 23, 2006. Each cohort had approximately 30 participants. All participants in the NMLP were
expected to attend within six months of assuming their new managerial position.
7. The concept note lists five components. The fifth component -- an assessment of participants'
implementation of action plans -- was dropped because the participants were not given action plans as
originally envisioned.
8. In addition, post training assessments were gathered from cohort 8, but pre-training assessments were
not available, as cohort 8 began in Module 1 in mid-February, just when the Knowledge and Learning
Board asked WBIEG for this evaluation and too late for pretraining assessments. The prospective
participants for cohorts 9 and 10 were not a "control" for longer than a month after launching their survey.
WBIEG administered the survey to each cohort one month before the start of Module 1.
9. This component was added after cohort 8 completed the program, thus precluding the opportunity to
interview cohort 8 participants' supervisors in advance.
10. Although this study is based on cohorts 8, 9, and 10, we included cohort 7 to maintain a balance
between data from control and from treatment groups. Ideally, the study would control for several
variables, including number of years since participation in NMLP, number of years with the Bank, and
number of months in position. The study would also compare externally recruited managers with internally
promoted managers. In compliance with the HR policy against providing data about individuals, however,
we do not have access to such control variables.
3
1.7 In reporting the results of all comparisons made in each of the aforementioned
components, we report only differences that are significant at a .05 level.
Evaluation instruments
1.8 This evaluation used three types of instruments: Questionnaires administered
immediately after completion of every module, surveys administered before the start and
after the completion of the program, and interviews conducted before and after the
program.
1.9 For Phase II, WBIEG constructed a revised customized level 1 questionnaire for
the newly designed Module 1, and used the same customized level 1 questionnaire for
Modules 2 and 3 as those used in Phase I. The survey instruments comprise (a) a pre-
self-assessment instrument for prospective participants, (b) a post self-assessment
instrument for graduates of the program, and (c) extracted questions from the 2005 Bank
Group Staff Survey.
1.10 The pre-assessment survey was a short (14-item) instrument that was repeated in
the post-assessment. It allowed participants to rate their management knowledge, skills
and behaviors on 14 areas specified by HRSLO. It was completed just before
participating in the NMLP.
1.11 In contrast to the pre-assessment survey, the post-assessment instrument was an
extensive questionnaire (closed- and open-ended questions) that measured indicators on
two levels: Management skills and behaviors of participants and the usefulness of the
program. The post-assessment instrument covered:
� Self-assessment of 14 key management skills and behaviors pre-identified by
HRSLO (identical to the pre-assessment survey items);
� Extent of participant use of and benefit from newly acquired knowledge and
skills;
� Degree to which the participants believed that the program made a difference in
specific management areas (e.g., communicating tough issues, improving
teamwork, leading change, etc.);
� Comparison of NMLP with other management training;
� Facilitators of and barriers to participants' use of behavioral knowledge and
skills; and,
� Recommendations for improvement.
1.12 The interview instruments included questions asked of supervisors of NMLP
participants one month before and three to nine months after the new managers
completed the program. The first interview asked supervisors to identify performance
4
areas where they hoped their managers would improve, and the second interview asked
supervisors whether the program improved the performance of participants.
1.13 Sources of data for all graphs and tables, with the exception staff survey data
(presented in Chapter 4), are participant surveys and supervisor interviews.
Evaluation questions
1.14 The evaluation sought to identify and measure the program's impact on
participants and addressed the following questions:
� Did the training fulfil the participants' learning needs?
� Did it achieve its stated objectives?
� Were the modules' content, activities, and materials of high quality and relevant
to the needs of the participants?
� Did the participants use the behavioral knowledge and skills presented in the
program?
� Did the participants' performance and behaviors improve after attending the
NMLP?
� What were the facilitators of and barriers to participants' use of behavioral
knowledge and skills?
� Did the NMLP participants perform better than others who did not attend the
program?
� How can the program be improved to address better the learning needs of new
managers?
Target population and response rates
1.15 For the Phase II study, WBIEG studied cohorts 8, 9, and 10 just before the
training, during the training (level 1 evaluations at the end of every module), and four
months after completion of the last module.11 Cohorts 8, 9, and 10 included 90
participants --cohort 8 had 27 participants, cohort 9 had 28, and cohort 10 had 35
participants.12
11. Cohort 8 participants could not be studied just before the training because they began their training on
February 14, 2005, and WBIEG and KLB finalized their agreement on the methodology of the Phase II
evaluation in April 2005.
12. The number of observations (Ns) reported in the level 1 analysis reflects 11 cases where participants
transferred to a future cohort to complete Module 2 and/or Module 3 due to unplanned business or personal
time conflicts. It is possible, therefore, for a participant to have started in cohort 7 for module 1 and
transferred to cohort 8 for module 2, and graduated in cohort 9. Because of the anonymity of
5
1.16 Of the 90 graduates of NMLP in cohorts 8, 9, and 10, 65 percent were from
IBRD, 33 percent were from IFC, and 1 percent was from MIGA. Eighty-three percent
were promoted from within the Bank, 10 percent were new externally recruited
managers, and 7 percent were existing managers. Twenty-nine percent were female.
1.17 The average response rate for level 1 evaluations of Modules 1, 2, and 3 across all
cohorts was 93 percent. Response rates for the pre- and post self-assessment
questionnaires were lower at 78 percent in advance of training and 79 percent after
training.13 Response rates for the interviews with supervisors were 100 percent pre-
NMLP and 91 percent post-NMLP.14
questionnaires, we base our level 1 findings on all questionnaires completed by respondents at the end of
each module, whether or not they ultimately transferred to another cohort. Such reporting, though
comprehensive, presents slight disparities between number of graduates from every cohort and the
corresponding level 1 "Ns" for every module.
13. The pre-assessment response rate was 96 percent for cohort 9 and 60 percent for cohort 10. Pre-
assessment questionnaires were administered by HRSLO on the first day of Module 1. Fourteen of 35
participants in cohort 10 did not complete the pre-assessment questionnaires. See appendix 4 for number
and rates of pre and postresponses to self-assessment surveys.
14. Response rates for post interviews with managers are based on the number of "eligible" supervisors
(supervisors who were managers of NMLP participants from program start through six months after
program completion). For cohorts 9 and 10, 22 managers were eligible. Twenty of the 22 managers were
available for interviews. Four of the 20 supervisors interviewed each had two participants in the program,
making the number of questionnaires 24.
6
2. EARLY EVALUATION RESULTS: MANAGERS' VIEW
2.1 To assess the quality, relevance, and effectiveness of the course, cohorts 8, 9, and
10 completed level 1 evaluations at the end of each module.15 This chapter provides an
overview of each module and reports the results of the level 1 evaluations.
MODULE 1: CRAFTING YOUR MANAGEMENT ROLE -- LEVEL 1 FINDINGS
2.2 Module 1 aims to provide insights about the skill of relationship management by
exploring the Bank's work environment, the participants' influencing skills, interpersonal
communication skills, and cultural competencies. As a result of recommendations of
Phase I evaluation, the emphasis of Module 1 in Phase II shifted from resource
management related topics to interpersonal development and influencing skills. The
module covers how managers should fulfill their Bank managerial accountabilities,
inspire trust, and manage complexities and priorities.16
2.3 The findings of the level 1 survey reflect the views of NMLP participants
immediately after completing the first module. Participants scored each survey item on a
5-point scale (1=low to 5=high/preferred behavior). Despite the changes made to the
content and delivery of Module 1, participants' ratings of the revised module remain
similar to those before the revision.17 Four of the module 1 indicators show a slight
improvement (from 4 to 8 percent more of the participants in Phase II rate Module 1
higher than participants in Phase I rated Module 1), while ratings on two of the indicators
show a slight decline.18
2.4 The highest ratings are found in the "usefulness and applicability of knowledge
and skills acquired from the module," with 84 percent of the respondents giving ratings
of 4 or 5. About three-fourths (76 percent) of the respondents gave 4 or 5 ratings for
"ability to achieve its announced objectives," while 70 percent gave ratings of 4 or 5 on
the "effectiveness of the module in helping them learn from other managers" and on the
"overall quality of the module." Only 58 percent indicated that the training in Module 1
fulfilled their learning with a 4 or 5 rating, and 10 percent gave ratings of 1 or 2. With
15. Appendices 2a-c provide level 1 questionnaires for Modules 1, 2, and 3.
16. Module 1 of Phase II differs from that of Module I in Phase I.
17. It is not possible to test for significant differences in mean ratings, due to lack of data from the Phase I
evaluation regarding the number of cases and the standard deviations in the ratings.
18. Indicators with positive change are: "Extent module achieves its stated objectives", "quality and
relevance of contents of Module," "extent the module fulfills learning needs," and "overall quality of
module." Two indicators with lower ratings in Phase II than I are: "Quality and relevance of materials
used" and "quality and relevance of method of delivery."
7
the exception of "applicability of knowledge and skills acquired" within the module
(mean score 4.17), scores were significantly below the Bank's average level 1 scores.
Figure 1: Participant ratings of Module 1 on use and effectiveness
Mean
To what extent are the Score
knowledge and skills you (4.17)
have acquired useful and 84% 14%
applicable to your job? 1%
To what extent did this (3.78)
module achieve its stated 76% 17% 7%
objectives?
How would you rate the (3.74)
overall quality of this 70% 25% 5%
module?
How would you rate the
effectiveness of this module 4% (3.89)
in helping you learn from 70% 26%
(and with) other managers?
To what extent did this (3.55)
module fulfill your learning 58% 31% 10%
needs?
Rating of 4 or 5 Rating of 3 Rating of 1 or 2
2.5 The overall quality of the training in Module 1 was rated 4 or 5 by 70 percent of
the participants, whereas Module 2 was rated 4 or 5 by 84 percent and Module 3, by 89
percent.
2.6 The quality and relevance of the content, materials, and delivery methods used
were not highly rated (figure 2). Although two of the three ratings were above the mid-
point, neither exceeded 4 and must therefore be considered average. Cohort 10 rated
materials particularly poorly with only 30 percent giving them a 4 or 5. Cohort 10 gave
fewer favorable ratings on all three items (by 10 to 15 percentage points) than did cohorts
8 and 9.
Figure 2: Participant ratings of Module 1 on quality and relevance
Mean
Score
How would you rate the
quality and relevance 70% 25% 5% (3.73)
of: Contents of
Module?
How would you rate the
quality and relevance of:
The course's methods of 62% 30% 8% (3.65)
delivery (Activities?)
How would you rate the
quality and relevance 43% 45% 12% (3.29)
of: Materials Used?
Rating of 4 or 5 Rating of 3 Rating of 1 or 2
8
2.7 On average, 28 percent of the participants found Module 1 to be too slow, 56
percent found the pace adequate, and the remaining 8 percent said it was too fast (Figure
3). Almost one-half of cohort 10 (43 percent) found the module to be too slow.
2.8 Qualitative data collected for both level 1 and impact surveys (four months after
program completion) suggest that participants' low ratings of this module could be partly
attributed to the slow pace of the module. Several managers stated that "there was too
much slack time," "[the module was] not paced fast enough" or "it could be done in three
days." Several participants noted that the plenary sessions were "too long" and "boring."
Figure 3: Participant ratings of Module 1 on pacing
Too Slow, 29%
Cohort 8 Adequate, 64%
Too Fast, 7%
Too Slow, 13%
Cohort 9 Adequate, 79%
Too Fast, 8%
Too Slow, 43%
Cohort 10 Adequate, 48%
Too Fast, 10%
2.9 In addition to standard level 1 items such as quality, relevance, and pace,
participants were asked to rate the effectiveness of the particular focus areas addressed in
each module (figure 4). There was a wide variation in ratings of focus areas -- from a
mean of 4.19 for "listening and inquiry skills" to 2.77 for "HR practices and WBG
resources."19
2.10 Module 1 participants gave high ratings to only two of the seven themes, or focus
areas -- "your current level of listening and inquiry skills" (84 percent 4 or 5) and "your
current level of advocacy/feedback skills" (79 percent 4 or 5) (figure 4). The module was
not at all successful in addressing human resource policy and practice issues at the Bank.
The mean score for this item was 2.77.
19. Note that "behaviors that inspire trust in others" was an evaluation item only for cohorts 9 and 10 (and
not for 8 because the unit on "inspiring trust" was not fully developed and implemented until cohort 9.
9
Figure 4: Participant responses to question: "To what extent was Module 1 effective in
increasing your awareness of:"
Mean
Score
Your current level of listening and inquiry skills 84% 14% (4.19)
1%
Your current level of advocacy/feedback skills 79% 18% (3.97)
3%
Behaviors that inspire trust in others 65% 31% (3.73)
4%
Your impact on others 60% 29% 12% (3.61)
Your role as a WBG manager 56% 26% 18% (3.49)
Your level of awareness multi-cultural issues at work
51% 38% 12% (3.49)
Human resource policies/practices and the available
WBG resources to help you address people 13% 55% 32%
management issues (2.77)
Rating of 4 or 5 Rating of 3 Rating of 1 or 2
MODULE 2: ACHIEVING BUSINESS RESULTS -- LEVEL 1 FINDINGS
2.11 The second module focused on helping managers (a) gain clarity about personal
vision, (b) align personal values with professional demands, (c) build interpersonal
competence and confidence, and (d) accept responsibility for outcomes.
2.12 Findings from level 1 survey reveal that Module 2 was more highly regarded by
participants than was Module 1 in every aspect except usefulness and applicability of
knowledge and skills which received equally high ratings across all three modules
(Module 2 ratings shown in figure 5).
Figure 5: Participant ratings of Module 2 on use and effectiveness
Mean
Score
To what extent are the 81% 14% 5%
knowledge and skills you (4.20)
have acquired useful and
applicable to your job?
To what extent did this
module achieve its stated 87% 9% 4% (4.31)
objectives?
How would you rate the
overall quality of this 84% 10% 6% (4.22)
module?
How would you rate the 3%
effectiveness of this module
in helping you learn from 84% 13% (4.26)
(and with) other managers?
To what extent did this
module fulfill your learning 79% 13% 8% (4.08)
needs?
Rating of 4 or 5 Rating of 3 Rating of 1 or 2
10
2.13 Ratings for the overall quality of Module 2 (mean 4.22) were higher than those
for Module 1 (mean 3.74).20
2.14 Ratings for course content (82 percent 4 or 5, mean of 4.06) and delivery methods
(85 percent 4 or 5, mean of 4.17) were higher than Module 1 (figure 6). The ratings on
quality and relevance of the materials used in Module 2 (3.73) were rated significantly
below than contents of the module and methods of delivery used in the module.
Figure 6: Participant ratings of Module 2 on quality/relevance
Mean
Score
How would you rate the
quality and the relevance of 82% 12% 6% (4.06)
Contents of Module?
How would you rate the
quality and methods of the 85% 9% 6% (4.17)
delivery (Activities?)
How would you rate the
quality and relevance of 63% 32% 5% (3.73)
Materials Used?
Rating of 4 or 5 Rating of 3 Rating of 1 or 2
2.15 The majority of cohorts 8 and 9 (83 percent and 84 percent, respectively) found
Module 2 to be adequately paced (figure 7). Opinions of cohort 10 participants on pacing
varied -- 18 percent thought it was too slow, 21 percent thought it was too fast, and 61
percent reported it as adequate.
Figure 7: Participant ratings of Module 2 on pacing
Too Slow, 8%
Cohort 8 Adequate, 83%
Too Fast, 8%
Too Slow, 0%
Cohort 9 Adequate, 84%
Too Fast, 16%
Too Slow, 18%
Cohort 10 Adequate, 61%
Too Fast, 21%
20. This difference in means is statistically significant at 5 percent level.
11
2.16 Compared to Module 1, Module 2 rated higher in terms of effectiveness in the
area of focus (figure 8 shows participant ratings of the six focus areas for Module 2).
Approximately three-quarters or more of the managers considered the training to be
effective in the areas covered. The module was most effective at increasing new
managers' awareness of their "impact on others" (mean of 4.12) and "what enhances
interpersonal effectiveness" (mean of 4.00).
Figure 8: Participant responses to question: "To what extent was Module 2 effective in
increasing your awareness of:"
Mean
Score
Your impact on others 81% 14% 5% (4.12)
What enhances your 81% 9% 10% (4.00)
intrapersonal-effectiveness?
Your current and desired (4.10)
levels of personal mastery 79% 13% 8%
The clarity your personal 78% 19% (4.03)
vision 3%
What impedes your 76% 16% 8% (3.92)
intrapersonal effectiveness?-
The alignment of your
personal values with 73% 20% 6% (3.94)
professional demands
Rating of 4 or 5 Rating of 3 Rating of 1 or 2
MODULE 3: LEADING CHANGE -- LEVEL 1 FINDINGS
2.17 Cohorts 9 and 10 completed the program simultaneously as they both attended the
same session for Module 3. The third and final module of the NMLP was designed to
help managers (a) become strategic leaders in a complex and decentralized environment,
(b) institute change as part of the Bank's leadership team and in their units, and (c)
accelerate learning and innovation in their units.
2.18 Module 3 results (figure 9) are similar to those from Module 2 in overall quality
of training, fulfilling learning needs, achieving stated objectives, usefulness and job
applicability, quality and relevance of training content, and course delivery. Module 3
outperformed Module 2 on the quality and relevance of the materials used, with 88
percent of the respondents giving a rating of 4 or 5 (figure 10).
12
Figure 9: Participant ratings of Module 3 on use and effectiveness
Mean
To what extent are the Score
knowledge and skills you 81% 13% 5% (4.11)
have acquired useful and
applicable to your job?
To what extent did this 1%
module achieve its stated 90% 9% (4.22)
objectives?
How would you rate the
overall quality of this 89% 7% 4% (4.19)
module?
How would you rate the
effectiveness of this module 74% 21% 5%
in helping you learn from (4.03)
(and with) other managers?
To what extent did this
module fulfill your learning 78% 15% 7% (3.96)
needs?
Rating of 4 or 5 Rating of 3 Rating of 1 or 2
Figure 10: Participant ratings of Module 3 on quality and relevance
Mean
Score
How would you rate the
quality and relevance of: 79% 17% 4% (4.04)
Contents of Module?
How would you rate the
quality and relevance of: 83% 13% 3% (4.21)
The courses?
How would you rate the
quality and relevance of: 88% 9% 3% (4.30)
Materials Used?
Rating of 4 or 5 Rating of 3 Rating of 1 or 2
2.19 The pacing of Module 3 was adequate for most participants although there were
differences between cohorts (figure 11). The majority of participants (79 percent of
cohort 8 and 66 percent of cohorts 9 and 10) found Module 3 to be adequately paced,
while 26 percent of participants from cohorts 9 and 10 rated Module 3 as too fast.
13
Figure 11: Participant ratings of Module 3 on pacing
Too Slow, 14%
Cohort 8 Adequate, 79%
Too Fast, 7%
Too Slow, 8%
Cohort 9 & 10 Adequate, 66%
Too Fast, 26%
2.20 Specific areas of focus in Module 3 are shown in figure 12. The module was
especially successful in two areas -- "approaches to organizational culture and decision
making" (89 percent 4 or 5 with a mean of 4.31), and "frameworks and approaches to
organizational strategy and flexibility" (74 percent 4 or 5 with a mean of 4.00).
Figure 12: Participant responses to question: "To what extent was Module 3 effective in
increasing your awareness of:"
Mean
Approaches to Score
organizational culture and 89% 8% 3% (4.31)
team decision making
Frameworks and approaches
to organizational strategy 74% 22% 4% (4.00)
and flexibility
Approaches to leading a
learning organization 74% 18% 9% (3.86)
Frameworks and
approaches to coalition 70% 23% 7% (3.95)
building and partnerships
Rating of 4 or 5 Rating of 3 Rating of 1 or 2
COMPARISON ACROSS MODULES 1, 2, AND 3
Summary quantitative findings
2.21 Table 1 compares Modules 1, 2, and 3 based on eight selected level 1 indicators:
(1) overall quality of training, (2) extent that the training fulfilled participants' learning
needs, (3) extent that the training achieved stated objectives, (4) extent that the training is
useful and applicable to your job, (5) effectiveness of this module in helping you learn
14
from other managers, (6) quality and relevance of the training content, (7) quality and
relevance of materials used, and (8) quality and relevance of course delivery method.
2.22 For six of the eight indicators listed in Table 1, Modules 2 and 3 rated higher than
Module 1.21 The six areas are "overall quality of training," "fulfilling learning needs,"
"achieving stated objectives," "quality and relevance of content," "quality and relevance
of materials used," and "quality and relevance of course delivery method." Module 3
was rated highest in "quality and relevance of materials." There was no statistical
difference between the three modules for "usefulness and job applicability of knowledge
and skills" for which all three modules received high average ratings of 4.1 and above.
Module 2 ranked first for pacing of topics, Module 3 second, and Module 1 last. Ten
percent of the respondents rated the pacing of Module 2 as "too slow," 74 percent rated
the pacing as "adequate," and sixteen percent rated it as "too fast."
Table 1: Selected level 1 indicators: Percentage of respondents with high ratings,
mean scores, t-test of mean differences
Means testing (t-test)
Question Module 1 Module 2 Module 3 1 vs. 2 2 vs. 3 1 vs. 3
1. How would you rate the overall quality % 4&5 70 84 89 t-stat. -3.79 0.31 -3.86
of training? N's 77 98 91 p-value 0.00 0.76 0.00
Means 3.74 4.22 4.19
StdDev 0.75 0.90 0.74
2. To what extent did this Module fulfill % 4&5 58 79 78 t-stat -3.95 0.95 -3.16
your learning needs? N's 77 99 91 p-value 0.00 0.34 0.00
Means 3.55 4.08 3.96
StdDev 0.82 0.94 0.86
3. To what extent did this Module % 4&5 76 87 90 t-stat -4.40 0.88 -3.98
achieve its stated objectives? N's 76 99 91 p-value 0.00 0.38 0.00
Means 3.78 4.31 4.22
StdDev 0.79 0.80 0.65
4. To what extent is the knowledge and % 4&5 84 81 81 t-stat -0.26 0.70 0.47
skills you have acquired useful and N's 77 99 91 p-value 0.80 0.49 0.64
applicable to your job? Means 4.17 4.20 4.11
StdDev 0.71 0.94 0.87
5. How would you rate the effectiveness % 4&5 70 84 74 t-stat -2.92 1.83 -1.04
of this Module in helping you learn N's 77 99 91 p-value 0.00 0.07 0.30
from other managers? Means 3.90 4.26 4.03
StdDev 0.80 0.84 0.89
6. Quality and relevance of training % 4&5 70 82 79 t-stat -2.80 0.13 -2.64
content N's 77 99 90 p-value 0.01 0.90 0.01
Means 3.73 4.06 4.04
StdDev 0.68 0.85 0.85
7. Quality and relevance of materials % 4&5 43 63 88 t-stat -3.71 -4.86 -8.66
used N's 75 98 91 p-value 0.00 0.00 0.00
Means 3.29 3.73 4.30
StdDev 0.71 0.82 0.77
8. Quality and relevance of course % 4&5 62 85 83 t-stat -4.15 -0.32 -4.46
delivery method N's 77 99 90 p-value 0.00 0.75 0.00
Means 3.65 4.17 4.21
StdDev 0.77 0.87 0.84
21. Module 1 was also rated the lowest of the three modules in Phase I evaluation. A statistical test of
difference between Module 1 in Phase I evaluation (cohorts 3 and 4) and Phase II (cohorts 8, 9, and 10)
would have been desirable, but was not possible for two reasons: first, the instrument for module 1 has
changed since Phase I, and second, for those items that are the same, standard errors of Phase I indicators
are not available.
15
Summary of qualitative responses
2.23 Level 1 questionnaires also asked four open-ended questions. The most frequent
responses are reported in Table 2.
� What did you find most useful about this module?
� What did you find least useful about this module?
� What are your recommendations for improving the effectiveness of the
facilitation team?
� What are your recommendations for improving the effectiveness of this module?
Table 2: Summary of level one open-ended questions.
Questions Module 1 Module 2 Module 3
Most useful � Role playing � Work in small groups � Role playing
� Case studies � Time for self-reflections, � Case studies
visualization, and awareness
Least useful Cultural diversity Time for self-reflections, Too fast
visualization, and awareness
Recommendations for � Must clarify their Team could improve by being more Could improve by being more
facilitation team roles and goals inclusive, interactive, and by knowledgeable about the
challenging the participants more. applicability of the training to
� Try to be more WB specific context.
inclusive and
interactive
Recommendations for Module could be Module could be improved by: Module could be improved by
module improved by: having more real-world and/or
� being more inclusive and WB specific case studies.
� being more interactive,
inclusive and
interactive with the � having more real-world and/or
participants, WB specificity, and
� having more real- � More time for
world and/or WB reflection/meditation.
specificity,
� increased pace and
reduced plenary
sessions, and
� Improved
facilities/room.
16
3. IMPACT EVALUATION RESULTS:
SELF-ASSESSMENTS BY PARTICIPANTS
BEFORE AND AFTER NMLP
3.1 This chapter addresses four questions: 22
� Did the participants utilize the behavioral knowledge and skills presented in the
program?
� Did the participants' performance and behaviors improve after attending the
NMLP?
� What were the facilitators of and barriers to the participants' utilization of
behavioral knowledge and skills?
� How can the program be improved to better address the learning needs of new
managers?
DID THE PARTICIPANTS USE THEIR NEW KNOWLEDGE AND SKILLS?
3.2 We gauge the use of management skills acquired in the NMLP in three ways:
(a) managers' self-assessment of the degree to which they were able to use their newly
acquired skills, (b) managers' rating of the extent to which they benefited from the
program, and (c) managers' self report of receiving favorable feedback after attending the
course.
3.3 In response to the question "Since your participation in NMLP, to what extent
have you been able to use the various skills and approaches you learned?" All
respondents reported that they had been able to make use of NMLP training "to some
extent," and about one-half (47 percent) said they had applied the training "to a
considerable extent."
3.4 In response to the question "Overall, to what extent have you benefited from
NMLP?" only two (of 72) respondents reported benefiting "to a limited extent" or "not at
all." Twenty-four percent of the participants reported that they benefited from the
22. This chapter is based on responses from 48 managers from pre-NMLP surveys (78 percent response
rate) and 72 managers who responded to post-NMLP survey (80 percent response rate). Also, as
mentioned earlier, our sample of observations for the pre-NMLP survey excluded cohort 8 because the
report was requested after cohort 8 began, hence the smaller number of responses relative to the post survey
responses. See appendix 3a and 3b for pre- and post survey instruments.
17
program "to a great extent," 54 percent to a "considerable extent," and 19 percent "to
some extent."
3.5 More than two-thirds of the participants (71 percent) reported receiving favorable
feedback on their management abilities since attending NMLP. The remaining 29
percent did not receive any feedback at all.
DID THE PARTICIPANTS NOTE A CHANGE IN THEIR OWN PERFORMANCE?
3.6 Improvement in participants' performance and behaviors was examined in two
ways: Through pre- and post self-assessment of the management skills and through post
assessment of the extent to which their participation made a difference in improving their
management skills.
Participant self-assessments of performance, before and after NMLP
3.7 The data for this analysis are based on responses from participants of cohorts 9
and 10 at the start of the program and four months after the program.23 Both surveys
were anonymous and confidential, and it is not possible to match the pre- and post
responses on an individual basis. A total of 48 participants completed the pre-assessment
and 49 participants completed the post-assessment for cohorts 9 and 10 (see appendix 4
for details on the response rates).
3.8 To measure program impact, participants were asked to rate the extent to which
they used the 14 preferred management skills and behaviors addressed by the NMLP.
The ratings were based on a 5-point scale broken down into these categories: "to a great
extent" (5), "to a considerable extent" (4), "to some extent" (3), "to a limited extent" (2),
and "not at all" (1). Figure 13 presents the share of participants rating themselves in the
highest two categories, before and after the NMLP.
3.9 In 9 of the 14 areas, a higher share of participants rated themselves in categories 4
and 5 following the NMLP than prior to the NMLP.24 Differences in mean ratings of
pre- and post-NMLP mirror the results of change in 4 or 5 ratings -- in 9 of the 14 areas,
managers' self-ratings were 8 to 22 percent higher than they were before attending
NMLP.25 Significance tests for the improvements shown in figure 13 are reported in
Table 3.
23. See appendices 3a and 3b for the pre- and post questionnaires. Also, in the absence of preprogram data
for cohort 8 we base this analysis on cohorts 9 and 10 only.
24. One of the areas shows a small negative shift and another two show a positive shift of less than 5
percent on the mean ratings (see table 3.) As shown in table 3, changes in all three areas are statistically
insignificant.
25. Percentage of improvement in each area was calculated with a simple percentage change formula of:
(post mean rating � pre mean rating)/pre-mean rating * 100.
18
Figure 13: Self-assessments of behaviors, skills, and knowledge, pre-NMLP and post-
NMLP, Cohorts 9 and 10
(Percent of respondents providing 4 = "considerable" or 5 = "great extent" ratings)
Understand the importance of creating partnerships to 90%
achieve results? 90%
65%
* Understand the value of different cultural perspectives? 88%
52%
* Know the principles of effective team leadership? 82%
* Understand the impact you have on others? 33%
80%
Understand what behaviors inspire trust--or, 63%
conversely undermine trust--in leaders? 76%
* Practice creating a clear vision of what you want to 35%
accomplish as a leader? 76%
* Know how to provide effective feedback to staff, with 52%
emphasis on actionable improvement? 76%
79%
Employ listening and inquiry skills?
75%
* Understand the Institution's expectations of you as a 42%
manager? 73%
Align the structure and processes of your team with the 73%
mission of the organization? 71%
46%
* Create a learning environment for your unit? 67%
* Pay attention to aligning your personal values with 57%
professional demands? 67%
* Know how to influence decisions that impact you or your 44%
team? 65%
Develop effective changes within the team to address 54%
organizational changes? 60%
Pre Post
* Indicates significant difference of means at .05 level.
3.10 The program was successful in the three lowest rated areas in the pre-NMLP. All
three areas benefited from an increase of over 30 percentage point in ratings of 4 or 5
after program completion. The three areas of exceptional improvement were "understand
the impact you have on others" (47 percentage point increase in 4 or 5 ratings), "practice
creating a clear vision of what you want to accomplish as a leader" (41 percentage point
increase in 4 or 5 ratings), and "understand the institution's expectations of you as
manager" (31 percentage point increase in 4 or 5 ratings).
3.11 Although participants benefited from a significant increase in ratings in the
aforementioned areas, their post ratings suggest that there is also room for further
improvement � since in the two areas of "practice creating a clear vision of what you
19
want to accomplish as a leader" and "understand the institution's expectations of you as a
manager" - where post ratings of 4 or 5 remain less than 80 percent.
Table 3: Participant self-assessments of managerial behaviors, skills, and knowledge before
and after NMLP (mean rating on 1-5 scale)
Group One-
Pre/Post Standard tailed P-
NMLP N's Mean Deviation Difference t-stat value
Understand the importance of Pre 48 4.13 0.570
creating partnerships to achieve 0.188 1.42 0.055
results? Post 48 4.31 0.719
Understand the value of different Pre 48 3.88 0.866
cultural perspectives? 0.492 3.08 0.001
Post 49 4.37 0.698
Know the principles of effective Pre 48 3.54 0.849
team leadership? 0.418 2.83 0.002
Post 49 3.96 0.576
Understand what behaviors inspire Pre 48 3.77 0.692
trust--or, conversely, undermine 0.188 1.36 0.087
trust--in leaders? Post 49 3.96 0.676
Understand the Institution's Pre 48 3.33 0.907
expectations of you as a 0.646 3.57 0.000
manager? Post 49 3.98 0.878
Understand the impact you have Pre 48 3.29 0.683
on others? 0.729 5.34 0.000
Post 49 4.02 0.661
Practice creating a clear vision of Pre 48 3.35 0.635
what you want to accomplish as a 0.625 4.64 0.000
leader? Post 49 3.98 0.692
Know how to provide effective Pre 48 3.54 0.683
feedback to staff, with emphasis 0.295 2.22 0.011
on actionable improvement? Post 49 3.84 0.624
Pre 48 4.13 0.733
Employ listening and inquiry skills? 0.042 0.26 0.397
Post 48 4.17 0.808
Align the structure and processes Pre 48 3.88 0.789
of your team with the mission of 0.064 0.36 0. 359
the organization? Post 49 3.94 0.944
Pay attention to aligning your Pre 47 3.64 0.942
personal values with professional 0.321 1.76 0.039
demands? Post 49 3.96 0.841
Create a learning environment for Pre 48 3.58 0.767
your unit? 0.286 1.80 0.035
Post 46 3.87 0.778
Know how to influence decisions Pre 48 3.42 0.767
that impact you or your team? 0.298 1.99 0.023
Post 49 3.71 0.707
Develop effective changes within Pre 48 3.52 0.850
the team to address organizational 0.125 0.69 0.254
changes? Post 48 3.65 0.934
3.12 The five areas without significant changes in ratings were "employ listening and
inquiry skills," "align the structure and processes of your team with the mission of the
organization," "understand what behaviors inspire trust--or, conversely, undermine
trust--in leaders," "develop effective changes within the team to address organizational
changes," and "understand the importance of creating partnerships to achieve results." T-
tests of the mean difference between the ratings before and after NMLP show that there is
no statistically significant difference in managers' knowledge and skills in these five
areas before and after NMLP. The difference between before and after is only a small
fraction of the standard deviation of the pre-scores presented in table 3.
20
Extent to which NMLP made a difference
3.13 In addition to a pre-and post self-assessment of management skills and behavior,
participants were asked to rate the extent to which they believed NMLP made a
difference in improving their skills in target areas pre-identified by NMLP as "key focus
areas of management improvement." Figure 14 lists the six areas of management skills
that NMLP aimed to improve and the distribution of ratings for each area. Participants
estimated the effects of the NMLP in three categories: A lot of difference, a slight
difference, and no difference.26
3.14 Of the 6 pre-identified target areas of management improvement (listed in Figure
14), the NMLP has made the most difference in "communicating tough issues." Though
rated highest relative to the other pre-identified target areas, only one-half (49%) of the
participants rated the program as making "a lot of difference" in this area. In another pre-
identified communication related area, "resolving differences among staff and with
stakeholders," only 35 percent of the participants rated it as making "a lot of difference."
These findings suggest that there is still room for improvement in the communication
related areas of the program. About one-fifth of the respondents indicated that the NMLP
"made no difference" in "managing stress when making tough choices" (17%) and
"encouraging my staff to attend learning events" (19 %)."
Figure 14: Extent to which NMLP made a difference in six focus areas
Communicating tough issues
with my staff 49% 47%
3%
Improving teamwork in my 45% 54%
unit
Encouraging my staff to
attend learning events 41% 39% 19%
Leading change in the Bank
39% 45% 13%
Group environment
3%
Resolving differences among
35% 61%
staff and with stakeholders
Managing stress when making
32% 51%
tough choices 17%
NMLP made a lot of Difference NMLP made a SLIGHT Difference NMLP made NO Difference
Note: Some bars do not add to 100 percent because some respondents chose "Not Applicable."
3.15 Asked if the NMLP had led to managing in different and better ways, 96 percent
(65 of 68 respondents) responded yes, and 54 respondents explained how they manage in
different and better ways. Participants shared seven areas of improvement, listed below
in order of importance, from most frequently cited to least frequently cited:
26. One manager chose "not applicable" for all the target areas except managing stress, and two managers
chose it for leading change in the Bank Group environment.
21
� Better understanding and considering the other person's point of view
� Better understanding of the importance of listening and practicing it
� Better understanding of the importance and requirements of a successful team
� Better articulating my vision, goals, and expectations
� More confidence in myself
� Enhanced balance between work and family life
� Deeper understanding of concepts and principles of management
3.16 Although our pre-post analyses show that there was no statistically significant
change in participants' self ratings of listening and inquiry skills (para. 3.12), ten of the
54 participants who responded how "NMLP led you to manage in different and better
ways" indicated that they have a better understanding of the importance of listening and
practicing it. Possible explanations for this distinction are that those ten participants who
noted their awareness in the importance of listening after the program, had over-rated
themselves before the program � when they were not yet aware of listening as a "skill"
per se; or that they under-rated themselves after the program. Another explanation could
be that three of the 10 who said that they had a better understanding of the importance of
listening skills belonged to cohort 8 which - due to lack of pre-course data - was not
included in the pre-post analysis.
WHAT ARE THE FACILITATORS AND BARRIERS FOR MANAGERS TO APPLY NEW SKILLS?
3.17 To identify and measure potential factors that facilitate or hinder new managers'
ability to practice NMLP management skills, participants were asked about the extent to
which a list of pre-identified facilitators and barriers could potentially influence their
ability to use the new management skills.
3.18 Five potential facilitators were identified: (a) staff readiness to accept change, (b)
personal commitment, (c) support from colleagues, (d) support from others who attended
NMLP, and (e) support from supervisors/managers. Respondents rated the extent to
which these factors served as facilitators for applying their new knowledge and skills on a
scale of 1 (not at all) to 5 (to a great extent) (figure 15).
3.19 By far the most important facilitating factor was "personal commitment" to
applying the new knowledge. The second most important facilitating factor identified by
the new managers was "support from their supervisors/managers" in implementing their
newly acquired skills.
3.20 It is noteworthy that new managers did not consider support from their NMLP
peers as an impetus for putting NMLP skills into use. Although the program aspires to
create cohorts of peers to support each other after NMLP, it may well be that the
22
participants do not have the time or occasion to maintain the support for one another after
completing the program.
3.21 Three potential barriers to the use of newly acquired management skills were
identified (figure 16): (a) irrelevant content of NMLP, (b) abstract/too conceptual nature
of NMLP, and (c) heavy volume of day-to-day work. Respondents rated these barriers on
a scale of 1 (not at all) to 5 (to a great extent).
3.22 Participants identified a heavy volume of day-to-day work as the greatest barrier.
Forty-one percent of the respondents thought this was a great or considerable barrier. In
contrast, a small number had concerns about the applicability of the skills to their
particular job (6 percent), or felt the course was too abstract or conceptual to be
applicable (9 percent).
Figure 15: Attributes that facilitate practice and use of NMLP
Mean
Score
Your personal 87% 13% (4.13)
commitment
Support from your (3.02)
supervisor/manager 42% 25% 28%
Staff readiness to accept
change 37% 41% 18% (3.10)
Support from colleagues 35% 39% 26% (3.10)
Support from others who 18% 33% 44% (2.40)
attended NMLP
Great/Considerable Some Limited/None
Note: Some bars do not add to 100 percent because the option "Not applicable" was chosen
23
Figure 16: Attributes that hinder practice and use of NMLP
Mean
Heavy volume of
my day-to-day 41% 23% 36% (3.07)
work
Abstract / too
conceptual nature 9% 9% 79% (1.69)
of NMLP
Irrelevant content 6% 13% 79% (1.69)
of NMLP
0% Great/Considerable Some Limited/None 100%
HOW CAN THE PROGRAM BE IMPROVED?
3.23 Ninety-three percent of the respondents indicated that they would recommend the
NMLP to other new managers, and 79 percent rated NMLP as "better than" or "one of
the best" management courses compared with other management training in which they
had participated (figure 17).
Figure 17: How the NMLP compares with other management courses
Can not compare,
15%
One of the worst, One of the best,
0% 36%
Worse than others,
3%
Same as others,
4%
Better than others,
43%
24
3.24 Sixty of 72 respondents made comments and recommendations. They made
general comments about the usefulness of the modules and provided specific
recommendations for improving the overall program in terms of the curriculum, content,
and duration.
Comments specific to the modules
3.25 Module 1. Fourteen of 60 respondents indicated that Module 1 was the least
useful of the three modules. Three people described the module as too generic and
another said that it was "a superficial recycling of other Bank courses." All 14
respondents suggested that Module 1 be changed significantly in design, content, and
delivery, or it should be dropped. Suggestions to make it more useful included cutting its
duration, adding more coaching, people management exercises, and resource
management skills.
3.26 Module 2. Participants gave mixed comments on Module 2. Five of the 10
participants who commented on this module thought it was useful. The remaining five
suggested that the module should be modified to include simulations to allow practice of
the theories and concepts covered in the module.
3.27 Module 3. Participants' comments on Module 3 were by far more positive than
those for Modules 1 and 2. Of the 26 participants who commented on Module 3, 23
indicated that it was the most useful and most stimulating module. Almost without
exception, however, all 23 respondents cautioned that the module was perhaps too
academic and needed more emphasis on applying the theories and cases presented to the
WB/IFC situation. Respondents suggested adding Bank/IFC case studies and allocating
more time for analysis and discussion, and relating the cases to their personal experience.
Recommendations to improve NMLP
3.28 Four months after the course, with time to reflect on and practice the course
content, participants offered two concrete recommendations to increase the usefulness
and impact of NMLP:
� Align NMLP with day-to-day management issues of the Bank. To increase
alignment, they made two suggestions: (a) allocate more time to "real
world/actual" Bank/IFC case studies that cover HR management, change
management, and resource management; and (b) develop action plans that
enable participants to practice and apply NMLP skills.
� Design follow-up refresher courses. Participants noted that refresher courses
would provide a systematic process for follow up. Some suggested combining a
refresher course with one-on-one coaching, and others said that follow-up
courses would help new managers assess their own progress and address the
skills that take longer to put into day-to-day practice.
3.29 Participants made mixed recommendations about the duration of the course.
Some suggested that the course should be cut to two weeks and others said that it was
25
"too fast." Asked to rate the duration of the course as too long, too short, or just right, 76
percent said that three weeks was just right, 16 percent said it was too long, and 6 percent
said it was too short.
26
4. IMPACT EVALUATION RESULTS:
THIRD-PARTY ASSESSMENTS FROM SUPERVISORS
AND STAFF SURVEYS
4.1 This chapter further explores two of the same evaluation questions addressed in
Chapter 3 from the perspective of NMLP participants. In this chapter two evaluation
questions are addressed from the perspective of supervisors and direct
reports/subordinates of NMLP participants.
� Did the participants' performance and behaviors improve after attending the
NMLP?
� How can the program be improved to better address the learning needs of new
managers?
4.2 The first part of this chapter presents the results of supervisors' assessments of the
NMLP based on their observed changes in behavioral and management skills of their
staff who were members of cohorts 9 and 10. The second part of the chapter presents
results from the 2005 Bank staff survey, comparing staff ratings for managers who had
completed the NMLP at the time of the staff survey with those who had not yet
participated in NMLP (but were about to do so).
SUPERVISORS' ASSESSMENTS BEFORE AND AFTER NMLP
4.3 This section analyses supervisors' opinions about whether the new managers in
cohort 9 and 10 achieved their objectives. It reports the supervisors' assessment of the
impact of NMLP on its participants, based on their observations of changes in the
performance of their new managers after attending the program. These assessments are
based on two sets of interviews with the supervisors -- once before their staff attended
NMLP and four months after they completed NMLP.27
4.4 The data used for this analysis are based on pre- and post interviews with 20
supervisors who reported on 24 NMLP participants from cohorts 9 and 10.28 Four of the
27. See appendices 5A and 5B for pre- and post-NMLP interview questions to the supervisors.
28. The number of supervisors "eligible" for post interviews (23) was far less than those interviewed in the
pre-NMLP phase (47). This is because 17 supervisors and seven new managers had changed positions by
the time the post-NMLP interviews were conducted. Three of the eligible supervisors for post interviews
did not respond to requests for interviews.
27
interviewees reported on two NMLP participants, each of which provided separate
interviews for each of their NMLP staff members.
4.5 In the pre-NMLP interviews, the supervisors were asked to identify (a) three
developmental outcomes that they would like to see the participant achieve over the
course of the NMLP, and (b) a list of management skills that they thought the participant
should attain to deal with the challenges of his or her particular work environment. On
average, supervisors listed three development outcomes and two management skills that
they expected their new manager to achieve over the course of the training program.29 In
total, supervisors identified 63 development outcomes and 50 management skills.
4.6 During the post-NMLP interviews the supervisors (a) indicated whether they had
noticed any changes in the previously identified managerial skills and behaviors of their
NMLP staff, (b) rated the extent to which his or her NMLP participant achieved the
previously identified development outcomes and management skills, and (c) assessed the
contribution of NMLP to the performance of his or her participant.
4.7 Four months after their new managers had completed the program, 18 out of 24
supervisors indicated that they had noticed changes in the managerial skills and behaviors
of their NMLP staff. The changes were revealed through less frequent requests for
advice, stronger leadership, more effective problem solving mechanisms, and improved
team management. Seventeen percent of the supervisors said they did not know whether
any change had happened, largely because they were unable to attribute any
improvements to NMLP because their staff were also benefiting from other training such
as coaching programs. Only two supervisors reported that they had not noticed any
changes.
4.8 Supervisors also rated the extent to which their NMLP participant had achieved
the pre-identified outcomes and skills. Supervisors' ratings were based on a 5-point
scale: broken down into these categories: "to a great extent" (5), "to a considerable
extent" (4), "to some extent" (3), "to a limited extent" (2), and "not at all" (1). The
overall mean rating (weighted average) for the extent to which the participants achieved
the pre-determined development outcomes was 3.54.
4.9 On average, supervisors indicated that NMLP participants had achieved 60
percent of the pre-determined developmental outcomes to a "great or considerable"
extent, 24 percent to "some extent," and 16 percent to a "limited extent" (figure 20).30
4.10 Figure 20 also shows the distribution of managerial skills ratings. Supervisors
indicated that NMLP participants had achieved 46 percent of the predetermined
management skills to a "great or considerable" extent, 42 percent to "some extent," and
29. The number of development outcomes listed by each supervisor ranged between one and seven; the
number of management skills listed by each manager ranged between one and four.
30. None of the outcomes were rated as having no achievement at all.
28
12 percent to a "limited extent."31 The mean rating for the extent to which the
participants achieved the predetermined management skills was 3.51.
Figure 18: Supervisors' perceptions of the extent of improvement in NMLP participants'
development outcomes and managerial skills, by degree of improvement
(Average percent of outcomes showing improvement)
Development
Outcomes 5=Great & 4=Considerable 3=Some 2=Limited
(Weighted Mean= 3.54) (60%) (24%) (16%)
Managerial
Skills 5=Great & 4=Considerable 3=Some 2=Limited
(Weighted Mean= 3.51) (46%) (42%) (12%)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Percentage of skill and development outcomes pre-determined by supervisors
4.11 Supervisors' responses to a general and all inclusive question --"Has the NMLP
contributed to the performance of your staff in their new managerial position" --
produced results that are in line with previous findings. The majority of the supervisors
(63 percent) said "yes," the NMLP has contributed to staff performance. One-third of the
supervisors said they could not attribute changes of participant performance strictly to
NMLP, and therefore said "don't know," and one supervisor said that NMLP has made
no contribution to the management capabilities of their NMLP staff.
4.12 Overall, the majority of the supervisors indicated that NMLP is (a) "worth the
sacrifice" of having the manager away for three weeks, and (b) important for new
managers because it helps them "settle in their position" and exposes them to a "network
of colleagues" that they would have otherwise not known. More specifically, several
supervisors indicated that the NMLP helped participants to "gain a deeper understanding
on how to use authority," "pay attention to areas of self-improvement", and "gain
confidence in their management role." Three supervisors said that they wish the program
had existed when they first became managers.
4.13 Supervisors provided four specific recommendations for improving the NMLP
value and effectiveness:
31. None of the management skills were rated as having no achievement at all.
29
� Provide regular follow ups to NMLP. Eight of the supervisors interviewed said
that for the program to be effective, participants need additional training in the
form of regular follow ups. The opinion of one supervisor summarizes this
recommendation: "The current NMLP provides a good base, but it needs
continuation and further deepening." A follow-up course would also keep the
support network for the new managers in operation.
� Ensure timely course delivery to new incoming managers. "Consider timing the
program's delivery to take place within the first six months after a new manager
is assigned to his/her new position." Supervising managers would like the
program to be available early in the manager's career.
� Tailor NMLP to Bank needs. As one supervisor said, "It is important to
recognize the need for tailoring NMLP material to the Bank's reality." This
would require incorporating case studies that are based on actual "real life"
cases regarding a number of issues, including but not limited to project
management, performance management, and conflict resolution. One-half of the
supervisors highlighted the need for aligning NMLP to the Bank's work to make
it more relevant and useful for Bank managers.
� Strengthen NMLP in three specific areas. Supervisors indicated that NMLP
could be even more effective than it has been to date if it would emphasize the
three topics that continue to be challenging issues in the Bank environment --
cultural diversity, conflict management (including delivering hard news), and
resource management.
NMLP STAFF SURVEY FINDINGS
4.14 The data used for this analysis were based on responses from subordinates of new
managers to determine the impact of the NMLP on the perceptions of subordinates about
their managers.
4.15 To assess the impact of NMLP from the perspective of subordinates, we studied
the results of the 2005 staff survey for two groups of NMLP participants: a quasi-control
group of new managers who had not yet completed NMLP in November 2005 when the
2005 staff survey was being administered (cohorts 9 and 10, with 39 managers rated by
subordinates), and a treatment group of new managers who had already completed
NMLP by the time of the staff survey (cohorts 7 and 8 with 31 managers rated by
subordinates).
4.16 Results of five staff survey questions related to the "my manager" section of the
staff survey were extracted by HR for both the control and treatment groups. Table 4
reports percent of favourable ratings for both the control and treatment groups.32 To
32. Testing the difference of mean ratings showed no statistically significant difference between the
treatment and control groups, not even for staff survey item 36.
30
maintain confidentiality, HR was unable to provide individual ratings for every manager.
The data obtained were aggregated based on ratings by subordinates for every manager.
For every manager we obtained (a) the average ratings (using a 1 to 5 scale), (b) percent
of "favorable" (4 or 5 ratings), and (c) the approximate number of subordinate responses
that were used to calculate the ratings (1 if n<10, 2 if n>=10 and n<20, 3 if n>=20).
Table 4: Staff ratings of their managers on 2005 Staff Survey, Cohorts 7 and 8 versus Cohorts 9 and 10
(difference in percent of favorable ratings)33
Percent rating favorable
Treatment group Control group
Staff survey item (Cohorts 7-8 ) (Cohorts 9-10) t-stat p-value
21: My manager encourages open and honest
discussion. 79.4 77.5 0.51 0.61
30: My manager displays the people
management skills to effectively lead the group. 68.3 66.8 0.32 0.75
31: My manager is supportive of our
workgroup. 84.3 83.2 0.33 0.74
33: My manager fosters collaboration with other
groups whose work relates to ours. 75.8 75.7 0.02 0.98
36: My manager provides me with timely
feedback to enhance my performance. 64.2 55.5 2.35 0.02
4.17 The unavailability of individual participant level data limited the scope of the
analysis. An ideal significance test would use the original raw scores (either 1 to 5 scores
or a binary favorable/unfavorable scale) to calculate chi-squares statistics.34 A
comparison of percent favorable ratings using chi-squares, however, is valid only if we
are comparing the fully disaggregated ratings of individual subordinates. Because we did
not have the exact number of subordinates for each rating, we could not make accurate
calculations.35 The scope of our analysis allowed only significance testing by way of
comparing unweighted percent favorable scores (4 or 5) using a two-tailed t-test.36
4.18 The results are not as optimistic as the pre- and post self-assessments. In four of
five cases, we note that there are no statistically significant differences between the
treatment and control groups (even at a 10 percent significance level; see table 4 last
column). The only survey item that shows a statistically significant difference in
favorable ratings is "My manager provides me with timely feedback to enhance my
performance".
33. Percent ratings represent average of averages.
34. Although it is highly unlikely that an "ideal" test would create results that conflict with those presented
above.
35. A simple test based on the conservative assumption that each manager has 10 subordinates creates
similar results to those shown above (i.e., only question 36 is significantly different).
36. A comparison of mean ratings using the average 1 to 5 scoring is not valid because the distribution of
ratings is not normal.
31
5. CONCLUSIONS
5.1 This evaluation has:
� Assessed the NMLP's success in reaching its goals for the three modules;
� Assessed the program's usefulness and gauged the extent to which managers
apply the knowledge and skills gained from the program in their day-to-day
work;
� Assessed the improvements in Module 1; and
� Provided recommendations for continuous improvement.
5.2 Based on this evaluation, the authors conclude that the NMLP is a helpful
program to the participants and should be continued. Recommendations to improve the
program for upcoming cohorts include:
� Improve quality and reduce duration of Module 1. Module 1 continued to
receive lower ratings than the other modules, even though the content of the
module in Phase II was different from that in Phase I.
� Align NMLP with the day-to-day management issues of the Bank and tailor
the program to Bank needs. Both NMLP participants and their supervisors
highlighted the need for aligning NMLP to the Bank's work. To increase
alignment, NMLP participants made two suggestions: (a) allocate more time to
"real world/actual" Bank/IFC case studies that cover HR management, change
management, and resource management; and (b) develop action plans that
enable participants to practice and apply NMLP skills.
� Design follow-up refresher courses. Participants as well as supervisors noted
that refresher courses are necessary in order to "provide a systematic
continuation and further deepening" of the program. A follow-up course would
also keep the support network in operation for the new managers.
� Ensure timely course delivery to new incoming managers. Supervising
managers would like the program to be available early in the manager's career
within the first six months after a new manager is assigned to his/her new
position.
� Strengthen NMLP in three specific areas. Supervising managers also
indicated that NMLP could be even more effective than it has been to date if it
would emphasize the three topics that continue to be challenging issues in the
Bank environment -- cultural diversity, conflict management (including
delivering hard news), and resource management.
32
APPENDICES
33
34
APPENDIX 1: SUMMARY OF PHASE I EVALUATION
METHODOLOGY, FINDINGS, AND RESULTS
Methodology
Phase I assessed the usefulness of the NMLP for cohorts 3 and 4 through three methods:
� Level 1 evaluations immediately following each of the three modules to assess
their quality;
� Self-assessments completed by the participants approximately six months after
Module 3; and
� Comparison of pre- and post assessments of participants by their subordinates
(one month before Module 1 and three months after Module 3) to assess changes
in managerial performance and behaviors as a result of the course.
Findings
Self-assessments coupled with pre and post subordinate assessment of the participants'
managerial performance yielded to four findings:
1. As compared to Modules 2 and 3, participants gave low ratings to Module 1 in
terms of its coverage and success in meeting its objectives;
2. Subordinates of the NMLP participants, who were surveyed before and after their
managers participated in the program, did not provide any evidence of improved
performance and behaviors by the participants as a result of their attendance;37
3. Participants who were already performing well according to their subordinates
before completing the NMLP continued to do well after the course, while
managers who had not performed well before the program did not improve as a
result of attending the NMLP; and,
4. Participants reported receiving favorable feedback on their management style
from their supervisors after attending the NMLP.
Recommendations
Based on these findings, the Phase I evaluation made the following recommendations:
a. Revise Module 1 to make considerable changes - including improvements
to the resource management component of the Module - or discontinue the
module,38 and;
37. "The "before" and "after" assessments of the new managers by their subordinates showed no
statistically significant changes in the perception of managerial performance and/or behaviors during the
time period covered by the study." (Zia, Hanson, and Gunnarsson 2004)
38. A key factor in the low ratings seen in the evaluation of Module 1 was the component on resource
management, which was the lowest rated of all Module 1 topics.
35
b. Design a systematic follow-up procedure, such as "refresher sessions."
Next Steps � Phase II
After a review of the findings, the Knowledge and Learning Board recommended that the
NMLP should be continued. The KLB suggested several improvements to the program
and requested a second evaluation.
36
APPENDIX 2A: END OF COURSE LEVEL 1 QUESTIONNAIRE �
MODULE 1
New Managers' Leadership Program
Crafting Your Management Role
Dear Participant:
The questionnaire below asks for your overall opinions about Module 1 � Crafting Your Management Role that
you have just completed. The feedback you provide is very important, as it will assist to improve the course quality.
To answer, please circle the number that best represents your response. Please print or write your comments
clearly. The information you provide is completely confidential. Thank you for your collaboration!
Background Information
1. Where do you mainly work? (Select one number)
1. IBRD
2. IFC
2. You are: (Select one number)
1. Promoted as Manager within the Bank
2. New externally recruited Manager
3. Other, specify: _________________________________________
3. What is your gender? (Select one number)
1. Female
2. Male
Module Outcomes: Relevance, Effectiveness, Potential Impact
Using the scale on the right, please rate each question below.
Not
Circle one number Low High Applicable
4. To what extent did the training achieve its 1 2 3 4 5 0
announced objectives?
5. To what extent did the training fulfill your 1 2 3 4 5 0
learning needs?
6. How would you rate the effectiveness of 1 2 3 4 5 0
this module in helping you learn from (and
with) other managers?
7. To what extent are the knowledge and skills 1 2 3 4 5 0
you have acquired useful and applicable to
your job?
8. How would you rate the overall quality of the 1 2 3 4 5 0
training?
37
9. How would you rate the quality and relevance of:
Not
Circle one number Poor Excellent Applicable
a. The materials used 1 2 3 4 5 0
b. The content of this module 1 2 3 4 5 0
c. The course's methods of delivery 1 2 3 4 5 0
(activities)
d. The pacing of the training module was: Too Slow Adequate Too Fast
1 2 3
10. To what extent was this Module effective in increasing your awareness of:
Not
Circle one number Low High Applicable
a. Your role as a WBG manager 1 2 3 4 5 0
b. Behaviors that inspire trust in others 1 2 3 4 5 0
c. Your current level of listening and inquiry 1 2 3 4 5 0
skills
d. Your current level of advocacy/ feedback 1 2 3 4 5 0
skills
e. WBG People Management policies and the 1 2 3 4 5 0
available WBG resources for People
Management
f. Multi-cultural issues at work 1 2 3 4 5 0
g. Your impact on others 1 2 3 4 5 0
Overall Comments and Recommendations
11. Please indicate which three (3) areas of knowledge/skills obtained from this module were the
most significant for you. (Please circle up to three numbers.)
a. Your WBG managerial role. f. Heightened awareness of how
b. Heightened awareness of your impact on others. attentive you are to multi-cultural
c. Methods of improving your listening/inquiry skills. issues.
d. Methods of improving your advocacy skills. g. Other __________________
e. Methods to inspire trust in others.
12. What did you find most useful about the module?
13. What did you find least useful about the module?
38
14. What are your recommendations for improving the effectiveness of the facilitation team (Lennox,
Fred, Carolyn, Allison, Vanessa, Art and Muriel)?
15. Please provide specific recommendations for improving the effectiveness of this module? What
could be done differently next time?
Thank you for completing the questionnaire.
39
APPENDIX 2B: END OF COURSE LEVEL 1 QUESTIONNAIRE �
MODULE 2
New Managers' Leadership Program
Achieving Business Results through Personal Mastery
Dear Participant:
The questionnaire below asks for your overall opinions about Module 2 � Achieving Business Results
through Personal Mastery that you have just completed. The feedback you provide is very important, as it will
assist to improve the course quality.
To answer, please circle the number that best represents your response. Please print or write your
comments clearly. The information you provide is completely confidential. Thank you for your collaboration!
Background Information
1. Where do you mainly work? (Select one number)
1. IBRD
2. IFC
2.
You are: (Select one number)
1. Promoted as Manager within the Bank
2. New externally recruited Manager
3. Other, specify: _________________________________________
3.
What is your gender? (Select one number)
1. Female
2. Male
Module Outcomes: Relevance, Effectiveness, Potential Impact
Using the scale on the right, please rate each question below.
Not
Circle one number Low High Applicable
4. To what extent did this module achieve its stated 1 2 3 4 5 0
objectives?
5. To what extent did this module fulfill your 1 2 3 4 5 0
learning needs?
6. How would you rate the effectiveness of this 1 2 3 4 5 0
module in helping you learn from (and with)
other managers?
7. To what extent are the knowledge and skills you 1 2 3 4 5 0
have acquired useful and applicable to your job?
8. How would you rate the overall quality of this 1 2 3 4 5 0
module?
9. How would you rate the quality and relevance of:
Not
Circle one number Poor Excellent Applicable
a. The materials used 1 2 3 4 5 0
b. The content of this module 1 2 3 4 5 0
c. The module's methods of delivery 1 2 3 4 5 0
(activities)
40
d. The pacing of this module was: Too Adequate Too Fast
Slow
1 2 3
10. To what extent was this Module effective in increasing your awareness of:
Not
Circle one number Low High Applicable
a. Your current and desired levels of 1 2 3 4 5 0
personal mastery
b. The clarity of your personal vision 1 2 3 4 5 0
c. The alignment between your 1 2 3 4 5 0
personal values with professional
demands
d. Your impact on others 1 2 3 4 5 0
e. What enhances your intra-personal 1 2 3 4 5 0
effectiveness
f. What impedes your intra-personal 1 2 3 4 5 0
effectiveness
Overall Comments and Recommendations
11. Please indicate which three (3) areas of knowledge/skills obtained from this module were the
most significant for you. (Please circle up to three numbers.)
d. Increasing your awareness/knowledge
a. Increasing your awareness/knowledge of your of your impact on others
current and desired levels of personal mastery e. Helping clarify what enhances your
b. Clarifying your personal vision intra-personal effectiveness
c. Aligning your personal values with professional f. Helping clarify what impedes your
demands intra-personal effectiveness
g. Other __________________
12. What did you find most useful about the module?
13. What did you find least useful about the module?
14. What are your recommendations for improving the effectiveness of the facilitation team (Anita,
Eamonn, Mindy, Paul, Guat Bee, Lennox and Allison)?
15. Please provide specific recommendations for improving the effectiveness of this module? What
could be done differently next time?
Thank you for completing the questionnaire
41
APPENDIX 2C: END OF COURSE LEVEL 1 QUESTIONNAIRE �
MODULE 3
New Managers' Leadership Program
Leading Change
Dear Participant:
The questionnaire below asks for your overall opinions about Module 3 � Leading Change that you have just
completed. The feedback you provide is very important, as it will help improve the course quality.
To answer, please circle the number that best represents your response. Please print or write your comments
clearly. The information you provide is completely confidential. Thank you for your collaboration!
Background Information
1. Where do you mainly work? (Select one number)
1. IBRD
2. IFC
2. You are: (Select one number)
1. Promoted as Manager within the Bank
2. New externally recruited Manager
3. Other, specify: _________________________________________
3. What is your gender? (Select one number)
1. Female
2. Male
Module Outcomes: Relevance, Effectiveness, Potential Impact
Using the scale on the right, please rate each question below.
Not
Circle one number Low High Applicable
4. To what extent did this Module achieve its 1 2 3 4 5 0
stated objectives?
5. To what extent did this Module fulfill your 1 2 3 4 5 0
learning needs?
6. How would you rate the effectiveness of this 1 2 3 4 5 0
Module in helping you learn from (and with) other
managers?
7. To what extent is the knowledge and skills you 1 2 3 4 5 0
have acquired useful and applicable to your
job?
8. How would you rate the overall quality of this 1 2 3 4 5 0
Module?
9. How would you rate the quality and relevance of:
Not
Circle one number Poor Excellent Applicable
a The materials used 1 2 3 4 5 0
b The content of this module 1 2 3 4 5 0
c The Module's methods of delivery 1 2 3 4 5 0
(activities)
42
d The pacing of this module was: Too Slow Adequate Too Fast
1 2 3
10. To what extent was this Module effective in increasing your awareness of:
Not
Circle one number Low High Applicable
a. Frameworks and approaches to organizational 1 2 3 4 5 0
strategy and flexibility
b. Approaches to organizational culture and team 1 2 3 4 5 0
decision making
c. Frameworks and approaches to coalition 1 2 3 4 5 0
building and partnerships
d. Approaches to leading a learning organization 1 2 3 4 5 0
Overall Comments and Recommendations
11. Please indicate which three (3) areas of knowledge/skills obtained from this module were the
most significant for you. (Please circle up to three numbers.)
a Frameworks and approaches organizational d Approaches to leading a learning
strategy and flexibility organization
b Approaches to organizational culture and team e Other __________________
decision making
c Frameworks and approaches to coalition building
and partnerships
12. What did you find most useful about the module?
13. What did you find least useful about the module?
14. What are your recommendations for improving the effectiveness of the facilitation team (Amy
Edmondson, David Upton, Frances Frei, David Garvin, Mark Moore, Tiziana Casciaro, Alan
MacCormack, Lennox and Allison)?
15. Please provide specific recommendations for improving the effectiveness of this module? What
could be done differently next time?
Thank you for completing the questionnaire
43
APPENDIX 3A: NEW MANAGERS' LEADERSHIP PROGRAM
IMPACT EVALUATION QUESTIONNAIRE � PRE-NMLP
QUESTIONNAIRE
Following are behaviors, skills and knowledge covered in the New Managers' Leadership Program. Please
provide a personal assessment of each based on your managerial skills at this time, prior to attending NMLP.
To what extent do you...
Not To a To To a To a
At Limited Some Considerable Great Not
All Extent Extent Extent Extent Applicable
a. Employ active listening skills (such as 1 2 3 4 5 0
maintaining eye contact, giving
undivided attention, or making
clarifying statements)?
b. Understand what behaviors inspire 1 2 3 4 5 0
trust--or, conversely, undermine
trust--in leaders?
c. Know how to provide effective 1 2 3 4 5 0
feedback to staff, with emphasis on
actionable improvement?
d. Know how to influence decisions that 1 2 3 4 5 0
impact you or your team?
e. Practice creating a clear vision of 1 2 3 4 5 0
what you want to accomplish as a
leader?
f. Understand the value of different 1 2 3 4 5 0
cultural perspectives?
g. Understand the impact you have on 1 2 3 4 5 0
others?
h. Pay attention to aligning your 1 2 3 4 5 0
personal values with professional
demands?
i. Align the structure and processes of 1 2 3 4 5 0
your team with the mission of the
organization?
j. Understand organizational changes 1 2 3 4 5 0
and develop effective changes within
the team?
k. Know the principles of effective team 1 2 3 4 5 0
leadership?
l. Understand the importance of 1 2 3 4 5 0
creating partnerships to achieve your
goals?
m. Understand your style of influence in 1 2 3 4 5 0
an organizational setting?
n. Create a learning environment for 1 2 3 4 5 0
your unit?
44
APPENDIX 3B: NEW MANAGERS' LEADERSHIP PROGRAM
IMPACT EVALUATION POST-NMLP QUESTIONNAIRE
1.Overall, how would you rate the New Managers' Leadership Program (NMLP) compared with
other management training in which you may have participated?
1. One of the best
Select 2. Better than others
3. Same as others
one
4. Worse than others
number
5. One of the worst
6. Can not compare
2.Overall, to what extent have you benefited from the NMLP?
1. To a great extent
Select 2. To a considerable extent
one 3. To some extent
number 4. To a limited extent
5. Not at all
3.Since your participation in NMLP, to what extent have you been able to use the various skills and
approaches you learned?
1. To a great extent
Select 2. To a considerable extent
one 3. To some extent
number 4. To a limited extent
5. Not at all
4.Following are behaviors, skills and knowledge covered in the New Managers' Leadership
Program. Please provide a personal assessment of each based on your managerial skills at this
time.
To what extent do you...
Not To a To To a To a
At Limited Some Considerable Great Not
All Extent Extent Extent Extent Applicable
a. Employ active listening skills 1 2 3 4 5 0
(such as maintaining eye
contact, giving undivided
attention, or making clarifying
statements)?
b. Understand what behaviors 1 2 3 4 5 0
inspire trust--or, conversely,
undermine trust--in leaders?
c. Know how to provide effective 1 2 3 4 5 0
feedback to staff, with
emphasis on actionable
improvement?
45
d. Know how to influence 1 2 3 4 5 0
decisions that impact you or
your team?
e. Practice creating a clear vision 1 2 3 4 5 0
of what you want to accomplish
as a leader?
f. Understand the value of 1 2 3 4 5 0
different cultural perspectives?
g. Understand the impact you 1 2 3 4 5 0
have on others?
h. Pay attention to aligning your 1 2 3 4 5 0
personal values with
professional demands?
i. Align the structure and 1 2 3 4 5 0
processes of your team with the
mission of the organization?
j. Understand organizational 1 2 3 4 5 0
changes and develop effective
changes within the team?
k. Know the principles of effective 1 2 3 4 5 0
team leadership?
l. Understand the importance of 1 2 3 4 5 0
creating partnerships to achieve
your goals?
m Understand your style of 1 2 3 4 5 0
influence in an organizational
setting?
n. Create a learning environment 1 2 3 4 5 0
for your unit?
5.To what extent did your participation in the NMLP make a difference in improving your
management skills listed below?
NMLP NMLP
NMLP made A made A
made NO SLIGHT LOT OF Not
Difference Difference Difference Applicable
a. Communicating tough issues with my staff 1 2 3 0
b. Managing stress when making tough choices 1 2 3 0
c. Resolving differences among staff and with 1 2 3 0
stakeholders
d. Encouraging my staff to attend learning events 1 2 3 0
e. Improving teamwork in my unit 1 2 3 0
f. Leading change in the Bank Group 1 2 3 0
environment
46
6.Has NMLP led you to manage in different and better Yes No
ways?
Please explain:
7.I have received favorable feedback from my supervisor and/or staff on my management abilities
since attending NMLP.
Yes No
8.To what extent did each of the following help you put into practice what you learned at NMLP?
To a To To a To a
Not Limited Some Considerable Great Not
At All Extent Extent Extent Extent Applicable
a. Staff readiness to accept 1 2 3 4 5 0
change
b. Your personal commitment 1 2 3 4 5 0
c. Support from colleagues 1 2 3 4 5 0
d. Support from others who 1 2 3 4 5 0
attended NMLP
e. Support from your supervisor/ 1 2 3 4 5 0
manager
9.To what extent did any of the following inhibit you from putting into practice what you learned at
NMLP?
To a To To a To a
Not Limited Some Considerable Great Not
At All Extent Extent Extent Extent Applicable
a. Irrelevant content of NMLP 1 2 3 4 5 0
b. Abstract / too conceptual nature 1 2 3 4 5 0
of NMLP
c. Heavy volume of my day-to-day 1 2 3 4 5 0
work
10Would you recommend the NMLP to other new managers?
Select 1. Yes
one 2. No
number 3. Not sure
If "No" or "Not Sure," please explain:
47
11Do you think that three weeks' duration of this course was:
Select 1. Too long
one 2. Too short
number 3. Just right
12What are your recommendations for improving the NMLP? (i.e., What should be done differently
next time? Are there other topics you feel should be addressed in the training?)
48
APPENDIX 4: RESPONSE RATES FOR NMLP PRE- AND POST
SELF-ASSESSMENTS
Total Number of Number of Preresponse Postresponse
Cohort Participants Preresponses Postresponses rate rate
8 27 NA 23 NA 85%
9 28 27 22 96% 78%
10 35 21 27 60% 77%
total 90 48 72 78% 80%
49
APPENDIX 5A: PRE-NMLP INTERVIEWS WITH SUPERVISORS OF
COHORTS 9 AND 10
INDIVIDUAL INTERVIEWS WITH THE REPORTING MANAGERS OF NMLP
PARTICIPANTS CONDUCTED BY HRSLO CONSULTANT:
1. What existing strengths do you see the participant bringing to their management role?
2. What 3 developmental outcomes would you like to see the participant achieve over
the course of the entire NMLP?
3. What 2 specific challenges does the current environment present and what
appropriate management skills would you like to see the participant attain to deal
with these specific challenges?
4. What type of support can you give the participant during the program?
50
APPENDIX 5B: POST-NMLP INTERVIEWS WITH SUPERVISORS
OF COHORTS 9 AND 10 ALUMNI
NMLP Participant: Supervisor:
Date of Interview: Number of contacts to get the interview:
1. Within the past 8 � 12 months, have you noticed any changes in the managerial skills and behaviors of
[participant name]?
YES___ NO___ Don't Know_____
If yes, what would these changes be?
2. When you were interviewed last year, prior to [participant's name] participation in the NMLP, you noted
several developmental outcomes that you would like to see him/her achieve as a result of the NMLP.
Here is the list of NMLP related outcomes that you identified last year (hand the list to interviewee). Please
review this list, indicate the extent to which [participant's name] has achieved each of these outcomes (by
checking the appropriate rating) and then circle the two most important development outcomes for this
manager.
Note to the interviewer: As indicated in the instructions sheet it is crucial that the skills listed below are aligned with the
course objectives and content.
EXTENT OF ACHIEVEMENT
1 2 3 4 5
Developmental Not at to a limited to some to a considerable to a great
Outcomes all extent extent extent extent
1.
2.
3.
4.
5.
6.
3. Last year you had also noted several management skills that you would like to see the participant achieve
to deal with the challenges of the work environment at that point in time.
Here is the list of NMLP related skills that you identified last year (hand the list to interviewee). As in the
previous question, please review this list, indicate the extent to which you have noticed improvement, and
circle the two most important management skills (i.e., skills that this manager needed to or needs to improve).
Note to the interviewer: Again, it is crucial that the skills listed below are aligned with the course objectives and content.
EXTENT OF IMPROVEMENT
1 2 3 4 5
Management Not at to a limited to some to a considerable to a great
Skills all extent extent extent extent
1.
2.
3.
4.
5.
6.
51
4. Has the NMLP contributed to this participant's performance in their new managerial role?
o YES. Please
explain.___________________________________________________________________
o NO. Please
explain.___________________________________________________________________
o I don't
know______________________________________________________________________
o Other_____________________________________________________________________
5. Is there anything else that you would like to add that would help us understand the value and
effectiveness of the NMLP?
52