Approach Paper
Learning in World Bank Lending
February 23, 2024
1. Background and Context
1.1 The World Bank has a comparative advantage in development knowledge.
External assessments, Independent Evaluation Group (IEG) evaluations, and client
surveys done by the World Bank and others consistently show that countries and
partners value the World Bank’s knowledge. In fact, the World Bank’s knowledge sets it
apart from other development organizations and helps it attract partners, funding, and
new business.
1.2 The World Bank aims to promote the use of the best global and country
knowledge available to inform operations. The World Bank’s knowledge contributes to
outcomes directly (by informing clients in their reform efforts) and indirectly (via World
Bank financing). Knowledge is embedded in World Bank–financed operations and is an
important part of the World Bank’s value proposition for clients. The World Bank also
learns from and synthesizes clients’ policy reforms, experiences, and outcomes.
However, credible, operationally relevant knowledge is critical to inform (i) the choice
and design of operations and (ii) course correction during project implementation.
1.3 Managing knowledge in a large organization like the World Bank is inherently
complex. The World Bank has responded in many ways over the years—for example,
through the concept of the “Knowledge Bank” (1996), the creation of the networks
(predecessors to the Global Practices [GPs]) in 1997, the 2010 knowledge strategy, the
2014 organizational reforms that created the GPs to strengthen global knowledge flows,
the 2017 Knowledge Management Action Plan and the creation of a central knowledge
management team led by a director (since disbanded), and the 2019 and 2020
adjustments to the reporting lines intended to ensure that global knowledge serves
country programs. More recently, the World Bank developed the Strategic Framework
for Knowledge (SFK) in 2021 (World Bank Group 2021). The SFK’s objectives are to help
the World Bank Group “realize its vision to deliver the best knowledge for development
solutions provided to client countries and for the global development agenda” (World
Bank Group 2021, 25) and to “move from being impactful most of the time to being
impactful all the time” (World Bank Group 2021, 30). The SFK identifies three pillars for
actions to improve the Bank Group’s knowledge contributions: (i) systems for
prioritization, including learning from projects, quality assurance, knowledge
management, and so on; (ii) incentives for staff (such as management signals, attention,
and accountability), performance and promotion criteria, and encouragement for staff to
1
engage in knowledge flows; and (iii) staff’s human capital, including hiring practices
and skill upgrading.
1.4 The new Knowledge Compact for Action is the World Bank’s latest approach to
articulate its vision and approach to knowledge. Still in draft form at the time of writing,
February 2024, the Knowledge Compact for Action aims to transform the World Bank’s
approach to knowledge, positioning it as a “21st century Knowledge Bank,” and to
create a dynamic knowledge ecosystem that facilitates knowledge flows within the
World Bank and externally. The draft proposes changes to knowledge products,
processes and systems, partnerships, training, and capacity building. It also discusses
the need for new or improved internal selectivity processes, knowledge budgeting
practices, and incentives for staff to generate, curate, and share knowledge. IEG is
engaged in discussions with the office of the senior managing director, which will help
inform the context for this evaluation.
Past Independent Evaluation Group Evaluations and Research
1.5 Past IEG evaluations have covered different aspects of knowledge. IEG’s two
previous evaluations of learning in World Bank operations (World Bank 2014, 2015)
found that for most staff, informal learning and tacit accumulation of knowledge
predominate and are driven by incentives, mindsets, and aspects of organizational
culture, such as group norms and diversity of teams. IEG’s evaluation Behind the Mirror:
A Report on the Self-Evaluation Systems of the World Bank Group (World Bank 2016) stressed
the role of staff values, motivations, and incentives regarding results measurement and
using self-evaluation. IEG’s evaluation Knowledge Flow and Collaboration under the World
Bank’s New Operating Model (World Bank 2019a) examined the relationship between
knowledge and the operating model, highlighting that although some GPs had coherent
and systematic approaches to managing and investing in knowledge, others did not, and
that differences in managerial signals and incentives could explain much of the observed
variation. Enhancing the Effectiveness of the World Bank’s Global Footprint (World Bank
2022) found that locating staff in country offices improves client relationships and brings
other benefits but poses challenges to a global knowledge flow because many of the
World Bank’s knowledge processes center on headquarters. IEG’s evaluations Data for
Development: An Evaluation of World Bank Support for Data and Statistical Capacity (World
Bank 2017a), World Bank Group Engagement in Upper-Middle-Income Countries: Evidence
from IEG Evaluations (World Bank 2017b), and The World’s Bank: An Evaluation of the
World Bank Group’s Global Convening (World Bank 2020c) confirmed that data and
knowledge are core sources of comparative advantage. Appendix B lists past IEG
evaluations’ key findings touching on knowledge.
2
1.6 Projects informed by knowledge inputs—both explicit and tacit—tend to
perform better. Econometric studies using IEG project ratings have linked the quality
and stability of the project’s task team leader (TTL) to project performance (Denizer,
Kaufmann, and Kraay 2013; Geli, Kraay, and Nobakht 2014; Moll, Geli, and Saavedra
2015). Ashton et al. (2021) reported that project design, including the estimated value
added of design staff and the presence of prior analytic work, predicts project success. A
key determinant of staff’s contribution is their experience with previous World Bank
projects, indicating the contribution that knowledgeable staff make. Foundational
knowledge work during project preparation matters for quality at entry because it not
only enhances World Bank’s understanding of local policy, capacity, and institutions but
also allows it to build trusting relationships and fine-tune procurement arrangements
(World Bank 2019b). Time pressures during preparation have a statistical association
with projects’ quality at entry, presumably because of less time to invest in knowledge
work (World Bank 2020b).
1.7 This evaluation will assess how knowledge interacts with World Bank financing
projects. It will build on the following three evaluations of learning and knowledge
flows in the World Bank: Learning and Results in World Bank Operations: How the Bank
Learns (World Bank 2014), Learning and Results in World Bank Operations: Toward a New
Learning Strategy (World Bank 2015), and Knowledge Flow and Collaboration under the
World Bank’s New Operating Model (World Bank 2019a). Whereas previous research and
evaluations (cited in chapter 1) established the importance of knowledge for Bank
performance, including positive links between knowledge and outcomes, this evaluation
will identify, in a granular manner, the types of knowledge the World Bank uses to
design and implement successful projects, how and when it uses the knowledge, and the
enabling, or explanatory, factors for use.
2. Purpose, Objectives, and Audience
2.1 This evaluation’s broad purpose is to assist the ongoing efforts of the World
Bank and the Committee on Development Effectiveness to improve learning in World
Bank–financed operations. The purpose and scope are informed by the SFK and the new
Knowledge Compact for Action, as well as interviews and engagements with managers
and directors from Regions, Practice Groups, and the Development Economics Vice
Presidency. This evaluation is timely and relevant because it is aligned with
management’s efforts to renew the World Bank’s leadership in development knowledge
and develop the new Knowledge Compass.
2.2 This evaluation will focus on identifying knowledge enablers that promote better
learning across the World Bank. Conversely, the evaluation will also identify barriers or
practices that might inhibit learning from knowledge. To do so, the evaluation will seek
3
to (i) identify what types of knowledge inputs were used (including how, when, and
why) to inform the design and implementation of World Bank–financed operations; (ii)
survey current knowledge management practices across operational units to identify
processes and practices that enable learning; (iii) review which of these could be
continued, reinforced, adjusted, or dropped (and under which conditions); and (iv)
provide recommendations to help operational units achieve more consistent learning in
operations. IEG plans for this evaluation to be the first of two evaluations on World
Bank knowledge, with the scope and objectives of the second evaluation to be decided.
2.3 IEG expects a wide audience for this evaluation given the strong interest in the
topic from inside and outside the organization. IEG added this evaluation to its work
program because of (i) the articulation of the Knowledge Compact for Action, (ii) the
strong interest from the Committee on Development Effectiveness in the knowledge
agenda, and (iii) demand from parts of World Bank management to understand what
more could be done to learn from World Bank–financed operations, which was one of
the SFK’s key challenges. IEG is particularly keen for this evaluation to reach and inspire
mid-level management in GPs and Regions, and the staff who support them, for
example, in development effectiveness, quality assurance, operations management, and
knowledge management roles. IEG also expects interest in the evaluation from World
Bank clients, partners, and observers.
3. Evaluation Definitions, Questions, and Scope
Definitions
3.1 The terms knowledge and learning are ambiguous and have various meanings
depending on the context. This evaluation will use the following working definition of
these terms:
• Knowledge input is the different types of knowledge available to project teams
that are used to inform the project design and implementation.
• Knowledge management processes include the broad set of activities conducted
by everyone in the organization to hold, arrange, apply, and expand knowledge
for the organization. They can be both formal processes set up by the
organization and informal ones used by teams and individuals.
• Knowledge enablers include opportunities, capacities, processes, and
motivations that are conducive to knowledge generation and use.
• Learning is the application of different types of knowledge to inform the design
and implementation of World Bank–financed operations.
4
Evaluation Questions
3.2 The evaluation will be guided by the high-level question, How can the World
Bank improve learning in its financing operations? This question aligns with SFK goals.
Three subquestions will inform the evaluative work.
Evaluation question 1: To what extent do different types of knowledge inputs inform
the design and implementation of World Bank financing operations?
3.3 This analysis will seek to map the types of knowledge inputs that went into a
sample of World Bank projects through the project cycle and determine the enabling
factors (knowledge enablers). The analysis will assess the extent to which the formal
processes associated with the project cycle, such as peer review, Mid-Term Reviews, and
TTL turnover, create spaces for learning. It will also assess informal processes, such as
exchange of tacit knowledge among staff. The evaluation will examine the extent to
which knowledge enablers, such as capacity, opportunities, and motivations, affect
learning. Examples of knowledge enablers include trust funds, knowledge partnerships,
and artificial intelligence. IEG expects this analysis to generate a wealth of granular
findings and hypotheses about what works under what conditions. For example, the
analysis will consider differences across lending instruments and countries’ stages of
development (further details are provided in appendix A). To arrive at a limited set of
robust findings, the evaluation will test the most important of these hypotheses using
methods such as quantitative techniques, text analytics, and direct observation, among
others.
Evaluation question 2: To what extent have different operational units put effective
knowledge enablers in place?
3.4 Having identified the types of knowledge that typically benefit World Bank
projects and their moments of use, IEG will survey current practices across operational
departments and assess where those knowledge inputs are being systematically enabled,
where they are not, and why. This work will consider the role of the GPs, in particular
the global units, and the role of the Regions in enabling learning.
Evaluation question 3: What would it take to have more consistent learning across
World Bank financing operations?
3.5 Having mapped the current knowledge enablers across the World Bank, the
evaluation will then be able to identify which of these could be continued, reinforced, or
adjusted and under which conditions effective practices could be scaled and
mainstreamed. The evaluation will also assess the experiences in taking forward IEG’s
earlier recommendations pertaining to knowledge. This will lead to conclusions and
5
recommendations on how to realize the SFK’s vision of consistent knowledge in World
Bank financing.
Scope
3.6 The focus of this evaluation is on learning in World Bank financing operations—
specifically, how the task teams and management in the GPs and Regions apply and
expand the knowledge available in lending operations. The evaluation will review
projects across all Regions and all financing instruments (investment project financing,
development policy financing, and Program-for-Results financing). Both country
knowledge and sector knowledge will be covered. Country knowledge is specific to the
development challenges and opportunities in the country—for example, related to
stakeholders, political economy, or the implementing agency. Sector knowledge,
sometimes referred to as global knowledge, is specific to the issue, sector, objective, and
key intervention—for example, about intervention design. The scope includes both
formal and informal knowledge processes and both tacit and explicit knowledge. Peer
reviews, advisory services and analytics, and other diagnostics are all part of the scope.
3.7 The evaluation period is 2014–23. IEG will review learning in operations
approved after 2014 to coincide with the creation of the GPs. The review of knowledge
enablers in operational units will focus on current practices.
3.8 This evaluation’s main focus is internal—on learning by World Bank staff and
management. The evaluation covers a wide range of knowledge inputs, including
learning from and with clients, partners, and academia in the context of financing
operations. That said, client learning and capacity building more broadly are outside the
evaluation’s scope because these are large topics with important outcomes that would
merit their own review. The evaluation covers only the World Bank rather than the
entire Bank Group for two reasons: (i) to keep the scope focused and manageable and
allow for detailed findings specific to different World Bank financing instruments,
Regions, and sectors and (ii) because it is unlikely that there would be synergies from
reviewing World Bank, International Finance Corporation, and Multilateral Investment
Guarantee Agency knowledge issues jointly.
4. Evaluation Design
Conceptual Framework
4.1 The team will be guided by the conceptual framework shown in figure 4.1, in
which different types of knowledge are generated or used differently at different
moments in time. The conceptual framework postulates that the generation and use of
knowledge inputs is influenced by factors such as the capacity of the organization,
6
teams, and individuals, including their skills and funding, along with the opportunities
they have to use knowledge in relevant processes, and their motivations to use it.
4.2 As shown in figure 4.1, knowledge can be explicit or tacit. Explicit knowledge is
in tangible forms (reports, books, and databases) and in writing. Conversely, tacit
knowledge is gained through experience and engagement with peers relying on
relationships to some extent.
4.3 The sector and country contexts are also relevant for project design and
implementation. Knowledge inputs are likely to be far more critical in (i) complex
settings (for example, in countries affected by fragility, conflict, and violence and those
undergoing political instability) and (ii) projects that are multisectoral or experiment
with novel approaches. Similarly, new client relationships and new sectoral issues
present more challenges than clients and issues that are familiar to the World Bank. The
learning needs also differ by lending instrument with development policy financing, for
example, underpinned by analytic work that is different from the diagnostics that
inform investment project financing. This evaluation will aim to capture these nuances.
4.4 Knowledge inputs are used for various purposes by a diverse set of staff in
financing operations. Knowledge can be used for strategic purposes to build a business
case for a project or to motivate a decision. Knowledge can also be used for instrumental
purposes—for example, for solving a problem, such as how to design a specific project
component. Instrumental use requires that task teams have access to the right
knowledge inputs at the right time, often in the design stage. Some knowledge use is
more conceptual in nature, informing thinking and promoting shared understanding of
an issue, and timing is less important.
Learning in financing is tied to the project cycle. The World Bank has several structured
entry points for learning during the project cycle, including quality at entry reviews,
Concept Note discussions during appraisals, decision meetings for approval, Mid-Term
Reviews at the time of implementation, regular Implementation Status and Results
Reports, and lesson learning through the Implementation Completion and Results
Report at closing.
7
Figure 4.1. Conceptual Framework for Learning in World Bank Financing
Source: Independent Evaluation Group.
8
4.5 Many factors—internal and external to the World Bank—influence learning in
financing. Capacity is shaped by the ability to generate knowledge, undertake analytics,
and reflect on what works and what does not. These are a function of funding, time, and
staff capacity—that is, the knowledge the organization holds in terms of intellectual
capital represented by its staff (including skills). Opportunities in the form of processes
and practices create space for learning and can be World Bank–wide centers of expertise
located in global units or trust-funded programs, or practices created by GPs and teams,
including mechanisms to codify tacit knowledge, handover notes, checklists, guidance
documents, knowledge packages, and mechanisms specific to a GP, such as the “Ask
Water” service. These can also be external, such as partnerships for learning with the
clients. Motivations are more intrinsic with staff being driven to learn from past projects,
global best practices, and so on. Some of the factors are outside the control of the World
Bank, such as the political, economic, and social contexts affecting operations, including
shocks, elections, and so on. At the heart of this evaluation is the assessment of how
these factors influence learning in lending for different instruments, at different points in
the project cycle and under different sector and country contexts.
4.6 This evaluation’s concepts align with the pillars of the Knowledge Compact for
Action. For example, explicit knowledge inputs referenced in figure 4.1 include what the
Knowledge Compact for Action refers to as “upgraded knowledge products” and
“extended core ASA [advisory services and analytics].” The opportunities and capacity
categories in figure 4.1 cover most of the other action areas of the Knowledge Compact
for Action, such as training and capacity building for staff and clients, knowledge
partnerships, and knowledge processes and systems.
Methods
4.7 The evaluation will use a mixed methods phased approach. Figure 4.2
summarizes the different techniques that will be leveraged for each evaluation question.
Appendix A details the methodological approach.
4.8 In phase 1, IEG will use case-based methods to review a stratified random
sample of World Bank projects approved over 2014–23. The sample of approximately 35
project case studies will include investment project financing, development policy
financing, Program-for-Results, and Multiphase Programmatic Approach projects to
allow for generalization by type of financing instrument. The sample will include both
active and closed projects to allow for analysis of knowledge inputs across the project
cycle. The sample will span all Regions and many sectors to ensure sufficient variety,
but IEG does not plan to disaggregate the case analysis by Region or sector. IEG will use
this case analysis to generate granular hypotheses about what types of knowledge
9
inputs are used, in what stage of the project cycle, how they are used, how this depends
on the financing instrument, and what factors enable learning.
4.9 In phase 2, IEG will test the most important of the hypotheses from phase 1. It
will use quantitative methods to add depth and robustness to the phase 1 findings. It
will use interviews and surveys to map out knowledge enablers across departments. It
will triangulate across all the various qualitative and quantitative methods to test
hypotheses and arrive at robust findings.
4.10 The team will extract and use data from different sources for each method. For
case studies, the team will review project records, including project documents, meeting
minutes, comment matrices, advisory services and analytics, and other knowledge
inputs recorded in the Operations Portal, and conduct semistructured interviews with
project teams and TTLs. For quantitative analysis, the team will leverage project-level
data from the World Bank’s internal systems, including, for example, databases of vetted
and actual peer reviewers, data on team composition and roles, and data on TTL
transitions and overlaps between successive TTLs (appendix A provides further details).
10
Figure 4.2. Methodological Design
Source: Independent Evaluation Group.
Note: ASA = advisory services and analytics; DEC = Development Economics Vice Presidency; DIME = Development Impact
Evaluation; EQ = evaluation question; GP = Global Practice; ML = machine learning; OPCS = Operations Policy and
Country Services; PCN = project Concept Note; TTL = task team leader.
11
Design Strengths and Limitations
4.11 The evaluation takes a systematic, sequenced approach, where qualitative and
quantitative analyses inform each other. Moreover, given the breadth of the topic, the
phased approach helps narrow the scope based on findings. The detailed evaluation
design will help ensure a level of granularity in our findings and recommendations. The
key challenge for the evaluation design is that knowledge is an intangible topic where it
can be challenging to establish a solid evidence base. Several factors contribute to this
challenge: there is not a clear definition for knowledge, and common terms are often
perceived in different ways by different stakeholders. It is hard to assess knowledge
quality and influence and to link knowledge inputs to projects’ outcomes.
5. Quality Assurance Process
5.1 The evaluation will follow IEG’s standard quality assurance process. The
evaluation will be peer-reviewed by Dr. Ruth Levine (former chief executive officer of
IDinsight, internationally recognized leader in impact evaluation, development finance,
and organizational learning), Ellen Goldstein (international development consultant and
former World Bank country director), and an additional reviewer to be identified. A
previous version of this Approach Paper was peer-reviewed by Dr. Ruth Levine, Ellen
Goldstein, and Ajay Narayanan (former manager, Group Internal Audit).
6. Expected Outputs
6.1 This evaluation is engaging closely with management’s efforts to develop the
new Knowledge Compact for Action. IEG will use engagements with operational
counterparts and the senior managing director’s office to inform the evaluation’s design
and execution, seek alignment with management’s ongoing work to develop the
Knowledge Compact for Action, explore joint work with operations, help the evaluation
land well, and support dissemination and implementation. IEG will seek to engage
counterparts from all Regions, GPs, the office of the senior managing director, and the
Operations Policy and Country Services.
6.2 Outputs. The evaluation expects to complement the standard evaluation report
with brief notes or audiovisuals on findings in key areas for easy absorption.
7. Resources
7.1 Team. The team will be led by Rasmus Heltberg (lead evaluator) and composed
of Pascal Saura (senior knowledge management officer, providing cross-support), Stuti
Sachdeva (evaluation officer), Harsh Anuj (data scientist), Farhana Sonia Ahmad
(consultant), Gaby Loibl (program assistant), and Maximillian Ashwill (developmental
12
editor). Maan Roanna Sayes will help with designing the graphics. Onno Ruhl (retired
World Bank staff) and Thomas Delahais (associate, Quadrant Conseil) are external
advisors to the team.
7.2 Oversight and timeline. Galina Sotirova (manager), Theo David Thomas
(director), and Sabine Bernabe (Director-General, Evaluation) will oversee the report.
The completed evaluation is scheduled for submission to the Committee on
Development Effectiveness in the first quarter of fiscal year 2025. The report has an
estimated budget of US$750,000.
13
Bibliography
Ashton, Louise, Jed Friedman, Diana Goldemberg, Mustafa Zakir Hussain, Thomas Kenyon,
Akib Khan, and Mo Zhou. 2021. “A Puzzle with Missing Pieces: Explaining the
Effectiveness of World Bank Development Projects.” Policy Research Working Paper
9884, World Bank, Washington, DC.
Denizer, Cevdet, Daniel Kaufmann, and Aart Kraay. 2013. “Good Countries or Good Projects?
Macro and Micro Correlates of World Bank Project Performance.” Journal of Development
Economics 105 (November): 288–302.
Geli, Patricia, Aart Kraay, and Hoveida Nobakht. 2014. “Predicting World Bank Project Outcome
Ratings.” Policy Research Working Paper 7001, World Bank, Washington, DC.
Knack, Stephen, Bradley C. Parks, Ani Harutyunyan, and Matthew DiLorenzo. 2020. “How Does
the World Bank Influence the Development Policy Priorities of Low-Income and Lower-
Middle Income Countries?” Policy Research Working Paper 9225, World Bank,
Washington, DC.
Moll, Peter, Patricia Geli, and Pablo Saavedra. 2015. “Correlates of Success in World Bank
Development Policy Lending.” Policy Research Working Paper 7181, World Bank,
Washington, DC.
Raimondo, Estelle. 2016. “What Difference Does Good Monitoring and Evaluation Make to
World Bank Project Performance?” Policy Research Working Paper 7726, World Bank,
Washington, DC.
World Bank. 2010. “Transforming the Bank’s Knowledge Agenda: A Framework for Action.”
Board Report 52989, World Bank, Washington, DC.
World Bank. 2014. Learning and Results in World Bank Operations: How the Bank Learns.
Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2015. Learning and Results in World Bank Operations: Toward a New Learning Strategy .
Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank
Group. Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2017a. Data for Development: An Evaluation of World Bank Support for Data and
Statistical Capacity. Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2017b. World Bank Group Engagement in Upper-Middle-Income Countries: Evidence from
IEG Evaluations. Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2019a. Knowledge Flow and Collaboration under the World Bank’s New Operating Model.
Independent Evaluation Group. Washington, DC: World Bank.
14
World Bank. 2019b. Results and Performance of the World Bank Group 2018. Independent Evaluation
Group. Washington, DC: World Bank.
https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/rap2018.pdf.
World Bank. 2020a. Management Action Record Reform: IEG’s Validation Report. Independent
Evaluation Group. Washington, DC: World Bank.
World Bank. 2020b. Results and Performance of the World Bank Group 2020. Independent Evaluation
Group. Washington, DC: World Bank.
World Bank. 2020c. The World’s Bank: An Evaluation of the World Bank Group’s Global Convening .
Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2022. Enhancing the Effectiveness of the World Bank’s Global Footprint. Independent
Evaluation Group. Washington, DC: World Bank.
World Bank Group. 2018. “Sustainable Financing for Sustainable Development: World Bank
Group Capital Package Proposal.” Prepared for the Development Committee Meeting,
April 21, 2018, DC2018-0002/P, World Bank Group, Washington, DC.
World Bank Group. 2021. Realizing the World Bank Group’s Knowledge Potential for Effective
Development Solutions: A Strategic Framework. Washington, DC: World Bank Group.
World Bank Group. 2023. “Ending Poverty on a Livable Planet: Report to Governors on World
Bank Evolution.” Prepared for the Development Committee Meeting, October 12, 2023,
DC2023-0004, World Bank Group, Washington, DC.
15
Appendix A. Methods
The evaluation will use a mixed methods phased approach. Initial phases will consist of
inductive and exploratory analysis aimed at generating hypotheses. These hypotheses
will be tested (deductively) through additional quantitative and qualitative analyses in
subsequent phases.
Approach to evaluation question 1: To what extent do different types of knowledge
inputs inform the design and implementation of World Bank financing operations?
This question will be answered in two phases.
Phase 1 will allow us to establish hypotheses on what types of knowledge inputs inform
World Bank–financed operations, to what extent, in what context, and why. The most
important of these hypotheses will then be tested and refined using mixed qualitative
and quantitative methods.
In phase 1, the evaluation will review a stratified random sample of approximately 35
World Bank projects for the period 2014–23. The team will ensure that the sample
includes projects in different sectors covering all Practice Groups, instruments
(investment project financing, development policy financing, and Program-for-Results),
country capacity, and project complexity (using intentional oversampling as needed).
The team will review both active and closed projects to allow for better analysis of
knowledge inputs across the project cycle and repeater projects to ensure that the
lessons learned from one project are used to inform subsequent World Bank
engagements. Evidence will come from data and documents in the Operations Portal,
including project documents, advisory services and analytics, Implementation Status
and Results Report ratings, Mid-Term Reviews, meeting minutes, and semistructured
interviews with project teams.
The evaluation will review the types of knowledge inputs, explicit or tacit, that inform
projects during preparation and implementation. In particular, the evaluation will focus
on how teams use these various types of knowledge inputs to inform project design,
quality at entry, Mid-Term Review, and restructuring. The evaluation will review to
what extent various knowledge inputs have influenced project design, for example
leading to simpler and clearer operations; to what extent knowledge inputs have
influenced Mid-Term Reviews and decisions to restructure projects or otherwise
improved implementation; and whether knowledge inputs have facilitated stakeholder
engagement over the course of the project cycle.
Different types of tacit and explicit knowledge inputs can inform project design and
implementation in several ways. The Independent Evaluation Group plans to assess
16
whether the knowledge input was actionable; in what ways the knowledge input
changed the task team’s understanding; what the task team did differently because of
the knowledge input; and to what extent the knowledge input resulted in a meaningful
change in the project design, its implementation progress, and its outcomes.
The analysis will cover formal knowledge processes, such as peer review and use of
reports and other analytics; informal knowledge processes, such as the exchange of tacit
knowledge between task teams and individuals; and the conditions and processes that
enable knowledge flow. To do this, the analysis will take the following steps:
• Develop granular findings for each case (each case being a World Bank financing
project), by triangulating between systematic document review and interviews
on what knowledge inputs were used and how learning occurred.
• Develop hypotheses across the sample of projects to identify common patterns
and potential enablers or hinderances to learning in lending. The team will
systematically review and compare case studies, identify key patterns, and
develop hypotheses.
For phase 1, the team has developed an evaluation tool to help answer evaluation
question 1. The tool captures information on types of knowledge inputted during formal
processes (including project Concept Note, decision meetings, use of analytics, and so
on) through cross-support and via informal channels (including insights from peers and
global leads). To further understand the variety of knowledge inputs used, it
disaggregates explicit knowledge into data, analytics, impact evaluations, World Bank
internal research, academic research, learning with partners, and so on, and gathers
insights on the extent to which lessons from the past and ongoing projects are used to
inform project design and implementation. The tool also aims to capture the extent to
which information is arranged (refer to approach for evaluation question 2 in this
appendix).
In phase 2, the team will leverage other methods and techniques, such as quantitative
analysis, machine learning, surveys, key informant interviews, and direct observation to
test and refine the findings and hypotheses from phase 1. Examples include quantitative
analysis of choice of peer reviewers, task team leader turnover rates, and project
document references based on data in the World Bank’s operational systems.
The team’s evaluation tool and broader approach to evaluation question 1 were
designed based on a pilot review of a sample of closed and ongoing education projects.
The findings from the pilot demonstrated that the proposed approach is viable. The
review of available project evidence (Concept Notes, meeting notes, project documents,
peer-review notes, and so on) allowed a broad mapping of knowledge inputs that
17
influenced project design and quality at entry of projects. The pilot found that project
teams had been able to access global technical knowledge via peer reviewers and
communities of practice or similar mechanisms and had been able to access country
knowledge via Country Management Units and country office staff. The pilot also
confirmed that document reviews need to be complemented with interviews with
project teams to arrive at sufficient information.
Approach to evaluation question 2: To what extent have different operational units put
effective knowledge enablers in place?
After identifying the types of knowledge inputs that typically benefit World Bank
projects, the team will survey current practices across operational units to assess to what
extent and how units enable robust knowledge inputs.
The team will then map out existing or missing knowledge management processes
across World Bank operational units in the Global Practices and Regions that provide
knowledge inputs across the dimensions of type (tacit, explicit, country, sector, or
operational) and audience (task teams, Global Practices, and Country Management Unit
management). The team will focus on knowledge priorities, partnerships, and systems
and capacities (including Global Solutions Groups, communities of practice, and other
initiatives designed to support learning in lending). To complete and validate the
mapping, the team will use focused surveys of knowledge management and learning
specialists and key informant interviews. The expected output is a comprehensive
mapping of knowledge processes and enablers and a stylized representation of
knowledge management maturity across the World Bank.
Conceptual Model for Knowledge
Underpinning the team’s examination of knowledge inputs is a simple conceptual
model according to which knowledge in organizations can be managed both as an input
(or stock) and as a process (or flow). The following conceptual model is adapted from
Wiig (1993), with insights from McInerney and Koenig (2011)1 and Mohajan (2017). In
this model, knowledge management consists of holding, arranging, applying, and
expanding knowledge for the organization (figure A.1). Knowledge is embedded in
people (staff, consultants, and experts inside and outside the organization)—what we
define as tacit—and in knowledge assets (files, books, databases, and so on)—what we
define as explicit knowledge in this evaluation.
1The model is more specifically derived from The Stages of KM (Knowledge Management) Development section in
chapter 1 and the Knowledge as Resource and Process section in chapter 3 (McInerney and Koenig 2011).
18
Figure A.1. Stages of Knowledge Management in Organizations
Knowledge as a process
Apply Expand
Arrange Hold
Knowledge as an input
Sources: Independent Evaluation Group; McInerney and Koenig 2011; Mohajan 2017; Wiig 1993.
Managing Knowledge as an Input
• Holding: Understanding the stock of knowledge that the organization holds, its
intellectual capital, through its staff and consultants (tacit) and through the entire
body of analytic work accessible to the organization (explicit). The stock of
knowledge is typically constituted through human resource management and
records of research and analytic and advisory activities.
• Arranging: Making specific parts of this knowledge ready to use by way of
organizing it. This is typically achieved through information technology systems
(including document and expertise management tools, collaboration tools,
taxonomies, search engines, and so on) and operational units (including business
units, professional family mappings, task teams, communities of practice, and
task forces, among others). In our conceptual framework (figure 4.1), these are
captured under knowledge enablers because these organizational practices make
knowledge relatively more accessible and easier to use.
Managing Knowledge as a Process
• Applying: Retrieving and applying specific parts of this knowledge where and
when needed for decision-making and day-to-day operations. This is about
connecting questions and answers and challenges with just-in-time solutions.
This is typically achieved through operational processes and mechanisms
19
designed to embed or inject just-in-time knowledge into the activities of the
organization, particularly financing operations. Checklists, peer reviews,
decision meetings, quality enhancement reviews, safe space meetings, and Mid-
Term Reviews are all examples of processes where knowledge is applied. These
processes also often contribute to expanding knowledge.
• Expanding: Growing the stock of knowledge available to the organization. New
knowledge is generated by the feedback loops that occur while applying
knowledge and through stand-alone, deliberate, and formal efforts to import
new knowledge into the organization. New knowledge may come from feedback
loops that recombine, synthetize, or aggregate existing knowledge (for example,
via all processes listed in the Managing Knowledge as a Process section). New
knowledge may also be created via external consultation, research programs,
analytic and advisory activities, self-evaluation, independent evaluation,
training, and learning programs.
Important processes for applying and expanding knowledge in the context of World
Bank financing include the following:
• Applying advisory services and analytics
• Convening peer reviews
• Using other aspects of quality assurance and contestability
• Using cross-support
• Incorporating sector-specific knowledge and learning mechanisms
• Leveraging advice and inputs provided by global leads, Global Solutions
Groups, communities of practice, and so on
• Creating and using checklists, guidance documents, and handover notes
• Organizing training
Approach to evaluation question 3: What would it take to have more consistent
learning across World Bank financing operations?
Having identified which knowledge management processes are effective for which type
of operations and under what conditions, the team could recommend which of these
should be continued, reinforced, or adjusted.
Although the methods will be refined once the answer to evaluation question 1 is
available, this part of the evaluation will rely on a triangulation and synthesis of existing
20
literature on knowledge management maturity levels, results from surveys and
interviews conducted for evaluation question 2, and so on.
References
McInerney, Claire R., and Michael E. D. Koenig. 2011. Knowledge Management (KM) Processes in
Organizations: Theoretical Foundations and Practice. New York: Springer.
Mohajan, Haradhan Kumar. 2017. “The Impact of Knowledge Management Models for the
Development of Organizations.” Journal of Environmental Treatment Techniques 5 (1): 12–
33.
Wiig, Karl M. 1993. Knowledge Management Foundations: Thinking about Thinking —How People and
Organizations Create, Represent, and Use Knowledge. Arlington: Schema Press.
21
Appendix B. Past Independent Evaluation Group Evaluations on
Knowledge
Table B.1. Independent Evaluation Group Evaluations on Knowledge and Relevant
Findings
Title Year Scope Relevant Findings
Learning and Results in 2014 These reports covered World Bank staff often rely on informal learning and
World Bank and learning that takes place gradual accumulation of tacit knowledge. Such
Operations: How the 2015 through World Bank learning and knowledge are based on observing and
Bank Learns projects. copying the behavior of others in the group. They
Learning and Results in depend on mindsets, group effects, and institutional
World Bank incentives. Staff value mentoring and learning from
Operations: Toward a peers. Therefore, the World Bank should focus on
New Learning Strategy making better use of informal learning and tacit
knowledge, and an updated strategy for learning
and knowledge sharing might be helpful in this
regard.
Behind the Mirror: A 2016 The report covered the The mandatory self-evaluation systems are seldom
Report on the Self- World Bank Group’s self- used for organizational learning. ICRs are seen as not
Evaluation Systems of evaluation instruments, useful and provide only generic lessons. Staff
the World Bank Group including ICRs. operational knowledge often comes from tacit
sources (which is insufficient because weaknesses in
documenting lessons and overreliance on personal
connections can lead to loss of important
knowledge). Self-evaluation systems do not exploit
dialogue and tacit knowledge formats to foster
operational learning. Self-evaluation systems would
benefit by being more flexible and geared toward
socializing learning.
World Bank Group 2017 This synthesis report The Bank Group’s analytic and advisory work has
Engagement in Upper- covered the outcomes been key in supporting reforms in upper-middle-
Middle-Income and lessons from the Bank income countries and valued by country
Countries: Evidence Group’s work in upper- stakeholders. The analytic and advisory work also
from IEG Evaluations middle-income countries. shapes the quality of the World Bank’s assistance,
particularly in quality at entry of development policy
financing and during crises. However, there is little
assessment of the outcomes of knowledge services,
and the potential for South-South knowledge
exchange has been underused.
Knowledge Flow and 2019 The evaluation covered Some GPs have coherent and systematic approaches
Collaboration under how well the World Bank’s to managing and investing in knowledge; others less
the World Bank’s New post-2014 operating so. Some GPs focus on learning by doing and tacit
Operating Model model stimulates knowledge flow to support operations but do not
knowledge flow and emphasize generating or curating knowledge,
enhances collaboration to innovation, client training, or global thought
deliver multisector and leadership. Differences in GPs’ attention to
multiservice tasks to knowledge often reflect the availability of trust funds
clients. and leadership support. Contestability in quality
assurance is uneven. The mechanisms designed to
pursue knowledge excellence have been met with
mixed results. Global leads have unclear roles and
unfunded mandates. A few GPs made their Global
22
Title Year Scope Relevant Findings
Solutions Groups work largely as intended, whereas
others recast or disbanded the model.
The World Bank should focus more on incentives,
culture, and collaboration mechanisms than on
structure. Incentives to enhance knowledge flow
could include senior management support for
knowledge excellence; metrics for knowledge
uptake, impact, quality, and influence; more
contestability in quality assurance; and nimbler
budgeting arrangements.
The World’s Bank: An 2020 The evaluation covered The Bank Group’s knowledge is key to its global role.
Evaluation of the which global issues the The Bank Group’s convening power as an
World Bank Group’s Bank Group convenes on, independent generator and broker of global
Global Convening the factors that drive its knowledge allows it to inform policy makers and
convening choices, and take a lead role in setting the agenda for global
the determinants of its discussions on development.
convening effectiveness.
Enhancing the 2022 Among other things, this An expected benefit of decentralization is that it
Effectiveness of the evaluation covered how helps integrate local knowledge into the World
World Bank’s Global staff decentralization Bank’s global knowledge network and informs World
Footprint affects the World Bank’s Bank strategies and operation, but decentralization
knowledge flow. also poses challenges to a global knowledge flow
when staff are away from headquarters for extended
periods. Knowledge Management is often
headquarters focused, and formal knowledge from
the field is less appreciated and used globally. The
evaluation recommended that the World Bank take
measures to safeguard knowledge flow and the
World Bank’s global nature.
Results and 2022 Among other things, the There is a good match between ASA topics and
Performance of the report presents a government policies, and the World Bank often uses
World Bank Group qualitative analysis of the ASA in its policy dialogue. However, there is limited
2022 use of ASA based on evidence on governments’ ownership and use of
Completion and Learning ASA findings and on the use of ASA in World Bank
Review Validationsa for 50 programs and projects, and what evidence there is
countries. shows mixed effectiveness, for example, because of
issues with ASAs’ timing and dissemination.
Sources: Independent Evaluation Group; World Bank 2014, 2015, 2016, 2017, 2019, 2020, 2022a, 2022b.
Note: ASA = advisory services and analytics; GP = Global Practice; ICR = Implementation Completion and Results Report;
IEG = Independent Evaluation Group.
a. The Completion and Learning Review Validation was called Completion and Learning Review Review before May 1,
2023. No change was made to the methodology.
References
World Bank. 2014. Learning and Results in World Bank Operations: How the Bank Learns.
Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2015. Learning and Results in World Bank Operations: Toward a New Learning Strategy .
Independent Evaluation Group. Washington, DC: World Bank.
23
World Bank. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank
Group. Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2017. World Bank Group Engagement in Upper-Middle-Income Countries: Evidence from
IEG Evaluations. Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2019. Knowledge Flow and Collaboration under the World Bank’s New Operating Model.
Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2020. The World’s Bank: An Evaluation of the World Bank Group’s Global Convening .
Independent Evaluation Group. Washington, DC: World Bank.
World Bank. 2022a. Enhancing the Effectiveness of the World Bank’s Global Footprint. Independent
Evaluation Group. Washington, DC: World Bank.
World Bank. 2022b. Results and Performance of the World Bank Group 2022. Independent Evaluation
Group. Washington, DC: World Bank.
24