Reference Annex
Illustrative Examples for Independent Evaluation Group
Validators
Implementation Completion and Results Report Reviews for
Investment Project Financing
Last Revision: May 2024

1
Contents
Abbreviations .................................................................................................................................................... iv

Appendix A. Implementation Completion and Results Report Review Checklist ....................1
Appendix B. Guidelines for Meeting with the Last Project Task Team Leader ..........................4
Appendix C. Split Rating ................................................................................................................................8

iii
Abbreviations
CBA cost-benefit analysis
CCT conditional cash transfer
CPF Country Partnership Framework
ERR economic rate of return
ICR Implementation Completion and Results Report
ICRR Implementation Completion and Results Report Review
IEG Independent Evaluation Group
MIDES Ministry of Social Development
M&E monitoring and evaluation
NPV net present value
OMVS Senegal River Basin Development Organization
PAD Project Appraisal Document
PDO project development objective
PFM public financial management
SBG school-based grant
SBM school-based management
SDP Supplier Development Program
SME small and medium enterprise
TOC theory of change
TTL task team leader
WUA water user association
All dollar amounts are US dollars unless otherwise indicated.

iv
Appendix A. Implementation Completion and
Results Report Review Checklist
Overall
Text is frank but diplomatic, with a neutral tone.
Review is critical, not just a summary of the Implementation Completion and Results Report (ICR).
Statements are substantiated with evidence.
Acronyms are minimized and spelled out the first time they’re introduced.
No typos or grammatical errors (English fixed for multilingual speakers).
Section 1: Project data
All required fields are populated.
The loan or credit amount is less than or equal to the total project costs, at appraisal and closing.
If there was any cofinancing, the cofinancers are mentioned; if cofinancers are mentioned, then there
is some cofinancing recorded.
If the actual is different from the planned, there is an explanation in section 2d.
If there was additional financing (check section 2d), it has been added to the actual (not appraisal).
Section 2: Objectives and components
2a. The objectives from the legal agreement are stated, with page numbers.
2a. Key outcome targets are mentioned, if any; the section does not list all of the key performance
indicators.
2a. If the project’s objectives were changed by formal restructuring (see section 2b), then the revised
objectives are also presented, with the source and page numbers.
2a. Both overarching and specific objectives are mentioned; for one phase of a series of projects, the
series of projects objectives and the phase-specific objectives are listed.
2b. If the project’s objectives were changed because of a formal restructuring, the revised objectives
should be in 2a; the relevance of objectives is assessed for the revised objectives (section 3a);
achievement of the original and revised objectives is assessed (section 4); efficiency of the original and
the revised project is assessed (section 5); and a weighted outcome rating is in section 6.
2c. Components are listed and summarized with the planned and actual expenditure (for investment
projects); any components added after approval are also listed with the same information.
2d. Differences between the actual amount (total or the credit or loan) and the planned amount are
explained (for example, cancellations, additional financing, favorable or unfavorable exchange rates,
and so forth).
2e. Extent to which the counterpart contribution was paid, and the timeliness is discussed.
2f. There is an explanation for any project extension(s).
Section 3: Relevance of objectives
Relevance of the objectives or outcomes is rated with respect to the extent to which an operation’s
objectives are consistent with current World Bank and country strategies (expressed in the Country
Partnership Framework; CPF) and are addressing significant constraints to development challenges.
The selection of the implementing agency, quality of preparation, monitoring and evaluation (M&E)
design, choice of indicators, project complexity, and other quality at entry material is not included.
Relevance of objectives is rated on a four-point scale.

1
Appendix A
ICR Review Checklist

Section 4: Achievement of objectives—efficacy
There is a separate subsection for each objective or outcome whose achievement is assessed. Each
subsection contains a field for the wording of the objective or outcome (the heading), a field for the
text explaining the assessment (the rationale), and a rating. If an objective was revised at restructuring,
there is a separate subsection for each revision.
The headings all represent outcomes or intermediate outcomes (the expected changes to result from
the project), not outputs or components.
The wording of the objectives or outcomes is taken from the legal agreement, in section 2a.
If the statement of objectives was expressed in terms of outputs, there is a discussion and justification
of the outcomes that will be assessed.
If the statement of objectives has multiple outcomes and intermediate outcomes, there is a heading
for each one.
Under each heading, the assessment and evidence for the achievement of the entire results chain
(outputs and outcomes) is presented, including intermediate outcomes.
In each case, there is a discussion of attribution of the results to the project and other factors that
might have affected the outcome beyond the project.
There is an overall efficacy rating.
Section 5: Efficiency
If there is an economic rate of return (ERR) or net present value (NPV), the table showing the coverage
of the ERR or NPV is completed (section 5a) and the Implementation Completion and Results Report
Review (ICRR) addresses the assumptions and their realism or points to lack of transparency regarding
the assumptions.
The Review presents evidence of cost effectiveness and efficient use of project resources, efficient
implementation, or efficient design.
Improved efficiency of the sector is not included as evidence of project efficiency.
There is a single rating for efficiency on a four-point scale.
Section 6: Outcome
The proposed outcome rating is consistent with the guidelines for combining relevance of objectives,
efficacy, and efficiency.
The rationale for the rating is explained in terms of results for the elements and couched in the
language of the harmonized criteria.
If the project was restructured to change the objectives, the outcome rating has been correctly
calculated (assessing the outcome rating for the entire project with the original and revised objectives,
then weighting the results according to the share disbursed before and after restructuring).
Section 7: Risk to development outcome
Identifies the major risks that could occur in the future, the likelihood that they may occur, and the
consequences if they do (for example, technical, financial, economic, social, political, environmental,
ownership of government or other stakeholders, institutional support, governance, and exposure to
natural disasters).
Section 8: Bank performance
Quality at entry incorporates comments from the relevance of objectives and M&E design, in addition
to other criteria. Deficiencies in M&E design detract from the rating.
Quality of supervision incorporates comments from the M&E implementation section. Deficiencies in
the latter reduce the rating.

2
Appendix A
ICR Review Checklist

Shortcomings in the identification of safeguard issues or compliance are taken into account in the
quality at entry and quality of supervision ratings, as appropriate. (See section 11 on safeguards.)
Shortcomings in fiduciary arrangements or performance are taken into account in the quality at entry
and quality of supervision ratings, as appropriate. (See section 11 on fiduciary arrangements.)
The overall Bank performance rating is correctly calculated from the two subratings and, when split
between the satisfactory and unsatisfactory scales, according to the outcome rating.
Section 9: Monitoring and evaluation
On monitoring, the M&E design considers the choice of indicators, whether they are adequate to
measure each element of the results chain, whether the implementation arrangements for M&E were
identified, and whether there were adequate baseline data.
On evaluation, M&E design discusses planned evaluations.
The M&E implementation section comments on the extent to which the M&E plan was fully
implemented and the data were collected in a timely manner and were of good quality.
The M&E use section provides evidence of the extent to which timely data were used to improve
project performance and results in the field.
Overall M&E quality is rated on a four-point scale; the three individual elements are not rated.
Section 10: Other issues
For investment loans, the safeguard category is mentioned, the presence of a mitigation plan (if
required), and whether the mitigation plan was successfully implemented (or the ICRR comments on
lack of information in the ICR in this regard).
If unanticipated impacts are mentioned, they are truly unanticipated, adequately evidenced,
attributable to the project, and of sufficient magnitude to be important.
Section 11: Ratings
When the Independent Evaluation Group (IEG) ratings differ from ICR ratings, there is an explanation
in the last column.
Section 12: Lessons
There are no more than five lessons, and it is clearly noted whether they are from the ICR or from the
IEG ICR reviewer.
The lessons clearly build on results evidenced elsewhere in the ICRR.
The lessons are not findings.
Section 13: Assessment recommended?
If an assessment is recommended, the reason given makes sense.
Section 14: ICR quality
Shortcomings in the ICR that were mentioned in the other sections are collected in this section.
If the quality is rated modest or low, the explanation points to one or more “fatal flaws,” not a series of
small errors.
If the quality is rated high, the reasons are well documented and there should not be more than small,
incidental shortcomings. (It shouldn’t simply say that it was frank and well written.)
Appendix B. Guidelines for Meeting with the Last
Project Task Team Leader
1. What is the purpose of the meeting?

The purpose of the meeting is twofold: (i) to gain a better understanding of the project
experience to improve the accuracy and quality of IEG’s ICRRs; and (ii) to ensure due
process by providing the project task team leader (TTL) and the IEG ICR reviewer an
opportunity to discuss the project experience. The meeting is explicitly not intended to
discuss any possible ICRR ratings.

This meeting is conducted before IEG sends the draft ICRR to the Global Practice. The
meeting with the TTL is different from the meeting that the Global Practice might
request to discuss the draft ICRR after receiving it from IEG (please see point 4 below for
further details on the timing of the meeting).

2. Who should initiate and attend the meeting from IEG’s side?

The ICR reviewer should initiate and attend the meeting from IEG’s side. As a general
rule, a new ICR reviewer (regardless of their seniority) should be accompanied by a
more experienced reviewer to the meeting in conducting their first ICRR. It will be the
responsibility of the hiring IEG staff (typically, the ICRR coordinator) to determine when
the new ICR reviewer is ready to plan and conduct their meetings with the last project
TTL without assistance from more experienced reviewers.

Depending on who will be attending the meeting from the Global Practice’s side, the
relevant ICRR coordinator or IEG manager may also choose to attend the meeting
(please see point 3 below).

3. Who should attend from the Regional director’s side?

The meeting should be held with the last TTL of the project. The meeting should not be
held with the ICR author alone, unless the last TTL and the ICR author are the same
person, or unless the last TTL specifically delegates to the ICR author responsibility for
the meeting behalf of the Global Practice. In the rare instances when the last TTL of the
project has left the World Bank, the IEG ICR reviewer, on consultation with the ICRR
coordinator, should contact the concerned sector manager for an alternative suggestion.
It would be up to the project TTL to invite other Global Practice staff at their discretion.

The IEG ICR reviewer should ensure that they have advance notice of the complete list
of attendees from the Regional director. If the list includes the practice manager, the ICR
reviewer should inform their manager, giving them the option of attending the meeting.

4
Appendix B
Guidelines for Meeting TTL

4. At what point during the ICRR process should the meeting be conducted?

The meeting should be conducted only after the ICR reviewer has prepared an advanced
draft of the ICRR, and after the feedback on the first draft is received from the panel
reviewer. The ICR reviewer is expected to indicate in the relevant sections of the draft
ICR that information will be sought to substantiate the assessment when submitting the
draft to the panel reviewer, along with the list of questions that they intend to ask.

In the rare instances when there is a delay in the availability of the last TTL, the ICR
reviewer (with the concurrence of the ICRR coordinator) may meet with the project TTL
after obtaining the panel reviewer’s formal clearance and sign-off on the draft ICRR. In
such cases, it will be the responsibility of the ICR reviewer to obtain an email indicating
the panel reviewer’s concurrence with any substantive changes made to the draft ICRR
after the meeting with the project TTL and to include that email in the ICRR package
going to the IEG manager.

5. What should the length of the meeting be?

The meeting should be between 30 minutes and one hour. This should also be clearly
indicated to the project TTL in the email inviting them to the meeting.

6. What should be discussed during the meeting?

The ICR reviewer should inform the meeting participant(s) that additional information
obtained during the meeting and their comments may be used in the ICRR. The ICR
reviewer should focus on missing or ambiguous information in the ICR that is necessary
to answer IEG’s evaluative questions, including any additional evidence that may be
needed to substantiate the ratings. In addition, the ICR reviewer should use the meeting
to confirm their understanding of the project context, gain a better understanding of the
factors that might explain the project’s (good or bad) performance, and probe what the
project TTL might have done differently had they had the option.

Under no circumstances should the ICR reviewer share the draft ICRR, or even discuss
the ICRR ratings, at the meeting. However, the ICR reviewer should be totally
responsive to any other questions, including questions about IEG evaluation
methodologies, posed by the project TTL.

The ICR reviewer should record any additional information (not already contained in
the ICR) obtained during the meeting with the project TTL in the draft ICRR, noting, “In
a meeting between IEG and the project task team, the project team stated that . . . .”

If an agreement is reached at the meeting that the project TTL will provide additional
information to IEG, the ICR reviewer will confirm the specific additional information to
Appendix B
Guidelines for Meeting TTL

be provided and the format in which it will be presented, both at the meeting itself and
in a follow-up email to the project TTL. With regard to the format, the ICR reviewer
should make it clear that the additional information is to be provided in a separate note
written specifically for this purpose, and not in the form of volumes of documents.

7. Should the meeting be conducted in person, by telephone, or by video?

Every attempt should be made to have a face-to-face meeting. If the project TTL is
traveling for an extended period or based in the field, which would make a face-to-face
meeting impossible (given time and budget constraints), the meeting should be
conducted by telephone, personal conference line, video chat (for example, MS Teams or
Webex), or a similar means that does not involve additional cost.

8. What are the instructions if the last TTL is unresponsive to attempts to set up a
meeting or if the TTL is traveling?

The ICR reviewer should prepare for this eventuality by contacting the project TTL and
checking their travel schedule early in the process and getting the draft ICRR ready soon
after it is assigned. If the TTL is traveling or if the TTL is based in the field office, an
audio meeting should be suggested. If the TTL is not responsive after three weeks to
setting up a meeting, the ICR reviewer may proceed with finalization of the draft. They
should inform the panel reviewer and the ICRR coordinator that repeated attempts to
meet with the TTL were unsuccessful.

9. Should IEG circulate an agenda to the project TTL before the meeting?

Yes. Good practice is to develop a list of questions in advance, and to share this list with
the TTL before the meeting.

10. Should a written record be kept of the discussion at the meeting?

It is mandatory to prepare a written summary of the general topics covered in the
meeting. The names of attendees, date, and time of the meeting should be noted in the
summary.

The summary of the meeting with the TTL must be attached to the Activity History in
the IEG ICRR Portal record for this ICRR.

If a meeting did not take place (because, for example, repeated requests for a meeting
were not answered), the IEG ICR reviewer enters a note explaining the reason for not
having a meeting with the TTL in the Activity History.

6
Appendix B
Guidelines for Meeting TTL

11. How will the IEG manager sending the ICRR to the Global Practice know that the
meeting has taken place?

The Activity History contains the record of the ICRR progress, including either the
summary of the meeting with the TTL or the note indicating the reason for not having a
meeting with the TTL.
Appendix C. Split Rating
Table C.1. Split Rating Scenarios and Examples
Split
Scenarios Rating
Changes in outcome indicators
If the change in the PDO indicator or target represented a lowering of the project’s scope or Yes
ambition (with same commitment level), this would support the decision to apply a split rating.
If the original indicator was relatively weak and there is a strong case to be made that the new No
indicator or target provides a better measure for achievement of the PDO, then this would
support a decision not to apply a split rating and instead assess the entire project based on the
revised indicator or target.
If the original indicator(s) (for example, dropped at restructuring) did not have targets, but the No
new indicator(s) had targets, it may be possible to make a case that this represented an
improvement in the validity of the indicators, and therefore this would be a reason not to
apply a split rating and instead to assess the project based on the revised indicators and
targets (assuming the level of ambition and funding both remain constant).
If the original indicator(s) were connected to outcome-level results, and new indicator(s) are Yes
connected to more intermediate outcome-level results or output-level results, then this likely
represents lowering the level of ambition of the project and therefore would support a
decision to apply a split rating (and likely would support a relatively low efficacy rating).
If project documentation or experience suggests that the reason for selecting a new indicator Yes
was mainly convenience (for example, using an indicator that was already being tracked but
that did not represent an improvement in validity), this would support a decision to apply a
split rating.
Scope of the project narrowed
If the scope of the project shrank and project commitments remained constant, this supports a Yes
decision to apply a split rating. Example: A project originally served five regions before the
revision and, after the revision, it served three regions.
If the target coverage declined or the definition of beneficiaries was narrowed (lowering the Yes
level of ambition with the same amount of funding), then this supports a decision to apply a
split rating. Example: A project aimed to reach 1 million mothers with a basic package of health
services before the revision and, after the revision, it aimed for half a million.
If the scope of the project shrank, project commitments decreased (for example, through Yes
cancellation of funds), and even if a good case is made that the lowering of project scope was
commensurate with the lower commitment size, then this would still support a decision to
apply a split rating.
Scope of the project expanded
If additional financing was provided to add to the scope of the project (for example, additional No
geographical areas, additional beneficiary groups, or some other kind of additional scope) and
the targets related to the increased scope were met, then this supports a decision not to apply
a split rating and instead to assess the project based on the revised targets. In such a case,
applying a split rating is likely moot and can overly complicate the reporting.
If additional financing was provided to add to the scope of the project (for example, additional Yes
geographical areas, additional beneficiary groups, or some other kind of additional scope) but
with evidence of mix achievements for original and more ambitious targets (for example,
targets for the geographical areas included in the original project were achieved, but targets

8
Appendix C
Split Rating

Split
Scenarios Rating
for the geographical areas added at restructuring were not achieved) then this supports a
decision to apply a split rating.
If some donor(s) or stakeholder(s) dropped out (therefore reducing the amount of funds or Yes
other resources available), and these stakeholder exit(s) resulted in scope shrinking or lowering
the level of ambition within the same amount of funding, then this supports a decision to
apply a split rating.
If some donor(s) or stakeholder(s) joined the project (therefore increasing the amount of funds No
or other resources available), and stakeholder entry resulted in increasing the scope of the
project or raising the level of ambition, then this can support a decision not to apply a split
rating and instead assess the entire project based on the revised outcomes and outcome
targets.
Scope remained the same
If additional financing is received but the project’s objectives and outcome targets are not No
revised, a split rating generally should not be performed. In such a scenario, the project must
be assessed against the original indicators and targets. The IEG ICR reviewer should note,
however, that additional financing not accompanied by an increase in the scope or ambition of
a project may be an indication of lowered efficiency, and therefore should be reflected in a
downgraded efficiency rating.
If activities were realigned or resources were reallocated to move the geographical coverage No
from one region to another, and the ICR made a convincing case that the level of ambition or
difficulty was similar, then this can support a decision not to apply a split rating and instead
assess the project based on revised objectives or under the revised geographical coverage.
If activities were realigned or resources were reallocated to change the target or beneficiary No
coverage, and the ICR made a convincing case that the level of ambition or difficulty was
similar, then this can support a decision not to applying a split rating and instead assess the
project based on objectives under the revised coverage or the revised definition of
beneficiaries.
Source: Independent Evaluation Group.
Note: ICR = Implementation Completion and Results Report; IEG = Independent Evaluation Group; PDO = project
development objective

Instances and Examples for Not Using Split Rating Approach
1. If overall outcome rating is clearly predetermined, for example:

o Similar efficacy ratings, original and revised targets achieved (Lao People’s
Democratic Republic: Health Governance and Nutrition Development,
P151425)

o Predominance of a disbursement weight

2. Appropriate adjustment of outcome targets unrelated to project scope or
ambition (Mali Skills Development and Youth Employment, P14586). One
associated outcome target pertaining to the entrepreneurship program was
Appendix C
Split Rating

reduced to allow team applications for business plan competition instead of only
individual applications.

3. Outcome indicators reformulated—better measures of project achievement—
unrelated to project ambition (Lao PDR Early Childhood Education Project,
P145544: change from absolute numbers to percentages, providing more context
and illustrating the relative change in improved coverage and quality of early
childhood education services).

C1. Applying Split Rating: Examples of Simple and Complicated Cases

Example C1.A. Simple Case: Nepal Community Action for Nutrition Project
(P125359)
One objective was dropped, a new one added, and the geographic scope was expanded,
with mixed achievements.

Table C.2. Nepal—Rating of Original and Revised Objectives
Objectives after
Rating Dimension Original Objectives Revision
Relevance of objectives High
Efficacy
Objective 1. Improve nutrition practices Modest Substantial
Objective 2. Improve attitudes Modest —
Objective 3. Provide emergency nutrition and — High
sanitation
Overall efficacy Modest Substantial
Efficiency Modest
Outcome rating Moderately unsatisfactory Moderately satisfactory
Outcome rating value 3 4
Amount disbursed (US$, millions) 4.8 26.8
Disbursement (%) 15 85
Weight value 3 x 15% = 0.45 4 x 85% = 3.40
Total weights 3.85 (rounded to 4.00)
Overall outcome rating Moderately satisfactory
Source: Independent Evaluation Group.
Note: — = not applicable.

10
Appendix C
Split Rating

Example C1.B. Complicated case (multiple revisions) Liberia Health Systems
Strengthening Project (P128909)
PDO was to improve the quality of maternal health, child health, and infectious disease
services in selected secondary-level health facilities.

First Restructuring (August 2014)
The PDO was revised to include an emergency response to the Ebola outbreak, at which
time no disbursements had yet occurred. [Just new objective—no need for split]

Second Restructuring (January 2017)
AF revised PDO by (i) adding primary health care and neonatal care, (ii) dropping the
infectious diseases services, (iii) dropping support to Ebola response, (iv) reducing the
outcome target for maternal and child death audits from 100 to 80 percent, and (v)
introducing a new indictor on neonatal death audits with outcome target set at
80 percent. [Split]

Third Restructuring (May 2019)
Project was restructured to extend the grant to allow full disbursement in view of
previous delays caused by the Ebola crisis. [Ignore]

Fourth Restructuring (April 2019)
Project was restructured to introduce changes in the results framework: (i) outcome
target for maternal death audits reduced from 80 to 65 percent, and (ii) outcome target
for neonatal death audits reduced from 80 to 45 percent. The restructuring paper stated
that end target changes reflected realistic estimates based on the current progress to
date. [Split]

Fifth Restructuring (May 2020)
The project’s closing date was extended. [Ignore]

Table C.3. Liberia—Rating of Original and Revised Objectives
Objectives after Objectives after
Original First Revision Second Revision
Rating Dimension Objectives 2017 2019
Relevance of objectives High
Efficacy
Objective 1: Improve quality of maternal Substantial Substantial High
and child health services at secondary
level
Appendix C
Split Rating

Objectives after Objectives after
Original First Revision Second Revision
Rating Dimension Objectives 2017 2019
Objective 2: Improve quality of maternal — High High
and child health services at primary level
Objective 3: Improve quality of infectious Negligible — —
disease services
Objective 4: Support emergency response Modest — —
to Ebola
Overall efficacy Modest Substantial High
Efficiency Substantial
Outcome rating Moderately Satisfactory Highly satisfactory
unsatisfactory
Outcome rating value 3 5 6
Amount disbursed (US$, millions) 6.78 6.22 17.6
Disbursement (%) 22.2 20.3 57.5
Weight value 3 x 22.2% = 0.67 5 x 20.3% = 1.02 6 x 57.5% = 3.45
Total weights 5.14 (rounded to 5.00)
Overall outcome rating Satisfactory
Source: Independent Evaluation Group.

Example C1.C. Kenya Nairobi Metropolitan Services Project (P107314)
The first restructuring was approved on April 20, 2017. The targets for PDO indicators
were revised as follows:

• The target for PDO indicator “direct beneficiaries” was increased from
1.5 million to 2.2 million, when Makuyu urban center was added as a beneficiary
of the project;

• The target for PDO indicator “people provided with access to improved
sanitation facilities” was decreased from 100,000 to 10,000, (the ICR reports that
this was because of the change in methodology as per the guidelines of the core
indicator; Restructuring Paper para 6);

• The PDO indicator “people in the Nairobi Metropolitan Region provided with
access to regular solid waste collection under the project” was deleted because
the project focus changed from collection to disposal; and

• The target for PDO indicator “people in the Nairobi Metropolitan Region
living within a one-kilometer radius of a location for a commuter rail station,
benefitting from improved access to that location” was decreased from 500,000
to 350,000 (the scope of the commuter rail stations was reduced to fit the reduced

12
Appendix C
Split Rating

budget because of foreign exchange fluctuations, which caused losses in total
project financing up to $30 million);

• A related new corporate core indicator, “People provided with access
to improved sanitation facilities—urban,” was added to the results framework.
The project’s closing date was extended from June 30, 2017 to May 31, 2019 (23
months) to complete all project activities that were delayed because of slow
implementation during the first few years.

The second restructuring was approved on February 20, 2019. The targets for PDO
indicators were revised as follows:

• The target for PDO indicator “direct beneficiaries” was increased
from 2.2 million to 3.2 million (Restructuring Paper);

• The target for PDO indicator “people provided with access to improved
sanitation facilities” was increased from 10,000 to 14,000;

• The target for PDO indicator “people provided with access to improved
sanitation facilities—urban” was increased from 10,000 to 14,000;

• The target for PDO indicator “number of people in urban areas provided with
access to all-season roads within a 500-meter range” was increased from
100,000 to 350,000; and

• The target for PDO indicator “people in the Nairobi metropolitan region living
within a one-kilometer radius of a location for a commuter rail station,
benefitting from improved access to that location” was decreased from 350,000
to 280,000. The closing date was extended from May 31, 2019 to March 31, 2020
(10 months) to complete the outstanding key infrastructure works, especially
markets and the Mitubiri landfill, and implementation of the associated
Resettlement Action Plan.

Split rating. As discussed above, the outcome targets were substantially changed during
the first two restructurings. Therefore, the review will use split rating methodology.

Outcomes compared against the original target:

• PDO indicator 1 was achieved. The project investments reached 3.5 million
“direct beneficiaries” (exceeding the original target of 1.5 million). Of these, an
estimated 1.75 million were females (at project design about 50 percent of
beneficiaries were targeted to be female).
Appendix C
Split Rating

• PDO indicator 2 was not achieved. About 49,000 people were provided with
access to “improved sanitation facilities,” substantially lower than the original
target of 100,000.

• PDO indicator 3 “People in the Nairobi Metropolitan Region provided with
access to regular solid waste collection” was dropped, because the project focus
changed from solid waste collection to disposal. No indicator for measuring solid
waste disposal was added.

• PDO indicator 4 was achieved. About 414,800 people were provided with access
to “all-season roads within a 500-meter range,” substantially higher than the
original target of 100,000.

• PDO indicator 5 was not achieved. At project closure, about 350,000 people in
the “Nairobi Metropolitan Region were living within a one-kilometer radius
of a location for a commuter rail station, benefitting from improved access to
that location,” lower than the original target of 500,000.

In sum, two out of four PDO indicators were achieved. The efficacy of this objective is
modest.

Outcomes compared against the revised target set at April 2017 restructuring:

• PDO indicator 1 was achieved. The project investments reached 3.5 million
“direct beneficiaries,” exceeding the revised target of 2.2 million. Of these, an
estimated 1.75 million were females (at project design about 50 percent of
beneficiaries were targeted to be female).

• PDO indicator 2 was achieved. About 49,000 people were provided with access
to “improved sanitation facilities,” substantially higher than the revised target
of 10,000.

• PDO indicator 2 (b) (added at 2017 restructuring) was achieved. About 43,200
people were provided with access to “improved sanitation facilities—urban,”
substantially higher than the target of 10,000.

• PDO indicator 4 was achieved. About 414,800 people were provided with access
to “all-season roads within a 500-meter range,” substantially higher than the
original target of 100,000.

• PDO indicator 5 was achieved. At project closure, about 350,000 people in the
“Nairobi Metropolitan Region were living within a one-kilometer radius of a

14
Appendix C
Split Rating

location for a commuter rail station, benefitting from improved access to that
location,” achieving the revised target of 350,000.

All PDO indicators were achieved. The efficacy of this objective is substantial.

Outcomes compared against the revised target set at April 2019 restructuring:

• PDO indicator 1 was achieved. The project investments reached 3.5 million
“direct beneficiaries” exceeding the revised target of 3.2 million. Of these, an
estimated 1.75 million were females (at project design about 50 percent of
beneficiaries were targeted to be female).

• PDO indicator 2 was achieved. About 49,000 people were provided with access
to “improved sanitation facilities,” substantially higher than the revised target
of 14,000.

• PDO indicator 4 was achieved. About 414,800 people were provided with access
to “all-season roads within a 500-meter range,” higher than the revised target of
350,000.

• PDO indicator 5 was achieved. At project closure, about 350,000 people in the
“Nairobi Metropolitan Region were living within a one-kilometer radius of a
location for a commuter rail station, benefitting from improved access to that
location,” substantially higher than the revised target of 280,000.

All PDO indicators were achieved. The efficacy of this objective is substantial.

Outcome Rating

Under the Original Outcome Targets
With high relevance of objectives, modest efficacy, and substantial efficiency, the overall
rating is moderately unsatisfactory (3).

Under the First Revised Outcome Targets

With high relevance, substantial efficiency and efficacy, the overall rating is satisfactory
(5).
Appendix C
Split Rating

Under the Second Revised Outcome Targets

With substantial relevance, substantial efficiency and efficacy, the overall rating
is satisfactory (5).

A split rating is applied based on the disbursement shares before and after the project
restructuring in 2017 and 2019, when a disbursement share was at 32.7 percent,
34.5 percent, and 32.7 percent respectively (total financing was $300 million).

The overall outcome rating is moderately satisfactory; the weighted value is 4 (0.327 × 3
+ 0.0.354 × 5 + 0.327 × 5 = 4.345).

C2. Relevance of Objectives Examples

Example C2.A. Myanmar Electric Power Project (P143988)
PDO: “To increase capacity and efficiency of gas-fired power generation and strengthen
the institutional capacity of the Ministry of Electric Power and the Myanmar Electric
Power Enterprise.”

Country and Sector Context

At the time of appraisal in 2013, Myanmar was among least developed countries in the
world, going through a political transition, coupled with economic reforms.

Despite substantial endowments, Myanmar’s energy sector was characterized by low
levels of access (only 29 percent of households); inadequate supply; and one of the
lowest rates of electrification in Southeast Asia. During the dry season, forced load
shedding was up to 70 percent of demand. There was low (20 percent) efficiency of
existing (aged) gas-fired power plants.

Alignment with Strategy

The project was the first International Development Association financing after World
Bank reengagement in Myanmar after more than two decades of absence and was
considered a cornerstone of the World Bank Group’s program in the energy sector
(Project Appraisal Document). The program aimed to alleviate the acute energy
shortages that prevailed in the country at the time and to set the power sector on a
sustainable development path. Therefore, the project’s development objectives were
highly relevant at appraisal, consistent with the engagement priorities for the country,
and remained so at project closing. The project was consistent with the priorities
outlined in the World Bank’s Interim Strategy Note (2012), especially pillar II, which
focused on building confidence in the ongoing reform process, and pillar III, which

16
Appendix C
Split Rating

focused on paving the way for resumption of a full country program. It was also closely
aligned with the Bank Group’s CPF for Myanmar, FY15–17 (which was extended to June
2019 and remained the latest strategy document at project closing). The PDO was
aligned with focus area 1, which was aimed at reducing rural poverty, and its objective
1.1 of “improved power generation and access to electricity,” which addressed both the
delivery of electricity services to people and expanded conventional and renewable
power generation. Under this objective, the government of Myanmar aimed to achieve
universal access to electricity by 2030. Toward this end it was developing a National
Electrification Plan to establish an enabling policy and institutional framework for a
nationwide electrification program, to which the Electric Power Project was expected to
significantly contribute. The project’s objectives were also consistent with the 2019
Systematic Country Diagnostic, which reiterated the current-day significance of CPF
objectives, as highlighted in Priority Area 1.4, Policy Action 1: “Accelerate delivery of
quality electricity services to reach all people in Myanmar by 2030.” The Systematic
Country Diagnostic also underlined the need for public institutions to be reformed and
made more effective, to pursue this broad and challenging agenda.

The project’s objectives were fully consistent with the government’s national
development goals, as outlined in the Myanmar Sustainable Development Plan for 2018–
30, particularly with pillar 2, strategy 3.6, “building priority infrastructure to facilitate
sustainable growth and economic diversification,” and pillar 3, strategy 5.4, “providing
affordable and reliable energy to populations and industries.” The project’s second
objective was especially aligned with strategy 5.4, which emphasized the importance of
energy development plans, electricity tariffs and subsidies, access to electricity, and
private sector participation for achieving the country’s national development goals.
Finally, the PDOs were consistent with the government’s plan to maintain low-carbon
development of the power sector, as outlined in the Myanmar Climate Change Strategy
and Master Plan for 2018–30, which underlined the use of renewable energy in the
future.

Given the strong consistency of the project’s development objectives with the
government’s sector development objectives and their close alignment with the Bank
Group’s strategy at appraisal and at closing, relevance is rated high.

Example C2.B. Armenia Education Improvement Project (P130182)
PDO: “To improve school readiness of children entering primary education, improve
physical conditions and the availability of educational resources in upper secondary
schools, and support improved quality and relevance in higher education institutions of
the Borrower.”
Appendix C
Split Rating

The PAD demonstrated that the project’s objectives were relevant to country context. At
the time of appraisal, considerable progress had been made in improving access to
preschool, general education, and higher education. Enrollment rates were nearly
universal for primary and lower secondary education; the preschool enrollment rate was
anticipated to reach 75 percent by 2015; and there had been significant growth in higher
education enrollment, from 19.6 percent in 2001 to 28.6 percent in 2008. More females
than males were enrolled in upper secondary and tertiary education. The government
had undertaken a wide range of reforms to improve quality of education at all levels,
including curriculum development, teacher training and certification, introduction and
integration of information technology, standardized performance assessments, and the
implementation of a per-student financing mechanism that produced substantial
efficiency gains. However, there were challenges remaining at all levels. Preschool
expansion was not on track to meet the government’s 2017 target of 90 percent
enrollment. Prior reforms had not yet produced demonstrable changes in student
performance at the general education level, with grade 4 and 8 test scores stagnant or
declining between 2003 and 2011. High school facilities were aging and, in some cases,
unsafe (lacking proper seismic stability and sanitary infrastructure). There was a
mismatch between the skills offered by institutions of higher education and those
required by the labor market.

The State Program of Education Development (2011–15) reflected these challenges,
emphasizing the need to (i) ensure that all children enter general education ready to
learn; (ii) build an education system that can provide high-quality education to all
students; (iii) establish a diverse tertiary education and research system responsive to
labor market needs; and (iv) improve productivity by endowing tomorrow’s labor force
with the appropriate set of skills. Armenia’s overall development strategy for 2014–25
prioritizes the development of human capital, with particular emphasis, under its
education pillar, on strengthening upper secondary education, increasing the efficiency
of basic education, reducing regional disparities in education access and quality, and
improving teacher training.

The project was aligned with two of the four key results areas identified in the World
Bank’s Country Partnership Strategy for Armenia at appraisal (2014–17): enhancing
human capital through better access to quality services, including health care, education,
culture, and basic infrastructure (which is reflected in all of the project’s objectives); and
expanding employment through high-productivity, decently paid jobs (which is
primarily reflected in the project’s objective to improve the relevance of higher
education). It was designed to build on two previous Adaptable Program Loans in the
education sector (First Education Quality and Relevance Project, 2004–09, and Second
Education Quality and Relevance Project, 2009–15). It remains aligned with the second

18
Appendix C
Split Rating

focus area of the current Country Partnership Framework (CPF 2019–23), Human
Capital Development and Equity, and specifically with its objective of “enhanced access
to good quality educational services for skills development and employability.” Planned
results under that CPF objective include increasing the share of children enrolled in
early childhood education, improving the quality of diagnostic tools implemented
across different areas of the education system, and creating a new cluster of higher
education institutions and research centers.

Within the context of World Bank and government strategy, the PAD’s description of
country conditions provided a compelling argument that the specific areas contained in
the objectives were important: preschool enrollment levels that were increasing but not
on track to meet government targets; substandard physical conditions and resource
availability in high schools, of particular concern given the recent extension of the
general education system from 10 to 12 years; and a mismatch between the skills being
acquired in tertiary education and the needs of the labor force. However, it did not
provide information or context that situated these areas as the most binding constraints
to achievement of education sector goals as outlined in the government’s strategy and
the World Bank’s current CPF. In other words, there was not clarity in project
documents on the reasons that these objectives were chosen over other possible
education objectives. There is no explanation, for example, for prioritizing preschool,
upper secondary, and tertiary education over the primary and lower secondary levels,
or for focusing on quality and relevance but not efficiency in higher education. There is
also insufficient justification for the scope and breadth of the project’s objectives,
spanning from preschool to higher education.

Furthermore, the second objective, to improve the physical conditions and availability of
educational resources in high schools, was output oriented; it did not specify
educational outcomes that were to result from planned improvements in high school
infrastructure and resources. A more outcome-oriented objective would have been
expected, particularly given the World Bank’s previous experience in the sector. The
Second Education Quality and Relevance (Adaptable Program Loan 2) project (2009–15,
P107772) had an objective to enhance school learning in general education; this project’s
second objective is not consistent with progress over time.

Although the objectives were clearly aligned with country context and with World Bank
and government strategy, there were significant shortcomings in their justification,
framing, and overall cohesion. Relevance of objectives is therefore rated modest.
Appendix C
Split Rating

C3. Theory of Change

Example C3.A. Transmission Efficiency Project
Figure C.1.Vietnam—Theory of Change for Transmission Efficiency Project

Source: Independent Evaluation Group.
Note: AMS = Asset Management System; EVN = Vietnam Electricity; GHG = greenhouse gas; IFRS = International Financial
Reporting Standards; kV = kilovolt; MVA = megavolt-ampere; MWh = megawatt-hour; NPT = National Power
Transmission Corporation; O&M = operation and maintenance.

The project’s inputs—International Bank for Reconstruction and Development loans and
technical assistance support—were to be used to finance the construction and
rehabilitation of high-voltage transmission lines and substations and upgrading of
existing substations, all including smart grid technology. The immediate output of these
activities would be an increase in the transmission capacity. Therefore, the first objective
to improve the capacity of electricity transmission services was at the output level rather
than outcome level. Assuming no electricity generation shortage, the project output of
increased transmission capacity would ease the overload on the transmission lines and
the substations that would lead to the project’s expected outcomes of fewer electricity
outages and load shedding because of fewer tripping in substations, and a decrease in
the amount of unserved energy, that is, lower technical losses in the system. A decrease
in the number and duration of electricity outages corresponds to an improvement in the
reliability of electricity supply. The lower unserved electricity amount corresponds to

20
Appendix C
Split Rating

improved efficiency and availability of electricity to meet increasing electricity demand
in the urban and industrial load centers. Improved reliability of power supply supports
long-term socioeconomic development and improved efficiency of electricity supply
lowers the amount of greenhouse gas emissions.

Overall, the causal pathways from inputs to outcomes were valid and direct, and the
achievement of the outcomes and project objectives could be attributed to the project’s
intervention because there was no other intervention in the project areas. The capacity-
building activity did not directly support the achievement of the project objectives.
However, increased technical and operational capacity of the National Power
Transmission Corporation—because of the implementation of an asset management
system, financial reporting on the basis of International Financial Reporting Standards,
and application of performance-based transmission tariff application—should be
expected to contribute to improved operational efficiency and technical capacity for
adequate maintenance of the transmission network that would improve the
sustainability of the electricity transmission services.

Example C3.B. Integrated Water Resources Management Project
Figure C.2.Senegal—Theory of Change for Integrated Water Resources Management
Project

Source: Independent Evaluation Group.
Note: OMVS = Senegal River Basin Development Organization; PDO = project development objective; WUA = water user
association
Appendix C
Split Rating

The ICR presents a diagrammatic visualization of the theory of change (TOC), depicting
the causal links among the various activities supported by the project, the intermediate
results, and ultimate outcomes. The activities included (i) restoration of the main
headwaters of the Senegal River and protection of river banks, (ii) control and
management of the proliferation of typha, along with improving access to drinking
water and strengthening of water user associations (WUAs), (iii) strengthening the
capacity of the Senegal River Basin Development Organization (L’Organisation pour la
mise en valeur du fleuve Sénégal; OMVS) and national agencies to lead management of
invasive species in the region. It was expected that these efforts would result in (i) the
implementation of soil and water resource protection approaches and the development
of diagnostic studies; (ii) the removal of typha from infested areas, installation of a
number of potable water stations, and training of WUAs in aquatic invasive species
management; and (iii) strengthening of capacity and information base to manage
invasive aquatic species. These intermediate outcomes could be expected in turn to lead
to (i) strengthened capacity of OMVS and local WUAs to improve environmental quality
(this was later modified at restructuring to reducing erosion in the Upper Basin); and (ii)
strengthened capacity of OMVS and local WUAs to improve water quality (later
modified at restructuring to reducing proliferation of typha in the Lower Basin). The
longer-term higher-level outcomes of this were expected to be the sustainable use of
land and water resources, enhanced agricultural productivity and income, improved
ecosystem conditions, and improved water sector capacity and culture to manage
environmental threats.

Although the activities appear appropriate to achieving the desired outcomes, it should
be noted that the TOC discussion in the ICR does not specifically analyze whether they
were of adequate scale to create a critical mass for change. The results framework
initially appeared to reflect the results chain and was appropriate to measure the
outcomes being sought. Modifications were introduced during restructuring, intended
to better align the key performance indicators with the revised PDO. One indicator that
was dropped at restructuring, however, related to the number of operational WUAs
created or strengthened—which appeared to undermine the link between the role of
OMVS and the WUAs and the end results in terms of improved environmental quality
or soil erosion.

22
Appendix C
Split Rating

Example C3.C. State Employment and Expenditure Project
Figure C.3. Nigeria—Theory of Change for State Employment and Expenditure Project

Source: Independent Evaluation Group.
Note: AuG = auditor general; CDD = community-driven development; IFMIS = integrated financial management
information system; IGR = internally generated revenue; IT = information technology; LDP = local development plan; MTEF
= Medium-Term Expenditure Framework; MTEP = medium-term expenditure plan; PF =; SHOA = State House of
Assembly.

The outputs such as, acquiring training materials, rehabilitating selected technical
vocational training schools, national level accreditation of courses taught at the these
schools, conducting training sessions, and developing strategies for skill development
programs, and starter packs for graduates of these schools, were expected to improve
the vocational skills of people in general, and youth in particular. These outputs would
increase the supply of skilled workers with vocational skills. The outputs such as,
strengthening the capacity of the civil society organizations at the subnational level to
design community infrastructure projects, would increase the demand for the skilled
workers trained under the project. In addition, the results of activities associated with
building community infrastructure were expected to lead to improved access to
socioeconomic services. The causal links between project activities, outputs, and
outcomes were clear. The intended results were monitorable.
Appendix C
Split Rating

The causal links between the project activities, outputs, and outcomes were logical. The
participating states had undertaken public expenditure management and financial
accountability reviews and prepared reform action plans. This project intended to assist
the states in achieving some of the recommended public expenditure management and
financial management plans. The outputs of activities such as submitting the public
finance legislation to the State House of Assembly, developing a medium-term
expenditure plan and publishing the annual budget, were expected to reduce the
deviation between annual expenditure and budgeted spending. Adopting new
procurement laws and establishing a specialized procurement cadre would enable the
adoption of standard bidding procedures and an increased percentage of public
contracts above a threshold awarded through a competitive process. The outputs of
activities associated with establishing a taxpayer database would increase internally
generated revenue. These outcomes would contribute to the medium- and long-term
result of improving the public financial management (PFM) of the four states.

C4. Efficiency Examples

Practice Area: Transport

Example C4.A. Nigeria Rural Access and Mobility Project Phase 2 (P095003)
Economic Analysis

At appraisal, an economic evaluation study based on the Road Economic Decision
Model developed by the World Bank for low-volume roads was conducted using data
gathered through traffic surveys for the first tier of 800 kilometers of rural roads to be
rehabilitated under the project. The Road Economic Decision Model uses the consumer
surplus approach to estimate road user costs savings, such as vehicle operating costs
savings, passenger time costs savings, and accident cost savings. For the economic
evaluation, costs were taken as rehabilitation costs and maintenance costs. The
assumptions used at appraisal were appropriate: 12 percent discount rate, 6 percent
traffic growth rate, and a 20-year evaluation period. The calculation resulted in an ERR
of 39 percent and an NPV of $155.57 million.

At project closing, a similar economic evaluation based on the Road Economic Decision
Model could not be conducted because of the COVID-19 pandemic, which resulted in
restricted movement of people and goods distorting traffic data. Alternatively, actual
average costs of road rehabilitation were compared with the average costs estimated at
appraisal. In Adamawa, the average cost of 1 kilometer of road rehabilitation was
$70,032, which was lower than the average cost of $97,406 estimated at appraisal.
Similarly, in Niger the actual cost of road rehabilitation was lower than the average cost

24
Appendix C
Split Rating

estimated at appraisal; $70,870 and $96,936, respectively. But in both Enugu and Osun
actual average costs of road rehabilitation were higher than average costs estimated at
appraisal: Enugu, $123,098 and $95,642; and in Osun, $68,103 and $62,320, respectively.

Operational and Administrative Efficiency

Project effectiveness was delayed by 14 months because of the lengthy approval process
of the Borrowing Plan by the national assembly. This delay, together with other delays
caused by external factors, such as a worsening security situation or governance issues
in some states, resulted in slow project implementation. The counterpart funds at the
state level, except in Osun, were insufficient to cover supervision and implementation
costs and process the payment of compensation to project-affected persons under
resettlement action plans. These caused further project implementation delays. The State
Project Implementation Units’ supervision of civil works was inadequate because of
weak technical capacity, which resulted in low-quality road and infrastructure works in
the first phase. Although, as pointed out by the project team to IEG, the civil works
quality was subsequently corrected and improved, this is not an efficient way to
proceed. Important technical assistance activities to reform the sector could not be
completed because of the absence of a road sector specialist and the focus of the State
Project Implementation Units on the civil works that had greater visibility and direct
benefits for the population. The phased approach to road rehabilitation works was
implemented with the expectation that the first stage of works would provide
experience to State Project Implementation Units and lessons for the second stage.
However, this approach created very complex procurement processes for both phases,
resulting in each state implementing approximately 100 contracts with limited
institutional capacity. 1

Overall, although the average costs of road rehabilitation were comparable to estimated
average costs at appraisal, there were significant shortcomings in the operational and
administrative efficiency, as explained in the previous paragraph. Hence, the project’s
efficiency in achieving the project objectives is rated modest.

Efficiency rating: Modest

1$601 million is the total of the variance from year to year across the six states between calendar
years 2014 and 2020.
Appendix C
Split Rating

Practice Area: Transport (Technical Assistance)

Example C4.B. Azerbaijan—Public Investment Capacity Building (P115396)
As a Technical Assistance Credit, this project was not subjected to economic analysis at
appraisal. The following aspects of project design and implementation contributed to
project efficiency.

Cost Effectiveness

There were no cost overruns. The unit costs mainly comprised remuneration and
reimbursable expenses of the trainers and the costs of training materials and
refreshments. Since the training sessions were delivered on government premises and at
training centers provided by the project, there were cost savings associated with not
renting hotel or other conference facilities. Also, where there was local expertise for
providing training, this was drawn on, which entailed cost savings associated with
minimizing air fare, hotel, and living allowances out-of-town trainers.

Increased scope due to savings. Based on competitive procurement processes of project
goods and services, the project benefited from savings in the amount of about $500,000,
or 6 percent of the total credit amount. Based on the request from the government, these
funds were used for additional capacity-building activities in relation to a vocational
education school at the Sumgait Chemical and Industrial Park.

Although the project’s initial focus was on training public servants from the relevant
ministries and agencies, at government request, other agencies were allowed to
participate in the training activities. This allowed training additional participants at
minimal incremental costs.

Administrative efficiency and cost-sharing arrangements. The Project Coordination
Unit of an ongoing World Bank–financed project (Azerbaijan—Education Project) was
designated to take on the implementation of this project. This contributed to reduction
of overall project costs because no unnecessary parallel institutions were created. The
supervision missions for this project were combined with other transport projects in
Azerbaijan and this ensured efficient use of the World Bank budget.

Time overrun. The closing date was extended twice, in total by 1.9 years, to allow the
government to use unallocated funds for additional capacity-building activities. If not
for these additional activities, which increased the scope of the project, the project would
have closed as scheduled.

26
Appendix C
Split Rating

Practice Area: Environment, Natural Resources and the Blue Economy

Example C4.C. Brazil Development of Systems to Prevent Forest Fires and
Monitor Vegetation Cover in the Brazilian Cerrado (P143185)
Economic and financial analysis. At appraisal, no cost-benefit analysis (CBA) was
conducted because the technical assistance nature of the project would not render
meaningful results (ICR, annex 4, page 50). At project closing, a CBA was conducted in
an external evaluation of the project to assess the impact on stakeholders and their
perception of project’s results by using the Social Return on Investment method for
measuring extra-financial value with 200 scenarios (ICR, para 63). The ex post CBA
revealed that every Brazilian real invested by the project generated a total monetary
return of more than five times the investment amount, and only one out of the total 200
scenarios had a cost-benefit ratio lower than one (ICR, para 63).

Aspects of design and implantation that affected efficiency. The project was extended
for 19 months because of the following two aspects. First, the implementation delays in
early years were caused by the implementing agency’s unfamiliarity with the World
Bank’s procurement and financial management procedures and requirements and
insufficient human resources contributed to the project extension (ICR, para 37). Second,
after the emergence of the COVID-19 pandemic, the need for social distancing measures
changed the delivery format of knowledge-exchange activities from face to face to online
or hybrid, negatively affecting the participants to acquire the necessary knowledge
swiftly (ICR, para 49). No additional financing was made.

The ex post CBA indicated that the project’s investments generated the total monetary
return more than five times the invested amount. The extension of project duration was
accompanied by results of a larger number of beneficiaries reached and an additional
annual deforestation map developed, without additional costs. The efficiency thus is
rated substantial.

Efficiency rating: Substantial

Practice Area: Finance, Competitiveness and Innovation

Example C4.D. SME Competitiveness Project (P147705)
Economic and Financial Analysis

Ex ante analysis. A detailed economic and financial analysis was conducted for each of
the components at appraisal with a comprehensive list of assumptions and external
factors for each of the components. The analyses used ex ante CBA framework to assess
and monetize costs and benefits of the project and associated externalities. Two types of
Appendix C
Split Rating

costs were associated with the project: (i) Direct financial outflows under the project
components and (ii) indirect cost of public funds needed to finance the project. On the
benefits side, the principal benefits from the project were development of the small and
medium enterprise (SME) sector through increased capacity of government agencies,
several hundred local business advisors, and SMEs; additional business with local and
foreign large companies; increased access to finance; and better monitoring and design
of SME support policies. The analysis covered financial and economic revenues to be
generated by increased SME competitiveness and development of both supply and
demand sides. The analysis arrived at NPV of $22.13 million in the baseline case. The
CBA was based on a series of assumptions, available statistical data, and relevant
literature. The project’s NPV was $22.13 million using a 5.5 percent discount rate (with
projected multiplier effect of 7) for baseline case with total SME support of $46 million.
Further calculations were made for a pessimistic case and a worst case. It was estimated
that future sales of Supplier Development Program (SDP) participants would grow 2.5
times over five years, including gains of 70 percent in the first year after graduation from
the program.

Ex post analysis. The analysis was conducted on the second component of the project
(SDP) only. It is the component with the largest share of financing after the
restructurings—approximately 40 percent of the full project costs (approximately
$5 million), including the portion of certification costs covered by SMEs ($0.4 million)
and the value of government cofinancing (less than $0.4 million). According to the ICR
(29), the review of changes in the performance of SDP beneficiaries suggests that project
investments in designing and implementing the SDP were highly efficient. For every
dollar spent by the project in reimbursement (de facto subsidization) of SMEs’ costs on
certification, the beneficiaries managed to expand their sales by $105. If full costs of
certification are considered (given the actual reimbursement ratio of 74 percent), the
respective ratio is estimated as $78 dollars in extra sales per $1 spent on certification.
However, these efficiency estimates are based on the limited number of SDP
beneficiaries that provided their actual data on incremental sales and costs of
certification. No counterfactual data available to enable comparison. The selection of the
SDP beneficiaries for the analysis was not random. The available data are only from the
SMEs, which agreed to provide information, as it was not obligatory for the beneficiaries
to provide such information in the project. Out of these 53 successful beneficiaries, 15
reported increased sales. In addition, 12 out of 15 had obtained reimbursement of the
project cost. These 12 SMEs received more than a combined amount of $510,000 in
reimbursement, which amounts to approximately 57 percent of the total reimbursement
financed under the project (ICR, 29). The ICR also reports that the SDP contributed gains
to government budget; based on the data submitted by the same sample of 12 SMEs, on
average for every $1 spent by the project on the reimbursement of certification costs, the

28
Appendix C
Split Rating

state budget received $6.40 in additional taxes paid by SDP beneficiaries. For two of the
other major project components (clusters and factoring), CBA was not conducted.

Administrative and Operational Efficiency

The cost of major activities was kept within the original budget estimates and savings
were realized during the procurement of consulting services to support cluster
development under subcomponent 2.2; furthermore, the devaluation of the Kazakhstani
tenge resulted in savings as well. However, there were some shortcomings in the
project’s administrative and operational efficiency. The project had significant
implementation delays resulting from delayed counterpart contribution, lack of
understanding among the involved institutions on the implementation of the project,
and changing priorities of the government. These delays accordingly affected
procurement of some of the goods and services and led to cancellation of significant part
of the activities particularly under component 1. Because of delays in counterpart
funding, the salaries of Project Implementation Unit staff could not be paid on time, and
hired staff had to leave their positions, which adversely affected the Project
Implementation Unit’s performance of its fiduciary and coordination functions.

Although the SDP component was efficient, it is not possible to assess efficiency of the
project as a whole because of lack of evidence and underachievement. Furthermore,
shortcomings of operational and administrative efficiency affect the rating.

Efficiency rating: Modest

Practice Area: Health, Nutrition, and Population

Example C4.E. Philippines Social Welfare Development and Reform II (P153744)
The PAD’s economic analysis used information and benefit incidence analysis derived
from the previous phase of the Pantawid Pamilya Conditional Cash Transfer (CCT)
program to highlight positive effects on poverty and inequality (13–14). On average,
without CCT grants, poor beneficiary households’ incomes were 35 percent short of
meeting the minimum income level needed to afford their basic needs. Pantawid cash
transfers helped beneficiaries close this gap. The estimated reduction of total poverty
and food poverty among beneficiaries of the CCT Program by up to 6.7 percentage
points was significant. At the national level, it was also estimated that the program
reduced both total poverty and food poverty by up to 1.4 percentage points in 2013,
from a baseline of 26.4 percent total poverty and 12.5 percent of food poverty without
the CCT. These estimates fell between those found in two large CCT Programs: Mexico’s
Prospera, which was estimated to have reduced total poverty by 1.8 percentage points;
and Brazil’s Bolsa Familia, by 0.5 percentage point. Pantawid also reduced the overall
Appendix C
Split Rating

poverty gap from 9.1 percent to 8.2 percent and contributed to reducing national income
inequality as measured by the Gini coefficient by 0.5 points (49.2 without the program
and 48.7 with the program in 2013).

The PAD also reported on efficiency gains accruing to the health and education sectors.
In terms of health, Pantawid Pamilya promoted facility-based deliveries (7 in 10 live
births in five previous years compared with only 5.5 in 10 live births among non-
Pantawid mothers). It also showed that Pantawid Pamilya promoted access to
professional postnatal care, where 80 percent of mothers were checked by a skilled
health professional after giving birth, compared with 59 percent for non-Pantawid
mothers. In terms of education, Pantawid households living near the poverty threshold
spent 206 Philippine pesos more per school-age child per year compared with non-
Pantawid households. Households’ higher investment in education translated into
higher gross enrollment rates of children in both age groups 6–11 years and 12–14 years,
and these increases made enrollment rates equally high among poor and nonpoor
children. Importantly, the likelihood of completing elementary education among
Pantawid Pamilya children of 13–16 years old (83 percent) was higher than among same-
age poor children who were not Pantawid Pamilya children (79 percent). The positive
effect was observed in both boys and girls.

In addition, the ICR offered arguments about financial sustainability. The World Bank’s
contribution over 2016–22 accounted for approximately 7 percent of the government
CCT budget; therefore, the loan would have a limited fiscal impact on the overall
Department of Social Welfare and Development budget and would not raise
sustainability risks. Loan proceeds were fully used.

Nevertheless, the ICR noted that in terms of delivering targeted assistance to children
from poor households, efficiency was not as high as originally expected in view of weak
targeting performance and the exclusion of pregnant women and young children (ICR,
17).

Efficiency rating: Substantial

Practice Area: Education

Example C4.F. Lao People’s Democratic Republic Second Global Partnership for
Education (P149130)
A CBA was undertaken at the time of appraisal, analyzing three scenarios: high impact
(gaining the equivalent of one full year of additional education), medium impact (half a
year), and low impact (two months). Under the three scenarios, using current returns to
education of 10 percent per year completed, the NPV of the benefits under the three

30
Appendix C
Split Rating

scenarios would be, respectively, almost 20 times, 10 times, and 2 times larger, than the
cost.

The ICR undertook an end-of-project CBA, also exploring three scenarios of learning
outcomes for the school-based grant (SBG) program (learning outcomes of children in
the reading readiness program were assumed to be the same across the three scenarios
at 0.61 years). The ICR’s learning outcome estimates were less ambitious than those
assumed in the PAD’s analysis. Under the ICR’s low estimate scenario (assuming no
improvement in learning outcomes), using actual component costs and a real discount
rate of 5 percent, the NPV of the project at completion is estimated at $15.35 million, and
the corresponding internal rate of return is estimated at 6.85 percent. Under its
intermediate scenario (assuming that improvement in learning outcomes for the school-
based management (SBM) program would be equivalent to 0.125 years of schooling), the
NPV increases to $79.48 million and the internal rate of return to 11.23 percent. The
optimistic scenario (assuming that improvement in learning outcomes for the SBM
program would be equivalent to 0.25 years of schooling), the NPV increases to
$144.07 million and the internal rate of return to 13.73 percent. All three scenarios
assumed improvement in the learning outcomes of children under the reading readiness
program by 0.61 years. The present values of the benefits of the two components (SBGs
and reading readiness program access) are greater than the present values of the
investments across all three scenarios (ICR, 52). The ICR notes that implementation
delays were factored into the CBA, employing a conservative assumption that the first
cohort of student beneficiaries started to enter the program three years after the project
was initiated.

There were implementation efficiencies that facilitated achievement of outcomes.
Virtually all (98 percent) of the Global Partnership for Education funds were used.
Savings achieved under revised component 2 (preprimary reading readiness program)
were reallocated to SBGs, allowing considerably more financing for schools during the
project’s life. Refinement of project targeting criteria increased the total number of
disadvantaged districts from 80 to 88, all of which benefited from higher levels of
government- and project-financed grants than originally planned.

However, spending of these grants (at 44 percent of total grants) on learning materials
by these schools fell short of the 60 percent threshold specified in the SBG manual. There
were also moderate implementation inefficiencies. As noted in the ICR (24), the project
experienced delays during the first two years of implementation. It took a full year to get
a consensus among development partners to agree on SBM training material, and the
delivery of SBM training took four months longer than anticipated. Procurement of
technical advisors to develop and implement the reading development pilot
Appendix C
Split Rating

intervention took approximately two years, and the reading pilot design was changed in
December 2017. Ultimately, project restructuring in June 2018 was undertaken to avoid
duplication of efforts with the national primary education curriculum activities of the
Australian Government Department of Foreign Affairs and Trade’s Basic Education
Quality and Access in Lao PDR program. With the agreement of the government of Lao
PDR, the World Bank, and the Lao PDR Education Sector Working Group, the project
moved its reading development focus away from primary school grades 1 and 2, and
instead developed a reading readiness program for five-year-old children in preprimary
education. The onset of the COVID-19 pandemic and the consequent closing of schools,
along with severe limits on the ability to conduct project activities, supervision, and
monitoring caused by the pandemic, resulted in unavoidable implementation delays,
despite efforts to mitigate them. The closing date was extended three times for 2.5 years.

Efficiency rating: Modest

Practice Area: Social Protection and Jobs

Example C4.G. Panama Strengthening Social Protection and Inclusion System
(P155097)
The project likely produced important economic benefits, though the ICR did not
attempt to quantify these benefits because of a lack of appropriate data. An impact
evaluation for the Social Cohesion Program (Programa Cohesión Social) indicated that
the productive inclusion programs supported by this operation, especially the Social
Cohesion Program, have had direct economic benefits resulting from increased
productivity among beneficiaries (ICR, 22). Similarly, better targeting of Ministry of
Social Development (Ministerio de Desarrollo Social; MIDES) CCTs, and continuous
CCT payments during the COVID-19 pandemic, contributed to strong indirect economic
benefits by reducing poverty among beneficiaries.

There were, however, important implementation inefficiencies. Throughout the life of
the project, implementation was delayed several times because of multiple factors that
affected efficiency significantly. The operation was effectively stalled for the first four
years of implementation (ICR, 23). The project team later added that there were mainly
four key factors behind this lack of progress.

• The first factor was a lack of sufficient budgetary allocation from the Ministry of
Economy and Finance (Ministerio de Economía y Finanzas). As per the National
Budgetary Law of Panama, the project had to wait to receive funds until it was
signed. The project received less than $2 million between 2017 and 2018, and
such a low budget allocation was mostly due to fiscal consolidation in the
country during that time.

32
Appendix C
Split Rating

• Second, there was internal reorganization in MIDES. Between 2016 and 2017,
MIDES had gone through an internal reorganization. A new directorate was
created to oversee all CCT programs. The Project Implementation Unit was
moved under this new directorate. Project coordinators were changed several
times, and decisions were stalled for a long period of time because of these
changes, including significant revisions in the project’s operation manual.

• The third factor was the delay in signing of an interinstitutional agreement
among MIDES, the Ministry of Labor and Workforce Development (Ministerio
de Trabajo y Desarrollo Laboral), and the National Institute of Vocational
Training for Human Development (Instituto Nacional de Formación Profesional
y Capacitación para el Desarrollo Humano), all of which were responsible for
implementing component 3 activities. The project team noted that it took almost
a year and a half to get this interinstitutional agreement signed.

• The fourth factor was the shift in governmental priorities toward addressing
extreme poverty. The government initially wanted to focus more on urban areas
by adding training and intermediation through the Ministry of Labor and
Workforce Development and the National Institute of Vocational Training for
Human Development. However, when the government changed, the focus
shifted again to rural and Indigenous areas. The change in government also
affected the implementation of component 2 in a significant way. The new
government converted all temporary social workers to staff in 2016 and created a
hold in implementation of activities.

Soon after the first restructuring, the fiscal situation in Panama deteriorated. Despite its
commitment to support the project with $15 million, the Ministry of Economy and
Finance could disburse only $4 million in the 2019 budget cycle. This was more than the
previous year’s disbursement but significantly less than the requirement to implement
all components as planned. Between 2019 and 2020, the fiscal situation was the key
challenge that affected project implementation adversely.

The 2021 restructuring helped improve implementation efficiency, despite some initial
challenges due to the COVID-19 pandemic. Even after the cancellation of $15 million
during the 2018 restructuring, the World Bank was able to rapidly provide financial
support to the country when the COVID-19 crisis hit. The ICR mentioned that MIDES
CCTs were at risk because of the tight fiscal situation after the initial months of the
pandemic, but project implementation picked up speed after the third restructuring in
2021. The World Bank was able to restructure the project and provided the needed
financial resources to secure payment for MIDES CCTs.
Appendix C
Split Rating

Efficiency rating: Modest

Practice Area: Governance

Example C4.H. Modernization and Innovation for Better Public Services In
Argentina (P157136)
The PAD provides a detailed breakdown of expected efficiency gains by project
subcomponents, focusing on the areas in which the benefits can be quantified: (i)
reduced transaction costs and time savings for processes related to SME certification and
civil registries (subcomponent 1.1), (ii) procurement (subcomponents 1.2 and 2.2), and
(iii) workflow management (subcomponent 3.3). The World Bank expected the project to
yield an NPV of $203.1 million at appraisal. The World Bank estimated the benefits of
reduced prices for commonly procured goods using framework agreements (financial
benefits) and in terms of time saved by citizens, businesses, and government staff
(economic benefits). The analysis assumed an exchange rate of 15 Argentine pesos per $1
and a 6 percent discount rate over 10 years and accounted for 10 percent in price and
physical contingency for unexpected delays. The subcomponent cost values provided in
annex 4 of the PAD (Economic and Financial Analysis) differ significantly (are higher)
from the main text of the PAD and ICR (after the 2019 restructuring), which provide cost
at the component level only. The difference could be attributed (although the PAD is not
explicit about it) to the inclusion in the analysis of estimated future costs to sustain the
interventions (which are not specified). The PAD does not provide an estimate of the
expected ERR. Using the simple standard formula for ERR calculation and the estimated
project cost of $88 million at appraisal yields an expected ERR of 130 percent.

Subcomponent 1.1: Improvements in government-to-citizen and government-to-
business services. The total estimated cost for this subcomponent per Annex 4 of the
PAD was $32.3 million (the estimated cost of the whole component 1 is $17.9 million
according to the PAD and the actual cost is $24.9 million per ICR), including
investments in the Citizen Digital Profile, selected services to SMEs, and the automation
of civil registry processes, and estimated future costs to sustain the intervention. The
benefits (financial and economic savings) were expected to yield an NPV of
$100.2 million from digitizing processes related to SMEs and civil registries, and
accounting for time and travel cost savings for SME representatives, additional fees that
can be generated as a result of processing higher volumes of SME certifications, and the
reduced transaction costs related to government staff time.

Subcomponents 1.2 and 2.2: Government-to-government services and electronic
procurement system. The total cost of two subcomponents was estimated at
$37.3 million in Annex 4 of the PAD (the estimated cost of components 1 and 2

34
Appendix C
Split Rating

combined, per PAD, is $38.8 million and the actual cost $50.8 million, per ICR),
including investments for the expansion of framework agreements, updated e-
procurement system, training of personnel, elaboration of a National Procurement
Strategy, and estimated future costs to sustain the intervention. The total value of
savings was expected at $41.9 million, resulting from reduced prices of commonly
procured goods using framework agreements.

Subcomponent 3.3: Information and communication technology infrastructure. The
total estimated cost for this subcomponent per Annex 4 of the PAD was $74.9 million
(the estimated cost of Component 3 at approval [PAD] was $49.9 million, and the actual
cost of $37.9 million after restructuring [ICR]) for investments in ARSAT’s capacity to
host information and communication technology services for government agencies,
consultancy services, software licenses, hardware infrastructure, operational costs, and
estimated future costs to sustain the intervention. Economic benefits from staff time
saved by electronic workflow management were expected at an NPV of $99.9 million.

The ICR provides a summary of efficiency gains (annex 4, tables 7–9) in the aggregate,
not breaking it down by subcomponents (as in the PAD), and notes that it was not
possible to replicate calculations similar to the PAD at closing because of the volatility of
the peso to dollar exchange rate throughout project implementation. Although this
would have been a valid explanation for possible discrepancies in expected and actual
efficiency gains as measured in US dollars, this does not explain why a similar analysis
(by subcomponent) was not done at completion.

The ICR notes that the project led to efficiency gains in services for government agencies
and users (citizens and firms), which are expected to outlast the project’s lifetime. The
government can provide administrative services more quickly, and deploying the e-
filing system and the remote access platform brought time savings. The ICR calculates
that the project generated an estimated $159 million in savings for citizens from the
efficiencies gained in digitizing government services and an estimated $136 million in
efficiency gains related to time savings for public sector employees (assuming the use of
digital service platforms results in 25 percent time savings for civil servants). With this,
the ERR for the project is calculated at 78.97 percent, using the savings related to
citizens’ use of digital services ($159 million). The ICR, however, does not provide
details on how it arrived at that number. The simple standard formula for ERR
calculation—(159(savings) − 78(project cost at closing) / 78(project cost at closing) ×
100)—arrives at 103 percent or 80.7 percent using the total appraisal cost of $88 million,
including borrower contribution.

Other nonquantified efficiency gains included (i) improved user interface with service
providers to ensure that citizens and businesses have easier, faster access to a wider
Appendix C
Split Rating

range of services and spend less time dealing with administrative processes; (ii) faster
and easier application and tracking of government administrative services, allowing
citizens and firms to track the status of their digital services through their online
accounts; (iii) reduction in environmental costs—reduced paper use and greenhouse
emissions due to travel savings; and (iv) improved business continuity due to the
automation and digitalization of public administrative services and critical government
records.

Overall, this review broadly concurs with the ICR conclusions regarding the clear
efficiency gains of the project. The ICR did not exactly follow the analytical path charted
at the appraisal (by subcomponents and specific activities), and the methods of arriving
at the final number are not specified. Nevertheless, the analysis presented in the ICR (at
the aggregate level) appears to be broadly acceptable and credible in terms of arriving to
an overall positive assessment and high ERR.

Efficiency rating: Substantial

Practice Area: Governance

Example C4.I. State and Local Governance Reform Project (P133045)
The project’s benefits are mostly intangible and cannot be quantified and compared with
the project’s costs in the conventional sense, because the core of the project focused on
long overdue institutional reforms and capacity building and training of state and local
government officials that cannot be priced. The audit, procurement, and other
institutional reforms of PFM systems in the participating states are expected to produce
significant and long-lasting monetary or fiscal benefits by stemming financial corruption
and generating fiscal savings. They could enable the financing of more and better social
services that directly reduce poverty. The economic benefits could be substantial and
accrue in the medium to longer term but cannot be attributed to any particular outcome
of the project.

Nevertheless, the efficiency of some aspects of the project can be gauged (though
imperfectly) through the following:

1. Implementation efficiency. Delivering training and capacity-building activities
locally led to significant savings in travel and boarding costs and enabled more
persons to benefit from the training. The skills generated for PFM are public
goods that will have spillover effects and efficiency gains in the medium term
because they contribute to better PFM and social service delivery.

36
Appendix C
Split Rating

2. Input costs. The single largest cost incurred on any activity under the project was
the implementation of the State Integrated Financial Management Information
System across the states. The cost for deploying this system in each of the five
states that implemented the activity was approximately $3.8 million. This
amount is much lower than the sum of $6.6 million estimated by IEG as the
average cost for the implementation of 87 integrated financial management
information systems across 51 countries (48 of which are countries eligible for the
International Development Association; World Bank 2021).

3. Project management costs, including capacity building under the first
component, 2 were reasonable given the context. At project closure, project
management costs were under 12 percent of total project costs compared with a
projection of 19 percent at inception. At 12 percent, the project’s management
costs were within generally accepted levels to be considered efficient, especially
in Nigeria’s partially fragile and conflict country context.

4. The cost/return ratio was favorable. The total project financing was €54.9 million,
of which €47.8 million was disbursed. During the implementation period (2014–
20), approximately $5.65 million was expended on the tax administration reform
subcomponent. During the same period, the total additional revenue collection
across the six states was approximately $601 million. 3 Although the project’s
investments in tax administration reforms may not have been the only
intervention or factor that led to this progress, they did make an important
contribution.

5. The project’s investments in tax administration reform were generally well
targeted. Additional revenue yield surpassed the total project investment in tax
systems and overall project investments by over 10,000 percent and
1,000 percent, respectively.

6. The project benefited from a consistent and largely field-based team throughout
implementation. This facilitated continuity in implementation support,
consistency in institutional memory, and quick decision-making during the long
project cycle. Although the project did not perform well on timeliness, the three

2As a cost saving measure, the Project Implementation Unit adopted direct sourcing and
provision of training and workshop venues and logistics, rather than out-sourcing to consultants.
3$601 million is the total of the variance from year to year across the six states between calendar
years 2014 and 2020.
Appendix C
Split Rating

restructurings did not escalate the total cost. Most of the extensions compensated
for the time lost between project initiation in 2012 and approval in 2014. The first
two restructurings aimed to accommodate the five-year term that was deemed
essential originally to implement the project. The last restructuring was
necessitated by the COVID-19 pandemic.

Overall, project efficiency was substantial. The intangible benefits of the institutional
PFM reforms and skills and PFM capacity building in the states will be invaluable for an
efficient PFM system that delivers more and better public services in the medium to
longer term. Although proof of better service delivery is not available for five states, one
concrete example in the ICR suggests that at least one state invested in new social
services: “Improved IGR [internally generated revenue] receipts enabled the Kano State
government over the same period (2014–21) to successfully upgrade to world class
status several health care facilities, including the Giginyu Specialist Hospital and
Pediatrics Hospital, Zoo Road, as well as investing in the rehabilitation of several
schools and teacher professional enhancement programs.” (ICR, page 19) Moreover, the
costs of delivering critical infrastructure like the State Integrated Financial Management
Information System were below average, the cost of project management was below
budget, and the increase in internally generated revenue in the states was significant.

Efficiency rating: Substantial

C5. Writing Lessons with Impact
Criteria for identifying and writing lessons with impact:

1. Actionable. Lessons need to be oriented to decision points and behavior change.

2. Relevant. Lessons need to be useful for other projects or programs.

3. Nongeneric. Lessons need to be detailed, distinct, and informative.

4. Specific. Lessons need to be project, program, or context specific.

5. Valid. Lessons flow logically from evidence that is available.

These criteria are discussed below in detail:

1. Actionable. Lessons need to be oriented to decision points and behavior change:

o The lesson provides information on when, and how to apply a lesson, and
who applies it.

38
Appendix C
Split Rating

o If there is enough evidence, include the effect that the decision had on project
preparation, design, or implementation.

o If there is enough evidence, include the effect that the decision had on
outcomes, effectiveness, or efficiency.

Weak example: Procurement mistakes by the project team can result in project
delays.

Better example: In contexts where project implementors lacks capacity and
expertise, without financial and procurement training for all implementing
partners at the beginning of each project, staff make costly procurement mistakes,
and this can result in delays in the completion of construction activities.

2. Relevant. Lessons need to be useful for other projects or programs and lead to
better decisions in similar circumstances.

o Avoid lessons that only highlight unique innovations or aspects of a
particular project or program or untested innovations.

o If novel approaches are being tested, identify whether sufficient evidence has
been provided and provide lessons that prevent others from making similar
mistakes.

Weak example: Cash-based transfers to pregnant adolescents in rural and
conflict-prone zones of South Sudan helped adolescents pay transportation costs
to health clinics.

Better example: Pilot efforts related to the access to health services of pregnant
adolescents in hard-to-reach areas can have limited impact if models cannot be
scaled or easily replicated—particularly cash-based transfers for populations in
vulnerable situation.

3. Nongeneric. Lessons need to be detailed, distinct, and informative. They must
contain enough detail to ensure uptake and future applicability.

o Lessons do not have to be “novel.” They can continue to emphasize
important project drivers such as improvement on project appraisal
processes, M&E, partner engagement, and financial efficiency.

Weak example: Transportation projects need to include indicators that measure
all outcomes of the project, including market access.
Appendix C
Split Rating

Better example: Transportation projects that only measure direct effects of road
construction, such as reduced travel time, will not be able to determine if they
achieved market access.

4. Specific. Lessons need to be project, program, or context specific. Include as
much information about the project, program, decision, and context as possible
while still being concise and clear about the lesson.

Weak example: Agricultural yields are being affected by the changes in climate
and targets need to adjust accordingly.

Better example: Agricultural yields were unreliable measures of project’s
impact due to droughts; unfortunately, targets were not adequately adjusted
throughout the project to compensate.

5. Valid. Lessons flow logically from evidence that is available.

o Avoid making sectoral or technical evidence leaps even if you have
additional information. Lessons need to be supported by specific project or
program evidence, not necessarily general sectoral evidence or emerging
practices.

o Avoid speculation—if something was found not to work, do not suggest
what should have been done unless there is a strong basis of evidence.

o Lessons are verifiable given that every lesson needs to be supported by
evidence.

Weak example: Teaching training programs increase test results and quality of
education.

Better example: Without sufficient and sustained professional development,
teacher training programs can fail to achieve sustained improvements in education
quality.

C6. Quality of Implementation Completion and Results Report
Examples

Example C6.A. Afghanistan Irrigation Restoration and Development Project
(P122235)
Quality of evidence. The ICR acknowledged that the results framework suffered from
weaknesses (see section 9a). Despite the ICR noting that data on different indicators

40
Appendix C
Split Rating

were triangulated through several sources to assess their validity, this Review has
concerns on the accuracy of crop yield data.

Quality of analysis. The ICR provided clear linking to the extent possible between
evidence and findings and used the evidence base to serve the arguments under the
different sections, in particular the discussion on outcomes.

Lessons. Lessons reflected the project experience and were based on evidence and
analysis.

Results orientation. The ICR included a comprehensive discussion on the achievement
of the PDO. However, discussion on outcomes could have been more balanced between
reporting on the achievement of outcome indicators and what the project actually
achieved in the field.

Consistency with guidelines. The ICR successfully used the available data to justify
most of the assigned ratings. Discussion of outcomes was adequate. There are concerns
on the robustness efficiency analysis, given the data inconsistencies and questions on the
accuracy of yield increments.

Conciseness. The ICR provided comprehensive coverage of the implementation
experience and candidly reported on shortcomings. The reporting on safeguards
included an explicit statement on compliance. The ICR commented on the status of the
final audit reports for the project. However, the discussion of M&E design could have
benefited from further details with regards to the results framework shortcomings.
Finally, the ICR did not explicitly state the reason(s) for the three extensions of the
closing date, did not discuss the risks that materialized during implementation, and did
not report on the appropriateness of the mitigation measures mentioned in the Project
Paper. Also, procurement was not explicitly discussed, and some acronyms were not
reflected in the list at the beginning of the ICR.

Overall, the quality of the ICR is rated substantial despite some shortcomings.

Example C6.B. Mekong Integrated Water Resources Management Project—
Phase II (P124942)
The ICR provided a comprehensive record of the project in keeping with Operations
Policy and Country Services guidelines. The narrative offered the context behind the
integrated water resource management at the country level and its contribution to the
regionwide support for sustainable use of the Lower Mekong Basin. The ICR provided
an exhaustive picture behind the World Bank’s presence in the country and the sector
(ICR, paragraphs 5, 6, 8, 24, 27, footnote 8).
Appendix C
Split Rating

The ICR was results oriented, with the TOC showing the reader how the components
and activities led to achieving the PDOs (ICR, paragraph 49), and how the Decision
Support System tools and modeling capacity supported the implementation of the Water
Law (ICR, paragraph 48). The ICR also justified the effective use of reduced resources to
achieve the outcomes (ICR, paragraph 43). Table 2 was particularly helpful in guiding
the discussion about changes made to the indicators to justify the TOC at closing.

The ICR was internally consistent, cross-referencing sections with the results achieved.
Candid assessment acknowledged the shortcomings attributed to early implementation
delays, the initial lack of clarity with regard to roles and responsibilities of various
entities (ICR, paragraph 52), and how this lack affected achieving the PDOs (ICR,
paragraph 51).

Annex 5 from the government’s comments on the ICR reinforced the drawback of
having five TTLs over the implementation period, resulting in insufficient hands-on and
tailored training in hydraulic modeling as part of using Decision Support System.

Lessons were drawn from project challenges.

There were very minor inconsistencies such as the reference to the 2012 Water Resources
Law in various places as 2012 Water Law (ICR, paragraphs 3, 48, and 77) or the 2013
Water Resources Law (ICR, paragraphs 27, 50). The task team acknowledged the
typographical error.

42
Appendix C
Split Rating

References
World Bank. 2022. “P151425, Lao People’s Democratic Republic: Health Governance and Nutrition
Development”. Independent Evaluation Group, Implementation Completion and Results Report
Review. World Bank, Washington, DC.

World Bank. 2022. “P145861, Mali Skills Development and Youth Employment.” Independent
Evaluation Group, Implementation Completion and Results Report Review. World Bank,
Washington, DC.

World Bank. 2022. “P145544, Lao PDR Early Childhood Education Project.” Independent Evaluation
Group, Implementation Completion and Results Report Review. World Bank, Washington, DC.

World Bank. 2018. “P125359, Nepal Community Action for Nutrition Project” Independent
Evaluation Group, Implementation Completion and Results Report Review. World Bank,
Washington, DC.

World Bank. 2022. “P128909, Liberia Health Systems Strengthening Project.” Independent
Evaluation Group, Implementation Completion and Results Report Review. World Bank,
Washington, DC.

World Bank. 2021. “P107314, Kenya Nairobi Metropolitan Services Project.” Independent Evaluation
Group, Implementation Completion and Results Report Review. World Bank, Washington, DC.

World Bank. 2021. “P143988, Myanmar Electric Power Project.” Independent Evaluation Group,
Implementation Completion and Results Report Review. World Bank, Washington, DC.

World Bank. 2022. “P130182, Armenia Education Improvement Project.” Independent Evaluation
Group, Implementation Completion and Results Report Review. World Bank, Washington, DC.

World Bank. 2021. “P21455, Nigeria: State Employment and Expenditure Project.” Independent
Evaluation Group, Implementation Completion and Results Report Review. World Bank,
Washington, DC.

World Bank. 2022. “P131558, Vietnam: Transmission Efficiency Project.” Implementation
Completion and Results Report. ICR00005828, World Bank, Washington, DC.

World Bank. 2022. “P131558, Vietnam: Transmission Efficiency Project.” Independent Evaluation
Group, Implementation Completion and Results Report Review. World Bank, Washington, DC.

World Bank. 2023. “P153863, Senegal Integrated Water Resources Management Project.” Independent
Evaluation Group, Implementation Completion and Results Report Review. World Bank,
Washington, DC.

World Bank. 2021. “P095003, Nigeria Rural Access and Mobility Project Phase 2.” Independent
Evaluation Group, Implementation Completion and Results Report Review. World Bank,
Washington, DC.
Appendix C
Split Rating

World Bank. 2017. “P115396, Azerbaijan—Public Investment Capacity Building.” Independent
Evaluation Group, Implementation Completion and Results Report Review. World Bank,
Washington, DC.

World Bank. 2023. “P143185, Brazil Development of Systems to Prevent Forest Fires and Monitor
Vegetation Cover in the Brazilian Cerrado.” Independent Evaluation Group, Implementation
Completion and Results Report Review. World Bank, Washington, DC.

World Bank. 2023. “P147705, SME Competitiveness Project.” Independent Evaluation Group,
Implementation Completion and Results Report Review. World Bank, Washington, DC.

World Bank. 2022. “P153744, Philippines Social Welfare Development and Reform II.” Independent
Evaluation Group, Implementation Completion and Results Report Review. World Bank,
Washington, DC.

World Bank. 2023. “P149130, Lao People’s Democratic Republic Second Global Partnership for
Education.” Independent Evaluation Group, Implementation Completion and Results Report
Review. World Bank, Washington, DC.

World Bank. 2023. “P155097, Panama Strengthening Social Protection and Inclusion System.”
Independent Evaluation Group, Implementation Completion and Results Report Review. World
Bank, Washington, DC.

World Bank. 2023. “P157136, Modernization and Innovation for Better Public Services In Argentina.”
Independent Evaluation Group, Implementation Completion and Results Report Review. World
Bank, Washington, DC.

World Bank. 2022. “P133045, State and Local Governance Reform Project.” Independent Evaluation
Group, Implementation Completion and Results Report Review. World Bank, Washington, DC

World Bank. 2022. “P122235, Afghanistan Irrigation Restoration and Development Project.”
Independent Evaluation Group, Implementation Completion and Results Report Review. World
Bank, Washington, DC

World Bank. 2021. “P124942, Mekong Integrated Water Resources Management Project—Phase II.”
Independent Evaluation Group, Implementation Completion and Results Report Review. World
Bank, Washington, DC