Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
Report Number : ICRR0021060
1. Project Data
Project ID Project Name
P117103 AF: National Solidarity Program III
Country Practice Area(Lead) Additional Financing
Social, Urban, Rural and Resilience P159307
Afghanistan
Global Practice
L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD)
IDA-H6030,TF-98459 30-Sep-2015 1,146,604,005.86
Bank Approval Date Closing Date (Actual)
29-Jun-2010 30-Sep-2015
IBRD/IDA (USD) Grants (USD)
Original Commitment 40,000,000.00 1,107,255,800.00
Revised Commitment 39,994,261.40 1,104,637,406.92
Actual 41,966,598.94 1,104,637,406.92
Prepared by Reviewed by ICR Review Coordinator Group
Hassan Wally Lauren Kelly Christopher David Nelson IEGSD (Unit 4)
PHPROJECTDATATBL
Project ID Project Name
Afghanistan: NSPIII Additional
P159307
Financing ( P159307 )
L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD)
0.00
Page 1 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
Bank Approval Date Closing Date (Actual)
29-Jun-2016
IBRD/IDA (USD) Grants (USD)
Original Commitment 0.00 0.00
Revised Commitment 0.00 0.00
Actual 0.00 0.00
2. Project Objectives and Components
a. Objectives
This is the third phase of the National Solidarity Program (NSP III). The Project Development Objective
(PDO) of this third phase, as articulated in the Emergency Project Paper (EPP, p. 7), was identical to the
PDO as stated in the Financing Agreement (FA, p. 4) and the Trust Fund Grant Agreement (p. 4), and aimed
to:
"build, strengthen, and maintain community development councils (CDCs) as effective institutions
for local governance and social-economic development."
b. Were the project objectives/key associated outcome targets revised during implementation?
No
PHEVALUNDERTAKENLBL
c. Will a split evaluation be undertaken?
No
d. Components
NSP III was supported by three components:
1. Capacity Building of Community Development Councils (CDCs) (appraisal cost: US$291 million,
actual cost: US$212.15 million). This component would continue to establish CDCs and to build their
capacity to: (a) function as a village level governance body for continued empowerment of village
communities, bringing their voices into the government decision making process for subnational
development planning, resource allocations and delivery of rural development programs, and (b) facilitate
communities' participation in the various sectoral programs operating in rural areas (rural roads, irrigation,
Page 2 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
horticulture and livestock, water supply, education, health, microfinance and etc), harvesting benefits of
synergy and complementarity among various programs, maximizing development outcomes and reducing
delivery costs. Capacity building activities would focus on: participatory planning of community
development priorities; community governance and accountability systems and processes, and
mechanisms for conflict resolution; participatory planning for broad-based community recovery from
conflict; gender awareness and enhancing women's effective involvement in the economic and social
activities of the communities; community participatory monitoring of programs being implemented in the
localities; basic bookkeeping and contracting for management of community-implemented investment
schemes; and communication with subnational and national government.
2. Community Grants for Economic and Social Development (appraisal cost: US$1,093 million,
actual cost: US$902 million). The component would provide block grants to 27,720 CDCs to fund priority
investment sub-projects for rural and social development. The first block grant would be given to the10,320
newly mobilized CDCs; and a second block grant would be given to 17,400 existing CDCs that were
operating under NSP-I and NSP-II, that have successfully utilized their first block grant and are maintaining
completed subprojects. These communities must also hold re-elections, update their CDP and prepare
community action plans before receiving a second block grant. The block grant amount would be
determined by the number of families in a community at the rate of US$ 200 per family, but with a
maximum ceiling of US$60,000 per community. The average block grant per community under NSP-II was
US$33,500 and this would be used as the basis for estimating the block grants needed for this project.
3. Project Implementation Support (appraisal cost: US$122.3 million, actual cost: US$142.7 million).
This component would cover expenditures associated with the Ministry of Rural Rehabilitation and
Development (MRRD) overall management (excluding the Facilitating Partners) and oversight of NSP,
including the salaries of all contractual staff engaged by MRRD for NSP, technical assistance needed to
support MRRD’s Program Management Office for NSP, and MRRD incremental operating costs pertaining
to NSP, costs associated with monitoring and evaluation of the program, and an agreed share of salaries
of MRRD contractual staff who perform certain headquarters functions for the MRRD as a whole.
Revised Components. The November 2015 restructuring introduced the Maintenance Cash Grant (MCG)
scheme, in support of the Government’s Jobs for Peace program, as well as scaling back program size to
reflect a shortfall in total projected funding for the NSP III, primarily from the ARTF.
Specific changes to the components included:
Component 1. The NSP’s Provincial Management Units (PMUs) took on the role of supporting CDCs in
developing maintenance plans to foster effective MCG use as the Facilitating Partners (FPs) contracts
were set to close at end-September 2016. PMUs (funded by Component 3) also took on the remaining
work of FPs who had withdrawn from communities for security reasons.
Component 2. A new category of MCGs was introduced. The projected number of first block grants to
newly mobilized CDCs was reduced slightly from 10,320 to 10,200. Second block grants to existing CDCs
would be reduced from 17,400 to 10,000. The MCG program aimed initially to cover 3,700 communities in
Page 3 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
12 provinces. An Additional Financing (AF) of $57.3 million in June 2016 aimed to expand the MCG
program to an estimated 4,200 communities in 17 provinces.
Component 3. This component was amended to cover expenditures by the Ministry of Rural Rehabilitation
and Development's (MRRD) Project Implementation Unit (PIU), the NSP, and its Provincial Management
Units to support implementation of the Maintenance Cash Grants, including overall management,
oversight, technical assistance, monitoring and evaluation, and the incremental operating costs and
construction of provincial administration buildings. Funds for MRRD institutional strengthening under this
component, up to $1.94 million were earmarked to support MRRD’s Change Management Program.
e. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Overall, there is a lack of clarity on programmatic costs and financing in the ICR. The following data was
made available:
Project Cost. This is the third phase of NSP. The total estimated cost for the entire program was
US$1.51 billion and the actual costs were US$1.23 billion, or 83.5% of the appraisal amount (according to
Annex 3 of the ICR, p.54). This amount differs from the actual dispersed amount reported in the ICR's
Basis Data sheet (p.2) which reports total actual disbursements to be US$1.15 billion.
Financing. For the third phase of NSP - the phase being assessed - the estimated IDA contribution was
US$40 million and the actual disbursed level is US$41.97 million according to the ICR Basic Data
Information Sheet (p.2). Other financing came from contributions made available through the ARTF-
Afghanistan Reconstruction Trust Fund (TF-98459) reported as US$1.11 billion in actual disbursements in
the ICR (Basic Data Information Sheet, p.2).
Additional Financing. In June 2016, the project received US$57.3 million in Additional Financing from
the ARTF funded by contributions from the United States Agency for International Development (USAID)
and the United Kingdom Department for International Development (DFID).
Borrower Contribution. The Community Development Councils (CDCs) were expected to contribute
US$164 million. The ICR did not report on the actual amount contributed.
Dates. The project was expected to close on September 30, 2015. The closing date was extended twice
to March 31, 2017. The two extensions were necessary to accommodate implementation delays due to
security concerns. Also, to ensure that all sub-project construction was fully completed, any remaining
balances in Community Development Council bank accounts were used or returned, and the Additional
Financing funds for the Maintenance Cash Grants were fully disbursed. The extension also ensured that
there was a smooth transition of institutional arrangements to support the implementation of the Citizens’
Charter Afghanistan Project - which was approved in October 2016 (ICR, p. 11).
Page 4 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
Restructuring. The project was restructured four times, all of which were Level 2 restructurings. The first
was on Sepmteber18, 2015 when the amount disbursed was US$41.97 million, in order to extend the loan
closing date. The second was on November 20, 2015, when the amount disbursed was US$41.97 million,
in order to introduce modifications to the Results Framework, change in components and cost, extend
the loan closing date, allow cost reallocation between disbursement categories, change in disbursements
arrangements, change in institutional arrangements; and change in implementation schedule. The third
was on June 29, 2016, when the amount disbursed was US$41.97 million, in order to approve an
Additional Financing worth US$57.3 million, change in components and allow cost reallocation between
disbursement categories. The fourth was on March 24, 2017, when the amount disbursed was US$41.97
million, in order to allow cost reallocation between disbursement categories.
3. Relevance of Objectives
Rationale
The PDO of NSP III was highly relevant. The third phase objective was focused on local governance.
Specifically, the PDO aimed to build, strengthen and maintain Community Development Councils as effective
institutions for local governance and social-economic development. This focus on strengthening and
maintaining institutional gains that had been made over the prior two phases was a highly relevant aim. The
pursuit of CDCs as "effective institutions for local governance" was very much in line with the core aim of the
program as it was conceived in the early 2000s that linked participation, ownership and trust in local
decision-making and development processes to wider stabilization and development aims. In Afghanistan,
"participation is securitization" (Lockhart, 2009). Service delivery (socio-economic development) is a critical
part of the trust quotient in a unitary state where funds flow from the capital; but services in the context of the
NSP (and the block grants that fund them) were also the "means" by which the program promulgated a more
inclusive (e.g, including women) and democratic (e.g. including the use of secret ballot) form of decision-
making. As stated in the Borrower's comments (ICR p.62), the "client" views "governance as identifying and
addressing the broad field of coordination problems related to development, including a general restructuring
of decision-making to achieve greater inclusion and participation, and ensuring a link between social action
and discussion/negotiation."
The focus on governance is central to the Country Partnership Framework (CPF) for Afghanistan (FY17-20).
The first pillar of the CPF calls for building strong and accountable institutions; and objective 1.3 aims to
improve service delivery through enhanced citizens’ engagement with the state. This is expected to be
achieved by building the capacity of CDCs to implement, plan and monitor local service delivery and hold
government to account. Service delivery aspects of the program were and continue to be in line with
Afghanistan's National Development Strategies, including during the project appraisal period. The National
Solidarity Program has been one of the Government's main implementation tools in support of rural
reconstruction and rural livelihoods promotion, in line with its development strategies.
Page 5 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
Rating
High
4. Achievement of Objectives (Efficacy)
PHEFFICACYTBL
Objective 1
Objective
Objective: to build, strengthen, and maintain community development councils (CDCs) as effective
institutions for local governance and social-economic development.
As stated, the PDO included two sub-objectives that will be evaluated separately:
(a) to build, strengthen, and maintain community development councils (CDCs) as effective institutions for
local governance, and
(b) to build, strengthen, and maintain community development councils (CDCs) as effective institutions for
social-economic development.
Rationale
Objective 1: Build, strengthen, and maintain community development councils as effective
institutions for local governance.
NSP aimed to establish and build the capacity of CDCs to function as village-level development governance
bodies for rural communities, bringing their voices into the government decision-making process for
subnational development planning, resource allocations and delivery of rural development programs.
As explained by the Government, local governance was defined and measured as CDCs being:
1. legitimate and representative institutions;
2 . the main decision-making body in the community - doing so in a transparent, participatory and inclusive
way;
3 . avenues for active women’s participation in decision making by ensuring their membership and
representation in the council, involvement in CDP planning and sub-project selection and;
4 . performing functional mandates such as CDC planning, social audits and implementation of community
development investments.
Measurement/Results
Page 6 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
Legitimacy and Representation were measured by an outcome indicator which the program met and
exceeded. Against an appraisal target, that a minimum of 70% of sampled communities would recognize
CDCs as the legitimate and representative institution of communities, by end-line, 96% of sampled
communities are reported to recognize their CDCs as the legitimate representative of decision making and
development of their communities (based on a sample size of 51% of CDCs, surveyed post-
implementation).This outcome should be weighed against the lower number of sampled CDCs (80%) who
are reported to have performed their functional mandates (such as conducting CDC meetings, sharing of
information, successful completion of sub projects and so forth). It is not clear how a CDC can be considered
an “effective institution of good governance” if it does not perform its functional mandates.
A second governance metric, introduced through the additional comment period, is voter turnout.
For the first 5,000 CDC elections under the Citizen Charter program, in all regions, male voter turnout has
been between 75-78 % of the eligible voter population while the women’s voter turnout has been even
higher ranging from 78-81 percent with the exception of the South due to conflict and local customs that limit
women’s mobility. This data provided on voter turnout, which supersedes activities under NSP, can be used
as a useful proxy for perceived legitimacy.
A third metric, is the measurement of “Confidence” in CDCs that is made available through an annual
Asia Foundation Survey of the Afghan People. The survey found that 61% of rural persons surveyed
expressed confidence in CDCs in 2010 and 58 % in 2017. The ICR notes that these levels were close to the
confidence levels expressed for community shuras/jirgas, much longer established traditional bodies, which
started and ended the period at 66%. The ICR does not address the incongruity between the MIS’s end line
findings and the Asia Foundation Survey. There is a wide variation between the 96% of all rural villages that
are estimated by the project to view their CDCs as legitimate and representative and the 58% that express
confidence. The fact that a greater percentage (66%) have confidence in traditional bodies – the
shuras/jirgas – raises a fundamental research question about the
Data provided by the Altai 2013 assessment of 100 CDCs in 6 Provinces and the Swedish Committee
for Afghanistan (SCA) further illustrate how, anticipatedly, the roles and functions of CDCs varied
depending on geography and the culture of the region in which NSP was working. Altai’s 2013
assessment found that CDC functions ranged from a narrow role in support of local development (Helmand)
to a broader role including dispute resolution and the establishment of linkages between the community and
higher levels of government (Badakhshan and Bamiyan). In Wardack province, which is heavily affected by
conflict, the SCA found few to no broader governance outcomes resulting from NSP. Customary structures
remained the primary source of all governance functions except in the delivery of development projects
undertaken by CDCs.
There is strong evidence that NSP enabled female community members to participate in decision making
related to community development activities (mainly the identification of their priority problems and selection
of sub projects) during the subproject cycle. During implementation, female community members
participated in decision making related to the identification and selection of sub projects in an estimated 79%
Page 7 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
of sampled communities. However, there is a need to better understand how and where female participation
in local decision-making persisted after project end.
The ability of CDCs to sustain their legitimacy over time will largely depend on their ability to access
adequate resources to perform their governance functions. IEG's Evaluation, World Bank Assistance to
Low Income Fragile and Conflict Affected States (IEG, 2013) and an internal Bank assessment (Bradley S.,
et al. 2015), point to the fact that the size and frequency of block grants are a strong determinant of the
effectiveness and sustainability of a CDC, as revealed under NSP. Under NSP III, large and infrequent block
grants undermined the ability of CDCs to perform their governance functions. CDCs are currently being
supported through MOUs signed under the Citizen Charter with the Ministry of Education, Health, Agriculture
and Energy, and through other donor financed programs (e.g.IFAD). The Government of Afghanistan has
also channeled funds through CDCs for cash for work programs in provinces, such as Kunduz, that have
faced rising insecurity and political uncertainty. However, over time, the lack of formal institutionalization and
clarity about the role of CDCs, and financing, threatens to undermine their ability to perform the full suite of
governance functions that were performed under NSP, including many of the progressive elements (e.g.
female participation in governance, secret ballot voting, social audits etc.).
Assessment of Outcome
The project substantially achieved its objective to build, strengthen, and maintain community development
councils as effective institutions for local governance.
Rating
Substantial
PHREVDELTBL
PHEFFICACYTBL
Objective 2
Objective
Objective 2: to build, strengthen, and maintain community development councils (CDCs) as effective
institutions for social-economic development.
Rationale
The NSP III objective required an analysis of the link between the enhanced effectiveness of CDCs as local
service delivery agents and the outcomes of the service delivery investments. Unfortunately, the ICR
bifurcated the analysis; it did not establish an evidentiary link between the relative performance of
better governed CDCs and the relative performance of the subprojects, apart from the difficulties
experienced in heavily insecure areas (where the link is tacitly made).
The ICR provided data on the subprojects, their quality, O&M, and the economic rates of return of a sample
of the sub-projects. It did not measure the socio-economic welfare of individuals or the communities.
Page 8 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
The quantified outputs were:
1. The number and percentage of CDCs that completed at least one sub-project (98%).
2. The number and percentage of sampled communities that had improved access to services (80%).
3. The number of actual versus planned subprojects completed (7,208 of 7,710 subprojects, or 93%)
4. The relative distribution of sector investments, including Transport (32%); Irrigation (25%) Water and
Sanitation (20%) plus other.
5. The number of roads that were constructed (1293 km) and rehabilitated (141); the number of water
subprojects that resulted in “improved” water points (39,449); the amount of land that was irrigated, or
provided with new or improved irrigation or drainage services (361,523 hectares); the number of improved
latrines (7,953); the amount of Kw of electricity generated through micro-hydro power sub-projects
(15,800Kw); the number of primary classrooms (2,995) and health centers (53) that were built or
rehabilitated; and the number of community centers that were constructed (4200). In addition, 12,830 works
were maintained or repaired under the MCG program.
6. The ICR (p.17) also reported that the construction buildings and other structures financed by the
project, monitored by communities, was "generally of high quality" but that the design and implementation of
some energy and irrigation sub-projects were sometimes beyond the capacity of CDCs and FPs.
7. O&M issues have begun to be addressed through the signing of MOUs with the Ministries of Health and
Education, to support the incorporation of schools and health facilities into district and provincial level
planning.
Evidence of Socio-Economic Development
The ICR reported on the functioning of the sub-projects at project close. It also included economic rates of
return analyses for a sample of the projects. There were, according to the ICR, 7,208 sub-projects completed
under NSP III, but the ERR used a sample of less than 1% of the total to derive average returns and there is
no explanation in Annex 4 as to how the sub-projects were selected - there was no discussion of the
sampling frame (e.g. the sample appears to be stratified, but did not appear to be randomized). In some key
areas, like transport and roads, there was no record in the ICR of the total number of sub-projects, so it
was not clear whether the sample sizes were adequate (22 irrigation projects and 33 transport projects were
analyzed in the ERR analysis, but the only data provided in the ICR concerns the number of ha irrigated or
the number of km of rural roads rehabilitated).
The ICR (p.17) used the Independent Impact Evaluation to report on socio-economic benefits since the
design of NSP III is very similar to NSP II, and since the IE was conducted at the same time as NSP III was
implemented. (i.e. the effects of NSP II found by the IE are likely to be similar to those for NSP III). The IE
showed that:
(i) The usage of protected water sources increased by 36% at midline, but that this declined to 5% points at
end line. Overall, these contributed a 5% reduction in the collection of water time (a gender informed
indicator).
(ii) In addition to finding that there was a small increase in access to education for girls, there was
also improved school attendance and quality of learning for girls.
Page 9 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
(iii) The ERR analysis of roads projects indicated that there was transport cost savings and time
savings (to local markets) and for children accessing school, while pointing to lower ERRs after applying a
sensitivity analysis in the Central and East regions. The IE found positive yet modest increments for
the overall village accessibility (in non heavily conflict affected areas) and also found that there was no
evidence that the rehabilitated roads increased the frequency of males visiting the district center (a value not
reported on in the ICR).
(iv). While the IE found no noticeable economic impacts of irrigation investments, the ERR analysis found
that the four regions covered in the analysis showed an ERR averaging above 25% with the sensitivity
analysis applied. This is an understudied area of the NSP, since irrigation is so closely tied to the
livelihoods of the program's rural beneficiaries and since water use allocation and decision-making
is so closely tied to the program's governance objectives.
(v) Another area of the program that deserves further elucidation is the innovative ways that the
program sought to access in insecure areas, by rolling out the High Risk Areas Implementation
Strategy. While unrelated to the rating of the objective, this is a process issue that can inform the way the
Bank implements sub-projects in many FCV environments. The ICR reported that it allowed for more
flexibility and better access to insecure areas. At the same time, over 370 people associated with the
program were killed and another 125 persons were kidnapped, with the highest numbers during NSP III.
And, even with the strategy in place, works in 3000 communities were left incomplete. There appears to be a
great deal of learning "left on the table" with regard to the implementation of the HRAIS which has the
potential to inform other FCV operations.
Even with these various shortcomings, the achievements of NSP in meeting its intended goals was
Substantial.
Rating
Substantial
PHREVDELTBL
PHOVRLEFFRATTBL
Rationale
IEG upgraded the internal Efficacy rating from Modest to Substantial by assigning, per the guidelines, two
internal Efficacy ratings of Substantial (for socio-economic outcomes and for governance).
Overall Efficacy Rating
Substantial
PHREVISEDTBL
5. Efficiency
Economic and Financial Efficiency
Page 10 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
ex-ante
The overall Economic Rate of Return (ERR) of the project was estimated at 18% under a 10 year project life,
including O&M and FPs costs and the NPV at US$543 million. The ex-ante economic analysis focused
on improved access to infrastructure and services. It attempted to evaluate the viability and sustainability of
the project, its impact on rural communities, and its cost-efficiency compared to other alternative government
programs. The analysis relied on data and studies done under NSP-II. No value was assigned to efficiency
gains achieved through strengthened CDCs, including service access that may have been achieved through
linkages to local government institutions.
The ex ante analysis projected the highest ERRs for irrigation subprojects (22%), followed by power, transport
and water supply and sanitation sub-projects (18%). A sensitivity analysis estimated that a 10% increase in
costs would reduce ERRs from 18 to 14% and a 15% increase would lower this down to 12%.
ex-post
The ex-post analysis is based on completed projects. it includes an analysis of the estimated ERR for US$832
million of block grant financed infrastructure, equaling 66% of the total project costs. Works in 3,000
communities were not finished by project completion due to insecurity.
An overall ERR was estimated at 41.4%, and a Benefit Cost Ratio (BCR) of 2.5:1. The ICR (Annex 4) provided
a detailed ex post economic analysis of the sub-projects financed by the project.
As compared to a projected 22% rate of return, the ERR for irrigation is estimated to be 61%. However, the ICR
does not provide sufficient information to validate this high ERR. This return is out of step with other ERR
projections for irrigation investments in both stable and unstable environments. There is no explanation in
the ICR as to how this average effect would be achieved over the projected return period. IEG assumes that an
average ERR is projected that smooths or calibrates the effects of extreme weather events (i.e. that the return
was not estimated using a drought year as a baseline, and a non-drought year to calculate the return). the ICR
also indicated that there were increased costs (as would be expected) in insecure areas, but there is no
differentiation between the ERR of certain sub-projects in high cost and hard to reach areas from lower cost,
stable areas.
As compared to an 18% ERR for transport, power, and water subprojects, a 30.8%, 40.2%, and
40.5% ERR was estimated for these subprojects, respectively. Again, the ICR did not provide an explanation as
to how the ERR was doubled. Other estimates include 15% ERR for community centers, 17% for schools, and
41% for local clinics.
Delivery Costs and Implementation Efficiency
Average delivery costs (28% excluding MCGs) were comparable to other large CDD programs and lower for
like programs in FCS and project management (8%) is below the FCS average (13%). Costs also compared
favorably with other donor programs of similar scale in Afghanistan (e.g. USAID).
Page 11 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
The project provided efficient financial mechanisms that featured a system of direct payment between MRRD in
Kabul to thousands of CDC bank accounts. At the central level, NSP III continued and enhanced the use of
float accounts managed by a Financial Management Agent, which reduced the average payment processing
time by 2 to 4 weeks compared with the earlier system of payments vis the MoF. The project also increased
efficiency during NSP III by merging FP contracts (95 into 28).
As noted in the ICR (Pgs. 19-20), there were implementation delays and increased costs in insecure areas (as
would be expected) and delays in block grant disbursements and facilitating partner payments (and to a lesser
extent staff salaries) to the lack of liquidity between fiscal years. Also, the PIU structure which uses contracted
staff within MRRD was costly and hard to sustain.
Administrative Efficiency
The project closed sixteen months later than expected due to implementation delays. The
project experienced significant delays after centralizing all recruitment into the Ministry’s HR Directorate instead
of the earlier arrangement through the project's HR Department. This change also resulted in interference in
recruitments especially for key staff in district and provincial level recruitment during the rollout of Maintenance
Cash Grants (MCGs) and closure of the project. Also, the sudden change of the financial management agent
resulted in delays in 2011. Finally, there were implementation delays due to a slow down in the procurement
of Facilitating Partners contracts for the second phase of roll-out of the repeater block grants.
Overall, and based on the information provided above, efficiency is rated substantial, however there are
questions that relate to the average effect of some of the estimated rates of returns for different areas and for
seasonal variance, especially for irrigation.
Efficiency Rating
Substantial
a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal
and the re-estimated value at evaluation:
Rate Available? Point value (%) *Coverage/Scope (%)
0
Appraisal 18.00
Not Applicable
66.00
ICR Estimate 41.40
Not Applicable
* Refers to percent of total project cost for which ERR/FRR was calculated.
6. Outcome
Page 12 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
The NSP III ICR includes the following internal ratings: Relevance (High) Efficacy (Substantial); Efficiency
(Substantial). The ICRR validated and maintained the High Relevance rating; Upgraded the Efficacy rating to
Substantial by incorporating additional data provided by the team on governance outcomes; and validated and
maintained the Efficiency rating of Substantial. Therefore, the overall outcome rating is Satisfactory.
a. Outcome Rating
Satisfactory
7. Risk to Development Outcome
For the short and medium term, the CDCs are expected to benefit from the Citizen's Charter National Priority
Program and from financing through the Citizens' Charter Afghanistan Project. The Citizens' Charter would also
benefit the CDCs through extending their mandate and coverage to include community organizations such as
health and education shuras, and extending coverage to cities. Also, through linking CDCs to sub-national
governance bodies such as district and provincial assemblies.
However, there are three significant risks that could negatively impact the sustainability of the development
outcome:
• Institutional. The lack of formal institutionalization and legal recognition of CDCs, beyond the CDC By-Law,
remained a concern for CDCs sustainability and effectiveness beyond the NSP. Also, competition for external
resources at the national level provides little incentive for line ministries to collaborate in strengthening CDCs
or funneling funds through them, as they are still seen largely as Ministry of Rural Rehabilitation and
Development (MRRD) structures. Also, CDCs' efforts to coordinate with higher levels of governance were
hindered by the continuing weakness of district-level governance. While the lowest tier of government and
formal local governance remains the district level, efforts have focused on strengthening the provincial level.
• Country-specific. Afghanistan experienced a steady decline in the security situation over the last several
years. The spread of violence and deteriorating security is expected to negatively impact the ability of CDCs to
perform their duties.
• Funding. The decrease in the flow of foreign funding to Afghanistan already constrained development
programs including NSP. The role of two thirds of the CDCs is under serious threat due to lack of funds. While
the Government increased financing to CDCs, it is doubtful that this will suffice given the low economic growth
and challenging fiscal situation in the country.
Page 13 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
8. Assessment of Bank Performance
a. Quality-at-Entry
The project was the third phase of the National Solidarity Program in Afghanistan. While the project aim
was pitched at an appropriate level for the third phase - that had placed governance at the core of the
program throughout – program designed undermined achievement against the project’s governance
objectives. The PDO was: to build, strengthen, and maintain CDCs as effective institutions for local
governance and social-economic development. To achieve effective the governance aims of the
program, CDCs would need to be strengthened and maintained (not just built). The Bank responded to
the Government’s priority of national coverage, and in doing so, it built CDCs in 90 percent of rural areas.
But in responding to Government demand, the quality of entry of the project – in terms of its ability to
achieve its PDO related governance aims - was partially undermined, since expanded coverage came at
the expense of more frequent and appropriately sized block grants. The ICR cites CDD literature that
states that new behaviors become institutionalized after at least three rounds while noting that the
NSP provided (large block grants) too infrequently (every 7-10 years) to institutionalize strong
performance. “The size and frequency of block grants weakened CDC effectiveness and sustainability,
and thus achievement of the ‘strengthened’ and ‘maintained’ elements of the PDO.”
M&E design included multiple products - a MIS, an IE that carried over from NSP II, several agency
studies etc. But M&E could have been strengthened by creating a tool that was designed to test the
project objective. Bank comments indicate that the so called "grain experiment" was not well received. In
that case, other quasi-experimental tests could have been administered to measure the link between well
performing CDCs, the development of a trust quotient, and the way that the project financed services
contributed to the perception of legitimacy and increased trust in the state. As shown in the ICR, the MIS
measured the quantifiable outputs, and while there were several studies, each used a different method,
covered different geographic areas, while some were conducted over two points in time, others were
static. After three project phases that used over US$1 billion to help develop more representative,
participatory and effective local governance in Afghanistan, the ability to report on these effects should
have been given more prominence.
Design was otherwise informed by some aspects of the existing impact evaluations in several ways : the
discussion of CDCs and the roles that they play in planning and community leadership; the need to
ensure that the roles and responsibilities of CDCs versus local elders and shuras are made clear to the
community (this was reflected operationally through improvements in mobilization and communications);
technical designs for infrastructure; training for engineers, mobilizers and monitors; and enhanced
operation and maintenance (O&M, through user fees, transferring health, education, on grid energy sub-
projects to the relevant line ministries etc.)
In line with the comments received that provide additional evidence on how design was strengthened
under NSP III, IEG adjusted the Quality at Entry Rating from MU to MS.
Page 14 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
Quality-at-Entry Rating
Moderately Satisfactory
b. Quality of supervision
The project was implemented under a challenging security situation. However, the task team conducted
monthly visits to Kabul (sometimes more), regular meetings by video and audio conference and
managed to ensure that even with a small task team in place, limited movements, withdrawal of FPs and
increasing risk, regular coordination was maintained with the Government team to ensure smooth
implementation. Management, gave a high priority to the program and provided quick security clearances
to the task team so that more handholding support and regular supervision could take place. Reverse
missions were only done when the team was too large to be in Kabul at the same time.
The Bank team simultaneously brought to closure the activities under NSP I, II as well as NSP III.
Supervision ensured that project financial management complied with the Bank 's procedures and
guidelines. Quarterly Interim Financial Reports were submitted on time and were of acceptable quality
and project audits were unqualified every year (see financial management section).
Supervision should have acted earlier in the implementation cycle to address weaknesses in safeguard
compliance. The project failed to properly document the implementation of the environmental
management plan; did not properly process and document land transfers; and according to the ICR,
there were safety issues concerning the construction of schools and community centers that may have
been at risk for floods and avalanches. The third part monitoring structure that is used by the Bank in
very insecure environments has many benefits from the point of view of data collection, but the ability to
identify safeguard issues and to ensure proper compliance is often a challenge in these working
arrangements. While the task team provided further information about safeguards, it did not supply the
needed evidence to validate that the above issues, raised in the ICR, were addressed.
Quality of Supervision Rating
Satisfactory
Overall Bank Performance Rating
Moderately Satisfactory
9. M&E Design, Implementation, & Utilization
a. M&E Design
It is important from the outset to recognize that Bank projects require reporting against their objectives,
not their indicators. As stated in several inter-linked sections of this assessment, the core aim of this
project was to strengthen local governance. The M&E system that was designed was only partially
Page 15 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
capable of measuring this. As also stated in this assessment, one of the weaknesses of the M&E
system was that it bifurcated the achievement of the two parts of the objective (governance and service
delivery) rather than utilizing data to establish causal links between good governance and effective
service delivery.
With that said, IEG recognizes the strength of NSP's M&E system which underpins the project's
performance rating and which was undertaken under extremely challenging circumstances. The MIS
provided data on CDC progress and subproject achievements. Data was collected by a mix of Third
Party Monitoring, post-implementation monitoring, community based monitoring and through 20 World
Bank, Government and external evaluations or studies to verify results. The data for PDO, intermediate
and core sector indicators was regularly collected by community monitors, FPs, district and provincial
staff and its reliability was checked by third party monitors. It is notable that the Government could
report upon what was achieved across 36,000 communities, owing to the work of monitors who
frequently traveled to all districts and Provinces under This effort was unparalleled among projects in
Afghanistan, and to a large extent, in bank operations in FCV.
b. M&E Implementation
The project's PIU included an M&E Department at the central level and monitoring staff in its 34 PMUs. The
project data was accessible at headquarters though an operational MIS with a decentralized data entry
system to the provincial level. all provinces submitted monthly monitoring reports. Facilitating Partners
complied quarterly Community Participatory Monitoring reports from thousands of communities. Database-
generated reports were combined to provide weekly, monthly and quarterly reporting to the
project management, government and donors on the overall program progress.
To ensure that PDO level indicators were achieved and data reported from the MIS was reliable, the WB
team adjusted the TORs of the third-party monitors in the last years of NSPIII. So as to go beyond verifying
activities at the CDC level towards reporting on the robustness and accuracy of the MIS to ensure that there
are no inaccuracies between what the TPM saw in the field and what is entered in the MIS (this is mentioned
on page 26 of the ICRR).
Also, some indicators were adjusted upwards to achieve more outcome oriented results reporting. For
example, the PDO ‘a minimum of 70 percent of sampled communities have successfully implemented
community investment sub-projects’ was replaced with a ‘minimum of 70 percent of sampled communities
have improved access to services. While access falls short of measuring use, it is a more robust indicator
linked to service delivery.
As would be anticipated, M&E implementation suffered from insecurity in some areas of the country which
prevented safe access to data. M&E activities were also undermined by non-completion of some studies
envisioned in the Emergency Project Paper. These included: a household study to understand how villagers
experience change related to NSP and to other political, social and economic developments in their
communities; an impact study on governance, state-building, security and peacebuilding; and an evaluation of
Facilitating Partners looking at deliverables, reporting, and performance in the field (ICR, p. 26).
Page 16 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
c. M&E Utilization
The project's M&E data on performance and results progress informed project management and decision-
making. The PIU management team relied on data and field experience to regularly update the operational
tools, procedures and policies, including three revisions to the operations manuals. Data and reports from
third party monitoring served as a feedback mechanism for the project. Original reports were shared with
ARTF donors to assure proper use of funds. The ICR relied on the project's M&E to report on different
aspects of the project.
M&E Quality Rating
Substantial
10. Other Issues
a. Safeguards
The project was an environmental category B. Environmental Assessment (OPBP 4.01) was the only
safeguard policy triggered at appraisal and reported in the Emergency Project Paper. However, the ICR (p.
27) reported that OP/BP 4.12 on Involuntary Resettlement was also triggered due to possible land
acquisition in connection with certain sub-projects. The sub-projects implemented by NSP
communities would be small-scale civil works with little impact on the environment. An Environmental and
Social Management Framework (ESMF) was prepared; and later updated in November 2015 during the
restructuring to include the Maintenance Cash Grants (MCG) work. A Resettlement Policy Framework (RPF)
in compliance with the operational policy was also prepared.
The ICR provided mixed messages on compliance where it reported that "the project generally maintained
compliance with Bank safeguard requirements, as confirmed by implementation support missions (ICR, p.
27)." However, it also reported that the project failed to properly process and document land donations or
other transfers as required in the ESMF and Operations Manual, and suffered from inconsistent
documentation of Environmental Safeguards Management Plans (ESMPs) across sub-projects. There were
also safety issues at a limited number of sites for rural infrastructure including several sites for schools and
community centers that were at risk for floods and avalanches. No information was provided on the
environmental impacts of the micro-hydropower stations financed by the project. These could potentially
have some environmental impacts including reduced oxygenation and soil erosion immediately downstream
of the turbine draft tubes. Finally, compliance with safeguard requirements was constrained by limited
capacity among assigned safeguard focal points, FPs, and CDCs (ICR, p. 28).
Page 17 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
b. Fiduciary Compliance
Financial Management. The project used float accounts managed by a contracted Financial Management
Agent. This arrangement enabled the project to disburse funds quickly and efficiently-where float accounts
reduced average payment processing time by two to four weeks, compared with the earlier system of
payment via the Ministry of Finance. According to the ICR (p. 28), the project financial management
complied with the Bank 's procedures and guidelines. Quarterly Interim Financial Reports were submitted on
time and were of acceptable quality. The project audits by the supreme audit of Afghanistan were unqualified
every year. At closing, there were no pending/unresolved deductions under either IDA or ARTF Grants.
However, financial management suffered from a three month delay in 2011, when the original Financial
Management Agent (Deloitte USA) exited Afghanistan due to changes in the Government security protocols
for international contractors. This situation was resolved after contracting GIZ International Services as the
replacement Financial Management Agent.
Procurement. There were procurement delays that impacted the FP contracts for the second phase of roll-
out and repeater block grant communities. Also, as noted above the sudden exit of the Financial
Management Agent caused procurement delays. According to the ICR (p. 28) "the project complied with
Bank procurement procedures and policies."
c. Unintended impacts (Positive or Negative)
The program made a strong contribution to an increased and more meaningful role of women in
decision-making processes, including through representation in CDCs and increased voice in
determining community investments. Half of all council seats were allocated to women. With the
establishment of gender quotas for CDC office bearers, the number of female officials increased from 5
percent under NSP IIIA grant communities to 40 percent in NSP IIIBs. Female sub-committees provided a
forum in which women could meet and express their views in the selection of subprojects. Project
indicators supporting this finding are: (i) Women made up 39 percent of CDC members nationally, against
a goal of 38 percent; and (ii) Among sampled women representatives, 79 percent took active part in
decision-making, against a goal of 70 percent. The NSP II IE also found that the creation of gender-
balanced CDCs increased female representation in local institutions by 38 percentage points and
increased the provision of governance services particularly to women.
The participation of women in village life as supported by NSP representatives was important progress in
a society where women are largely absent from public fora. Culturally, Afghan women have limited formal
public roles, and are restricted in their participation in community decision-making processes. With the
onset of Taliban rule in the mid1990s, women were banned from paid employment and were prohibited
from leaving their homes without a mahram (escort). This imposed a severe restriction on women’s
mobility, and their ability to be active participants in public life. Their representation in CDCs and their
participation in decision-making as reflected in the program indicators points to a notable contribution
towards closing the gender gap in Afghanistan.
Page 18 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
The extent of progress in gender inclusion varies from place to place, as there is considerable variation in
the cultural context across rural Afghanistan. In some areas, the role of women in village leadership and
decision-making remains limited, due to continued enforcement of conservative gender norms. For
example, a recent evaluation of an FP’s work in Wardak province found limited progress in female
inclusion due to a strong presence of the Taliban in customary governance structures. Similarly, there was
underreporting of the female population in certain Provinces such as Uruzgan and Zabul which made it
difficult to include them or report on their involvement.
Women generally did not participate in the construction or installation of works under the block grants or
MCGs due to social norms. This affects their ability to benefit from the daily wages, especially for those
who don’t have able-bodied males to participate on behalf of the household.
At the institutional level, female staff representation remained low across the length of the program
(similar to other development projects in the country). At project closing, women represented 18 percent
of staff at HQ level and 11 percent at field level. Female staffing in the FPs increased only after the
introduction of specific allowances for female social mobilizers in the FP to enable them to travel.
d. Other
---
11. Ratings
Reason for
Ratings ICR IEG
Disagreements/Comment
Outcome Satisfactory Satisfactory ---
By Phase III, effectiveness
was still undermined by a
design that required full roll
out, but that was restricted by
infrequent block grants.
Design also could have
Bank Performance Satisfactory Moderately Satisfactory
included tools to better test
and report on governance
outcomes, including by using
the effective delivery of the
means (services) to test the
aim (governance).
Quality of M&E Substantial Substantial ---
Quality of ICR Substantial ---
12. Lessons
Page 19 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
There is a need to manage expectations by the Government and donors about the peacebuilding
impact that a Community Driven Development (CDD) project can have, including one that has been
implemented over several phases with a relatively large donor investment. CDD can build trust within
communities and among them toward the government. But its ability to influence exogenous forces
(recruitment, insurgency) remains untested. Likewise, there is a need to preserve the neutrality–and not
politicize–Community Development Councils by using them for military or security purposes, so that the
legitimacy that has been earned by the better performing councils does not become eroded.
Not all problems can be solved at the community level, so there is a need for close coordination with
line agencies and district and provincial levels of government for effective service delivery. Line
agencies must be responsible for actions beyond communities’ control, such as ensuring teacher and health
worker quality, providing engineering expertise, ensuring appropriate operations and maintenance interventions
and coordinating subprojects’ linkages with higher level infrastructure trunk networks (roads, sewerage, etc.) In
the case of NSP III, appropriate platforms were needed to allow public service providers to coordinate with CDC
and CCDC leadership or sub-committees at the local level, underpinned by closer coordination at the center.
There are trade-offs involved in implementing a national CDD project via a parallel PIU structure - and
any transition into Government systems should be a gradual one. NSP’s structure of contracted staff paid
at higher salaries that civil servants (but consistent with Government consultant salaries), operating largely
independently within the ministry, may have allowed it to recruit and retain more qualified staff, operate more
effectively and delivery services in rural areas where the civil service does not have a presence. But–even with
recent Afghanization and convergence towards civil service norms–the higher costs undermine sustainability
and limit the impact of institutional strengthening efforts, if still concentrated within a PIU rather than the ministry
as a whole. Furthermore, such convergence/transition processes must be a gradual one to avoid impacting
project performance, as for example happened with HR functions on the NSP III.
NGO participation in CDD programs can be essential, but careful training, selection and contract
management are needed to ensure consistent implementation performance. FPs played a key role in the
success of NSP III, as with earlier phases. However, gaps in training, selection of less-experienced NGOs, and
output-based contracts that may have prioritized subproject completion and block grant utilization over
participatory aspects apparently contributed to uneven performance under NSP III.
13. Assessment Recommended?
Yes
Please explain
An expanded assessment is recommended, which may be done on desk, if security considerations prohibit a
field visit. The ICR references 20 different studies, that together with the MIS, and an array of stakeholder
interviews could underpin a strong post project assessment. The assessment should be conducted to
Page 20 of 21
Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review
AF: National Solidarity Program III (P117103)
synthesize the cumulative knowledge of the three phases of the program, as well as any new information
available through the implementation of the Citizen Charter.
14. Comments on Quality of ICR
The ICR provided good coverage of project activities, but lacked details on project shortcomings. The ICR
tone was mostly positive despite the challenges faced by the project during implementation. Discussion of
outcomes was logical and the ICR provided good justification for the assigned efficacy ratings. However, the
ICR reported outcome rating as satisfactory despite a modest overall efficacy.
The ICR could have improved on the following points:
• Provide a clear statement of compliance for each of the triggered safeguard policies.
• Discuss the risks and effectiveness of risk mitigation measures during implementation.
a. Quality of ICR Rating
Substantial
Page 21 of 21