Document of
                                   The World Bank
                                 FOR OFFICIAL USE ONLY


                                                              Report No: ICR00004588




            IMPLEMENTATION COMPLETION AND RESULTS REPORT
                                       TF-16354

                                         ON A

                                        GRANT

                            IN THE AMOUNT OF US$ 15 MILLION



                                        TO THE


                            ISLAMIC REPUBLIC OF AFGHANISTAN

                                       FOR THE
       NON-FORMAL APPROACH TO TRAINING EDUCATION AND JOBS IN AFGHANISTAN
                                   PROJECT

                                     April 30, 2019




Education Global Practice
South Asia Region
                            CURRENCY EQUIVALENTS

                     (Exchange Rate Effective Jan 24, 2019)


                                Currency Unit = Afghani
                                Afghani 75.90 = US$1
                                    US$ 1.30 = SDR 1


                                FISCAL YEAR
                          December 21 – December 20




      Regional Vice President: Hartwig Schafer
             Country Director: Shubham Chaudhuri
Senior Global Practice Director: Jaime Saavedra Chanduvi
            Practice Manager: Mario Cristian Aedo Inostroza
         Task Team Leader(s): Nathalie Lahire, Palwasha Mirbacha
         ICR Main Contributor: Gerard Peart
                   ABBREVIATIONS AND ACRONYMS



AFN       Afghani (Currency)
AREDP     Afghanistan Rural Enterprise Development Project
ASDP      Afghanistan Skills Development Project
CMU       Country Management Unit
CPF       Country Partnership Framework
DG        Director General
DGSD      Directorate General of Skills Development
DPs       Development Partners
ESCs      Employment Service Centers
ESMF      Environment and Social Management Framework
FM        Financial Management
GDP       Gross Domestic Product
GIRoA     Government of the Islamic Republic of Afghanistan
GIZ       German Agency for Technical Cooperation
GMF       Grant Management Firm
GRM       Grievance Redressal Mechanism
HR        Human Resources
IADC      Italian Agency for Development Cooperation
IC        International Consultant
ICR       Implementation Completion and Result Report
IEs       Impact Evaluations
IFR       Interim Financial Report
IRR       Internal Rate of Return
ISN       Interim Strategy Note
LMIS      Labor Market Information System
LOU       Letter of Understanding
M&E       Monitoring and Evaluation
MoF       Ministry of Finance
MOLSAMD   Ministry of Labor, Social Affairs and Martyred and Disabled
MTR       Mid-Term Review
NC        National Consultant
NGOs      Non-governmental Organizations
NOSS      National Occupational Skills Standards
NPP1      National Priority Program 1
NPV       Net Present Value
NSDP      National Skills Development Program
ODA       Official Development Assistance
OP/ BP    Operational Policy/Bank Procedure
ORAF      Operational Risk Assessment Framework
PAD       Project Appraisal Documents
PDO       Project Development Objective
PIM       Project Implementation Manual
SEA      Safeguards and Environmental Assessment
SGs      Savings Groups
SMEs     Small and Medium Enterprises
STEP     Systematic Tracking of Exchanges in Procurement
TA       Technical Assistance
TOC      Theory of Change
TPs      Training Providers
TTTI     Technical Teachers Training Institute
TVET     Technical Vocational Education and Training
UNESCO   United Nations Educational, Scientific and Cultural Organization
USAID    US Agency for International Development
WB       World Bank
Table of Contents
DATA SHEET .......................................................................................................................... 1
I.    PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5
      A. CONTEXT AT APPRAISAL .........................................................................................................5
          Context .................................................................................................................................................... 5
          Theory of Change (Results Chain) ............................................................................................................. 5
          Project Development Objectives (PDOs) .................................................................................................. 7
          Key Expected Outcomes and Outcome Indicators .................................................................................... 7
          Components............................................................................................................................................. 7
      B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .......................................9
          Revised PDOs and Outcome Targets; PDO indicators; and Components .................................................. 9
          Other Changes ......................................................................................................................................... 9
          Rationale for Changes and Their Implication on the Original Theory of Change ....................................... 9

II.   OUTCOME .................................................................................................................... 10
      A. RELEVANCE OF PDOs ............................................................................................................10
          Assessment of Relevance of PDOs and Rating ........................................................................................ 10
      B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................10
          Assessment of Achievement of Each Objective/Outcome ...................................................................... 10
          Justification of Overall Efficacy Rating.................................................................................................... 11
      C. EFFICIENCY ...........................................................................................................................12
          Assessment of Efficiency and Rating ...................................................................................................... 12
      D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................13
      E. OTHER OUTCOMES AND IMPACTS (IF ANY)............................................................................13
          Gender ................................................................................................................................................... 13
          Institutional Strengthening .................................................................................................................... 13
          Poverty Reduction and Shared Prosperity .............................................................................................. 14
          Other Unintended Outcomes and Impacts ............................................................................................. 14

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 14
      A. KEY FACTORS DURING PREPARATION ...................................................................................14
      B. KEY FACTORS DURING IMPLEMENTATION .............................................................................16
IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 20
      A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................20
          M&E Design ........................................................................................................................................... 20
          M&E Implementation ............................................................................................................................ 21
          M&E Utilization ...................................................................................................................................... 22
         Justification of Overall Rating of Quality of M&E ................................................................................... 22
     B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................22
     C. BANK PERFORMANCE ...........................................................................................................23
         Quality at Entry ...................................................................................................................................... 23
         Quality of Supervision ............................................................................................................................ 24
         Justification of Overall Rating of Bank Performance............................................................................... 25

V. LESSONS AND RECOMMENDATIONS ............................................................................. 25
ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 28
ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 35
ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 37
ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 38
ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 41
ANNEX 7. RESTRUCTURING AGREED AT THE MID-TERM REVIEW .......................................... 42
ANNEX 8. RELEVANCE OF PDO ............................................................................................. 44
ANNEX 9. THE BUSINESS GRANT SCHEME (SUB-COMPONENT 3.2)........................................ 45
ANNEX 10. COMPONENT-WISE ACTIVITIES AND RESULTS ..................................................... 49
ANNEX 11. IMPLEMENTATION AGRANGEMENTS.................................................................. 51
ANNEX 12. PROCUREMENT OF MAJOR TA PACKAGES .......................................................... 55
ANNEX 13. THE NATEJA PROJECT FUNDED BY IADC .............................................................. 58
ANNEX 14. MONITORING AND EVALUATION........................................................................ 60
     The World Bank
     Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)




DATA SHEET


BASIC INFORMATION

Product Information
Project ID                                              Project Name

                                                        Non-Formal Approach to Training Education and Jobs in
P146015
                                                        Afghanistan Project

Country                                                 Financing Instrument

Afghanistan                                             Investment Project Financing

Original EA Category                                    Revised EA Category

Partial Assessment (B)                                  Partial Assessment (B)



Organizations

Borrower                                                Implementing Agency

Islamic Republic of Afghanistan                         Ministry of Labor and Social Affairs


Project Development Objective (PDO)
Original PDO
The Project Development Objective is to increase the potential for employment and higher earnings of targeted
young Afghan women and men in rural and semi-urban areas through non-formal skills training.




                                                                                                         Page 1 of 60
       The World Bank
       Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



FINANCING

                                    Original Amount (US$)      Revised Amount (US$)          Actual Disbursed (US$)
 World Bank Financing
                                               15,000,000                   5,529,046                    5,529,046
 TF-16354
 Total                                         15,000,000                   5,529,046                    5,529,046
 Non-World Bank Financing
                                                           0                           0                                0




 Borrower/Recipient                             3,000,000                              0                             0
 Total                                          3,000,000                              0                             0
 Total Project Cost                            18,000,000                   5,529,046                    5,529,046


KEY DATES

Approval                Effectiveness          MTR Review             Original Closing         Actual Closing
24-Mar-2014             11-Apr-2014            24-Nov-2016            30-Dec-2018              31-Oct-2018



RESTRUCTURING AND/OR ADDITIONAL FINANCING

Date(s)                      Amount Disbursed (US$M) Key Revisions
15-Oct-2018                                      6.01 Change in Loan Closing Date(s)



KEY RATINGS

Outcome                                 Bank Performance                       M&E Quality
Unsatisfactory                          Unsatisfactory                         Modest


RATINGS OF PROJECT PERFORMANCE IN ISRs

                                                                                                      Actual
 No.               Date ISR Archived           DO Rating                   IP Rating              Disbursements
                                                                                                     (US$M)
 01                   28-Jun-2014              Satisfactory               Satisfactory                           1.00
 02                   24-Dec-2014              Satisfactory               Satisfactory                           1.09



                                                                                                             Page 2 of 60
     The World Bank
     Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



03                    23-Jun-2015          Moderately Satisfactory    Moderately Satisfactory        1.70
04                    13-Jan-2016          Moderately Satisfactory    Moderately Satisfactory        1.94
05                    11-Aug-2016          Moderately Satisfactory    Moderately Satisfactory        2.50
                                                  Moderately
06                    07-Feb-2017                                    Moderately Unsatisfactory       3.88
                                                 Unsatisfactory
                                                  Moderately
07                    30-Aug-2017                                    Moderately Unsatisfactory       5.59
                                                 Unsatisfactory
08                    09-Apr-2018                Unsatisfactory           Unsatisfactory             6.01
09                    24-Oct-2018                Unsatisfactory           Unsatisfactory             6.01


SECTORS AND THEMES

Sectors
Major Sector/Sector                                                                                   (%)


Education                                                                                            100
       Public Administration - Education                                                               19
       Workforce Development and Vocational Education                                                  81


Themes
Major Theme/ Theme (Level 2)/ Theme (Level 3)                                                         (%)
Private Sector Development                                                                            100
         Jobs                                                                                        100

Human Development and Gender                                                                          102
         Education                                                                                     34
                     Access to Education                                                               17

                     Education Financing                                                               17

         Labor Market Policy and Programs                                                              68
                     Labor Market Institutions                                                         34

                     Active Labor Market Programs                                                      34




                                                                                                 Page 3 of 60
       The World Bank
       Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



ADM STAFF
Role                                          At Approval                         At ICR

Regional Vice President:                      Philippe H. Le Houerou              Hartwig Schafer

Country Director:                             Robert J. Saum                      Shubham Chaudhuri

Senior Global Practice Director:              Jesko S. Hentschel                  Jaime Saavedra Chanduvi

Practice Manager:                             Amit Dar                            Mario Cristian Aedo Inostroza
                                                                                  Nathalie Lahire, Palwasha
Task Team Leader(s):                          Leopold Remi Sarr
                                                                                  Mirbacha
ICR Contributing Author:                                                          Gerard Peart




                                                                                                         Page 4 of 60
          The World Bank
          Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)




I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES

 A. CONTEXT AT APPRAISAL

Context

1.       At the time of Project appraisal (2013), Afghanistan had 12 million young people aged 18-35 years, constituting
35-40 percent of the total population. The real GDP growth rate for the period 2003-12 had been 9 percent annually.
The country had experienced years of significant post-conflict political instability and insecurity that had adversely
affected its economy and society. There were concerns that instability and insecurity might heighten once international
military forces left in 2014 (as they were then scheduled to do); and that the global financial crisis might negatively affect
economic growth. The economy’s growth potential was thought to be constrained by the low levels of literacy and
technical skills particularly among the young. Average real GDP growth was anticipated to be 4-6 per cent annually for
the period 2013-18.

2.        Over 80 percent of Afghan workers were employed in the non-formal economy; the vast majority of them lacked
basic technical and literacy competencies, as suggested by employer surveys. Each year, approximately 400,000 young
men and women entered the labour market with few qualifications. As a result, the bulk of young people were
unemployed or underemployed and unable to break out of a low income, low productivity trap. At the same time, the
Project Preparation Facility Grant conducted surveys indicating there was substantial demand for literate unskilled/semi-
skilled labor; though employers had few or no links with Training Providers (TPs).

3.        Demand for non-formal training was thought to be robust, and tracer studies conducted by National Skills
Development Program (NSDP) in 2012 in the provinces of Badakhshan, Bamiyan, Samangan, Kandahar and Khost found
high levels of graduate placement. There were concerns about the quality of non-formal technical and vocational
training, as well as a mismatch between course offerings and market requirements. In the non-formal sector,
apprenticeships were the predominant form of skills training, as well as a major supply channel for semi-skilled and
skilled labor. However, apprentices were found to lack basic literacy and numeracy competencies; to have inadequate
training in their fields; and to work in suboptimal conditions. There was low implementation capacity at the Ministry of
Labor, Social Affairs and Martyred and Disabled (MOLSAMD), with inadequate staffing on skills development and the
lack of a credible labor market information system.

4.       To address these issues, the Government of the Islamic Republic of Afghanistan (GIRoA) had policies emphasizing
skills development. These included the Afghanistan National Development Strategy (ANDs 2009-13), the MoLSAMD
Strategic Plan (2013-15), and the National Priority Program 1 (NPP1) which all included measures to strengthen skills
development and employment policies with a view to promoting job-rich growth.

Theory of Change (Results Chain)

5.      The Theory of Change (TOC) has been constructed from the results framework and project description. The
project planned a set of activities pertaining to skills training and business development that in turn would increase work-
and business-related knowledge, skills and resources of youth. In Component 1 (Improving the Quality of Non-formal
Training and Labor Market Outcomes of Trainees), training providers would be incentivized to ensure that their students


                                                                                                                       Page 5 of 60
         The World Bank
         Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



are certificated, find work or start a business. In Component 3 (Piloting Entrepreneurship among Unskilled and Illiterate
Young Afghans), savings groups members would be trained and receive business development advice; eligible youth
would receive business grants, training and support; and the informal apprenticeship system would be analyzed with a
view to providing training. In parallel, under Component 2 (Project Management, Capacity Building of MoLSAMD and
M&E), the capacities of the National Skills Development Program (NSDP) and MoLSAMD would be built in various
functions including fiduciary, labor information systems, and monitoring and evaluation. (Details on the components
are in the next section). These activities would increase NSDP/MoLSAMD capacities in project management, M&E and
labor market analysis. Together, the outputs of all the components would increase the potential for employment and
higher earnings among young women and men in areas targeted by the project. These outcomes would in turn
contribute to longer-term outcomes pertaining to increased human capacity, economic activity and gender equality. The
ToC has also been captured in the diagram below.




                                                                                                                  Page 6 of 60
            The World Bank
            Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



     Project Development Objectives (PDOs)

6.    The PDO was to increase the potential for employment and higher earnings of targeted young Afghan women
and men in rural and semi-urban areas through non-formal skills training.


     Key Expected Outcomes and Outcome Indicators

7.      The PDO was to be achieved through two expected objectives, which had four PDO indicators.

        Objective 1: Increased potential for employment among targeted young Afghan women and men in rural and
        semi-urban areas. Three linked indicators:

        •     1.1: Share of trainees who find employment within 6 months of graduation, with respect to control groups
              (to be increased from a baseline of 0 percent to 15 percent)
        •     1.2: The amount of incentive(s) disbursed to training providers (to be increased from zero to USD 4M)
        •     1.3: Number of beneficiaries and share of female beneficiaries (to be increased from zero to 44,500; female
              share to be increased from 20 percent to 30 percent)

        Objective 2: Increased potential for higher earnings among targeted groups. One linked indicator:

        •     2.1: Increase in trainees’ earning, with respect to control groups (to be increased from 0 to 15 percent)

8.      According to the PAD, the indicators 1.1, 1.2 and 2.1 pertain only to results of Components 1 (Improving the
Quality of Non-formal Training and Labor Market Outcomes) and Sub-Component 3.1 (Encouraging Rural Employment);
while the indicator 1.3 also covers Component 3.2 (Promoting Entrepreneurship among Afghan Youth).

     Components

9.      Component 1: Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees. A total
cost of US$ 7.7 million was estimated at appraisal; US$ 1.05 million were actually disbursed.

10.      The component was designed to induce non-formal training providers to improve the quality of their training
and the placement of their graduates. An incentive of US$ 100 was to be granted to non-formal TPs for each graduate
who was certified or found a job within six months of graduation or started a business. 1 TPs were to be selected from
across the country, with some preference given to those who offered courses exclusively for women or had a high
percentage of women in their regular training programs. Teachers were to be trained at the Technical Teachers’ Training
Institute (TTTI) and contracted to provide tailored short-term courses at some of the non-formal TPs. 2 Courses were to
be aligned with the National Occupational Skills Standards (NOSS), which were to be used for graduates’ certification. 3

        1 PAD, pp. 7 and 24. A timeframe for starting the business was not indicated in the PAD. The design of the incentive scheme was
        modified in the Project Implementation Manual (PIM), with USD 50 being allocated upon certification (passing the NOSS skills test) and
        USD 50 upon employment or self-employment. See paragraphs 136 and 137.
        2 The TTTI was to be established under a Bank-financed project, the Afghanistan Second Skills Development Project (ASDP II).
        3 The NOSS were being developed by NSDP. They are publicly available competency standards for occupations, and should guide the




                                                                                                                                       Page 7 of 60
          The World Bank
          Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)




11.     Component 2: Project Management, Capacity Building of MOLSAMD and M&E. A total cost of US$3.6 million
was estimated at appraisal; US$ 2.04 million were actually disbursed.

12.      This component was designed to build MOLSAMD capacities
in procurement, financial management, Labor Market Information                           The National Skills Development Program
System (LMIS) and monitoring. Capacities were also to be built at
the NSDP, 4 a national program responsible for skills training with                  At project preparation, the NSDP was one of the
multiple projects (see Annex 11), including the Bank-supported                       National Priority Programs still operating that had
                                                                                     been created by a former President of the GIRoA
Afghanistan Second Skills Development Project (ASDP II). Its overall
                                                                                     in 2004. It provided technical assistance to the
management capacity was to be built (including in the areas of
                                                                                     TVET system as well as vocational education
channeling financial incentives, fiduciary, and safeguards); as were                 trainings nationwide with the support of MOF and
its capacities in monitoring and evaluation, including the sustained                 Development Partners including the World Bank.
monitoring of project indicators, the conduct of third-party                         It functioned as an independent structure and
validation exercises, and the design and implementation of                           separate program at MoLSAMD, with project staff
randomized experiments for evaluating impact. The component                          hired for fixed periods to carry out program
also covered project management costs including salaries,                            activities; its functions were technical and
incremental operating costs and expenses for monitoring and                          fiduciary, giving it substantial control over
evaluation.                                                                          program design and execution.


13.      Component 3: Piloting Entrepreneurship among Unskilled
and Illiterate Young Afghans. A total cost of US$ 3.7 million was estimated at appraisal; US$ 2.43 million were actually
disbursed.

14.      There were three Sub-Components. The first Sub-Component 3.1 (Encouraging Rural Employment) was
designed to provide customized training in hard and/or soft skills as well as functional literacy to members of Savings
Groups (SGs); 5 the training to be customized based on the findings of a baseline survey. The possibility was also held out
that interested SG members would receive guidance on establishing small businesses and links to finance programs. The
TPs (providing the customized training) were to receive US$ 100 for each trainee who was certified or found a job within
six months of graduation or started a business.

15.     The second Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) was designed to provide
grants of US$ 500 on a competitive basis to 1,890 individuals of three provinces to help them start their own business,
and to 630 individuals to expand an existing business. 6 The grants were targeted at females and males aged 18-35 years
with an entrepreneurial mindset who were mostly illiterate and unemployed/underemployed. The applicants were to
develop business plans with assistance from MOLSAMD. The Project was to provide successful candidates with
counselling and mentoring support during implementation of their plans, as needed.

        development of training materials so that courses provided by any TP enable students to demonstrate their ability to meet the
        standards. The certification was to be conducted by an agency to be created under the ASDP II.
        4 As per NATEJA Grant Agreement, MOLSMD was to maintain throughout the period of the project the NSDP within MOLSAMD and it

        will be directly responsible for implementation of the Project, coordination of implementation arrangements among departments,
        agencies, ministries and development partners involved in the Project and provide the technical and capacity building support to Project
        stakeholders and beneficiaries.
        5 The SGs were set up in five provinces as part of a Bank-financed project, the Afghanistan Rural Enterprise Development Project

        (AREDP).
        6 Proposed three provinces were Kabul, Nangarhar and Balkh.



                                                                                                                                        Page 8 of 60
            The World Bank
            Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)




16.      The third Sub-Component 3.3 (Feasibility Study on Apprenticeship Program) was to carry out a feasibility study
to assess whether there was a need for short-term practical apprenticeship training and how such training might be
piloted using various options. If the study found that piloting was viable, then the most promising option was to be
finalized and implemented in selected provinces.

 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE)

    Revised PDOs and Outcome Targets; PDO indicators; and Components
   There were no significant changes

    Other Changes

17.      There was an agreement between the GIRoA and the Bank at the mid-term review (November 2016) to
restructure the Project, to address poor project implementation and unsatisfactory progress towards the intermediate
and PDO indicators’ targets. The restructuring would have led to a revision of the PDO and PDO indicators including
targets; a redesign of certain features of Component 1 (Improving the Quality of Non-formal Training and Labor Market
Outcomes of Trainees); the cancellation of Sub-Components 3.1 (Encouraging Rural Employment) and 3.3 (Feasibility
Study on Apprenticeship Program) and the re-allocation of unspent funds from these two sub-components to Component
1. 7 However, subsequent discussions between and within the Bank Country Management Unit and senior officials of the
GIRoA focused on the relative merits of restructuring as per the MTR recommendations versus an early closing of the
Project altogether and a cancellation of the remaining funds. During the year and a half following the MTR, it was
consequently unclear whether the Project would be restructured or closed. In that time, progress was made on
Component 2 (Project Management, Capacity Building of MoLSAMD and M&E) and Sub-Component 3.2 (Promoting
Entrepreneurship among Afghan Youth), but little was achieved on the other components which all had certain design
constraints that were to be addressed through a restructuring based on the MTR recommendations. Finally, in July 2018,
the MoF formally requested the Bank to close the Project early and cancel unspent funds. A restructuring paper to this
effect was prepared to enable an amendment to the Grant Agreement, and the Project closed two months early on
October 31st, 2018. See also Annex 7.


Rationale for Changes and Their Implication on the Original Theory of Change

18.      The early closure and cancellation of unspent funds were undertaken due to the Project’s poor implementation
rating, high overall risk, and failure to make adequate progress towards its objectives. At the same time, both the GIRoA
and the Bank were motivated to engage in a broader discussion on Afghan Youth as a new emerging priority of the
country.




        7See Section III.B and Annex 7 for an explanation of why it was proposed that sub-components 3.1 and 3.3 be cancelled, and that the
        bulk of their activities were not implemented.

                                                                                                                                      Page 9 of 60
           The World Bank
           Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



II. OUTCOME

 A. RELEVANCE OF PDOs

    Assessment of Relevance of PDOs and Rating

19.     Relevance of objectives is rated high. At closing (as at preparation), the PDOs remain relevant to the Bank’s
Country Partnership Framework (CPF - FY 17-20). That is, in their focus on improved employment and earnings through
targeted vocational training, the PDOs directly support CPF Pillar 3 (Social Inclusion), Objective 3.1 to improve human
development including through improved access to inter alia skills training. The intended outcomes by the end of the
CPF in education include an improvement in the quality and employability of graduates from skills training institutes. The
Project was explicit in its intention that the components would improve access to skills training (including technical,
business and literacy skills); while also improving the quality of the training itself through strengthened curricular inputs
and capacity building of vocational trainers. The PDOs also support the WBG’s gender strategy in Afghanistan, which
involves inter alia empowering women economically in key markets, enhancing their roles as producers in the rural
economy, and increasing opportunities for employment and entrepreneurial activities. The PDO and intermediate
indicators, as well as the component activities, were designed to ensure that the Project worked to empower women in
non-formal and rural economy, by providing them with the skills and finance required to enhance their potential to
develop businesses and find skilled employment. The PDOs are also aligned with the 2017-2020 Afghanistan National
Peace and Development Framework (ANPDF). See Annex 8.

 B. ACHIEVEMENT OF PDOs (EFFICACY)

    Assessment of Achievement of Each Objective/Outcome

20.    The assessment is organized around the two key outcomes. With respect to the first outcome - increased potential
for employment among targeted young Afghan women and men in rural and semi-urban areas - using the three PDO
indicators:

       •    The share of trainees under Component 1 (Improving the Quality of Non-formal Training and Labor Market
            Outcomes) and Sub-Component 3.1 (Encouraging Rural Employment) who found employment within 6
            months of graduation, with respect to control groups, remained at the baseline value of 0 percent (as against
            a target of 15 percent) because no TPs/trainees were approved for inclusion in the Project under those
            components. However, this indicator does not consider employment gains under Sub-Component 3.2
            (Promoting Entrepreneurship among Afghan Youth). As amplified in Section IV.A, given the obvious alignment
            between the employment promise of the PDO and the employment gains intended and achieved under Sub-
            Component 3.2, these gains must be considered. As a result of the business development grant, training and
            mentoring, the 2,482 Sub-Component 3.2 beneficiaries (of whom 30% female) achieved an employment rate
            of 88 percent, a statistically significant 13 percentage points higher than the control group. The gains for
            women were even more impressive, with a statistically significant 33 percentage points higher than the
            control group.
       •    The amount of incentive(s) disbursed to TPs remained at the baseline value of zero (as against a target of USD
            4M), again because no TPs/trainees were approved to receive incentives under the Project.


                                                                                                                     Page 10 of 60
            The World Bank
            Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



        •    The number of beneficiaries increased from zero at baseline to 2,482 (as against a target of 44,500); while the
             share of female beneficiaries at closure was 30 percent, meeting the end target. The beneficiaries were solely
             business grant recipients under Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth).
             Given the Afghan context, the achievement of the gender target on this basis is significant.
        •    Given that there was some progress against the outcome as measured by employment and the number of
             beneficiaries, and further that a substantial portion of them were women (thereby meeting the 30% gender
             target), the rating for achievement of this outcome is modest.

21.    With respect to the second outcome - increased potential for higher earnings among targeted groups- using the
PDO indicator:

        •    No TPs/trainees were approved for inclusion under Components 1 and 3.1 of the Project. That is, there was
             no increase in trainees’ earning (as against a target increase of 15 percent) as measured by the PDO indicator.
             However, as argued above, gains accruing to Component 3.2 beneficiaries must be considered. With respect
             to earnings, the 2,482 beneficiaries of Component 3.2 earned approximately USD 11.70/month more in net
             income than the control group as a result of the Grant Scheme; a statistically significant amount equivalent
             to 17 percent of their pre-Scheme monthly earnings. Thus, the rating for achievement of this outcome is also
             modest.

22.      Three of the four indicators - pertaining to employment (indicator 1.1), incentives (1.2), and income (2.1) - are
not fully satisfactory measures of the Project’s development outcome, in that they did not capture the employment and
income resulting from the incentives and training provided under Sub-Component 3.2 (Promoting Entrepreneurship
among Afghan Youth). The ICR team is of the view that the outcomes achieved under Sub-Component 3.2 contribute to
the PDO achievement, albeit on a modest scale in relation to the overall outcomes intended by the PDO. Further, on the
strength of the design and results of the Scheme, the Italian Agency for Development Cooperation (IADC) is funding a
similar intervention with 2,000 beneficiaries; while the MoLSAMD has included in its budget request for the fiscal year
1398 funds to enable such grants for graduates of TVET programs. That is, having looked beyond the Results Framework
to other available data and evidence and found positive evidence of achievement against the PDO outcome, the ICR Team
judges that the result is sufficient to warrant an efficacy rating of modest. See also Annexes 9 and 10.

23.     While the ICR focuses on the extent to which the project’s objectives were achieved, there is evidence that the
project contributed albeit modestly to higher-level outcomes as laid out in the TOC. The impact evaluation for Sub-
Component 3.2 (Promoting Entrepreneurship among Afghan Youth) found that grant recipients (of whom 30 percent were
female) established or expanded businesses that – in line with ANDS Pillar 3 and its cross-cutting gender goal - increased
economic activity in response to increased human capacity and skills sets and business services; increased employment
among the poor and vulnerable; and increased gender equality.

     Justification of Overall Efficacy Rating


24.      Overall Efficacy is rated modest because there was some progress towards the targets for the first and second
outcomes. As no one underwent training under Component 1 (Improving the Quality of Non-formal Training and Labor
Market Outcomes of Trainees) or Sub-Component 3.1 (Encouraging Rural Employment), there was no possibility to award
incentives to the TPs or for trainees to find employment and increase their income; as a result, there was no progress on



                                                                                                                    Page 11 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



any of the three key PDO indicators insofar as beneficiaries of these two components might have been concerned. 8
However, there were 2,482 persons (of whom 30 percent female) under Sub-Component 3.2 (Promoting Entrepreneurship
among Afghan Youth) who received a grant to start/expand a business, as well as training and follow-up support. This
resulted in higher levels of employment and income vis-à- vis a control group, and contributed somewhat to higher-level
outcomes of the National Development Strategy. The PDO indicators pertaining to employment and income should have
been phrased to capture the results of the Sub-Component 3.2, in the same manner as the (core) PDO indicator pertaining
to beneficiaries. The positive outcomes from Sub-Component 3.2 on employment and income are modest but solid
because they are based on difference in difference measures using randomized treatment and control groups. . Further,
the strength of the Business Grant Scheme led to a similar IADC-funded initiative with 2,000 beneficiaries; while the
MoLSAMD is budgeting for the replication of the Scheme. Together, these results warrant assigning a modest rating to
efficacy.

  C. EFFICIENCY

     Assessment of Efficiency and Rating

25.      Efficiency of successfully implemented activities is rated negligible. The net present value (NPV) and the internal
rate of return (IRR) for the project were calculated for that part of the operation (Sub-Component 3.2 - Promoting
Entrepreneurship among Afghan Youth or the Business Grant Scheme) that led to measurable outcomes; operational
dimensions of efficiency were also analyzed for all components. The NPV is USD 1.7 million and the IRR is 12.6 percent
under the base case, ranging from 8.4 percent to 16.4 percent using a sensitivity analysis; stringent low case assumptions
indicate that the positive result is robust. Given the pilot character of the Sub-Component 3.2 activities, these returns are
in line with sectoral expectations. 9 The costs used to calculate the NPV/IRR included all Project expenditures for Sub-
Component 3.2 and a portion of Component 2 (Project Management, Capacity Building of MoLSAMD and M&E); together,
these accounted for 48 percent of all Project disbursements by closing. The returns on the remaining expenditures could
not be calculated, in that as per Project design, they should have been set against the returns generated by Component 1
(Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees) and Sub-Component 3.1
(Encouraging Rural Employment); neither of which successfully completed activities. In the absence of such successes,
the returns on Component 1 are likely to have been negligible and on Component 2 to be indeterminate. If one were to
set all Project expenditures against the returns of Sub-Component 3.2, then the Project has a negligible, below-discount
IRR (1.7 percent) and negative NPV.

26.      In terms of operational efficiencies, there were substantial implementation delays and a failure to successfully
complete that affected Component 1 and Sub-Component 3.1. These were associated with an overall funds utilization
rate of 37 percent, as well as substantial turnover in project personnel. Component 2 activities were completed within
the overall project schedule, though with some initial delays; this component might have been better designed to consider
the change in implementation arrangements agreed during negotiations. Sub-Component 3.2 experienced delays at the
outset (to recruit the Grant Management Firm), though the Project team took advantage of this to strengthen the design
of the Scheme by running a small pilot; this laid the foundation for the Sub-Component effectiveness and substantial
efficiency gains during the second, larger phase. (See Annex 9). In general, costs were contained within budgets though
there were cases of unnecessary over-runs.

         8 The funds expended on Component 1 were used to map, screen and recommend TPs; draft and adopt NOSS- and competency-based
         curricula; and produce written instruction materials. See Annex 10.
         9 As explained in Annex 4, Sub-Component 3.2 activities entailed relatively high learning and overhead costs. Were the grant scheme to

         be scaled up, the IRR would likely be in the region of 26 percent (base case).

                                                                                                                                      Page 12 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)




  D. JUSTIFICATION OF OVERALL OUTCOME RATING

27.       The overall outcome rating is rated unsatisfactory because the PDO relevance is rated high, efficacy is rated
modest, and efficiency is rated negligible. This rating corresponds to best judgement based on information received for
the ICR, in that:

         •     The PDOs remain relevant to the Country Partnership Framework (FY 2017-2020), as they support one of
               current CPF Pillars as well as WBG’s gender strategy in Afghanistan;
         •     There was only modest progress towards achieving the two development outcomes. The business grants
               led to positive results in terms of employment and income, which contributed in fact to the PDO. As
               mentioned earlier, the results were not captured by the PDO indicators because of shortcomings in the
               results framework. Despite the positive outcomes under the business grant scheme, these results were
               modest in relation to the Project’s overall targets.
         •     While those parts of the operation that generated an outcome have an IRR of 12.6 percent, the overall Project
               IRR was negligible.

  E. OTHER OUTCOMES AND IMPACTS (IF ANY)


   Gender
28.     Under Sub-Component 3.2 (Business Grant Scheme), the 30 percent of the 2,482 beneficiaries who were female
benefited from the business grant, training (including a module specifically focused on gender issues in business) and
counselling; and as a result, had significantly higher levels of employment and income as compared to a control group.


   Institutional Strengthening

29.      The NSDP was integrated into MoLSAMD as part of the Project. Nearly all NSDP functions were transferred to the
relevant units/departments of MoLSAMD, including technical, HR, fiduciary and M&E functions; in many cases, budgeted
posts were created to absorb the functions provided by NSDP staff. The capacities of the relevant units/departments were
built through a combination of strengthened operating procedures, skills transfer by NSDP staff (who transferred to the
relevant unit/department), and training. The transferred functions and capacity strengthening are discussed in Annex 11.
As amplified below, the integration created implementation difficulties for NSDP that were not foreseen during
preparation; and further strengthening measures are still required to enable MoLSAMD to implement projects.

30.     The Project strengthened training programs, which may have benefited non-beneficiaries. The production and
dissemination of 162 NOSS and 45 competency-based curricula, and written instruction materials for 7 trades; the
adoption by 70 non-formal TPs of NOSS for the courses in which they train; and the training of 673 non-formal TP
vocational trainers: all may have contributed to an improvement in the quality and effectiveness of trainings at non-formal
TPs around the country. This in turn may have led to improved learning and employment outcomes for students.




                                                                                                                    Page 13 of 60
              The World Bank
              Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



   Poverty Reduction and Shared Prosperity

31.     The Sub-Component 3.2 (Business Grant Scheme) beneficiaries had higher levels of employment and income than
the control group. The treatment group had an employment rate of 88 percent, 13 percentage points higher than the
control group; and earned approximately USD 11.60/month more in net income than the control group. The beneficiaries
were all small business operators and given the characteristics of the target group are among the semi-urban and rural
poor. The beneficiaries also contributed more to employment in their community, as compared to the control group;
thus, contributing to shared prosperity. 10

   Other Unintended Outcomes and Impacts

32.     Enrolment of beneficiaries in Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth), the
Business Grant Scheme required that they open a bank account and present a government-issued ID (Tazkira). Most of
the beneficiaries had never opened a bank account before, and this led to a form of ‘financial inclusion’ that conferred
banking benefits beyond receiving the incentive grant itself. Many of the beneficiaries – including a large proportion of
the women – did not have a Tazkira and were required to get one in order to receive the grant. Aside from accessing the
grant, having the Tazkira allows beneficiaries to vote and is a form of ‘civil inclusion’.

33.      The Italian Agency for Development Cooperation (IADC) funded a project using the same design as the Bank-
supported NATEJA; it is on a smaller scale, covers different geographic zones for training and grant activities, and supports
Component 2 (Project Management, Capacity Building of MoLSAMD and M&E) activities that are complementary to those
of the Bank-supported NATEJA. While it is too early to quantify its outcomes/impacts, its activities, if effective, should
result in improvements in employment and incomes for beneficiaries. 11

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME

  A. KEY FACTORS DURING PREPARATION

34.     The preparation phase benefited from several factors. The sector was quite well understood, with defined
challenges for the Project to address. This can be attributed to a body of research and experience that underlay the
development of the ANDS (2008-13) and its associated NPP1, as well as the MOLSAMD Strategic Plan (2013-15); to all of
which the Project was aligned. More recent research on SMEs, TVET and apprenticeships was available and consulted by
the preparation team. 12

35.     The team built on the experience of the Bank, particularly the Bank-financed ASDP I and II (both focused on skills
development) and AREDP; as well as of other development partners, notably the GIZ, USAID and UNESCO. Indeed lessons
learned from the ASDP I project as well as the 2013 Institutional Analysis commissioned by the Bank guided the design

         10 “While businesses in both groups tended to be small and a majority of both groups had no employees, of those who did employ
         individuals other than themselves beneficiaries were more than twice as likely to employ at least one person and similarly more than
         twice as likely as non-beneficiaries to employ more than one person. Importantly, while most employees for both groups were
         members of the beneficiaries’ household, it was more common among beneficiary business to employ individuals outside of their own
         household.’ Altai Consulting 2018 (v.10), p. 4.
         11 See Annex 13 (The NATEJA Project funded by IADC).
         12 The 2012 USAID survey of SMEs in six cities and the 2013 GIZ survey of Master Trainers and apprentices.



                                                                                                                                      Page 14 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



of the NATEJA project, particularly that the demand for non-formal training was high, that non-formal trainings could be
effective in terms of generating higher levels of employment and income (as demonstrated by tracer studies), that the
quality of non-formal trainings could be usefully strengthened, and that capacities in management, monitoring and
evaluation at MoLSAMD (including NSDP) also required strengthening: all elements that were targeted in the project.
Similarly, the AREDP project had learned that one of the constraints SGs members faced in creating more value out of
their savings was a lack of literacy and hard/soft skills; again, this was to be addressed by the Project.

36.     The project design was relatively straightforward, with a clear operational logic. The results framework was for
the most part aligned with operational objectives, had a relatively straightforward mechanism to generate data, and
included Third Party Verifications and IEs. The IEs for Sub-Components of Component 3 were designed in detail and laid
a foundation for operational rigor that benefited activity implementation. A level of readiness existed on the part of the
NSDP, which was responsible for implementation, in that it was already established, staffed and operating, with
experience using Bank procedures. Further, a detailed PIM was in place at the time of effectiveness.

37.     Certain factors during the preparation phase laid the foundation for subsequent challenges. There were
important dimensions of uncertainty and instability in the political and security spheres, of which one may highlight here
impending elections and systemic governance weaknesses, as well as the drawdown of international security forces and
the recurrence of security incidents. These were articulated in the Operational Risk Assessment Framework (ORAF), and
mitigation measures were proposed. 13

38.      There was an important yet last-minute change to implementation arrangements. Whereas in the PAD the NSDP
was identified as being responsible for implementation, and it was assumed that the NSDP would preserve its then-
current autonomous status and mode of operation, during negotiations the MoF successfully advocated for integrating
the NSDP into the Directorate General of Skills Development (DGSD) in MoLSAMD and this was entered into the Grant
Agreement. The change created uncertainty among the NSDP staff responsible for Project implementation. Further, it
was a substantial modification of the implementation arrangements, a modification that had not been appraised. While
the project design included capacity building of MoLSAMD (including NSDP), it was predicated upon keeping NSDP’s then-
current status and receiving financial contributions in support of Project operations from MOF under the discretionary
budget. The new arrangements could have reasonably been expected by the Bank to constitute a significant challenge
requiring more planning and capacity building to take into account effects on implementation procedures and capacities
that would need to be transferred as part of integration, based on the findings it had received from an institutional
analysis of MoLSAMD that had been commissioned by the Bank in the previous year and completed in early 2013. 14 At
the same time, the new arrangements were in line with a general move within the Bank towards institutionalizing
implementation responsibilities within permanent structures of the civil service, in keeping with the Bank’s Interim
Strategy Note for 2012-14 in force at that time, which highlighted a concern about how foreign aid had created a ‘second
civil service’. Weaknesses in planning, implementation and monitoring capacity in MOLSAMD and NSDP were
acknowledged as risk factors in the ORAF; though the only mitigation measure identified was to recruit international
advisors. See also Annex 11.

39.     The principal component (Component 1 – Improving the Quality of Non-formal Training and Labor Market

        13Some of the measures were vague or undefined, though perhaps it was not reasonable to expect more. For instance, in response to
        security incidents the mitigation measures were ‘NSDP to tailor and adapt implementation arrangements to the security situation in the
        targeted provinces.’
        14 Institutional Analysis of Ministry of Labor, Social Affairs, Martyrs and Disabled. January 2013.




                                                                                                                                     Page 15 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



Outcomes of Trainees) included an experimental dimension that introduced an element of risk that was not recognized,
and moreover did not explicitly identify priority, eligible trades. The incentive was tied to graduate certification,
employment or starting a business; and was predicated upon sufficient graduates being employed with their fees being
paid either by other donors or themselves. In the end, the Project had difficulty finding a sufficient number of TPs
enrolling a sufficient number of graduates across a wide range of trades. 15

40.      Project design relied on the AREDP to facilitate working with the SGs (under Sub-Component 3.1 - Encouraging
Rural Employment); and on the ASDP II, to establish the TTTI, which would train instructors at the TPs, and to develop a
certification mechanism, which would be used to certify trainee graduates for both Component 1 and Sub-Component
3.1. Given that the ASDP II cancelled the certification mechanism, the MTR recommended that the certification
requirement under NATEJA be dropped.

 B. KEY FACTORS DURING IMPLEMENTATION

41.       The political process. Not long after effectiveness, elections were held, the results of which were delayed; and
when the new government was formed, there were delays in appointing Ministers. The new Minister for MoLSAMD took
up her post in mid-2015, and naturally it took some time for her to become familiar with the project. These events made
it difficult for the project to get decisions approved during the opening 12-18 months, including for the procurement of
large TA packages that were vital to implementation, as well as to make progress on integrating NSDP into DGSD in
MoLSAMD. Further, the new President decreed a freeze on hiring given the GIRoA’s unexpectedly large fiscal deficit,
encouraging MOLSAMD to use NSDP staff for non-Project tasks. Partly as a result, there were relatively few activities
during the first year of the Project, and no progress on any of the intermediate indicators.

42.       NSDP integration. The decision at negotiations to integrate the NSDP into DGSD/MoLSAMD contributed to the
implementation difficulties that the project experienced. First, it took up substantial time from NSDP senior
management. Second, the new lines of authority meant that NSDP staff working in technical, fiduciary and M&E functions
no longer answered directly to the NSDP Director, but rather to the Heads of the relevant MoLSAMD directorate or unit.
Whereas it was originally thought the NSDP functions would be integrated into the DGSD such that all the implementing
functions would come under one Directorate, in the end the functions were integrated across multiple Directorates. This
in itself complicated the tasks for senior NSDP management, as they had less authority to take decisions or otherwise act
to ensure the timely completion of work in these areas; and had to liaise with multiple heads of Directorates, each with
equal authority and answering to two different Deputy Ministers and (in the case of M&E) to the Office of the Minister.
In practice, NSDP staff often needed to consult with the Deputy Ministers and the Office of the Minister to ensure
coordination and decision-making. Third, some of the procedures followed by the MoLSAMD units/directorates were
reported to be more cumbersome and time-consuming. In some cases, they did not conform to conventional project
management procedures. For example, NSDP senior management was not able to sign off on contract deliverables, which
sometimes needed to be approved by the M&E unit in the General Planning Directorate. Fourth, persons occupying posts


        15By the time of the MTR, 78 TPs had been short-listed as the first batch to sign LoUs with MoLSAMD; they enrolled approximately
        6,300 students, as against the Project’s final target exceeding 40,000. Further, approximately 2,700 of these students were to be
        enrolled in courses that MoLSAMD was concerned would not lead to employment, viz. project management, secondary academic
        degrees, IT skills, and English language. (See Annex 13 for further details.) Various factors likely contributed to the relatively restricted
        number of TP/enrollees and range of trades, including TPs withdrawing from consideration after delays and increased demands on the
        part of MoLSAMD; cutbacks in donor funding for vocational training; and a reduced field presence of donors including NGOs providing
        training. These factors are examined in the next section (III.B).



                                                                                                                                             Page 16 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



for the NATEJA project were tasked with building capacities of the units/Directorates to which they were attached. While
this is laudable, the ICR Team was informed that ‘skills transfer’ – both prior to and after integration – was often taken
to mean ‘do work of the Ministry beyond that pertaining to the NATEJA project.’ Finally, the uncertainties and
implications of integration (including lower salaries) may have contributed to turnover among NSDP staff and persons
occupying posts tasked with NATEJA implementation, which naturally resulted in further delays. 16 A case in point is the
Program Director post, which was vacant from mid-2016 until closure. In that period the project was managed by an
Acting Director who was more technical than managerial, and who had to seek approval from the Deputy Minister for
decisions, creating delays.

43.     Component 2 (Project Management, Capacity Building of MoLSAMD and M&E) had been designed to build
technical capacity within the Ministry. However, the design focused on building capacities in areas that were not directly
involved with the selection, operation and incentivizing of TPs and trainees; that is they focused rather on procurement,
financial management and LMIS and monitoring. Progress was made in building capacity in these areas, and indeed the
Project exceeded the quantitative target of the relevant intermediate indicator (150 MOLSAMD/NSDP staff trained as
against a target of 100). However, as Project implementation and NSDP integration went on, the number of staff eligible
burgeoned well beyond initial expectations and it became clear that capacity building needs greatly exceeded
Component resources. As a result, the capacity building was spread too thinly and the persons occupying the posts that
had originally been in mind when the Project was designed did not get the in-depth strengthening of knowledge and skills
required to ensure quality in Project implementation. 17 See also Annex 11.

44.      Recruitment of TA. Implementation arrangements foresaw the placement of TA to assist in key areas of Project
implementation, and the ORAF noted that international advisors were a risk management measure to address weak
MoLSAMD capacity. For instance, Component 1 (Improving the Quality of Non-formal Training and Labor Market
Outcomes of Trainees) provided for an International Advisor for Management of Training Providers, whose terms of
reference included ‘assisting NSDP senior management team in managing and supervising the firms contracted for
implementation’ and ‘ensuring that the NATEJA project components, and particularly Component 1…are delivered on
time …’ 18 This Advisor was not recruited. It is perhaps significant that the components and activities with good success
(e.g. Sub-Component 3.2 - Promoting Entrepreneurship among Afghan Youth) and the LMIS under Component 2 were
fortunate in having successfully recruited a full-time TA (international) responsible for ensuring implementation. The
failure to recruit TA, particularly International Consultants (ICs), seems to have been due in part to a MoLSAMD
preference for National Consultants (NCs) and in part to the security situation (discussed below). 19 There were also
significant difficulties associated with recruiting TA firms; this is discussed below.

45.     The progressive tightening of eligibility criteria. This affected Component One (Improving the Quality of Non-
Formal Training and Labor Market Outcomes of Trainees) in particular. After the Inspection Agency had been recruited
and started its screening of TPs, the Ministry questioned the Agency’s procedures for hiring field workers and introduced
new and complex TPs screening and registration procedures. While the Ministry was motivated by a concern to ensure
that the TPs were of good quality, the increasing informational and regulatory demands upon the TPs (including the need

        16 For example, there were two NATEJA Program Directors throughout project implementation. The first stayed in post for
        approximately 1.5 years; the second for 9 months. There was no Director from July 31, 2016 until closure. The second-in-command
        post (Program Coordinator) was occupied until end July 2016, and vacant thereafter until closure.
        17 See also ISR dated October 24, 2018.
        18 Project Implementation Manual, p. 97. Italics added.
        19 With respect to the International Advisor for Management of Training Providers, the government insisted on recruiting a National

        Consultant instead. Discussions between the government and the Bank on this issue took time and eventually became moot in light of
        the effective suspension of Component 1 activities after the MTR; and this critical post was never filled.

                                                                                                                                   Page 17 of 60
          The World Bank
          Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



to officially register, and therefore also pay the AFN 30,000 registration fee) encouraged many of them to withdraw their
application. The Ministry was also concerned about the preponderance of non-technical skills courses and introduced
new restrictions (such as excluding IT and English language). There was also a legitimate concern that the TPs were
largely semi-urban, effectively closing off access to a segment of the population targeted by the Project.

46.      Altogether, these contributed to delays in signing agreements with TPs and some of the TPs dropping out. In
preparation of a first batch of participating TPs, around November 2015, the Inspection Agency initially mapped 250 TPs;
inspected 175; and invited 134 to apply to participate in the incentives scheme. Faced with new compliance demands,
only 88 TPs submitted a formal application, of which 78 were recommended for approval by the responsible MoLSAMD
committee. 70 of these TPs actually signed a LoU (though not MoLSAMD) in May 2016. Yet MoLSAMD then introduced
more compliance demands, creating more delays. This led more TPs to withdraw, and it took until December 2017 to
finalize a shortlist of 24 TPs enrolling approximately 2,000 trainees who were willing to re-sign the LoU as the first batch
of TPs. At this point, it was too late to complete activities on time envisaged in the project, and the decision was taken 7
months later to cease Component 1 activities and close the Project early.

47.      Post-MTR uncertainty. The Project implementation and Bank supervision teams were fully cognizant of the design
and implementation challenges facing the project, and at the mid-term review (MTR) MoLSAMD and the WB agreed on
a restructuring. However, despite MoLSAMD’s initial agreement to restructure at the MTR, there ensued a lengthy period
of discussions – not resolved until December 2017 and formally confirmed until July 2018 – within and between WB, MoF
and MoLSAMD as to whether the Project should be restructured or closed. (See Section I.B and Annex 7 for
details). During this extended period of uncertainty (January 2017- December 2017) as to the Project’s fate, the decision
to restructure seems to have been taken and then overturned twice. The agreement to restructure was captured in the
Mid-Term Review Aide Memoire (November 22-26, 2016), the Management Letter from the Midterm Review dated
January 17, 2017. During the Bank mission in February 12-14, 2017, the Advisor to the Deputy Minister at MoLSAMD
overturned the MTR recommendations and ‘conveyed to the mission the desire to terminate the NATEJA project’ (as per
the mission’s Back-To-Office-Report). Discussions continued within the Bank between the CMU which advocated early
closure and the GP which disagreed. Despite the communication during the February mission, the Bank received a letter
from the Minister of Finance dated August 16, 2017 formally requesting the restructuring based on the recommendations
of the MTR ‘fully endorsed by both MoLSAMD and the MoF. The discussions continued between the GIRoA and the CMU,
which culminated in an agreement in December 2017 to close the Project. It took until July 2018 for MoLSAMD to request
MoF to close the Project which was quickly followed up by a formal MoF request to the Bank to close the Project early
and cancel unspent funds. A restructuring paper to this effect was prepared to enable an amendment to the Grant
Agreement, the Project closed on October 31st, 2018.

48.      There was little impetus to implement the Project’s main component (Component 1 - Improving the Quality of
Non-formal Training and Labor Market Outcomes of Trainees); all parties agreed that it could no longer be implemented
as originally designed, although views differed between the CMU which judged that MOLSAMD had not demonstrated
ability or willingness to implement the Project effectively and the Education GP which believed restructuring could turn
around the implementation performance. This uncertainty also made it difficult to keep project positions filled (see also
above).

49.     Fiduciary factors. There were shortcomings in procurement that negatively impacted the project’s
implementation. The procurement plan had nine major consultancy packages, eight of which should have been awarded
and disbursing within the first year. However, despite having the support of a national and international procurement
advisor, fifteen months into the implementation period, only one contract had been awarded (for the Social and


                                                                                                                    Page 18 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



Environmental Assessment); resulting in a downgrading of the rating from Satisfactory to Moderately Satisfactory. After
thirty months, only two further major consultancies had been recruited – the Inspection Agency for the TPs of Component
1 (Improving the Quality of Non-Formal Training and Labor Market Outcomes of Trainees) and the Grant Management
Firm for Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) – and these alone had required more
than 300 days between technical proposal and contract award. 20 Only one further TA package was subsequently
procured (for the Sub-Component 3.2 Impact Evaluation). This, along with other procurement shortcomings, 21 led to the
rating being further downgraded to Moderately Unsatisfactory in August 2016, where it remained until closure. The
delays in initiating the procurement for the Training Provider for Sub-Component 3.1 (Encouraging Rural Employment),
followed by the challenges in identifying a suitable and qualified service provider despite multiple re-advertisements,
were such that by the MTR no trainings had taken place; this led to the MTR recommendation that this Sub-Component
be dropped. See Annex 7.

50.     Non-fiduciary factors contributed to the procurement difficulties. In particular, the procurement function was
hampered by the non-execution of predicate activities, for example the TA firms to conduct the Special Purpose Review
and the Feasibility Study. Further, the security situation makes international firms and ICs reluctant to bid on Afghan
contracts. See Annex 12.

51.    The FM performance rating was downgraded from Satisfactory to Moderately Satisfactory in December 2014
(where it remained until closure), due to low budget credibility and weak fixed assets management. However, there
were no FM issues that undermined Project performance.

52.     Recipient financing. The government committed to provide USD 3.0 million to cover salaries of technical teacher
trainees and governmental support staff. In the event, roughly USD 1.5 million were allocated through the discretionary
budget to the NSDP during the project period; but none of the funds were used for salaries related to the Project or for
support staff. As noted in the ISR of February 2017 (in which Counterpart Funding was downgraded to Moderately
Unsatisfactory from Satisfactory), the lack of recipient financing resulted in a disproportionate expenditure on project
operating costs and overstretched the NATEJA team, leading to implementation delays.

53.     Technical capacity and strong design. The preceding discussion must not obscure the fact that when the Ministry
supported Project implementation as per design, there were strong results. In particular, Sub-Component 3.2 (Promoting
Entrepreneurship among Afghan Youth) very nearly reached the target written in the project description, providing grants
and training/counselling to 2,482 individuals to start or expand a small business. 22 Had the project been closed soon
after the MTR, this sub-component would not have achieved its results.

        Factors subject to World Bank control

54.     The Project benefited from proactive and relatively stable Bank supervision. The Bank worked closely to support
senior NSDP management. It intervened for example to facilitate the work of TA firms (e.g. the Inspection Agent, the
GMF and the Impact Evaluation firm), to ensure they understood project requirements, were able to navigate issues with
MoLSAMD, and kept reasonably well to their deliverables schedule; and regularly provided them with recommendations
to improve the quality of their work and outputs. The Bank’s supervisory role is discussed further in Section IV.C.


        20 The fourth major TA contract - to conduct the Impact Evaluation (for Component 3.2 only) - was signed roughly on time.
        21 See Section IV.B.
        22 As against the 2,582 target written in the project description (but not included in the results framework).



                                                                                                                                    Page 19 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



55.      There were five supervision missions throughout the Project’s 4.5 year life (of which one virtual). These took
place during the first three years, when the bulk of implementation activities took place, as well as at the mid-term review
(MTR). The Bank was proactive in supporting the MTR, and followed up quickly with the government on how to proceed
with restructuring. (The subsequent delay is discussed below). The missions were supplemented with email and other
communications as well as the regular supervision and support provided by the Bank technical, fiduciary and safeguards
staff based in Kabul.

56.     The security situation severely restricted the Bank’s ability to supervise, with meetings and visits beyond the
Ministry being very difficult to organize. The Bank mitigated these restrictions as it could, for example by organizing MTR
meetings with beneficiaries at the Bank’s premises.

        Factors outside the control of government and/or implementing entities

57.     The security situation. This deteriorated after effectiveness and remained fraught throughout the
implementation period. This led inter alia to the resignation of some international TA; made it difficult to recruit
international TA firms and ICs; and impacted the mobility of Project staff, hampering their ability to implement and
supervise activities.

58.      Changing sectoral landscape. In the post-preparation period, Official Development Assistance (ODA)
disbursements to Afghanistan for vocational training declined each year from USD 32.7 million in 2013 to USD 24.5 million
in 2017. 23 The number of international armed forces was reduced after 2014. At the same, there was an increased
demand from government to provide ODA through the budget. Together, these factors encouraged the donor
community (including NGOs) to reduce their presence in the field, and to cut back their funding of non-formal training
providers. This in turn reduced the number of students and the range of courses offered. This may have contributed to
the difficulties encountered in Component 1 (Improving the Quality of Non-formal Training and Labor Market Outcomes
of Trainees) to find a sufficient number of TPs enrolling sufficient students across a range of technical trades. With
respect to Sub-Component 3.3 (Feasibility Study on Apprenticeship Program), the research by GIZ and its subsequent
decision to introduce an informal apprenticeship program at a large scale meant that there was no longer a strong
rationale for implementing this component’s activities. 24

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME

 A. QUALITY OF MONITORING AND EVALUATION (M&E)

    M&E Design

59.    The results framework did not encompass project outputs and outcomes in one important respect: there was no
intermediate indicator for Sub-Component 3.2, 25 while the phrasing of the PDO indicators was such that they measured


        23 OECD International Development Statistics, https://stats.oecd.org/qwids. Accessed January 10, 2019.
        24 As noted above, had restructuring proceeded this component would have been dropped from the Project.
        25 The intermediate result indicator one (‘Number of new micro and small-scale business (sic) established and are running’) might have

        covered C3.2 activities, but the final target of 70 – in contrast to the 2,520 businesses targeted as stated in the project description for
        C3.2 – and the indicator description (‘New micro and small-scale business that are established and running post-graduation from
        components 1 and 3’, emphasis added) make it clear that only C1 and C3.1 activities were included here.

                                                                                                                                           Page 20 of 60
          The World Bank
          Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



only increases in the potential for employment and income generated by Component 1 and Sub-Component 3.1. That is,
the PDO indicators restricted increases in potential to those generated as a result of Component 1 and Sub-Component
3.1 training activities; whereas Sub-Component 3.2 gave grants, training and support to persons to start or expand
businesses, and in that way increase their potential for employment and income. This particularity of the results
framework was a weakness in that it did not capture Sub-Component 3.2, which was the most successful of the Project’s
components with employment/income-related activities, and contributed to positive outcomes in terms of employment
and income.

60.     Having said that, the Results Framework indicators captured much of the Project’s outputs and outcomes; the
indicators were generally well phrased; 26 and the data sources, methodologies and definitions were for the most part
clear and adequate. Special monitoring exercises were well articulated, including third-party monitoring (to assess quality
of TPs selection and transparency of beneficiary selection process), an independent assessment of trainee competencies
(to monitor the quality of training), and special purpose reviews (to monitor the cash incentives transactions). The design
of impact evaluations were clearly described in the PAD for Sub-Component 3.1 (Encouraging Rural Employment) and
Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth). 27

   M&E Implementation

61.      An elaborate software-based Management Information System (MIS) for the Project was developed. It was
formally housed at the M&E unit of MoLSAMD’s General Planning Directorate, and could have been used to monitor
Project activities and other activities of MoLSAMD and the ESCs; with web-based applications for data entry being feasible
for TPs, ESCs and other actors. 28 However, the MIS was never operationalized, due in part to the fact that the main
activities for which it was required were not implemented (see Annex 14); the results framework indicator target was
consequently not met. This did not seriously hamper implementation monitoring of those activities that were executed.
While the MIS was being developed, the NSDP staff and TA working on Component 1 (Improving the Quality of Non-formal
Training and Labor Market Outcomes of Trainees), Component 2 (Project Management, Capacity Building of MoLSAMD
and M&E) and Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) built their own monitoring
databases and mechanisms that were regularly consulted and used. The M&E Unit of the Planning Directorate also
monitored Sub-Component 3.2 activities. An evaluation of Sub-Component 3.2 was successfully conducted using a
randomized selection mechanism, an impressive accomplishment in any case but particularly so in the Afghan context.
Some planned M&E activities were not conducted, mainly due to the non-execution of predicate activities (Annex 14).

62.     More generally, the Project supported MoLSAMD’s monitoring functions as per design. In particular, a Labor
Market Information System (LMIS) was developed with project support. It is fed internally by the MoLSAMD using data
pertaining to ESCs operation and the registration of work permits for Afghan nationals and for foreigners; only the work
permits data are currently being entered, and only one ESC (in Kabul) is currently operational. It is to be fed externally
from five other ministries, using for example data on graduates and the labor market; though this aspect is not yet
operational. The LMIS also incorporates household and enterprises’ survey data. In this regard, the Project supported
the Human Resource Demand and Supply Survey (2016), which provided data for decision-making. 29

        26 Some of the indicators could have been strengthened. For instance, intermediate result indicator four (number of MoLSAMD/NSDP

        staff trained) is an activity indicator, and might have been replaced with a higher-level indicator of strengthened capacity.
        27 Cf. pp. 34-37 of the PAD.
        28 The applications would have enabled the tracking of registered TPs, trainees and grant recipients.
        29 The supply side gathered data from households and TVET institutions. The demand side gathered data from formal and informal




                                                                                                                                 Page 21 of 60
          The World Bank
          Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)




   M&E Utilization


63.    While the software-based Project MIS was not operationalized, other monitoring and evaluation mechanisms
were utilized that enabled activities to be monitored and decisions taken. For instance,

   • Monitoring of Component 1 led MoLSAMD to reconsider selection criteria for TPs.
   • Sub-Component 3.2 activities were closely monitored for quality, leading to improvements and timeliness during
   the second stage of implementation. Further, the IE design for Sub-Component 3.2 was closely followed during the
   post-applications selection phase; and this contributed to the rigor of the selection process.
   • The LMIS is being used to register work permits, while results from the labor demand and supply survey were
   used for example by MoLSAMD to decide on priority areas for training and by the Ministry of the Economy to design
   new private sector development programs.

   Justification of Overall Rating of Quality of M&E

64.      The ICR assigns a modest rating to the quality of M&E for the following reasons. First, the results and indicators
identified for the M&E framework captured the TOC’s key results with mechanisms designed for the most part to generate
valid, reliable and accurate information on a timely basis. However, there was a significant design shortcoming in that the
PDO outcome indicators should have been phrased to capture the outcomes of Sub-Component 3.2 (in addition to
Component 1 and Sub-Component 3.1). Second, there were some shortcomings in implementation, particularly in regard
to operationalizing the project MIS and to carrying out certain M&E activities. This was due in part to the fact that the
project did not implement its primary activities in Component 1 and Sub-Component 3.1. These shortcomings did not,
however, substantially hinder the project’s capacity to monitor and evaluate those activities that were implemented.
Third, monitoring data generated by the Project were used to introduce course corrections and control quality in the
execution of activities, particularly for Sub-Component 3.2.

 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE

65.      The following safeguard policy was triggered at appraisal: (a) Environmental Assessment (OP/BP 4.01). The
project developed an Environmental and Social Management Framework to lay out the specific requirements, processes,
and responsibilities for ensuring that the activities implemented by the Project will not have negative impacts on the
environment and people. As per the provisions of the ESMF, the project was required to undertake a Social and
Environment Assessment prior to commencement of any activities proposed under the project. An international firm
with a local joint venture was hired to conduct the required assessment. The firm produced a draft SEA report, which
was shared with the World Bank for review. The Bank team reverted with comments to address specific shortcomings
of the report. However, the report was never shared again for the Bank’s review, and as a result was not considered
finalized and cleared by the Bank’s safeguards team, as the comments were not addressed by the firm. The report was
approved by the project TTL for the purpose of due payments to the firm. The project was also required to prepare a
safety manual. However, there is no evidence of its completion as it was not shared with the World Bank for review.


        businesses including SMEs. Data were collected manually and later entered into the databases. The LMIA plans to periodically update
        the data using its own surveys, with direct data entry using a web-based application; as well as findings from Central Statistical Office
        (CSO) surveys.

                                                                                                                                         Page 22 of 60
          The World Bank
          Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



Institutional arrangements for a Grievance Redress Mechanism were established under the project, mainly pertaining to
Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth), however the Bank Safeguards Team was not
provided with evidence as to its functionality, operation and effectiveness.

66.     During the MTR, which took place in November 2016, the Bank’s safeguards team did not take part in the
meetings. The client reported that a dedicated Environment Specialist and a Social & Gender Specialist were hired for
overseeing and implementation of the safeguards compliance under the project. The contracts for the mentioned staff
were valid until June 2018. However, there are no records of any reports that have been shared by the client on the
monitoring and supervision of the safeguards compliance.

67.     The project maintained FM arrangements throughout most of the implementation period, except for the internal
audit report which was not conducted regularly nor shared with the Bank on a timely basis. The Bank otherwise received
all FM reports on a timely basis. There were no ineligible expenditures, nor are there any overdue IFRs or external audit
reports. All the fixed assets purchased from grant proceeds are to be transferred to MoLSAMD before the end of
December, 2018.

68.      There were significant shortcomings in procurement that affected project performance. These pertained in the
main to delays in the procurement, or the non-procurement, of TA packages; see Section III. B and Annex 12. Other
procurement shortcomings included the plan not being regularly updated, the procurement of luxury goods, and a failure
to reflect the plan in the STEP tool. The rating was consequently further downgraded to Moderately Unsatisfactory in
August 2016, where it remained until closure.

 C. BANK PERFORMANCE

    Quality at Entry


69.      The Bank team worked well to identify, facilitate the preparation of, and appraise the operation, thus laying a
foundation for the Project to achieve its planned development outcomes and acting in a manner consistent with the
Bank’s fiduciary role. The project content and strategic approach were well defined and integrated into the government’s
sectoral and national strategies. The project was for the most part technically well designed. The activities built on
national good practices, leveraged the experience of other projects, and targeted a gap in the sector that promised to
benefit an important segment of the poor and vulnerable population.

70.     The project was perhaps overly innovative in that its centerpiece incentives scheme – to reward TPs if their
graduates were certified, found employment or started a business – was predicated upon there being sufficient students
enrolled in trades with good employment prospects; and this in turn relied upon the students’ fees being paid either by
themselves or other DPs. This left the project open to a risk that was not explicitly recognized but nonetheless seems to
have been realized, namely that a downturn in the economy or DPs funding, or an increased security-related reluctance
on the part of NGOs to operate in the field, would reduce the number of students training at non-formal TPs. Further,
while the project targeted the semi-urban and rural poor, it did not formally identify priority trades. This resulted in some
TPs being identified by the Inspection Agency that offered courses that would not (according to MoLSAMD) lead to
employment and many of which were not actually skills training. Finally, the project design was predicated upon the
successful implementation of other projects, leaving it vulnerable should these encounter difficulties. These factors
created a substantial challenge for the component most critical to the Project’s development outcome (Component 1 -


                                                                                                                    Page 23 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees). Whereas the Project intended to
enroll over 40,000 students under Component 1, by the time of the MTR the Project had managed to finalize a first-batch
list of 78 TPs enrolling approximately 6,300 students, predominantly in semi-urban areas. 44 percent of these TPs
(enrolling approximately 2,700 students) offered courses in areas that MoLSAMD was reluctant to finance. 30 Further, the
project had no means to certify TP trainees, as this element of the ASDP II did not materialize. (See also Annex 13).

71.     The Project was satisfactorily appraised from both the economic and financial perspectives. The fiduciary
arrangements within NSDP and MOLSAMD as set out in the PAD were thoroughly appraised, with risks identified and
mitigating measures identified.

72.     The implementation arrangements as appraised in the PAD built upon experience previously accumulated by the
NSDP. However, at negotiations it was agreed that the NSDP would be formally integrated into the DGSD/MoLSAMD.
This was a significant change to the implementation arrangements, and complicated the implementation task for NSDP
(see Section III.A and Annex 11). The implementation arrangements arguably should have been re-appraised prior to
completing negotiations, and more capacity building under Component 2 should have been incorporated with clear
targets pertaining to expected results.

73.      The design of the M&E arrangements was generally satisfactory, though arguably the PDO and its indicators
should not have been restricted to outcomes deriving only from non-formal skills training, thus excluding similar benefits
from the Business Grants Scheme. The risk assessments were suitable (and later proven correct), as were mitigation
measures; though there are limits to which mitigation is feasible, particularly with regard to security. Finally, the proposed
Bank inputs and processes were adequate, and included regular supervision missions and reporting using a team with the
full range of relevant expertise.

    Quality of Supervision

74.     The Bank carried out five supervision missions (of which one was virtual) up to the immediate post-MTR period;
this was one mission short of the bi-annual missions schedule foreseen in the PAD. Thereafter, implementation was
largely suspended (except for Sub-Component 3.2). Throughout implementation, the Bank augmented its missions
through Kabul-based staff conducting technical and fiduciary supervision, as well as regular email and other
communications. Team leadership was quite stable, with two TTLs from the preparation stage to closure, and one co-TTL
based in Kabul who was a constant team member since the beginning of Project preparation. The task team included the
necessary range of skills to supervise the Project in terms of technical quality and implementation and fiduciary
arrangements. (See Annex 2).

75.      The Bank’s Supervision Team concentrated throughout on the two development outcomes of the PDO, as can be
seen by the focus of the AMs on linking implementation performance (of activities) to their impact on the PDO, and the
continued dialogue with government to ensure that deadlines were met. The Team intervened regularly to support the
implementing agency on technical matters, as can be seen for example on the detailed guidance provided on the work of
the Inspection Agent (Component 1), the design of the first stage of Sub-Component 3.2, and ensuring quality sampling
for the IE. Supervision mission findings were communicated clearly to Bank management and to the GIRoA, the latter
both in writing and through meetings with senior GIRoA officials. Agreed actions points were included in the AMs with
deadlines and responsibilities, and these were subsequently monitored and followed up for implementation by the Team.


        30   Such as project management, secondary academic degrees, IT, and English language.

                                                                                                                     Page 24 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



The performance reporting was generally of satisfactory quality, with some exceptions noted below.

76.     The fiduciary aspects were generally well supervised. The Team identified early on certain shortcomings in
fiduciary performance and procedures, and provided feedback and guidance on remedies to the client; with the exception
of the Aide Memoire dated December 13-17, 2015 in which the FM and procurement paragraphs are blank. The
supervising of safeguards might have been stronger. The safeguards team did not approve the SEA, did not follow up the
non-receipt of a safety manual, did not have input to the GRM (which was only established in mid-2017), and has no
records of client reports on safeguards compliance; nor did it participate in the MTR.

77.     At the MTR, the Bank was proactive in getting stakeholders to analyze implementation progress, identify lessons
learned, and propose recommendations for restructuring. It followed up quickly with a mission to clarify with the
government how best to implement the MTR recommendations. A protracted period ensued in which the GIRoA and the
CMU discussed both internally and with one another how to act on the MTR recommendations, vacillating between
acceptance and rejection, during which the Bank continued to supervise locally ongoing activities. This extended period
of discussion and decisions being taken, reversed and then again reaffirmed and reversed was unproductive if not
paralyzing. (See Annex 7). It would have been better for the GIRoA and CMU to act decisively on the MTR
recommendations, either by implementing them (as initially agreed) or execute legal remedies to either close or partially
suspend the project which was performing unsatisfactorily. 31

    Justification of Overall Rating of Bank Performance

78.     The ICR rates the overall Bank performance as unsatisfactory. At entry the Bank supported the government to
achieve a project design albeit with major shortcomings in its technical aspects and implementation arrangements.
During implementation, the Bank provided supportive results-focused supervision despite a very challenging
implementation environment, with moderate shortcomings in supervision regularity and significant shortcomings in
supervising safeguards; and was party to an extended post-MTR discussion that unproductively delayed a decision on the
MTR recommendations.

 D. RISK TO DEVELOPMENT OUTCOME

79.       The risks to the employment and income gains experienced by beneficiaries of Component 3.2 (Promoting
Entrepreneurship among Afghan Youth) are primarily economic and social in nature. Any deterioration of the economy
may hinder the beneficiaries’ capacities to maintain employment and generate income through their business. Further,
the female beneficiaries who started up or expanded an existing business require a certain degree of social mobility for
their continued success. Any move towards social/religious norms that restrict women’s mobility will likely negatively
impact the female beneficiaries’ ability to maintain their levels of business-related income and employment. With respect
to MoLSAMD’s budgeted replication of the Business Grant Scheme for TVET graduates, there is the risk that government
ownership of the replication might waver, or the Ministry will not have the necessary capacity to sustain the Scheme
successfully; these would be compounded by any budget cutbacks resulting from an economic downturn and/or further
fiscal tightening.

V. LESSONS AND RECOMMENDATIONS


        31   A partial suspension would have allowed Component 3.2 to complete. As noted above, this component yielded positive results.

                                                                                                                                     Page 25 of 60
             The World Bank
             Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



80.     Skills development should continue to be a priority area for cooperation between the GoA, the Bank and
Development Partners. The sectoral challenges remain; the factors subject to government control identified in the ICR
that impeded Project performance can be resolved; and sectoral interventions can be effective, as demonstrated by Sub-
Component 3.2 and other Development Partner-supported projects (e.g. those supported by GIZ), not to mention prior
Bank-supported operations (e.g. ASDP). Any future operations should respect certain technical restrictions. First, priority
trades should be identified at the outset (a lesson already incorporated into the ASDP II). Second, given the challenging
context, the design should be conservative. Innovations (such as Component 1, Improving the Quality of Non-formal
Training and Labor Market Outcomes of Trainees) should be first done on a pilot basis. The inter-linking of projects - such
that the implementation of one depends on the successful implementation of another - should be minimized.

81.      MoLSAMD’s capacities to implement projects still need to be strengthened such that, first, the departments/units
carrying out implementation functions for a project – whether technical, fiduciary or M&E – answer to one overall
authority below the level of Minister with powers of coordination and final decision-making. Second, the Ministry
requires augmented capacity to implement large-scale activities and manage TA firms, and should hire capable TA
accordingly even if it requires recruiting internationally. Third, the power to sign off on contract deliverables should be
given to the head of the department/unit managing the contract, consulting with the M&E unit. Fourth, the M&E function
should be further strengthened, ensuring at a minimum that it reports systematically on its monitoring of project
activities to the overall project authority and the relevant technical departments/units; that it develops a system to work
with other departments/units to monitor any trainings for logistics, quality and effectiveness; and that it is able to
technically manage TA contracted to undertake evaluations. The government and the Bank may also consider project
implementation solutions that are sometimes used in challenging contexts, such as a Project Coordination Unit
embedded within the structure of MoLSAMD. 32

82.      The Project implementation arrangements were significantly modified at negotiations in response to concerns
expressed by MoF, with deleterious consequences. These concerns should have been addressed earlier in the preparation
process, to have sufficient time to design a more robust set of implementation arrangements and capacity building
package. Concerns of all stakeholder Ministries should be explicitly addressed during the preparation process, to ensure
there is broad government ownership at the outset.

83.      The Business Grants Scheme shows evidence for a cost-effective way to create higher levels of employment and
income, including for female entrepreneurs. Provided the rigorous conditions of the Business Grants Scheme can be
replicated, this scheme should be scaled up. Among the conditions contributing to success were: effective
implementation arrangements, in this case a full-time international consultant working on behalf of the Ministry, with
both technical and operational responsibilities including managing the TA firm; an openness and flexibility to improve
the model based on lessons learned during an initial, small run; substantial field-level staff available to support the firm’s
local activities; a gender dimension for training and support; and impact evaluation parameters designed at the outset
that imposed a certain rigor and quality on activities implementation. 33 Further, a similar scheme could be usefully

        32 Other countries in the region have used (and in some cases continue to use) education/training-sector PCUs despite having far fewer
        challenges in terms of security, political instability and management capacity. For example, Tajikistan used a PIU from the mid-1990s
        until 2011, at which point it introduced split implementation arrangements (half implemented by a PIU, the other by a PCU with TA-
        augmented units of the Ministry) followed later by full implementation by the Ministry with TA-augmented units working as a PCU. In
        Kyrgyzstan, a PIU was used until 2013 after which it began to function as a PCU and place its TA in units of the Ministry, while still
        keeping an independent PCU Director.
        33 For instance, the need to have a relatively large number of similar persons in the treatment and control groups led to rigor in the

        scoring and sorting of applicants. This in turn helped to identify beneficiaries who were genuinely interested to start up or expand a



                                                                                                                                      Page 26 of 60
              The World Bank
              Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



    combined with technical/vocational training (whereby eligible graduates receive a business grant), 34 tested on a limited
    scale, and scaled up if found to be relatively cost-effective. Indeed, MoLSAMD has included in its budget request for the
    fiscal year 1398 (2019) funds to enable such grants for graduates of the TVET program.

    84.     There is substantial value to cooperation between DPs within an overall strategic context. The Project would
    have been able – had the restructuring recommended at the MTR been implemented – to benefit from GIZ’s work on
    apprenticeships; while the IADC-supported NATEJA benefited from the Project’s design, lessons learned during Project
    implementation, as well as the strengthened capacities and procedures created by the Project. The projects supported
    by the Bank, GIZ and the IADC all work within the same strategic context. Such strategically-bound cooperation should
    continue, and would further benefit from a strengthened policy and strategy framework that covers all aspects of the
    TVET landscape.

    85.      It took a year and a half before a decision was made to finally close early the project and cancel the remaining
    allocation. This prolonged period of uncertainty hampered staff morale and did not alleviate implementation bottlenecks.
    The project partners should take advantage of World Bank guidelines to change course when it becomes clear that some
    project design aspects need to be modified. Within the Bank, the Global Practice and CMU should clearly and consistently
    convey a joint position when communicating with the client.

    86.      The results framework failed to include PDO indicators that adequately captured the PDOs. This indicates the
    need to pay careful attention during the concept, preparation and appraisal stages to the PDO and the wording of the
    indicators to capture all project activities contributing to similar outcomes.

.




            business. See Annex 9 (C3.2 Business Grant Scheme).
            34 This was in effect one of the recommended restructuring changes agreed upon at the MTR, which proposed that C1 graduates be

            eligible for the C3.2 grant. See Annex 7 (Restructuring Agreed at the Mid-Term Review).

                                                                                                                                      Page 27 of 60
    The World Bank
    Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                                                ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS


A. RESULTS INDICATORS

A.1 PDO Indicators


 Objective/Outcome: The project focused on two key outcomes. The first outcome was to increase potential for employment among targeted young
 Afghan women and men in rural and semi-urban areas.
                                                                                                    Formally Revised          Actual Achieved at
 Indicator Name                    Unit of Measure Baseline                Original Target
                                                                                                    Target                    Completion

 Share of trainees who find        Percentage      0.00                    15.00                    0.00                      13.00
 employment within 6 months
 of graduation, with respect to                    31-May-2014             31-Dec-2018                                        31-Oct-2018
 control groups


 Comments (achievements against targets): There was no progress against the indicator for employment and incentive to the training providers
 as component one (Improving the quality of non-formal Training and Labor Market Outcomes of Trainees) was not implemented. However,
 the results of the Business Grant Scheme (Sub-Component 3.2: Promoting Entrepreneurship Among Young Afghans) is justified to be captured
 as outcome for this indicator. Therefore, employment for the beneficiaries increased 13 percentages in respect to that of control group.

 Objective/Outcome: The second outcome for this project was to increase potential of higher earnings among targeted groups.
                                                                                                    Formally Revised          Actual Achieved at
 Indicator Name                    Unit of Measure Baseline                Original Target
                                                                                                    Target                    Completion

 Increase in trainees’ earnings,   Percentage      0.00                    15.00                                              17.00
 with respect to control
                                                   31-May-2014             31-Dec-2018                                        31-Oct-2018


                                                                                                                                       Page 28 of 60
   The World Bank
   Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



groups


Comments (achievements against targets): The increased in earnings of the trainees with respect to control group is calculated from component
3.2 of Grant Scheme and results are presented in endline report of the impact evaluation, showing 17 percent of increased monthly earning
with statistic significance. Exceeding the original target by 2 percentage points. The beneficiaries acknowledge 55% increase as an average
revenue for last month before the survey was conducted.

Unlinked Indicators
                                                                                               Formally Revised        Actual Achieved at
Indicator Name                  Unit of Measure Baseline                  Original Target
                                                                                               Target                  Completion

The amount of incentive(s)      Amount(USD)      0.00                     4000000.00                                   0.00
disbursed to training
providers                                        31-May-2014              31-Dec-2018                                  31-Oct-2018


Comments (achievements against targets): Since component 1 of the project did not proceed as planned and no TPs were hired under the
project, this indicator remain at the baseline value of zero against the target.

                                                                                               Formally Revised        Actual Achieved at
Indicator Name                  Unit of Measure Baseline                  Original Target
                                                                                               Target                  Completion

Direct project beneficiaries    Number           0.00                     44500.00                                     2482.00

                                                 31-May-2014              31-Dec-2018                                  31-Oct-2018

  Female beneficiaries          Percentage       20.00                    30.00                                        30.00

                                                 31-May-2014              31-Dec-2018                                  31-Oct-2018




                                                                                                                                 Page 29 of 60
    The World Bank
    Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



Comments (achievements against targets): The direct project beneficiaries include beneficiaries sub-component 3.2 the Business Grant Scheme.


A.2 Intermediate Results Indicators

 Component: Component One: Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees.
                                                                                                Formally Revised      Actual Achieved at
 Indicator Name                  Unit of Measure Baseline                 Original Target
                                                                                                Target                Completion

 Number of teachers from         Number          0.00                     50.00                                       0.00
 selected non-formal training
 providers who receive                           31-Mar-2014              31-Dec-2018                                 31-Oct-2017
 training from TTTI/DMTVET
 or any other accredited
 teacher training center


 Comments (achievements against targets):


                                                                                                Formally Revised      Actual Achieved at
 Indicator Name                  Unit of Measure Baseline                 Original Target
                                                                                                Target                Completion

 Number of selected non-         Number          0.00                     70.00                  0.00                 0.00
 formal training providers
 that adopt NOSS for                             30-May-2014              31-Dec-2018                                 31-Oct-2018
 improvement of their
 teaching quality


 Comments (achievements against targets):




                                                                                                                                Page 30 of 60
   The World Bank
   Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)




Component: Component Two: Project Management, Capacity Building of MOLSAMD and M&E
                                                                                                 Formally Revised    Actual Achieved at
Indicator Name                 Unit of Measure Baseline                  Original Target
                                                                                                 Target              Completion

MoLAMD/NSDP staff trained      Number           0.00                     100.00                                      150.00
under NATEJA
                                                31-Mar-2014              31-Dec-2018                                 31-Oct-2018


Comments (achievements against targets):


                                                                                                 Formally Revised    Actual Achieved at
Indicator Name                 Unit of Measure Baseline                  Original Target
                                                                                                 Target              Completion

MIS system established and     Text             Existing MIS system      Fully functional MIS                        A Labour Market
functional                                                               system.                                     Information System
                                                                                                                     (LMIS) was developed
                                                                                                                     with the Project
                                                                                                                     support along with
                                                                                                                     biometric staff
                                                                                                                     attendance system.

                                                31-Mar-2014              31-Dec-2018                                 31-Oct-2018


Comments (achievements against targets): An elaborate software-based MIS was developed for M&E in General Directorate of Planning for
project activity monitoring, labor market assessment, employment center services with addition of web-based application for Training
Providers.


Component: Component Three: Piloting Entrepreneurship among Unskilled and Illiterate Young Afghans



                                                                                                                              Page 31 of 60
   The World Bank
   Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                                                                                            Formally Revised       Actual Achieved at
Indicator Name                 Unit of Measure Baseline                  Original Target
                                                                                            Target                 Completion

Number of new micro and        Number           0.00                     70.00                                      1871.00
small scale business
established and are running                     31-Mar-2014              31-Dec-2018                                31-Oct-2018


Comments (achievements against targets): The Grant Scheme endline survey completed results show that an estimate of 75% of small micro
and small scale businesses established and running by the closing of the project.




                                                                                                                              Page 32 of 60
    The World Bank
    Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



B. KEY OUTPUTS BY COMPONENT

   Objective/Outcome 1 Increased potential for employment among targeted young Afghan women and men in rural and semi-urban areas.
                                                                        1. Share of trainees who find employment within 6 months of
                                                                        graduation, with respect to control groups (to be increased from a
                                                                        baseline of 0 to 15 percent).
                                                                        2. The amount of incentive(s) disbursed to training providers (to be
   Outcome Indicators
                                                                        increased from 0 to USD 4 million).
                                                                        3. Number of beneficiaries and share of female beneficiaries (to be
                                                                        increased from 0 to 44,500; female share to be increased from 20
                                                                        percent to 30 percent).
                                                                        1. Number of teachers from selected non-formal training providers
                                                                        who receive training from TTTI/DMTVET or any other accredited
                                                                        teacher training center.
   Intermediate Results Indicators
                                                                        2. number of selected non-formal training providers that adopt NOSS
                                                                        for improvement of their teaching quality.
                                                                        3. MOLSAMD/NSDP Staff trained under NATEJA.
                                                                        1. There was no progress against the indicator for employment and
                                                                        incentive to the training providers as component one (Improving the
                                                                        quality of non-formal Training and Labor Market Outcomes of
                                                                        Trainees) was not implemented. However, the results of the Business
                                                                        Grant Scheme (Sub-Component 3.2: Promoting Entrepreneurship
                                                                        Among Young Afghans) is justified to be captured as outcome for this
   Key Outputs by Component
                                                                        indicator. Therefore, employment increased 13 percentages (against a
   (linked to the achievement of the Objective/Outcome 1)
                                                                        target increase of 15 percent).
                                                                        2. The number of beneficiaries increased from zero at baseline to
                                                                        2,482 (as against a target of 44,500); while the share of female
                                                                        beneficiaries at closure was 30 percent meeting the end target. The
                                                                        beneficiaries were business grant recipients under C3.2 (Promoting
                                                                        Entrepreneurship among Afghan Youth).


                                                                                                                                          Page 33 of 60
 The World Bank
 Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



Objective/Outcome 2 Increased potential for higher earnings among targeted groups (to be increased from 0 to 15 percent)

                                                                      1. Increase in trainees’ earning, in respect to control groups (to be
Outcome Indicators
                                                                      increased from 0 to 15 percent)
                                                                      1. Number of new micro and small scale business established and are
Intermediate Results Indicators
                                                                      running.
                                                                      1. There was no TPs/trainees were approved for inclusion under
                                                                      Project Component One (Improving the Quality of Non-formal
                                                                      Training and Labor Market Outcomes of Trainees) and Sub
                                                                      Component 3.1 (Encouraging Rural Employment). However, the
                                                                      results of the Business Grant Scheme (Sub-Component 3.2: Promoting
                                                                      Entrepreneurship Among Young Afghans) is justified to be captured as
Key Outputs by Component
                                                                      outcome for this indicator. Therefore, beneficiaries’ earnings
(linked to the achievement of the Objective/Outcome 2)
                                                                      increased 17 percentages (against a target increase of 15 percent).
                                                                      2. 1871 new micro and small-scale business were established and
                                                                      were verified running by firm responsible to collect assessment data
                                                                      for Sub-Component 3.2 (Promoting Entrepreneurship Among Young
                                                                      Afghans).




                                                                                                                                          Page 34 of 60
    The World Bank
    Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                      ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION


 A. TASK TEAM MEMBERS

Name                                                      Role
Preparation
Supervision/ICR
Nathalie Lahire, Palwasha Mirbacha                        Task Team Leader(s)
Rahimullah Wardak, Muhammad Abbass Rahimi                 Procurement Specialist(s)
Syed Waseem Abbas Kazmi                                   Financial Management Specialist
Najla Sabri                                               Social Specialist
Mabruk Kabir                                              Team Member
Sayed Mujtaba Shobair                                     Environmental Specialist
Ahmad Khalid Afridi                                       Team Member
Tariq Ashraf                                              Social Specialist




 B. STAFF TIME AND COST


                                                              Staff Time and Cost
 Stage of Project Cycle
                               No. of staff weeks                        US$ (including travel and consultant costs)
 Preparation
 FY14                          23.865                                                                   116,866.84
 FY15                          0                                                                             292.18
 FY17                          1.525                                                                       7,928.17

 Total                         25.39                                                                    125,087.19
 Supervision/ICR
 FY14                          2.025                                                                     17,652.28
 FY15                          28.300                                                                    96,805.89
 FY16                          36.000                                                                   114,496.87



                                                                                                        Page 35 of 60
   The World Bank
   Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



FY17                          17.662                                                      69,431.44
FY18                          16.772                                                      84,839.69
FY19                          9.612                                                       38,344.43
Total                         110.37                                                     421,570.60




                                                                                         Page 36 of 60
    The World Bank
    Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                                   ANNEX 3. PROJECT COST BY COMPONENT

                                         Amount at Approval         Actual at Project     Percentage of Approval
Components
                                                    (US$M)           Closing (US$M)                      (US$M)
Improving the Quality of
Non-formal Training and
                                                         7.70                   1.05                      13.6%
Labor Market Outcomes of
Trainees
Project Management,
Capacity Building of                                     3.60                   2.04                      56.6%
MOLSAMD and M&E
Piloting Entrepreneurship
among Unskilled and                                      3.70                   2.43                      65.6%
Illiterate Young Afghans
Total                                                   15.00                   5.52               36.80 percent




                                                                                                     Page 37 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                                               ANNEX 4. EFFICIENCY ANALYSIS



 1.      This annex presents a cost-benefit analysis of the NATEJA using the net present value (NPV) and
 the internal rate of return (IRR); and analyses certain dimensions of operational efficiencies. The NPV/IRR
 analysis focuses on the costs and returns of Sub-Component 3.2 (Promoting Entrepreneurship among
 Afghan Youth), the only component that achieved a clear outcome and has data on returns.

 2.          The following assumptions and data sources were used to estimate the net present value:

 3.      The benefit is the incremental income earned by beneficiaries as a result of the Grant. This was
 determined by first calculating the difference in monthly net income between the treatment and control
 group individuals at baseline (AFN 630 greater for treatment) and at endline (AFN 1,500 greater), and then
 subtracting the former from the latter (AFN 870, or USD 11.60). This showed that in relation to the control
 group, individuals in the treatment group had increased their monthly net income by USD 11.60 on
 average, at the time of the impact evaluation (IE). 35

 4.       The incremental income generated by the Grant at the time of the IE is assumed to be constant
 (in real terms) for a period of 15 years, in the base case. The costs are the Project expenditures for all of
 Component 3 (there were virtually no expenditures on either Sub-Component 3.1 or 3.3) and for a
 proportion (20 percent) of Component 2. These costs constituted 48 percent of all Project disbursements.
 The discount rate is 5 percent and the rate of inflation is 2.9 percent. 36

 5.       Using the above, the NPV is USD 1.7 M and the IRR is 12.6 percent. Under the alternative cases
 examined in the sensitivity analysis (see table below), the IRR ranges from 8.4 percent to 16.4 percent. In
 the low case, the increased income earned by the beneficiaries is assumed to last only 12 years, declining
 in value each year by 2 percent; and the discount rate is 6 percent. In the high case, the increased income
 is assumed to last 20 years, increasing in value each year by 2 percent; and the discount rate is 4 percent.
 That the harsh assumptions of the low case still yield an IRR above discount indicates that the positive
 result is robust.

                                            Sensitivity Analysis
                                    Low case: income gains Base case: income                       High case: income
                                    erode 2 percent            gains constant in real              gains increase 2
                                    annually for 12 years;     terms for 15 years;                 percent annually for 20
                                    discount rate 6 percent discount rate 5 percent                years; discount rate 4
                                                                                                   percent
   NPV (USD M)                      0.4                            1.7                             4.6
   IRR (percent)                    8.4                            12.6                            16.4

 6.     The real IRR of the Business Grant Scheme is likely much higher. The calculations above are based
 on what was in effect a pilot scheme with relatively few beneficiaries, and therefore had learning costs

 35   Altai Consulting, 2018. Endline Assessment of Promoting Entrepreneurship Among Afghan Youth, Final Report (v. 10, p. 16).
 36   The inflation rate is the average for the previous four years.

                                                                                                                           Page 38 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



 and high overheads in relation to the number of grant recipients. The number of beneficiaries could have
 been scaled up with a disproportionately small effect on operational costs. (This can be seen by the fact
 that Phase 2 took three months to complete, while Phase 1 – the learning phase – took over a year, with
 far fewer beneficiaries). If one were to implement the scheme such that overheads were limited to a 12
 percent surcharge (a common target for project overheads) on the value of the grant, then the IRR would
 be 26.7 percent in the base case.

 7.       Project efficiency can also be analyzed from the perspective of operational efficiency, particularly
 timing against schedule, efficiency of design, and costs against budget. Component 1 implementation
 was inefficient in that the recruitment of the Inspection Agency took 15 months. Then, despite mapping
 250 TPs, the Agency and Ministry could not agree to sign any letters of understanding between MoLSAMD
 and a TP, and no training took place. As a result, the component expenditures yielded no returns in terms
 of the outcomes defined by the Project, viz. certification, employment or starting a business consequent
 to training. There were nonetheless some training-related benefits pertaining to the development of
 NOSS, curricula and training materials, as well as the training of TPs’ trainers (see Annex 10); though there
 are no data available to calculate any returns these activities may have ultimately generated for students
 affected beyond the Project.

 8.      The implementation of Component 2 activities was relatively efficient in that most of the capacity
 building activities were completed within the overall planned schedule. However, the design of the
 capacity building turned out not to be fully adequate in that it was originally predicated upon NSDP having
 the main implementing responsibility; whereas at negotiations it was agreed that NSDP would be
 integrated into MoLSAMD, thereby effectively making various units/departments of MoLSAMD
 responsible for implementation. This would have required a larger program of capacity building. (See
 Section III and Annex 11).

 9.       Sub-Component 3.2 implementation was overall efficient. Activities began late particularly due
 to significant delays in recruiting the Grant Management Firm (GMF), whose contract was signed only in
 December 2015. The effect of this delay was alleviated by the Project team at the Ministry (led by the IC
 hired for Sub-Component 3.2), which worked for nearly a year prior to GMF hiring to prepare and initiate
 a first phase of activities. (See Annex 9). The design and implementation of the first phase including the
 placement and strengthening of the provincial ESCs, the selection of a first round of beneficiaries (775),
 and the disbursement of grants took approximately 15 months; whereas the second round (which
 included the majority of beneficiaries, or 1,707) took roughly three months. The initial lengthy period of
 design and implementation was due in part to the fact that such a grant scheme had not been previously
 implemented by NSDP, as well as to GMF recruitment delay.

 10.      In general, actual costs were in line with budget forecasts, though there instances of unwarranted
 excess (e.g. the luxury procurement of desks, see Section IV.B and Annex 12. This annex does not
 undertake a cost-benefit analysis of Component 1 or (for the most part) Component 2. With respect to
 Component 1, the costs had no obvious returns in that the activities did not result in graduates being
 certified, employed or starting a business. One might argue that the NOSS, curricula and training provided
 to certain TPs resulted in a return, but there are no data one might use to test this hypothesis. With
 respect to Component 2, the capacity building of MoLSAMD arguably resulted in returns, but again there
 are no data one can use to test this hypothesis; as per Project design, the returns were to be found in


                                                                                                         Page 39 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



 Component 1 and Sub-Component 3.1, but neither of these achieved any results. Nonetheless, a
 proportion (20%) of Component 2 expenditures were attributed as costs for the purpose of doing the Cost
 Based Analysis for Sub-Component 3.2, as arguably a proportion of the capacity building did benefit Sub-
 Component 3.2 activities. If one were to use all Project expenditures as costs to be set against the returns
 of Sub-Component 3.2 (and without attributing any further returns to the costs beyond those of Sub-
 Component 3.2), then the Project IRR is a neglible 1.7% with a negative NPV.




                                                                                                        Page 40 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                              ANNEX 6. SUPPORTING DOCUMENTS (IF ANY)

 Altai Consulting, 2018 (September). Endline Assessment of Promoting Entrepreneurship Among Afghan
 Youth. Final Report. Kabul: Altai Consulting.

 Islamic Republic of Afghanistan, MoLSAMD, 2018 (July). Implementation Completion and Results Report
 for NATEJA. Kabul: MoLSAMD

 Pearl Horizon, 2017 (October). NATEJA Final Report of Grant Management Firm (C3.2). Kabul: Pearl
 Horizon.

 World Bank (various). Implementation Status and Results Reports. Archived: 28 June, 2014; 24
 December, 2014; 23 June, 2015; 13 January, 2016; 11 August, 2016; 7 February, 2017; 30 August, 2017;
 9 April, 2018; 24 October, 2018.




                                                                                                    Page 41 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                      ANNEX 7. RESTRUCTURING AGREED AT THE MID-TERM REVIEW

 1.     Given the constraints experienced during the first 2.5 years of implementation, the GIRoA and
 the Bank agreed at the MTR to restructure the Project. The ICR Team found the recommendations to be
 sound. The main agreements included:

     •   Component One: (i) deploying additional NGOs directly by MoLSAMD to deliver skills training, (ii)
         focusing on rural areas and inclusion of unemployed, underemployed, illiterate youth (15-35 year
         olds), including marginalized populations (IDPs, refugees and returnees), (iii) revising the incentive
         structure for TPs to cover the cost of training up to US$ 300 per trainee inclusive of administration
         cost, in addition to the provision of a job placement incentive of US$ 100, (iv) the verification of
         employment outcomes by a third party after six months, (v) opening the entrepreneurship grant
         to skills training beneficiaries under a revised modality. It was further agreed that the target
         number of trainees would be reduced.
     •   Component Three: Sub-Component 3.1 to be dropped because of limited implementation, and
         the mandate to be transferred to the AREDP program. (By the time of the MTR, only the following
         activities had been completed: signing of MoU with AREDP, preparation of a database of Savings
         Groups, and selection of a treatment group from existing baseline information.) There had been
         significant delays in initiating procurement of the TP. Subsequently, the Project was not able to
         identify a suitable and qualified service provider despite multiple re-advertisements. Sub-
         Component 3.3 to be dropped because GIZ had executed a feasibility study and was piloting an
         apprenticeship program. Under Sub-Component 3.2, the Business Grant Scheme was to be
         extended to Component 1 beneficiaries.

 2.       The agreement to restructure was captured in the Mid-Term Review Aide Memoire (November
 22-26, 2016), the Management Letter from the Midterm Review dated January 17, 2017, and finally the
 letter from the Minister of Finance dated August 16, 2017 formally requesting the restructuring based
 on the recommendations of the MTR ‘fully endorsed by both MoLSAMD and the MoF’. However, despite
 MoLSAMD’s initial agreement to restructure at the MTR, there ensued a lengthy period of discussions –
 not resolved until December 2017 and formally confirmed until July 2018 – within and between WB,
 MoF and MoLSAMD as to whether the Project should be restructured or closed. During this extended
 period of uncertainty (January 2017- December 2017) as to the Project’s fate, the decision to restructure
 seems to have been taken and then overturned twice. In between the January Management Letter and
 the August MoF letter there was a Bank mission (February 12-14, 2017), during which the Advisor to the
 Deputy Minister at MoLSAMD overturned the MTR recommendations and ‘conveyed to the mission the
 desire to terminate the NATEJA project’ (as per the mission’s Back-To-Office-Report). Discussions
 continued within the Bank between the CMU which advocated early closure and the GP which
 disagreed. Despite the communication during the February mission, the Bank received a letter from the
 Minister of Finance dated August 16, 2017 formally requesting the restructuring based on the
 recommendations of the MTR ‘fully endorsed by both MoLSAMD and the MoF’.

 3.     The discussions continued between the GIRoA and the CMU, which culminated in an agreement
 in December 2017 to close the Project. It took until July 2018 for MoLSAMD to request MoF to close the



                                                                                                          Page 42 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



 Project, 37 which was quickly followed up by a formal MoF request to the Bank to close the Project early
 and cancel unspent funds. 38 A restructuring paper to this effect was prepared to enable an amendment
 to the Grant Agreement, the Project closed on October 31st, 2018.




 37In a letter to MoF, dated July 24, 2018 (No. 333/252).
 38The request was made in a letter from the Minister of Finance dated July 31, 2018. The letter further refers to a ‘discussion
 between the World Bank and MoLSAMD on December 19, 2017’ during which an ‘agreement was reached to complete the
 ongoing activities and close the project, with canceling of the remaining grant.’

                                                                                                                             Page 43 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                                       ANNEX 8. RELEVANCE OF PDO



 1.      Beyond the linkages with the CPF, the PDOs have been well integrated into national policies and
 strategies. At the time of appraisal, the PDOs supported the Afghan National Development Strategy
 (ANDS, 2008-13) and its associated National Priority Program 1 (Sustainable Decent Work Through Skills
 Development and Employment Policies for Job-Rich Growth, 2012) as well as the MOLSAMD Strategic Plan
 (2013-15). The PDOs:

     •   Directly supported ANDS Pillar 3 (Economic and Social Development), particularly strategies
         pertaining to vocational training, skills development and job creation, as well as the cross-cutting
         strategy to empower women;
     •   Indirectly supported ANDS Pillar 2 (Governance, Rule of Law and Human Rights), particularly
         strategies aimed at strengthening public sector institutions and governance and administrative
         capabilities;
     •   Were aligned with the objectives of Sub-Component A.2 of the NPP1 (to develop a sustainable
         LMIS), which foresaw building MOLSAMD capacities to design, manage and analyze national
         surveys and their results; Sub-Component B.2 (to increase access to improved occupational
         literacy among employed, unemployed and underemployed populations), which aimed to
         increase access to occupational literacy training based on an improved curriculum; and Sub-
         Component C.1 (to expand and strengthen MOLSAMD’s General Skills Development Directorate
         and the National Skills Development Program), which aimed inter alia to increase access to
         improved informal skills training, in particular for targeted vulnerable groups.

 2.      At the time of writing (November 2018), the PDOs continue to support the operative national
 strategy, the 2017-2020 Afghanistan National Peace and Development Framework (ANPDF). In particular,
 the PDOs support the following ANPDF strategies:

     •   Creating jobs (in the agricultural sector), increasing yields and opening markets for farmers
         (indirectly).
     •   Increasing labor productivity and investing in human capital. Investments in this sector are to
         better align education with private sector requirements and increase the current skills base. This
         will be achieved by expanding vocational and technical education, and investing in women’s
         education and market employment.
     •   Reforming the public sector, including building a responsible and effective public administration
         system (again, indirectly). This will be achieved inter alia by establishing, maintaining and
         improving professionalism, expanding the use of performance-based management, and
         improving accountability and transparency.
     •   Empowering women, using a gender strategy that includes ensuring full access to education,
         and advancing women in business.)




                                                                                                        Page 44 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                           ANNEX 9. THE BUSINESS GRANT SCHEME (Sub-Component 3.2)



 1.      This Scheme provided a USD 500 grant to 2,482 persons to start up a new small business, or to
 expand an existing small business. 30 percent of beneficiaries were women. The Scheme achieved two
 notable outcomes:

 2.      Beneficiaries earned a monthly net profit of USD 11.60 greater than a control group of non-
 beneficiaries. 39

 3.     The unemployment/underemployment rate of beneficiaries was 12 percent vs 25 percent for
 non-beneficiaries. 40

 4.        To achieve this, Sub-Component 3.2 successfully completed several activities.

      •    An awareness-raising campaign attracted 17,000 applications with business plans, in three
           provinces (Kabul, Balkh and Nangrahar).

      •    After screening, 2,482 individuals (of which 30 percent female) were awarded an incentive grant
           of USD 500 to start or expand an existing small business, just short of the target of 2,582 written
           in the project description. 41

      •    Beneficiaries also received business training, which for the women included a module on gender
           aspects of business; as well as post-training counselling and mentoring.

      •    A well-designed impact evaluation was conducted that included two rounds of data collection
           from beneficiaries and a control group; it found that beneficiaries had higher levels of
           employment and income as compared to the control group. The levels exceeded the targets of
           the related PDO indicators.42

 5.      The outcomes suggest that the scheme is worth scaling up in-country and replicating elsewhere.
 The ICR Team met and corresponded with the major actors involved with the design, implementation and

 39 That is, the monthly net profit in the month prior to the post-intervention interview was USD 11.6 greater for the treatment
 group, after deducting the differential that existed between them at baseline. In brief, this amount represents the income
 benefit to the beneficiaries of the grant, and is the equivalent of a 138% increase in income. It is equivalent to an annual rate of
 return of 27.8% on the investment, presuming the month prior to the post-intervention interview was representative of the
 months between the grant being disbursed and the post-intervention interview. The interviews took place roughly 18 months
 after grant disbursement. See Altai Consulting, 2018 (v.10) p. 15. The parameters of the evaluation were designed during
 Project preparation (PAD, pp. 36 ff.), enabling the careful construction of treatment/control samples during implementation, as
 well as the gathering of baseline and endline data.
 40 Altai Consulting 2018 (v.10), p. 14. The following activities were taken as proxies for unemployment/ underemployment:

 casual labor (4% for treatment; 15% for control) and household chores including fetching water (8%/10%). The employment
 rate for the treatment group was 88% vs. 75% for the control, which represents a difference of either 13 percentage points or
 17.3 percent.
 41 There was no intermediate indicator associated with this component. See Section IV.A.
 42 See Annex 9 (The Business Grant Scheme (C3.2)).



                                                                                                                               Page 45 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



 evaluation in order to capture salient aspects of the design and the implementation arrangements, for
 the benefit of those who might be interested in scaling up and replication

 6.       The sequence of main activities was as follows:

      •   Awareness raising
      •   Applications (including business plan) and uploading data
      •   Screening step one: scoring applications and filtering out non-qualifiers
      •   Scoring and ranking of applications to identify sufficient number of interviewees
      •   Screening step two: interviews and selection of those eligible to be in lottery
      •   Lottery to allocate eligible applicants into two groups: beneficiaries (treatment) and non-
          beneficiaries (control). A certain number of persons were also identified to be kept in reserve, in
          case of drop-out prior to grant receipt.
      •   Further confirmation of selected beneficiaries, and replacement as needed.
      •   Baseline data collection
      •   Signing of Letter of Understanding between MoLSAMD and beneficiary.
      •   Transfer of funds to recipient’s bank account. (In many cases, this required the beneficiary to
          open a bank account, something the Project did not help with).
      •   Initiation/expansion of business activities
      •   Training
      •   Counselling and mentoring
      •   Endline data collection

 7.       The Ministry hired a particularly able long-term IC who was responsible for the Scheme
 implementation, including the vital task of managing the TA firm hired to execute activities. The success
 of the Scheme all testify to his technical capacities, team management, and sound judgement; endurance,
 persistence and flexibility in the face of challenging and changing circumstances, taking full responsibility
 for the Scheme from inception through to its conclusion; ability and willingness to spend time in the field;
 and having the right ‘human touch’ and skills in managing large numbers of people with competing
 agendas and interests.

 8.       The IC had ample time prior to the start of the work of the Grant Management Firm (GMF), which
 he used to work out design and implementation details, working from parameters of the PAD and PIM;
 was able to facilitate the hiring of a small team of people – and subsequently manage frequent turnover
 - in each targeted province to help with awareness-raising and the screening of applications; and began
 to recruit a small, first (or pilot) batch of beneficiaries. This enabled the GMF to hit the ground running,
 as it were, when it came on board; and moreover, enabled the Ministry and the GMF to acquire the
 necessary capacities and proper design in preparation for the second, much larger batch. The learning
 curve and benefits of this ‘pilot’ can be seen in the timelines; while the processing of the first batch took
 approximately one year from start to finish, the second batch was completed in three months.

 9.     The Project was able to deploy sufficient numbers of field-level people to assist in the
 implementation of activities, as and when needed. This took two forms. First, the team mentioned in the
 previous point were in fact staff of the provincial Employment Service Centres, which nominally were
 responsible for certain functions pertaining inter alia to the registration of job openings and job-seekers

                                                                                                         Page 46 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



 and matching job-seekers with employers; they also performed valuable service for the Scheme in terms
 of raising awareness and scoring applications. Second, the GMF itself hired staff. In this regard, the
 capacity of the GMF to adjust staff levels with relatively little friction was instrumental to the Scheme
 success. The Scheme activities required varying levels of staff according to the intensity of activities (e.g.
 vetting of applications and interviewing of applicants were labour intensive, whereas post-disbursement
 activities were not). The GMF was able to manage such staff fluctuations relatively smoothly to ensure
 timely and efficient implementation, whereas the Ministry’s comparatively cumbersome hiring
 procedures would have entailed delays and/or under/over-staffing.

 10.      Keeping control of the message during awareness raising proved difficult, leading to some
 misunderstandings that in turn required time to dispel. Better quality staff and a tighter discipline in
 crafting, delivering and monitoring the communication of key messages to potential applicants might have
 alleviated these difficulties.

 11.      The impact evaluation (IE) was designed at the outset of the Project, prior to any implementation
 activities, so that it was clear what minimum numbers were required in terms of beneficiaries and
 controls. Further, aside from the intrinsic value of having a well-designed IE, the need for substantially
 similar control and sample populations imposed a certain rigor on the scoring and sorting of applicants.

 12.     All applications were scored, and all applicants who got through this stage were interviewed in
 order to further screen their application and determine their suitability as a candidate. The screening of
 applications required some work in order to get the process right, as it was difficult to achieve
 standardization across individuals responsible for scoring. One of the main purposes of the interview was
 to identify the degree to which applicants were serious in their intent to start/expand a business, and to
 gauge their reliability as an investor. Given the number of applicants, the interview process was time-
 consuming. However, the implementing team is convinced that this was instrumental in weeding out
 individuals who would not have used the grant for their stated purposes, as well as individuals who were
 not particularly well suited to taking on the responsibilities of running a business. The screening process
 was a substantial task, yet it seems the MoLMSAMD/GMF treated it seriously and were subsequently
 rewarded with the outcomes mentioned above. There were over 18,000 applications, and approximately
 5,000 individuals were identified through the interview process as being eligible for the lottery which
 selected grant recipients. (Those not winning the lottery became the control).

 13.       The LoU was a means of imparting to the beneficiaries the seriousness of their commitment. The
 official character of the letter, the explaining to the signatory of the clauses, including penalties, and so
 on – as the IC explained, these were all commitment reinforcers.

 14.    All participants received training on business skills, with women receiving a supplementary
 module on gender issues in business.

     •   A great deal of thought went into the training and support received by the participants,
         particularly to account for their time constraints and illiteracy. Therefore, it was decided that the
         training would be short, lasting only one afternoon; to be supplemented by phone counselling
         and, in some cases, mentoring visits. Further, the training materials were designed to convey as
         much information pictorially as possible; included practical forms, such as checklists; and were


                                                                                                          Page 47 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



         given to the participants as a handbook. Theory was minimized, focusing on more practical
         aspects of business. Finally, pedagogy was in part story-based, as it was found that participants
         responded well to learning from story.

     •   The training was followed up by phone counselling, during which counsellors would contact the
         grant recipients to enquire about progress and information needs. The IC noted that the moral
         support provided by these phone calls was as important as any advice given, and that the
         recipients appreciated knowing that someone was following up who was sympathetic to the
         struggles they were going through. On a pilot basis, a sub-group of beneficiaries (based in Kabul)
         received visits from mentors operating in a similar line of business.

     •   It was planned to deliver the training at the moment the grant was received, as per usual practice.
         However, due to implementation slippages, the training was delivered a few months after the
         grant was disbursed. This may have been a blessing in disguise, in that the training was now
         deeply relevant to the participants; they were able to relate the course contents to their business
         experiences since receiving the grant. In this way, it is possible they were able to extract more
         utility from the training than if they had received it at the same time as receiving the grant.

 15.     The ratio of mentees to mentor was roughly 5. The mentors were identified through a local small
 business association. Each mentor received an honorarium, perhaps just enough to cover their expenses
 associated with visiting the mentee on-site.

 16.    Further detailed lessons and recommendations can be found in the GMF’s final report. (Pearl
 Horizon, Final Report, October 1, 2017).




                                                                                                       Page 48 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                             ANNEX 10. COMPONENT-WISE ACTIVITIES AND RESULTS

 1.      This Annex presents some of the activities carried out under the Project. Many of them yielded
 valuable outputs, even though they did not result in PDO-level outcomes.

 2.      Component 1: Improving the Quality of Non-formal Training and Labor Market Outcomes of
 Trainees, completed valuable activities, though these fell short of an outcome.

      •   An Inspection Agent was contracted to map, vet and recommend non-formal TPs. It mapped
          250 TPs; inspected 175; and invited 134 to apply to participate in the incentives scheme. 88 TPs
          submitted a formal application, of which 78 were recommended for approval by the Inspection
          Agency. (The number selected would later drop to 24, for reasons examined in Section III.B.i).
      •   Two general guidelines were drafted and adopted on the development of NOSS and curricula;
          162 National Occupational Skills Standards (NOSS)-based curricula were either developed or
          revised; 45 competency-based curricula were developed; and written instructional materials
          (WIM) were produced for 7 trades.

 3.    Component Two aimed to strengthen MoLSAMD capacities that were required to facilitate the
 implementation of Component 1 and Component 3 activities, which were required to achieved the PDO
 outcomes. In this regard:

      •   150 MOLSAMD/NSDP staff were trained in financial management, procurement, curriculum
          development, labor market analysis, and entrepreneurship. This exceeded the target value for
          the intermediate indicator (100).
      •   The NSDP (including NATEJA operations) was integrated into the General Directorate Skills
          Development (DGSD), and capacities were built across various units of MoLSAMD that assumed
          the integrated NSDP functions. However, this integration was not part of the Project as appraised
          in the PAD and was included at negotiations at the behest of MoF. 43 The last-minute decision to
          integrate the NSDP constituted an extra activity of some importance and a significant modification
          to implementation arrangements, which affected implementation performance. These issues are
          discussed in detail in Annex 11.
      •   Three provincial Employment Service Centers (ESCs) were strengthened, each with 5 posts. They
          provided valuable support to Sub-Component 3.1. However, it is not clear that the ESCs ever
          functioned effectively in terms of their core mission, viz. registering job openings and job-seekers,
          and matching the two. Further, the posts were paid for under the Project, and were never
          formally established and budgeted under the local government authority (taskheel). Hence they
          are no longer operational.
      •   A Labor Market Information System (LMIS) was developed with Project support. It is fed internally
          by the MoLSAMD using data pertaining to ESCs operation and the registration of work permits for
          Afghan nationals and for foreigners; only the work permits data are currently being entered, and

 43A legal covenant pertaining to the development and implementation of an integration action plan was included in the Grant
 Agreement. In the event, the plan was not quite fully executed, and the covenant was considered to have been partially
 complied with at closure.



                                                                                                                         Page 49 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



          only one ESC (in Kabul) is currently operational. 44 It is to be fed externally from five other
          ministries, using for example data on graduates and the labor economy; though this aspect is not
          yet operational. Further, the Project supported the Human Resource Demand and Supply Survey
          (2016), which provided valuable data for decision-making. 45
      •   70 non-formal TPs adopted the National Occupational Skills Standards for their courses, meeting
          the intermediate indicator target. 46
      •   673 Vocational Trainers working at non-formal TPs were trained on NOSS and curricula
          content. 47
      •   The capacity building of MoLSAMD’s M&E function is discussed in Section IV.A, particularly as it
          relates to the Project.

 4.      Component 3.1 completed some preparatory activities, including a MoU with AREDP to train
 members of their SGs; and the updating of the SGs membership database as well as the identification of
 a control group for impact evaluation purposes. However, a suitable TP was never selected despite
 multiple re-advertisements. 48 It was recommended at the mid-term review (November 2016) that the
 component would be dropped, any activities to be continued would be transferred to the AREDP, and the
 unused funds would be re-allocated to C1.49

 5.      Component 3.3 was not implemented. During the second year of the Project, it was agreed that
 the GIZ study on apprenticeships obviated the need for the component’s planned feasibility study. There
 were some discussions to use the GIZ findings to design and implement a pilot program, but GIZ’s
 subsequent work to support the development of an apprenticeships program led to a recommendation
 at the mid-term review to cease activities under this component and use the unallocated funds
 elsewhere. 50




 44 Further, there are two ESCs currently being supported by the IADC-financed NATEJA project; they will also be unsustainable
 once the project completes, unless the government changes its policy about taksheel financing of the ESCs. None of the ESCs
 are currently providing data to the LMIS. The LMIS has the capacity through web-based applications to populate databases for
 job-seekers, employers registration, job-seekers training and capacity building, and job-placement. The ICR Team was not able
 to receive security clearance to visit the site where the LMIS is operational.
 45 The supply side gathered data from households and TVET institutions. The demand side gathered data from formal and

 informal businesses including SMEs. Data were collected manually and later entered into the databases. The LMIA plans to
 periodically update the data using its own surveys, with direct data entry using a web-based application; as well as findings
 from Central Statistical Office (CSO) surveys.
 46 The value for the intermediate indicator ‘Number of selected non-formal TPs that adopt NOSS for improvement of their

 teaching quality’ was registered as 70 in the ISRs from August 2016 to August 2018. However, the final ISR of October 2018
 registered a value of 24, presumably to reflect the final number of TPs shortlisted by the Inspection Agent.
 47 The value for the intermediate indicator ‘Number of teachers from selected non-formal TPs who receive training from

 TTTI/DMTVET or any other accredited teacher training center’ remained at zero for all ISRs, because no TPs were formally
 selected. The trainers used by the Project were NOSS experts on contract with MoLSAMD. The original Project intent was for
 the training to be conducted by the TTTI that was to be established under the ASDP II, but this component dropped from the
 ASDP II at restructuring. It is not possible to determine if any outcome resulted from the trainings, which were not monitored
 for effectiveness.
 48 See Annex 12.
 49 This was to be part of a restructuring that in fact never took place.
 50 Again, through a restructuring that never took place.



                                                                                                                          Page 50 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                                    ANNEX 11. IMPLEMENTATION AGRANGEMENTS

 1.      The Project implementation arrangements as laid out in the PAD differed in some important
 respects from those stipulated in the Grant Agreement. This Annex describes these differences and
 analyzes their implications.

 2.       During Project preparation and appraisal,
                                                                         The National Skills Development Program
 the implementation arrangements identified the
 NSDP as being responsible for implementation,
                                                                     At Project preparation, the NSDP was one of the
 including fiduciary and monitoring and evaluation                   National Priority Programs still operating that had
 functions. It was also stated that the NSDP would                   been created by the former President of the
 ‘strive to build capacity among their counterparts                  GIRoA in 2004. It provided technical assistance to
 in the civil service.’                                              the TVET system as well as vocational education
                                                                     trainings across the country with the support of
 3.       However, during negotiations, the MoF                      MOF and DPs including the WB. It functioned as a
 objected to these arrangements and advocated for                    parallel structure and separate program at
 greater implementation responsibility to be given                   MoLSAMD, with project staff hired for fixed
                                                                     periods to carry out program activities; its
 to MoLSAMD. The MoF’s underlying concerns
                                                                     functions were technical and fiduciary, giving it
 derived from its own evaluation, which were
                                                                     substantial control over program design and
 summarized by a consultant hired by the Bank to                     execution. It had implemented previous WB-
 do an institutional analysis of MoLSAMD as                          supported operations (e.g. the ADSP), with
 follows: ‘The evaluation report questioned                          satisfactory ratings. It had four regional sub-
 transparency of the bidding procedures and                          offices, and in the years leading up to 2015
 contract award process.’ The analysis also noted                    averaged 70-100 staff.
 that ‘the NSDP methodology for impact monitoring
 has been highly doubted by MoF … Concern was
 expressed about the NSDP monitoring and outcomes, particularly in relation to tracer studies and
 findings that around 86% of vocational training graduates have found employment shortly after
 completion of training… the lack of third-party evaluation appears to have undermined credibility of
 these tracer studies.’ 51 On the basis of these concerns, MoF had ‘refused to disburse required amounts
 for covering of NSDP’s operating costs. 52

 4.        As a result, it was formally agreed (through the Grant Agreement) that an action plan would be
 designed and implemented ‘designed at integrating NSDP with the Directorate General for Vocational
 and Educational Training within MoLSAMD.’ 53 (This Directorate later became the General Directorate
 for Skills Development, DGSD). Further, the Grant Agreement contained a legal covenant to the effect
 that the action plan would be prepared and implemented.

 5.        A plan was subsequently developed and agreed, with some delay. 54 The steps are provided in

 51 These same studies were used in the PAD to justify project design.
 52 All quotes in the paragraph are from ‘Institutional Analysis of Ministry of Labor, Social Affairs, Martyrs and Disabled.’ World
 Bank, January 2013. Pp. 40-41.
 53 Afghan Reconstruction Trust Fund Grant Agreement of April 11, 2014, p. 5 (I.A.1c).
 54 The Grant Agreement stipulated that the action plan would be prepared and furnished to the Bank within six months of Grant




                                                                                                                             Page 51 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



 the table below and include remarks about the extent to which the step was implemented. 55

   Step   NSDP unit.        To be                By                Actual and remarks
                            integrated into
   1      TPs               DGSD         TPs     31   March,       NSDP unit disbanded, with no posts transferred. All
          registration      registration unit    2015              NSDP documents and procedures for registration
                                                                   were handed over. MoLSAMD registration fees
                                                                   remain at 30,000 AFN/year, with only international
                                                                   NGOs registering.
   2      Curriculum        DGSD        –        31 December,      NSDP still retain 7 staff, and works closely with DGSD
          development       curriculum           2015              department.
          and               development
          standards         department
   3      M&E               DGSD - M&E           30   August,      NSDP unit disbanded. DGSD has no M&E unit, so 5
                            General              2015              NSDP posts transferred to General Planning
                            Planning                               Directorate (of which one was vacant at time of
                            Directorate          31 April, 2016    writing). No training provided to DGSD, though
                            (MoLSAMD)                              persons occupying the transferred posts did transfer
                                                                   their skills to the department through their regular
                                                                   work.
   4      Procurement       DGSD – General       30 April, 2015    NSDP unit disbanded after November 2016, with two
                            Administration                         posts transferred. There is no procurement function
                                                                   within DGSD, so both NSDP posts were transferred
                            Directorate of       31 May, 2016      (with loss of grade and salary) to Directorate of
                            Procurement                            Procurement. Both persons occupying NSDP posts
                            (MoLSAMD)                              received procurement training through the World
                                                                   Bank, and have worked to transfer skills to other
                                                                   colleagues within Directorate.         Procedures of
                                                                   Directorate are somewhat similar to what they were
                                                                   under NSDP, though there are concerns that they take
                                                                   longer to execute and that persons nominated to bids
                                                                   evaluation committee may not always have the
                                                                   requisite technical capacities. Further, government
                                                                   procedures do not conform to those approved for
                                                                   World Bank-supported operations, and transferred
                                                                   staff needed to explain these differences to the
                                                                   Directorate. Note that of the nine major TA packages,
                                                                   only one remained to be procured (the impact
                                                                   evaluation) after the integration process.
   5      Finance           DGSD – General       30 December,      NDSP unit disbanded. Function was integrated, with
                            Administration       2015              similar issues identified elsewhere (more time-
                                                                   consuming, concerns about technical proficiency).
                            Finance     and
                            Accounts             31 May, 2016
                            Directorate

 Agreement signing (April 11, 2014); to be thereafter implemented within a period of around 12 months (or any other later date
 agreed in writing with the Bank) after the date of the Grant Agreement. In the event, the parties agreed to develop the plan by
 19 February 2015, with a view to it being fully implemented by May 2016.
 55 The remarks are based in part on interviews with NSDP staff and MoLSAMD staff who were previously with NSDP.



                                                                                                                           Page 52 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                            (MoLSAMD)
   6      HR                DGSD – General       31 August,         NSDP unit was disbanded.            Integrated into
                            Administration       2015               Directorate, as DGSD has no HR function. Roughly
                                                                    similar procedures as before, though concerns were
                            HR Directorate                          expressed that they are more time consuming and
                            (MoLSAMD)            31 May, 2016       that recruitment panels not as technically proficient.
                                                                    Procedures different in an important respect:
                                                                    recruitment is done against a step 1 level, to be
                                                                    possibly renegotiated subsequently; rather than
                                                                    recruiting to a higher-level step given candidate’s
                                                                    experience.
   7      Others            Skills Support       31 May, 2016       The new Directorate was not established. Therefore
                            Directorate,                            certain functions remain within NSDP, including
                            DGSD (to be                             project development and cooperation/coordination
                            newly                                   with DPs. NSDP’s IT unit was disbanded and functions
                            established)                            integrated with ICT Directorate of MoLSAMD.

 6.     Given the uneven implementation of the action plan, the legal covenant was considered to have
 been partially complied with at closing. 56

 7.     The new implementation arrangements had two potent characteristics. First, it created work that
 had not been identified in the PAD and that moreover created uncertainty among the NSDP staff
 responsible for the Project implementation.

 8.       Second, it was a substantial modification of the implementation arrangements that had been
 appraised. Stated differently, the new implementation arrangements were not appraised. The new
 arrangements could have reasonably been expected by the Bank to constitute a significant challenge. An
 institutional analysis of MoLSAMD had been commissioned by the Bank in the previous year and
 completed in early 2013. 57 It found that inter alia ‘the management of the Ministry remains weak.
 Business processes are complex and involve multiple decision-makers. The senior managers tend to
 micromanage and make both small, operational and strategic decisions. The MoLSAMD’s structure is
 complex, large and inefficient. Several directorates across the Ministry and employment service centres
 have not produced any outputs for years, signaling a lack of commitment and accountability, especially at
 directorate’s and unit’s levels.’ 58 The same report concluded that ‘the rationale for the Bank’s continued
 support to NSDP is also strong’, and provided various recommendations to strengthen the NSDP, none of
 which entailed integration within MoLSAMD. 59 In this light, arguably the Bank should have insisted at
 negotiations to take further time to discuss and design the implementation arrangements with MoF and
 MoLSAMD, and to re-appraise them as needed including identifying suitable mitigation measures.

 9.      The new implementation arrangements likely contributed to some of the implementation delays
 and difficulties that the Project experienced. First, it took up substantial time from NSDP senior

 56 As per the final ISR dated October 24, 2018. In prior ISRs, the Project was considered to be in compliance with the covenant
 because the final date for plan implementation had not yet been reached.
 57 Islamic Republic of Afghanistan. Institutional Analysis of Ministry of Labor, Social Affairs, Martyrs and Disabled. January

 2013.
 58 Ibid, p. 5
 59 Ibid, pp. 45 ff.



                                                                                                                            Page 53 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



 management. Second, the new lines of authority meant that persons working in technical, fiduciary and
 M&E functions no longer answered directly to the NSDP Director, but rather to the Head of the relevant
 MoLSAMD directorate or unit. This in itself complicated tasks for senior NSDP management, as they no
 longer had the same authority to take decisions or otherwise act to ensure the timely completion work in
 these areas. The issue was further complicated by the fact that integration did not proceed fully as initially
 planned, whereby the departments/units with implementing functions would come under the authority
 of the DG of the DGSD; in this scenario, all final decision-making authority would have resided in one post.
 In the event it was decided that certain functions would be transferred to departments/units that
 answered to different DGs (e.g. HR, finance, procurement and M&E) who in turn did not all answer to one
 Deputy Minister (DMs) but rather to two DMs and (in the case of M&E) the head of the Office of the
 Minister. This meant that the NSDP Director had to liaise with several DGs, and in practice often needed
 to consult with different Deputy Ministers and the Office of the Minister to ensure coordination and
 decision-making. Third, the procedures followed by the MoLSAMD units/directorates were reported to
 be more cumbersome and time-consuming. In some cases, they did not conform to normal project
 management procedures; for example, NSDP senior management was not able to sign off on contract
 deliverables, which sometimes needed to be approved by the M&E unit in the General Planning
 Directorate. Fourth, persons occupying posts for the NATEJA project were tasked with building capacities
 of the units/Directorates to which they were attached. The ICR Team was informed that ‘skills transfer’ –
 both prior to and after integration – was often taken to mean ‘do work beyond that pertaining to the
 NATEJA project.’ Finally, the uncertainties and implications of integration (including lower salaries) may
 have contributed to turnover among NSDP staff and persons occupying posts tasked with NATEJA
 implementation, which naturally resulted in further delays.

 10.      The ICR Team has recommended that further activity be conducted in this sector. However, the
 extent to which how well NSDP integration has enabled MoLSAMD to implement projects must first be
 analyzed thoroughly, perhaps as part of appraisal of any future operations. As more than one NSDP staff
 member noted to the ICR Team, the new arrangements have resulted in the ‘disintegration’ of NSDP
 rather than its integration into MoLSAMD. Given NATEJA’s implementation performance, there is a prima
 facie case to be made that future implementation arrangements would need to be strengthened.

 11.     The above notwithstanding, one must be careful not to overstate the negative effect of
 integration upon implementation timeliness. For example, delays in procurement led the Bank to
 downgrade the procurement rating from Satisfactory to Moderately Satisfactory to Moderately
 Unsatisfactory, all by August 2016 – prior to the transfer of the NSDP procurement posts to MoLSAMD’s
 Directorate of Procurement.




                                                                                                          Page 54 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                           ANNEX 12. PROCUREMENT OF MAJOR TA PACKAGES

 1.       There were nine major TA packages (firms) in the procurement plan at effectiveness. The failure
 to recruit the bulk of these packages led to repeated down gradings of the procurement rating in the ISRs,
 and is part of the explanation as to why the Project performed poorly. This Annex describes the packages,
 attempts to reconstruct the reasons for why each package was or was not successfully recruited, and
 identifies lessons to be learned.

   No./     Contract description   Remarks
   Comp.
   1/C1     Inspection Agent for   Contracted (OSDLR) with significant delay, in September 2015.
            TPs
   2/C2     Institutional Capacity Was pending during first year of implementation due to integration of
            Building               NSDP into DGSD. Thereafter, the main ToRs for this package included
                                   guidelines (SOP) for TVET course implementation; updating the NSDP
                                   strategy; and manuals on HR and finance procedures. In the event,
                                   the NSDP revised its SOP and strategy with GIZ support, and manuals
                                   were developed with support from Bank-financed TA. Upon
                                   recommendation of the Bank, MoLSAMD dropped the package.
   3/C3.1   Training of SGs        There were significant delays in recruitment. Firms expressing interest
                                   did not meet the criteria, resulting in multiple re-advertisements.
                                   Eventually a short-list was established, and RFP responses were
                                   received in December 2015. A firm was recommended by Evaluation
                                   Committee. The WB rejected Committee’s recommendation based on
                                   technical reasons pertaining to qualifications of Committee
                                   membership. At this point, the MTR took place and government/WB
                                   agreed there was no longer sufficient time to recruit the TA, conduct
                                   the trainings and complete the component activities. The MTR
                                   recommended the component be dropped as part of a restructuring,
                                   and hence no further action was taken to recruit a firm.
   4/C1,    Awareness              Firm was selected, with significant delay. In the meantime, the
   C3.1,    Campaign               materials and awareness-raising activities were done by
   C3.2                            NSDP/MoLSAMD and Component 1 TA Firm for Component 1; and by
                                   ESCs (MoLSAMD) for Sub-Component 3.2. No progress was made in
                                   Sub-Component 3.1, and so no awareness-raising activities were
                                   required. Hence there was no need to hire a firm, and package was
                                   dropped.
   5/C3.2   Grant Management       Contracted December, 2015 (Pearl Horizon). Work was to be
                                   completed by June 2017, but four-month extension (without extra
                                   charge) was granted to complete tasks added to the contract.
   6/C3.2   Impact Evaluation      Contract with TA firm signed December, 2017 (Hawak Vision) and
                                   terminated March 2018 due to poor performance on first deliverable.
                                   New contract signed June 2018 (Altai Consultancy Services), and final
                                   payments to be made before end of 2018 once final draft is submitted.
   7/C3.3   Feasibility Study      Package was re-advertized in December 2015 as no firms who had

                                                                                                       Page 55 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                                        expressed interest met qualifications, at which time changes were
                                        introduced to budget and scope of work. At MTR it was recommended
                                        to drop Sub-Component 3.3, the activities of which were being
                                        implemented by GIZ. Therefore, the package was dropped.
   8/ C1,   Special Purpose             The Special Purpose Review was intended to track (on a sample basis)
   C3.1,    Review                      the disbursement of grants to beneficiaries: TPs in Component 1 and
   C3.2                                 Sub-Component 3.1, and small business operators in Sub-Component
                                        3.2. First round of procurement in 2015 cancelled by Bank due to no
                                        actual progress made on the ground by Project (ISR of June, 2015).
                                        Subsequent initiation of recruitment did not begin until mid-2016 (ISR
                                        of August 2016). However, because little or no progress was made in
                                        Component 1 and Sub-Component 3.1, and it was eventually decided
                                        not to recruit the firm and the package was dropped.
   9/all    Social and                  Contract signed in November, 2014 and completed in May of 2015.
            Environmental
            Assessment

 2.       The various delays referenced in the table above led to the ISR rating to be downgraded from
 Satisfactory to Moderately Satisfactory in June 2015 (at which point only one package had been procured,
 for the SEA). It was further downgraded to Moderately Unsatisfactory in August 2016, when the ISR noted
 that only three TA packages had been contracted (SEA, Inspection Agency for Component 1 and GMF for
 Sub-Component 3.2), there had been no progress in contracting the TA for Sub-Component 3.1 (TP for the
 SG members), and the recruitment process for the Special Purpose Review had only recently been
 initiated. In February 2017, the ISR rated was again Moderately Unsatisfactory because of extensive
 delays in initiating and finalizing major contracts, the procurement plan not being updated on a regular
 basis, and the plan not being reflected in the STEP tool; and it remained Moderately Unsatisfactory until
 Project closure.

 3.       Though executed on a somewhat more timely basis, the procurement of certain goods was still
 extensively delayed.60 The Bank also noted other shortcomings such as the procurement of luxury goods
 in some cases. 61 One can also note that some of the equipment procured was to be used to facilitate
 Project implementation (e.g. vehicles, furniture and IT equipment) yet many of the Project activities were
 not implemented, which would suggest that the goods did not yield their full value.

 4.      The ICR Team was informed by the Bank in-country fiduciary team that MoLSAMD is considered
 to have good procurement capacities. They were one of the first ministries to get accreditation from the
 national procurement authority to procure on their own, and their accreditation has been regularly
 renewed without incident. Despite this, its performance on NATEJA was rated Moderately Unsatisfactory
 at closure for reasons provided above, with the main recurring reason being delays. There are several
 underlying reasons, only some of which are strictly speaking related to fiduciary capacity. First, there
 were shortcomings in the proper execution of procurement tasks, as evidenced for example by delays,

 60E.g. Aide Memoire, November 22-26, 2016.
 61Aide Memoire, June 8-15, 2015. For example, 12 desks each costing AFN 24,000 or approximately USD 330 at then-current
 rates.



                                                                                                                     Page 56 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



 failure to regularly update the procurement plan, the procurement of luxury goods, and not using the
 STEP tool. Second, the procurement function was hampered by the non-execution of predicate activities,
 for example the TA firms to conduct the Special Purpose Review and the Feasibility Study. Third, the
 security situation makes international firms (and ICs) 62 reluctant to bid on Afghan contracts.




 62For instance, C3.1 foresaw the recruitment of an IC. Despite multiple attempts at recruitment, in the end an IC was not
 found.

                                                                                                                             Page 57 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                             ANNEX 13. THE NATEJA PROJECT FUNDED BY IADC

 1.        The IADC is funding a project that is in all respects identical to the Bank-financed NATEJA Project
 (indeed it has the same name), except that it does not include Sub-Component 3.3 (the activities of which
 are being ably implemented through a GIZ-supported project). That is, the IADC used the PAD as its
 template, and financed Component 1 and Component 3 activities in geographic zones not covered by the
 Bank-financed NATEJA Project; while supporting Component 2 activities in complementary fashion to the
 Component 2 activities financed through the Bank-supported Project. It uses the same implementation
 arrangements as those contained in the Grant Agreement (though with different project staff, of course),
 viz. an integrated NSDP is jointly responsible with relevant units of the MOLSAMD for implementation. At
 the time of writing (December 2018), the IADC NATEJA project is in its third year of implementation, and
 is proceeding satisfactorily according to IADC representatives; there are no plans for restructuring or early
 closure.

 2.       The ICR Team met with the IADC team responsible for the project, and discussed the contrasting
 fates of the Bank-financed and IADC-financed NATEJA projects, particularly with respect to Component 1.
 The following factors may help to explain the contrast:

     •   During the first 2.5 years of implementation, the Bank-financed NATEJA Project had to manage
         both NSDP integration and implementation. The challenges involved with this dual task are
         discussed in Section III.B. By the time the IADC project became effective in January 2016, some
         of these challenges had been alleviated, leading to a smoother implementation process.

     •   Also, during the first couple of years of implementation, there was some instability in the senior
         management at MoLSAMD. The Minister who was in place during Project preparation was
         replaced post-elections, with some delay; the new Minister assumed her duties in mid-2015, and
         naturally it took some time for her to become familiar with the Project. (This Minister was in turn
         replaced in mid-2017.)
     •   The IADC-financed project delayed its implementation in order to get extra funding from the
         Italian government to cover operational costs of the NSDP, given shortfalls in MoF funding on the
         discretionary budget. The Bank-financed Project, on the other hand, never received the amounts
         foreseen from the discretionary budget; the funds provided from the discretionary budget were
         both less than planned, and were used to support non-project activities. As noted in the body of
         the ICR text, this contributed to implementation delays.

     •   In brief, the IADC may have benefited from better timing, in that their project effectiveness came
         after the period of post-election uncertainty and when the Ministry was growing in
         implementation capacity.

     •   The IADC encountered difficulties in finding sufficient TPs offering a broad range of trades in which
         they trained, much as the Bank-financed NATEJA did. In response, they widened their geographic
         coverage. (The Bank-financed NATEJA also developed responses, as captured in the MTR




                                                                                                         Page 58 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



             restructuring recommendations; 63 but these were not implemented). As a result, they were able
             to enroll TPs in their first phase of training as follows:

                 o   MoUs signed with 60 TPs, of which 39 had started training at the time of writing.
                 o   Total enrolment of the 39 TPs is approximately 12,000. Some of the TPs have multiple
                     training sites.
                 o   The following 21 trades are offered: Iron Work, Mobile Repair, Electric Work, Bed
                     Sheet Making, Leather Bag Making, Shoe Making, Motor Cycle Repair, Backyard Poultry,
                     Carpet Weaving, Beauty Parlor, Tailoring, Khamak Dozi, Mora Dozi, Charma Dozi, Handy
                     Craft Making, Rug Knitting, Food Processing, Ball Making, Carpentry, Jewelry Making and
                     Yarn Spinning.

        •    This is in contrast with the first-batch TPs with whom the Bank-supported Project was ready to
             sign MoUs. Of the 78 TPs on the list for signing LoUs at the time of the MTR, the ICR Team was
             able to view data for 43 TPs on enrollment and courses offered.

                 o   43 TPs enrolling 3,472 students in the following 24 trades: tailoring, auto mechanic,
                     plumbing, metal work, electrician, ball-making, carpentry, carpet weaving, computer,
                     BCS/BBA/BSF (secondary academic degrees), poultry, hand crafts, IT, embroidery, English
                     language, journalism, charma dozi, food processing, candy making, flower making,
                     leadership, agriculture, curtain sewing, and project management.
                 o   19 of these TPs enrolling 1,497 were offering courses in areas that MoLSAMD was
                     concerned would not lead to employment, viz. project management, secondary academic
                     degrees, IT skills, and English language.
                 o   If one takes these 43 TPs to be broadly representative of all 78, then this would
                     correspond to first-batch TPs enrolling approximately 6,300 students; with 44 percent of
                     the TPs enrolling approximately 2,700 students in courses that MoLSAMD was concerned
                     would not lead to employment.




 63   See Annex 7.

                                                                                                        Page 59 of 60
The World Bank
Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)



                                    ANNEX 14. Monitoring and Evaluation



 3.       This annex amplifies some of the M&E activities that were implemented, and others that were
 not.

 4.       The M&E Unit of the Planning Directorate monitored the following Sub-Component 3.2 activities.

      •   It monitored the applications and screening process, and provided feedback to the ESCs and GMF.
      •   It monitored the trainings and provided feedback to the GMF. However, the monitoring was not
          based on a standardized methodology, and no formal reports were filed.
      •   It interviewed 10 percent of Sub-Component 3.2 beneficiaries and provided feedback to the GMF.
          However, the raw data generated from the questionnaires was not entered into a database and
          analyzed, and no formal report was produced.

 5.      A biometric staff attendance registration system was developed and operationalized for NSDP
 and later scaled up to cover all MoLSAMD employees at the Ministry in Kabul. However, it was no longer
 operational at the time of the ICR mission (November, 2018).

 6.       Some of the planned M&E activities were not executed.

      •   The independent third-party monitoring exercises, the independent assessments of trainee
          competencies, and the impact evaluations for Component 1 or Sub-Component 3.1. This was
          due to the fact the underlying activities that were to be monitored were themselves not carried
          out.
      •   The special purpose reviews (to audit the disbursement of incentives) were not carried out
          because it was judged that – in the absence of incentives being disbursed in large quantities under
          Component 1 - the scale of Sub-Component 3.2 incentives was too small to justify the SPRs
          expenditure. (See Annex 12).
      •   The IE workshops to build NSDP capacity to undertake IE analysis.

 7.      The failure to operationalize the Project MIS had to do with the fact that the individual hired to
 design and operate the MIS was not able to transfer his capacities to the two persons working at the M&E
 unit at MoLSAMD. He reported leaving a User’s Manual. Further, a solution was not found as to where
 to host the server for the MIS. The original intention was to have it hosted by the Ministry of
 Communications and Technology, but when the MIS was ready to go live the Ministry was encountering
 cyber security problems and did not want to assume responsibility for the database. The Project Team
 approached private companies, but MoLSAMD insisted that the NATEJA staff responsible for the MIS
 personally guarantee the data’s security, which the staff refused to do. At this stage, it was becoming
 clear that the Project was very possibly going to be closed early, and consequently the main activities for
 which the MIS was required (under Component 1 and Sub-Component 3.1) were not going to be
 implemented. Therefore, there was no longer an impetus to resolve these operational issues.




                                                                                                        Page 60 of 60