ICRR 14348 Report Number : ICRR14348 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/19/2014 Country : Jordan Project ID : P100534 Appraisal Actual Project Name : Jordan Employer US$M ): Project Costs (US$M): 8.33 4.66 Driven Skills Development Project L/C Number : L7561 Loan/ US$M): Loan /Credit (US$M): 7.50 3.37 Sector Board : Education Cofinancing (US$M): US$M ): Cofinanciers : Board Approval Date : 06/03/2008 Closing Date : 09/30/2013 09/30/2013 Sector (s): Vocational training (69%); Central government administration (31%) Theme (s): Education for the knowledge economy (50% - P); Improving labor markets (50% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Pia Schneider Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Loan Agreement (2008), the original project development objective (PDO) was "to assist the Borrower in improving its technical and vocational education training (TVET) sector through the harmonization of policies pertaining to the development of said sector and the realignment of such policies with the operational functions of E-TVET, including the development of employer participation in the formulation of said policies and in carrying out institutional development reforms as well as in the design and delivery of a skills development program." The Project Appraisal Document (2007) has a slightly different wording of project objectives, although the meaning is essentially the same: To enhance the internal and external efficiency of the E -TVET sector by making it more flexible and demand ii) driven through the development of employer community participation in (i) sector policy formulation, (ii) iii) skill development program design and delivery . institutional development and reform, and (iii) The wording of the Project Appraisal Document is used in this Review, as it is more outcome oriented than that in the Loan Agreement. The objective was revised in March 2013. The revised objective is: To realign the E -TVET sector with the National Employment Strategy by enhancing the enabling conditions ( i) TVET institutional development; and (ii) for employer participation in : (i ii) skills development program design and delivery by the Vocational Training Corporation (Restructuring Paper 2013). During this Level-1 restructuring, four PDO indicators were dropped, one new PDO indicator was added, and the two remaining PDO indicators were revised . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 04/04/2013 c. Components: The four components supported by the project are presented below, showing project cost estimated at appraisal, after the restructuring, and actual project costs . These amounts do not include the front -end fee for the IBRD loan, which is included in the total project costs reported above . 1. E-TVET System and Council Development with Employer Participation (Appraisal: US$ 1.90 million. Revised: US$ 0.90 million. Actual: US$ 0.73 million). This component was to support institutional development of the E-TVET Council and its Secretariat and implementation of key aspects of employment-related planning, development and training with TVET stakeholders through : (i) E-TVET Council orientation and capacity development; (ii) E-TVET sector policy, planning, and organizational development with active employers ’ participation; (iii) E-TVET system performance assessment with active employers ’ participation; and (iv) E-TVET sector promotion and awareness among TVET stakeholders . 2. Restructuring of Vocational Training Corporation (VTC) VTC ) (Appraisal: US$ 6.14 million. Revised: US$ 6.14 million. Actual: US$ 2.93 million). This component was to increase completion rates in training courses and to provide incentives for improved job relevance and quality of training through : (i) Establishment of the mandate, governance structure, and regulatory framework for the new Skills Development Agency; (ii) Organizational restructuring and staff development; and (iii) Reorientation of the training delivery model to reflect employer defined competencies . 3. Training and Employment Fund Development (Appraisal: US$ 0.06 million. Revised: US$ 0. Actual: US$ 0). This component aimed to make the Fund more responsive to labor -market skill needs. It was to finance technical advisory services and related activities to support the establishment of a demand -driven funding mechanism that facilitates and finances skills -upgrading of in-service employees, and provide prospective employees with relevant pre -service and in-service training. 4. Project Management . Implementation and Information Technology (Appraisal: US$ 0.71 million. Revised: US$ 0.55 million. Actual: US$ 0.65 million). This component was to finance support for project implementation arrangements and the development of the IT system and hardware . During the restructuring in 2013, project activities that were no longer relevant were eliminated : Component 1 Cancelled support and capacity building activities for the E -TVET Council and its Secretariat, as well as support for performance-based planning and realignment of budget allocations . Cancelled activities that were premature, including employment demand projects, development of a medium-term expenditure framework, and support to the National Qualification Framework . Added new technical assistance activity to support implementation of the National Employment Strategy. Component 2 Technical assistance was added to develop an M&E system to track vocational training interventions . Component 3 Component was dropped due to lack of clarity about immediate future of the Fund, its mandate and sources of financing. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost The actual project cost was US$ 4.66 million, compared to the original appraised amount of US$ 8.33 million including front-end fee for IBRD. Revised project cost following the restructuring was US$ 7.59 million. Actual costs were lower because of delays in implementation and cancellation of activities during project restructuring. Financing The project was financed by a US$ 7.50 million IBRD loan, of which US$ 3.37 million (45%) actually disbursed. During the March 2013 restructuring, US$ 1.21 million of the loan was cancelled, and the revised loan amount became US$ 6.29 million. The amount undisbursed at project closure was US$ 2.92 million. Borrower Contribution The actual borrower contribution was US$ 0.96 million, which is 72% of the appraised amount of US$ 1.33 million. Dates March 2013: Level-1 restructuring, six months before closure, to ensure the project's relevance to Jordan's National Employment Strategy which was formally launched in June 2012. Disbursement at this point was US$ 1.86 million (55% of the actual loan amount). The following changes were introduced : IBRD loan amount reduced and US$ 1.21 million cancelled. Project development objectives and components were revised (see above) and the M&E design was changed. Funds were reallocated across disbursement categories . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Original objective : High Revised objective : High Both the original objective of enhanced internal and external efficiency of the E -TVET sector through employer participation and the revised objective of a realigned E -TVET sector with the National Employment Strategy are in line with the government's National Agenda (2006-2015), which was in 2012 replaced by the National Employment Strategy. The Agenda proposes actions to improve employment, the availability of entry -level workers with the skills needed in the labor market, and training programs to upgrade skills of the employed . The objectives are relevant considering the problems identified in the National Agenda, including that the vocational training sector had produced poorly trained and uncommitted workers, a lack of private sector collaboration, fragmentation of training programs, and the need for a holistic approach . The National Employment Strategy aims to increase labor participation and create more and better jobs . The objectives are also relevant for the government's Executive Development Program which highlights a knowledge -based economy and harmonized education outputs with labor market requirements . The objectives are also relevant in the context of the Bank's Country Partnership Strategy (2012-2015), which aims to support job creation through developing skilled human resources in science and technology in line with private sector demand. The objectives address key challenges in education such as low enrollment in vocational education (around 13%) and the need to realign vocational education to the demands of the labor market . The objectives are also relevant in a context of high youth unemployment rates and an unmet demand by employers for individuals with vocational skills . b. Relevance of Design: Original design : Modest Revised design : Substantial The results framework in the original design included relevant activities to strengthen and coordinate technical and vocational education and training reform, and it supported employer -driven programs to attract more youth and ensure their transition into the labor market . The design focused on reforming governance, program development, and financing of the Vocational Training Corporation (VTC) serving students at risk of exclusion . These activities are logically linked to the objective of a more efficient E -TVET sector. However, the design had weaknesses. The original project objective was too ambitious given the project's time frame . The third component received only symbolic support from the project which made it less focused; this component was cancelled during the restructuring . Also, the design did not adequately plan for the financing of E -TVET, which raises concerns about sustainability . Finally, the design proved to be too complex in a context of fading government ownership for this project, and had to be revised . The revised design had a less ambitious objective of an aligned E -TVET sector with the National Employment Strategy, which is now hosted by the King Abdullah Fund . It also included fewer activities, thereby increasing project focus. The design is relevant for the work of other organizations, including the International Labor Organization and the European Union . Project components are logically linked to the objective . Still, the revised design was too ambitious given the limited project time left and lack of government ownership . 4. Achievement of Objectives (Efficacy): The project aimed at setting up a new governance framework and a new set of strategic incentives to change the way TVET programs are conceived, designed, set up, delivered and evaluated (ICR, p. 17). Original objective and design : To enhance the internal and external efficiency of the E -TVET sector by making it more flexible and demand driven through the development of employer community participation . Modest . There is no evidence that the E -TVET sector became more efficient, including changes to the rate of return of vocational education and changes to its contribution to the labor market and economy . The project has helped develop relevant tools in collaboration with employers, including an employer -driven Occupational Profile Manual which has been adopted by the Center of Accreditation and Quality Assurance, a E -TVET sector performance assessment system, a TVET communication strategy, an employer -driven modality to identify priority sectors for new training programs, and a baseline survey on public perception of the TVET . However, so far these tools have not been used to produce a governance framework and policy directives that could affect sector efficiency. Revised objective and design : To realign the E -TVET sector with the National Employment Strategy by enhancing the enabling conditions for employer participation in : (i ii) ( i) TVET institutional development; and (ii) skills development program design and delivery by the Vocational Training Corporation . Modest . There is insufficient evidence of project contribution to enhanced enabling conditions for employer participation. The project helped in the preparation of background work for the three pilot training programs and some enabling conditions were introduced in the Vocational Training Corporation . Enrollment increased but cannot be attributed exclusively to the project . The E-TVET council has not been effective in providing employer-driven policy directives to its stakeholders . Cabinet approval is needed of the new by -laws for the Vocational Training Corporation to allow the introduction of the revised organizational structure . Outputs relevant for original and revised objectives There were preliminary steps taken toward piloting of employer driven business, training, and governance models in three Vocational Training Corporation (VTC) centers, but no actual piloting was conducted . So far, the organizational structure, audit report and staffing plans were accepted in early 2013; but the VTC by-laws are still awaiting Cabinet approval . The completion rate of VTC graduates was 73% in 2013, surpassing the 70% target; however, project attribution is not possible. New output indicators added during restructuring : Development and introduction of a TVET Sector Performance Assessment System (PAS) was not met and related reports have not been provided . So far, PAS and data collection tools were developed and approved in early 2013, and a second assessment report was received in September 2013. There is no information on increased public awareness for E -TVET; Terms of Reference have been drafted but the survey has not yet been carried out to collect relevant information . The enrollment rate in VTC increased by 12% from 2012 to 2013; however, this increase cannot be attributed to the project as most activities under component 2 were delayed to 2013 or have not yet been implemented due to political challenges (see Section 9a). The female enrollment rate in VTC increased by 36% from 2012 to 2013 for regular courses and decreased by 42% for short-term courses over the same time period . Project attribution is not possible as component 2 activities were delayed to 2013 or have not yet been implemented . The following output indicators were not measured and were dropped during the 2013 restructuring: Enhanced understanding of occupational opportunities . Employer driven occupational profiles developed . Restructured VTC centers establish performance indicators . Financial resources allocated against performance . Policy objectives define feedback from TVET stakeholders . Autonomous governance model in place . Satisfaction among employers with training delivered through VTC . Increased % of non-leavers from 10th grade in total of enrollment. Outcomes relevant for original and revised objectives : Number of employer-driven occupational profiles developed and endorsed by the E -TVET Council increased from 0 in 2009 to 6 in 2011, meeting the target of 6. Organizational audit, staffing plan, and organizational structure are planned to be introduced in VTC once the by-laws are approved. So far, organizational structure, audits and staffing plans were accepted in early 2013, and VTC by-laws were sent to Cabinet for endorsement in July 2013. New outcome indicator added during restructuring : The number of employer-driven skills development programs designed increased from 0 in 2009 to 3 in 2013. However, these programs were not delivered and implemented, and therefore the target was not met . The following outcome indicators were not measured and /or were dropped during the 2013 restructuring: Functional E-TVET Council. (An E-TVET Council law was passed in 2009.) Improved employer satisfaction of ETVET graduates entering employment . Improved VTC internal efficiency through increased trainee /instructor ratio increased from 6.4 in 2006 to 12 in 2013. Number of firms receiving training through TEF (10 in 2009). 5. Efficiency: The economic analysis at appraisal anticipated that the project would contribute to higher employment among TVET graduates due to higher relevance of their skills . This would lead to higher lifetime earnings . In addition, it was expected that unit cost per graduate would decline as a result of the reorganized TVET system, and completion rates would improve. Based on these assumptions, and using a discount rate of 7 percent, the Project Appraisal Document estimates a benefit /cost ratio between 2.0 and 4.0. Rates of return are not reported in the ICR. There is no evidence that these anticipated benefits are materializing . The ICR (p. 18) identifies some possible efficiency gains from project benefits, including the elimination of supply-driven courses at the VTC which may reduce cost . The ICR estimates possible benefits of about US$ 4.3 million if the reform tools supported by the project are eventually implemented . Project funds were cancelled during the restructuring, and the project closed on time, which contributed to the efficient use of project funds . However, there were substantial delays in implementation, and lengthy government and Bank procedures substantially delayed the processing of project restructuring, which negatively affected the efficient use of project funds . Efficiency is rated Modest . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re-estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Project under the original objectives : Moderately Unsatisfactory Relevance of the original project objective is High, while the relevance of the original design is Modest . Achievement of the objective is Modest under the original PDO . There is no evidence that E-TVET sector became more efficient. Efficiency is rated Modest. Modest ratings on Relevance of Design, Efficacy, and Efficiency are indicative of significant shortcomings in the operation's preparation and implementation, resulting in an Outcome rating of Moderately Unsatisfactory . Project under the revised objectives : Moderately Unsatisfactory Relevance of the revised project objective is High, while the relevance of the revised design is Substantial . Achievement of the objective is Modest under the revised PDO . There is no evidence that the TVET sector became more aligned with the National Employment Strategy . Efficiency is rated Modest. Modest ratings on Efficacy and Efficiency are indicative of significant shortcomings in the operation's preparation and implementation, resulting in an Outcome Rating of Moderately Unsatisfactory . According to OPCS/IEG guidelines for restructured projects, the final outcome rating is determined according to the percentage of the credit that disbursed before and after project restructuring . This IBRD loan had disbursed 55% of actual loan amount at the time of the first restructuring . Therefore, the combined outcome rating is Moderately Unsatisfactory. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Several factors contribute to a high risk to project development outcomes . The institutional risk is high. Although government commitment has increased as shown by strategic documents such as the National Agenda and the National Employment Strategy, a high risk remains with respect to the practical implementation of relevant reforms in the TVET sector. Ownership and commitment are yet to be demonstrated for the E -TVET Council to adopt the project deliverables and fulfill its mandate . In addition, the Cabinet still needs to approve the relevant Vocational Training Program (VTC) by-laws to enable employer participation and deliver the VTC training program in a consistent manner. a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: a. Quality at entry: The Bank during project preparation recognized the high risk of declining government ownership and weak leadership to project implementation . To mitigate this risk, the team situated the project within the government's National Agenda and supported institutional capacity building and ownership among stakeholders. The Bank also worked closely with other donors to ensure consistent policy messages . The Bank invested extensively in country and sector reviews with the support of a PHRD grant from Japan to inform project design. However, there were significant shortcomings . Although the Bank identified a high-risk environment, the team went ahead with a design that was too ambitious for this context and that later required restructuring. It was not realistic to propose a sector council that can issue policy directives within the non-obliging legal framework of the E-TVET Council law. The team did not sufficiently involve the E-TVET Fund under the project, and this component was later cancelled . The M&E framework was too complex, and data on most indicators could not be collected (see Section 10). Project design did not plan for relevant indicators to assess the contribution of project activities to the objectives of improved internal and external efficiency. at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: Supervision was frequent and timely and provided consistent support to the borrower . There was continuity in the Bank team, which contributed to institutional memory . The team identified implementation problems and recommended actions to add ress them to the government. Supervision missions continued to recommend improvements to data collection and reporting . The Bank team was proactive. During the Mid-Term Review in May 2011, the team recognized the need for restructuring . However, the processing of restructuring was delayed by the Ministry of Planning and International Cooperation and by lengthy management reviews within the Bank, as a result of which the restructuring package was only approved by the Board two years later in March 2013 and six months before project closure . The Bank's decision to cancel project funds and not to extend the project was adequate in the context of insufficient government ownership and progress. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: During project preparation, the government had a clear vision and showed high political commitment to adopt an institutional approach towards skills development . However, during implementation, insufficient political commitment to the project and weak leadership negatively affected the attainment of project objectives : As chair of the E-TVET Council, the Ministry of Labor (MOL) failed to motivate cooperation by the Ministry of Education (MOE) and the Ministry of Higher Education (MOHE) in addressing sectoral policy directives. The E-TVET Council was dysfunctional, and the two Education Ministries did not become effective partners to the Council's work . Similarly, the office of the Prime Minister did not use the National Agenda and the National Employment Strategy to motivate better cooperation by the MOE and MOHE . There were frequent changes in the MOL leadership (six Ministers) and priorities. As a result of these changes, recommendations made by the Bank to address project performance were not acted upon . The MOL did not fulfill the Council’s mandate in addressing sector fragmentation and moving towards a responsive E-TVET sector, and there was absence of Government leadership in eliciting E -TVET Council’s engagement on broader TVET sector strategy . The MOL made only initial progress in aligning training programs of the VTC, and VTC management is characterized by reactive rather than proactive leadership . The government did not pursue the implementation of the National Agenda, which in 2012 was replaced by the National Employment Strategy . There was insufficient government leadership in focusing the E -TVET Fund on priority skill gaps of employers, There were delays by the Ministry of Planning and International Cooperation in approval of rebidding the technical assistance for VTC restructuring and in moving ahead with project restructuring . There was failure to provide timely budget financing for activities (including those financed by the loan ) under the budget ceilings imposed by the Ministry of Finance during the final year of project implementation. Government Performance Rating Unsatisfactory b. Implementing Agency Performance: The project was implemented with the support of the Development Coordination Unit (DCU), the Secretariat of the E-TVET Council, and the VTC. The DCU was inadequately staffed at the beginning and there were frequent changes in staffing . Financial management was satisfactory to the Bank; however, procurement was weak and slowed down project implementation (see Section 11). Ineffective management by the Ministry of Labor, the E-TVET Council, and the DCU negatively affected technical assistance . This included a reluctance to revisit terms of reference for consultancies, insufficiently skilled evaluation committees, and inadequate supervision of consultants . These weaknesses contributed to delays and the need to rebid contracts. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Both the original and revised design had weaknesses . The original M&E design included indicators that were not clearly defined and not linked to project activities, thus they were not measured and were dropped during the restructuring. The original M&E design did not include relevant efficiency indicators . The revised M&E design added indicators that improved without relevant project activities implemented, and therefore attribution is an issue. b. M&E Implementation: The initial M&E actions were not carried out as agreed, and the M&E officer in the Development Coordination Unit (DCU) resigned. Thereafter, progress reports were prepared and baseline studies carried out as planned . The restructuring in March 2013 made M&E less demanding. Most initial indicators were dropped and new indicators added. The DCU monitoring and evaluation officer managed the implementation of component 1, including technical assistance to the development of the decision support system and sector performance assessment, as well as overseeing implementation of the VTC management information system . c. M&E Utilization: M&E findings were mainly used to inform project implementation and the design and implementation of the public information and awareness efforts . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project did not trigger any safeguard policies . This was a Category C operation . b. Fiduciary Compliance: Financial Management . According to the ICR (p.12) reporting was timely and satisfactory to the Bank . Audit opinions were unqualified. An exceptional 6-month grace period was granted following project closure to process payment for all contracts . Procurement . Initially, procurement was satisfactory to the Bank . There were subsequent delays caused by the termination of technical assistance contracts (ICR, p.12). Slow decision making by stakeholders contributed to stagnation in procurement and disbursement . The Ministry of Finance budget restrictions completely stalled the procurement process. These shortcomings seriously affected project implementation . c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development High High Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Unsatisfactory Government commitment was lacking Unsatisfactory and capacity at the implementation agency weak, which contributed to serious delays and cancellation of activities. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (pp. 23-25) identifies several valuable lessons : It is important to allow for enough time and a gradual change process . Complex changes need a gradual implementation process, buy-in from all stakeholders, and leadership capacity among relevant stakeholders . In this case, a paradigm shift from supply -driven to demand-driven E-TVET programs was more difficult to implement than originally planned. Prob lems with technical assistance can severely slow down progress . In this case, several delays in project Problems implementation resulted from problems with technical assistance, including the rejection of inception reports and the need to rebid one major contract . Better results were achieved when all stakeholders participated in the formulation of relevant terms of reference and in the implementation of activities . Training on Bank procurement and financial management supports smooth implementation . The borrower's ICR also identifies the need for training of all relevant stakeholders in Bank fiduciary rules during project preparation. Early training will provide a clear understanding on the additional resources needed in terms of staff and time to follow Bank rules . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is well written and concise. It is candid in its assessment, contains good analysis, and is appropriately critical. The ICR Is internally consistent and consistent with guidelines . a.Quality of ICR Rating : Satisfactory