Document of World Bank Report No.: 97132 PROJECT PERFORMANCE ASSESSMENT REPORT INDONESIA MANAGING HIGHER EDUCATION FOR RELEVANCE AND EFFICIENCY PROJECT (IBRD-47890 - IDA-40770) June 18, 2015 IEG Public Sector Evaluation Independent Evaluation Group ii Currency Equivalents (annual averages) Currency Unit = Indonesian Rupiah (IDR) 2005 US$1.00 IDR 9,470 2013 US$1.00 IDR 9,718 2014 US$1.00 IDR 12,264 (Dec. 1, 2014) Abbreviations and Acronyms ADB Asian Development Bank ICR Implementation Completion Report AHELO Assessment of Higher Education IDA International Development Learning Outcomes Association ASEAN Association of Southeast Asian IEG Independent Evaluation Group Nations IEGPS IEG Public Sector Evaluation CAS Country Assistance Strategy IMF International Monetary Fund CPS Country Partnership Strategy IT Information Technology DGHE Directorate General Higher MOEC Ministry of Education and Culture Education M&E Monitoring and Evaluation DGHE-IU Implementation Unit of DGHE for MTR Mid-Term Review project NISHE National Information System DUE Development of Undergraduate Higher Education Education OECD Organization for Economic FDI Foreign Direct Investment Cooperation and Development GDLN Global Development Learning OU Open University Network PAD Project Appraisal Document GOI Government of Indonesia PBC Performance Based Contract GPA Grade Point Average PDO Project Development Objectives HE Higher Education PPAR Project Performance Assessment HELTS Higher Education Long Term Report Strategy QUE Quality of Undergraduate HEI Higher Education Institution Education project-Indonesia HEI-IU Implementation Unit of individual TA Technical Assistance HEI for project TTL Task Team Leader IBRD International Bank for WB World Bank Reconstruction and Development Fiscal Year Government: January 1 – December 31 Director-General, Independent Evaluation : Ms. Caroline Heider Director, IEG Public Sector Evaluation : Mr. Nicholas David York (Acting) Manager, IEG Public Sector Evaluation : Mr. Mark Sundberg Task Manager : Mr. Erik A. Bloom iii Contents Principal Ratings ................................................................................................................. v Key Staff Responsible......................................................................................................... v Preface............................................................................................................................... vii Summary ............................................................................................................................ ix 1. Background and Context................................................................................................. 1 2. Objectives, Design, and their Relevance ........................................................................ 4 Relevance of Project Objectives and Design .................................................................. 6 3. Implementation ............................................................................................................... 7 4. Achievement of the Objectives ..................................................................................... 10 Objective 1 .................................................................................................................... 10 Outputs ...................................................................................................................... 10 Outcomes .................................................................................................................. 10 Objective 2 .................................................................................................................... 11 Outputs ...................................................................................................................... 12 Outcomes .................................................................................................................. 13 5. Efficiency ...................................................................................................................... 14 6. Ratings .......................................................................................................................... 16 Outcome ........................................................................................................................ 16 Risk to Development Outcome ..................................................................................... 17 Bank Performance ......................................................................................................... 17 Borrower Performance .................................................................................................. 18 Monitoring and Evaluation ........................................................................................... 19 7. Lessons .......................................................................................................................... 21 References ......................................................................................................................... 23 Annex A. Basic Data Sheet ............................................................................................... 25 Annex B. Timeline of Events-Indonesia ........................................................................... 28 Annex C. List of Persons Met ........................................................................................... 30 Annex D. Borrower Comments ........................................................................................ 34 This report was prepared by Maurice Boissiere, who assessed the project in September 2014, under supervision of Task Leader, Erik A. Bloom. The report was peer reviewed by Roberta Malee Bassett and panel reviewed by Monika Huppi. Viktoriya Yevsyeyeva provided administrative support. iv Tables Table 1.1. Higher Education Institutions by Type (2010) .................................................. 2 Table 2.1. Outline of Project Components at Approval...................................................... 5 Table 2.2. Logical Framework of Project ........................................................................... 7 Table 3.1. Appraisal and Actual Costs by Component (US$ million equivalent) .............. 9 Table 4.1. Distribution of Subproject Grants in Component 2 of Project ........................ 12 v Principal Ratings ICR* ICR Review* PPAR Outcome Moderately Satisfactory Moderately Satisfactory Moderately Satisfactory Risk to Development Significant Significant Significant Outcome Bank Performance Moderately Satisfactory Moderately Satisfactory Moderately Satisfactory Borrower Moderately Satisfactory Moderately Satisfactory Moderately Satisfactory Performance * The Implementation Completion Report (ICR) is a self-evaluation by the responsible Bank department. The ICR Review is an intermediate IEGWB product that seeks to independently verify the findings of the ICR. Key Staff Responsible Division Chief/ Project Task Manager/Leader Sector Director Country Director Appraisal Richard Hopper/Susiana Christopher Thomas Andrew Steer Iskandar Completion Ratna Kesuma Luis Benveniste Stefan Koeberle vi IEG Mission: Improving World Bank Group development results through excellence in evaluation. About this Report The Independent Evaluation Group assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the Bank’s self-evaluation process and to verify that the Bank’s work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEG annually assesses 20-25 percent of the Bank’s lending operations through field work. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or Bank management have requested assessments; and those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEG staff examine project files and other documents, visit the borrowing country to discuss the operation with the government, and other in-country stakeholders, and interview Bank staff and other donor agency staff both at headquarters and in local offices as appropriate. Each PPAR is subject to internal IEG peer review, Panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible Bank department. The PPAR is also sent to the borrower for review. IEG incorporates both Bank and borrower comments as appropriate, and the borrowers' comments are attached to the document that is sent to the Bank's Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public. About the IEG Rating System for Public Sector Evaluations IEG’s use of multiple evaluation methods offers both rigor and a necessary level of flexibility to adapt to lending instrument, project design, or sectoral approach. IEG evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEG website: http://worldbank.org/ieg). Outcome: The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance includes relevance of objectives and relevance of design. Relevance of objectives is the extent to which the project’s objectives are consistent with the country’s current development priorities and with current Bank country and sectoral assistance strategies and corporate goals (expressed in Poverty Reduction Strategy Papers, Country Assistance Strategies, Sector Strategy Papers, Operational Policies). Relevance of design is the extent to which the project’s design is consistent with the stated objectives. Efficacy is the extent to which the project’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared to alternatives. The efficiency dimension generally is not applied to adjustment operations. Possible ratings for Outcome: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Risk to Development Outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). Possible ratings for Risk to Development Outcome: High, Significant, Moderate, Negligible to Low, Not Evaluable. Bank Performance: The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loan/credit closing, toward the achievement of development outcomes. The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for Bank Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Borrower Performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible ratings for Borrower Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. vii Preface This is a Project Performance Assessment Report (PPAR) for the Indonesia Managing Higher Education for Relevance and Efficiency project. The project was approved on June 9, 2005, became effective on December 20, 2005, and closed on December 31, 2012, 18 months after the original closing date. The project was financed through IDA Credit 40770 in the amount of US$30 million, IBRD Loan 47890 in the amount of US$50 million and a government contribution of US$34.5 million equivalent. This PPAR was prepared by Maurice Boissiere, IEG consultant. The findings are largely based on a two-week mission to Indonesia from September 15 - 27, 2014. The mission met with education and government authorities in Indonesia as well as private business people. The mission visited six universities in the islands of Java and Sumatra. A list of persons met is in Annex C. The mission also examined: (a) World Bank project files; (b) project related reporting and evaluation; and (c) education studies with data by government, other development partners, and civil society organizations, as well as the relevant research literature. IEG gratefully acknowledges the logistical assistance and support of the staff in the Directorate General of Higher Education, the universities that were visited, and the World Bank Jakarta office. Following standard IEG procedures, a copy of the draft PPAR was sent to the relevant government officials and agencies for their review and feedback, and comments received from the Government have been included in Annex D. ix Summary The objective of this PPAR is to assess the development effectiveness and lessons arising from the Indonesia Managing Higher Education for Relevance and Efficiency Project (IMHERE). The project was approved on June 9, 2005, became effective on December 20, 2005, and closed on December 31, 2012, 18 months after the original closing date. The project at approval was to be financed through IDA Credit 40770 in the amount of US$30 million, IBRD Loan 47890 in the amount of US$50 million and a government contribution of US$34.5 million equivalent. At closing, the actual Bank contribution was US$72.1 million, with US$6.9 million cancelled, and a government contribution of US$22.4 million. Country and Sector Background As the project was undergoing preparation in 2004, Indonesia was going through a transition towards a more open and decentralized government. These changes resulted from reforms introduced after the Asian financial crisis in 1997-98 and affected delivery of public services, including education at all levels. Economic policy is now aimed at developing a more competitive and technologically oriented economy and raising the growth rate back to the range of 8 to 9 percent. This project was intended to contribute to the development of these critical goals, in particular through the development of high level professional skills demanded in the labor market. The structure of higher education has evolved rapidly over the past three decades. Starting out with only three public universities at independence, the Higher Education (HE) sector has developed to the point that there are now 83 public Higher Education Institutions (HEIs) and more than 3,000 private ones. HE enrollment was around 3.0 million in 2004, giving a participation rate of 12 percent. Enrollment in 2012 reached about 5.0 million for a participation rate of 24 percent, and it is estimated that the number of university graduates doubled between 2004 and 2012. The Higher Education Long Term Strategy (HELTS) for 2003-2010 focused on developing the autonomy and accountability of the HEIs so as to facilitate the improvement of quality and relevance within the sector. Competitive and performance based grants would be used to steer the behavior of the institutions in the direction of better quality and efficiency. The Project The project’s development objectives (stated identically in the Credit Agreement and Project Appraisal Document) were: “to create an enabling environment for the evolution of autonomous and accountable public higher education institutions, and to develop effective support mechanisms for the improvement of quality, relevance, efficiency and equity of higher education”. The project as approved had three components: (1) Higher education system reform and oversight to support implementation of the new Higher Education Long Term Strategy (2003-2010) to develop autonomous and accountable institutions, (2) Grants for subprojects to improve academic quality and institutional performance, and (3) Project Management to support implementation of the project by the DGHE Implementation x Unit. The design of the competitive grants part of the project was such that it involved mainly the public university segment of the HE sector, with some participation of private universities in grants for improving quality of teacher training programs. Diploma level training institutions, public or private, did not participate. However, the governance and accreditation part of the project was designed to impact the whole HE system. The relevance of project objectives is rated high. The objectives were aligned with the development priorities of Indonesia as well as the Bank’s strategy, both at time of approval as well as at completion. The project objectives are also aligned with the Higher Education Long Term Strategy, which stresses autonomy, accountability, quality, relevance, efficiency and equity. The relevance of project design was aligned with the PDO by allowing for appropriate decentralization and accountability of higher education through a new and strengthened legal framework which also promoted the goals of the HELTS. The project components and outputs follow a logical framework in which the grant programs of various kinds contribute to the project objectives, although the project design could be stronger with respect to risk mitigation measures and monitoring and evaluation. The relevance of project design is rated substantial. A number of issues and challenges arose during project implementation. Developing the legal framework for the HELTS, which was supported by the project objectives, was a major activity that culminated in the passage of a new Higher Education Law in 2009. However the law was revoked by the Constitutional Court in 2010. A new version of the law was drafted and passed by the legislature and signed into law by the President in August 2012. The new law stipulated that more regulations were to be drafted by the Ministry of Education, which would then complete the new HE legal framework. The decision by the Ministry of Finance to prohibit the use of block grants, and had been allowed in previous Bank supported projects, gave rise to difficulties because it required a return to the less flexible annual line-item budgeting. These implementation bottlenecks were analyzed and resolved in the Mid-Term Review, which resulted in a Level II restructuring in September 2009, meaning that project objectives were not modified, but activities were modified along with an extension of the project closing date by 18 months. The achievements related to Objective 1 (creating an enabling environment for the evolution of autonomous and accountable public higher education institutions) are rated as substantial. The outputs include producing major background studies to support the development of the Higher Education legal framework and the new HE Law, which was passed in 2012, after the 2009 version was over turned by the Constitutional Court in 2010. Other outputs include the development of a new accreditation framework based upon institutional accreditation as opposed to the previous academic program accreditation, which were becoming too numerous and burdensome. These outputs, inter alia, contributed to the major outcomes, such as the new legal and regulatory environment. The capacity of HEIs to exercise autonomy along with accountability made significant progress as a result of the program of grants for institutional capacity. The achievements related to Objective 2 (developing effective support mechanisms for the improvement of the quality, relevance, efficiency and equity of higher education) are xi rated as modest. While acknowledging the difficulty of measuring such complex objectives, four proxy indicators were selected for the Subcomponent 2.1, which involved the subprojects for improving academic quality, relevance and equity. Grade point average showed an increase of six percent from 2.99 to 3.13 between 2009 and 2012. Time to find the first job declined from 8.5 months to 5.8 months. Time to graduate also declined from 56.7 months to 51.6 months. It was not possible to monitor the equity in the detail that was originally proposed in the PAD; during implementation is became clear that the baseline value was not valid. This misinterpretation was corrected at MTR, and it appeared that objective at completion was only partly achieved. Given shortcomings in the M&E framework, such as lack of comparison group or lack baseline trends for a few years, attribution of outcomes to the project is difficult. The project’s efficiency is rated substantial. The project has produced significant benefits for beneficiaries and the Indonesian economy. Depending on assumptions, the net present value of the benefits to students would vary from US$47 million to US$164 million. This excludes efficiency gains to the universities as well as economic benefits for research and development. Bank performance is rated moderately satisfactory. Quality at entry is moderately unsatisfactory mainly due to shortcomings in risk assessment related to the legal framework as well as shortcomings in M&E design. Quality of supervision is rated moderately satisfactory reflecting proactive superision in recognizing implementation problems and addressing them during the MTR and restructuring. Borrower performance is rated as moderately satisfactory. The Government performance is moderately unsatisfactory. Initially, Ministry of Finance and National Planning Agency played key roles in support of DGHE in preparing the project. Yet the Ministry of Finance decision to apply the letter of the Finance Law (2003) resulted in serious bottle necks during implementation, despite having agreed to the block grant arrangements. The DGHE played the role of overall implementation agency, along with implementation units of the HEIs. The performance of DGHE in supporting the procurement of the HEI grants was weak in the early stages. The performance of the implementation agencies is rated moderately satisfactory. The overall project outcome is rated moderately satisfactory. This is based upon relevance of the project objectives being high and the relevance of project design being substantial. The achievement of the first objective was substantial and the second one was modest. The project’s efficiency, in both external and internal measures, was substantial. The risk to development outcome is rated significant. Lessons A number of lessons can be highlighted based upon the experience of this project. If legislation is essential to the achievement of reform objectives, it also essential to have a risk assessment plan that outlines what measures could be taken if the legislation is not passed. In this project, legislation was essential to project success, which was made clear in early project preparation documents and then mentioned in risk xii assessment table of the PAD. Many people with whom the mission met thought that the project should not be held responsible for passing legislation since what happens in the legislature is beyond control of the project. While all stakeholders agreed that placing conditionality on the approval legislation was neither desirable nor feasible, the project design could have including more contingency planning. In situations where procurement is decentralized, it is never too early to start procurement planning and training to build up decentralized capacity. Procurement became a serious challenge early in the project. The HEIs were not ready to assume responsibility for procurement, especially given complicated Bank guidelines. Despite the fact that low procurement capacity was highlighted in the PAD, the project planned little in the way of capacity building. Delays in procurement can be serious enough to risk achievement of grant objectives and therefore the project objectives. The lack of capacity was one of the leading reasons that the tracer survey could not be carried out during project implementation, which weaked the M&E system. If preparation grants are available, pilot grants and procurements can be undertaken to test out the grant procedures and to harmonize Bank procurement and national procurement guidelines where possible. M&E planning should also begin early in project preparation, even during the analytic phases of sector work to establish what trend data is available as potentially useful as indicators. The indicators chosen for measuring achievement of PDOs, as is often the case, presented difficulty in attributing outcomes to project interventions, although arguments in terms of plausibility can still be made. No control groups were used or attempted. In such situations, which are common for complex projects, it is useful to assemble baseline trend data prior to the project interventions, as opposed to a single data point at the start of the project, such as GPA or time to first job. Also, it is important to assess the capacity of the country to measure important variables.Then resources needed to bolster this capacity could be included in the project. In addition, procurement can play a major role in developing an M&E system. The lack of procurement capacity delayed the implementation of the tracer survey, which is a critical element in higher education. Caroline Heider Director-General Evaluation 1 1. Background and Context 1.1 Country and Economic Context. When the Indonesia Managing Higher Education for Relevance and Efficiency Project (IMHERE) was undergoing preparation in 2004, Indonesia was going through a transition towards a more open and decentralized government. These changes resulted from reforms introduced after the Asian financial crisis in 1997-98 and affected delivery of public services, including education at all levels. The economy recovered from the 1997-98 financial crisis as a result of strong Government of Indonesia (GOI) reforms and financial support from the IMF, the Bank, and bilateral partners. From 2003 to 2012 the country had economic growth on average 5.7 percent annually versus an average of 8.0 percent from 1990-1996. However, the next global financial crisis in 2008-09 impacted Indonesia. Economic policy is now aimed at developing a more competitive and technologically oriented economy, raising the growth rate back to range prior to the financial crisis. This project was intended to contribute to the development of these critical goals. 1.2 Indonesia is the fourth most populous country in the world. However, 60 percent of the population live on the island of Java and another 22 percent on the larger island of Sumatra. The capital Jakarta, on the island of Java, makes up one of the world’s largest mega-cities having a population of roughly 10 million and much more if the larger metro- area is included. Population growth has slowed down to about one percent annually, although there is still some demographic momentum built into the labor force growth, which presents the challenge of job creation for a young cohort for some years to come. 1.3 Since independence in 1945, despite periods of low economic growth and high inflation, the economy of the country has generally made progress. Indonesia is now a middle income country with Gross National Income per capita around US$3,420 in 2012 and a population of 250 million, spread over a wide archipelago. With a Gross Domestic Product of US$884 billion (2012, at market exchange rates), Indonesia is also the largest economy in Southeast Asia and 10th largest global economy. With the start-up of the Association of South East Asia Nations (ASEAN) Economic Community in 2015, there will be more freedom of trade as well as movement for labor, requiring economic adjustments for Indonesia. Overall, this gives Indonesia considerable potential, but to take full advantage of this potential, many observers cite the need for increasing and upgrading its infrastructure for transport and telecommunications, reforming the regulatory environment for business, and improving its human capital, especially high-level professional and technical.1 1.4 Higher Education Sector. The structure of tertiary education has evolved rapidly over the past three decades. Starting out with only three public universities at the time of independence, the HE sector has developed to the point that there are now 83 public Higher Education Institutions (HEIs) and more than 3,000 private ones. Both private and public 1 Shortage of high level human capital as a constraint on growth was a persistent theme in discussions with individuals and organizations during the PPAR mission. Recent analytical work confirmed this, such as “Tertiary Education in Indonesia” (World Bank 2014), “Avoiding the Trap” (World Bank 2014), “Impact of U.S. Foreign Direct Investment on the Indonesian Economy” (USAID 2013), and “How Could Indonesia Improve its Productivity and Competitiveness” (Employers Association of Indonesia. 2013). 2 HEIs are required to the meet the standards of the National Higher Education Accreditation Board, which started in 1994 by a decree of the Ministry of Education and Culture (MOEC). 1.5 Tertiary enrollment was around 3.0 million in 2004, giving a participation rate of 12 percent, which was low compared to Southeast Asian countries. Enrollment in 2012 reached about 5.0 million for a participation rate of 24 percent. Private institutions enrolled 2.9 million, 58 percent of total enrollment. The number of university graduates doubled between 2004 and 2012. The structure of the HE sector is complex as shown in table 1.1. Table 1.1. Higher Education Institutions by Type (2010) Type Public Private Academies 0 1,015 Polytechnics 27 140 Advanced schools 2 1,314 Institutes 6 47 Universities 48 412 Total 83 3,011 Source: (World Bank 2014 based on MOEC data). 1.6 There are five categories of HEIs in the table above:2 (1) academies, which offer single profile training programs at the diploma level (below that of a university degree), which can range from a one year course (Diploma 1) to 3 years (Diploma 3); (2) polytechnics, which are often attached to a university and also provided a diploma level of training; (3) advanced schools, which also offer single profile training but usually at a high level of training (Diploma 4 for four years, almost equivalent to a university level in the single field); (4) institutes that can be a technological equivalent of a university; and (5) fully comprehensive universities, such as the older established universities, offering degrees from bachelor to doctoral level. Community colleges, which were recently authorized by the 2012 HE Law, are in the process of being established. The Open University, which has more than 500,000 students enrolled, caters mostly to working adults, most of whom are teachers seeking to upgrade their credentials. 1.7 Until the late 1990s administration of the HEIs was centralized within the DGHE, but as the trend toward decentralization developed after the financial and political crisis in 1997- 98, in HE this took the form of aiming for more institutional autonomy for the long established HEIs, with the aim of the less strong ones evolving towards autonomy over time. The rationale is that more institutional autonomy, which is a trend in many countries, would allow the HEIs to improve their quality and respond better to the economy and labor market and broader social needs, such as serious inequality of access to HE with the lower income quintiles, rural areas, and the outer islands being very under-represented in both public and 2 Many observers divide the university segment of HE into 3 tiers: the elite public universities (about 5 to 7), another 47-49 public universities of mixed quality, and another set of institutions administered by other government agencies. Although private universities are relatively recent, a few have been established for decades and are recognized for quality (see D. Suryadarma and G.Jones, 2013). 3 private HEIs. These developments in the mid-1990s became known as the New Paradigm for HE, and were formalized in the Higher Education Long Term Strategy (see below) and are cited in the Bank project documents. 1.8 Since autonomous public HEIs were established by special decree in 1999, the legal framework had become complex and marked by contradictions among laws and regulations, especially some regulations related to financial management of HEIs. Line-item budgets were the norm during the period of central control by DGHE, but block grant funding was introduced selectively during the 1990s for some donor financed projects involving competitive grants for specific purposes, like research. The Higher Education Long Term Strategy (HELTS) of 2003-2010 emphasized that a new legal framework was needed for HE so as to provide autonomy along with accountability to allow for a more effective response by HEIs to economic and social demands. This led up to the passage of the HE Law of 2009, which was soon overturned in 2010 by the Constitutional Court. Another law was drafted and passed by Parliament in July 2012 and signed by the president in August 2012. 1.9 HE sector within context of overall education system. The HE sector must be viewed in the context of the other sectors of the whole education system. Indonesia expanded lower levels of education rapidly during the late 1970s and 1980s, with its well- known school building program based upon financing from oil revenues. Primary education is now universal, with a gross enrollemt role of 112 percent (2010), up from around 80 percent in 1970. The gross enrollment rate for senior is around 63 percent (2010), up from 15 percent (1970). 1.10 The Bank did an analysis of education expenditures that indicated there is much room for improving how money is spent (World Bank 2013). The Indonesian constitution (amended in 2002) stipulates that 20 percent of the budget must be allocated to education at all levels and types. However, the government budget overall is constrained by the low level of tax revenues collected relative to Gross Domestic Product (15.2 percent). The decentralization of basic education finance since the early 2000s has also created a complex system of transfers that does not promote effectiveness of education spending. Within these complex issues of education finance, the appropriate allocation at all levels is a critical challenge. More autonomy for the HEIs can help in that they would have more scope for revenue generation and charging appropriate tuition, although increases in tuition have been resisted to some extent by some sectors of the public. 1.11 World Bank and other donors in education. The Bank has been involved in the HE sector since 1980 when the first HE project was initiated (see Time Line of events in Annex B) and thus had developed considerable experience with Indonesian HE. The initial projects were standard investment operations with pre-determined components and inputs, with a great deal of overseas fellowships to build up HE faculty. As the Bank developed its higher education policy (see “Higher Education: the Lessons of Experience” (1994)) and Indonesia’s thinking grew more sophisticated based upon experience, the New Paradigm era came about with substantial agreement on the main elements of reform. 1.12 This project is the fourth in a sequence of New Paradigm era projects (starting around the mid-1990s), which initiated new policies such as competitive grants financing and 4 encouraged more HEI autonomy and capacity development. These projects were: University Research for Graduate Education for US$109 million (approved 1994, closed 2001), Development of Undergraduate Education for US$102 million (approved 1997, closed 2002), and Quality of Undergraduate Education for US$98 million (approved 1998, closed 2004). Improving the quality and relevance of higher education was an objective in all of these projects, as well as efficiency. These projects were all completed with a satisfactory rating, despite the impacts of the 1997 Asian financial crisis which led to their actual costs being below the appraisal cost. Lessons learned in these projects, especially with regard to competitive grants for research, academic programs and institutional capacity were useful for the IMHERE project. These projects focused on the public universities and institutes, which comprise but one segment, albeit a very important one, of a larger HEI sector as described above. IMHERE, however, did allow private universities to compete for grants supporting teacher training programs. 1.13 In addition to the support for HE projects referred to above, the Bank has also been involved in the education sector as a whole, covering primary and secondary education, and recently providing more support for early childhood education. Quality of education at all levels became a concern, especially after a number of assessments, both domestic and international, showed that despite expansion of access and coverage, much more needed to be done in terms of learning achievement. In addition to an active portfolio of projects, the Bank has conducted an extensive program of analytical work across all levels of education. 1.14 The Asian Development Bank (ADB) is active in the education sector as a whole, ADB supported only one project in HE since 2000, the Technological and Professional Skills Development project (2000 -2006) which involved competitive grants. The major bilateral donors include Australia, Japan and USA. Australia is beginning a program of support to private HEIs for improvement in quality and relevance through a program of competitive grants to institutions based upon proposals they would submit. Japan has been active mainly in the scientific, technical and medical fields. Most recently its largest project was to support University of Indonesia and Institute of Technology in Bandung. The United States has had a number of programs with the HE sector. 2. Objectives, Design, and their Relevance 2.1 The project’s objectives (stated identically in the Credit Agreement and the Project Appraisal Document (PAD)) were: “to create an enabling environment for the evolution of autonomous and accountable public higher education institutions, and to develop effective support mechanisms for the improvement of quality, relevance, efficiency and equity of higher education”. The PDO provides a clear statement of objectives, which can be divided into two main objectives for the purpose of this assessment. Objective 1 involves the creation of an enabling environment for the evolution of autonomous and accountable public HEIs, and Objective 2 involves the development of effective support mechanisms for the improvement of quality, relevance, efficiency and equity. The project components and subcomponents are clearly aimed at “creating an enabling environment” and “developing effective support mechanisms”. There were no formal definitions given of these concepts in the project documents, but the indicators adopted and the description of activities implicitly 5 give specific operational meaning to objectives, such as improved learning indicated by increased Grade Point Averages (GPA). 2.2 The project as approved had three components: (1) Higher education system reform and oversight to support implementation of the new Higher Education Long Term Strategy (2003-2010) to develop autonomous and accountable institutions; (2) Grants for subprojects to improve academic quality and institutional performance; and (3) Project Management to support implementation of the project by the DGHE Implementation Unit. The design of the competitive grants part of the project was such that it involved mainly the public university segment of the HE sector, with some participation of private universities in grants for improving quality of teacher training programs. Diploma level training institutions, public or private, did not participate. However, the governance and accreditation part of the project was designed to impact the whole HE system. Table 2.1. Outline of Project Components at Approval Component 1: HE System Component 2: Subprojects for Reform & Oversight ($8.8 Academic Programs and Component 3: Project million) Institutions ($97.9million) Management ($7.8 million) (a) Legal Framework 2.1 Academic program Training and TA to support background work improvement grants (competition DGHE-IU to carry out project (b) Financial Management stratified by level of HEI oversight. within DGHE & public HEIs capacity). Subproject Grants for 3.1 Training for DGHE in its (c) NISHE capacity, including priority fields of science, new role. tracer survey (d) Policy engineering and teacher training. 3.2 Training HEI- evaluation Implementing Units for accountability. Accreditation shift to 2.2 Institutional capacity institutional basis instead of development subprojects for academic programs. autonomous and non-autonomous Support for the Accreditation HEIs Board. 2.2a. Competitive institutional subprojects for non-autonomous HEIs 2.2b. Institutional subprojects for autonomous HEIs 2.2c. Performance Based Contracts for autonomous HEIs. 1.3 Open University Revitalization Plan Source: Based upon Project Appraisal Document. 2.3 Component 1: Subcomponent 1.1 would modernize higher education sector oversight and management by implementing the legal framework for Higher Education Long Term Strategy. This would involve: (a) developing a revised legal framework for sector governance, (b) improving financial management within the DGHE and individual public HEIs, (c) expanding data collection capacity of the National Information System for Higher 6 Educations (NISHE) to include tracer studies of graduates, and (d) supporting a strategy to scale up reforms based upon evidence of effectiveness, especially financing innovations. 2.4 Subcomponent 1.2 would support a transition to emphasize institutional accreditation and licensing of professional fields as compared to the previous system of accreditation of study programs, which created overwhelming volume of reviews. This would make the load more manageable and is more aligned with the move to institutional autonomy. The National Higher Education Accreditation Board (BAN-PT, Indonesian acronym in project documents) had received support for this from ADB, which would be as part of this project. Professional licensing would be integrated into the BAN-PT accreditation system. BAN-PT would also acquire its own institutional autonomy, including generating revenue to support its mission. 2.5 Subcomponent 1.3 would develop and adopt a comprehensive revitalization plan for the Open University (OU) of Indonesia. The OU also reaches students in remote areas where HE options are limited. There were 35 Regional Learning Centers (RLCs) that deliver texts and exams across the country. The strategy and plan to revitalize the OU would be developed through this subcomponent in broad consultation with stakeholders. 2.6 Component 2: This component would have four categories of subproject grants: (1) Subcomponent 2.1- competitive grants for subprojects to improve quality in priority academic programs of engineering, science and teacher training (for which private HEIs were eligible to support improvement of teacher quality in basic and secondary education), (2) Subcomponent 2.2a -competitive grants for subprojects to strengthen institutional management in non-autonomous public HEIs, (3) Subcomponent 2.2b -proposal based subproject grants for strengthening institutional management at autonomous HEIs, and (4) Subcomponent 2.2c- performance based contract (PBC) grants for autonomous HEIs that successfully completed a grant under subcomponent 2.2b, qualifying them for a more advanced form of institutional funding that would be extended as autonomy progressed. The grant proposals also would address equity concerns by showing how the HEIs would recruit and provide scholarships for students from lower income families or from rural areas. 2.7 Component 3: This component would provide support to the Implementation Unit in DGHE to assist in overall project management, strategic planning procurement and financial management. The implementation units in each HEI also would need support and training in these same areas, but at the institutional level. Relevance of Project Objectives and Design 2.8 The relevance of project objectives to the country and sector strategies is rated high. The PDO was consistent with Indonesia’s development priorities at the time of appraisal and remained so until project closure. The Bank’s CAS in 2003, and updated again in the Country Partnership Strategy (2012), emphasized the need for improving the level of human resources in the context of developing economic competitiveness and technology innovation. The HELTS emphasized developing HEI autonomy and accountability as an effective way to allow HEIs to respond to economic and social needs while improving quality and relevance of HE programs. The project also addresses the CAS 7 objective of improving governance through building more effective and efficient public sector institutions. The more recent sector work underscores the continued relevance of the project objectives to the on-going challenges for the HE sector and the overall economy. 2.9 The relevance of project design is rated substantial. The project design allowed for appropriate decentralization of the HE sector through a new and strengthened legal framework while at the same time developing the capacity of individual HEIs to respond to incentives provided in the grants programs. In order to clarify the relevance of design to objectives, Table 2.2 below outlines the simplified Logical Framework, showing the causal chain from the bottom row of inputs to the next row of project outputs to top row of outcomes. In this framework, the outcomes are directly related to the two PDOs and the outputs are what the project components/subcomponents are responsible to produce. Some outputs can contribute to both PDOs, such as the institutional accreditation, which can help improve both accountability and quality. However, project design could have been stronger on the risk mitigation measures and also monitoring and evaluation. Table 2.2. Logical Framework of Project Objective 1: Create an enabling Objective 2: Develop effective support environment for the evolution of mechanisms for improvement of quality, autonomous and accountable public relevance, efficiency, and equity of HE. HEIs. Improved legal environment for Improved learning as indicated by GPA, autonomy and accountability. Less time to get job, Outcome Increase in number of autonomous HEIs. Less time to graduate, Improved accountability capacity via Greater participation of poor in higher improved audits and transparency. education Background studies to support drafting of Research new legal framework. Improved teaching Revised HE law and implementing More scholarships for poor students. Outputs regulations. New system of Institutional accreditation. Subprojects for institutional capacity Development of strategic plan to modernize development. and revitalize the Open University. New accreditation system Grants for sub-projects to equipment, Training, technical assistance and computers, training. Inputs computers provided where needed. Grants for student aid Technical assistance Source: Derived by author from PAD with some modification. The PAD treats the HE Law as an outcome, but here it is treated as an output. Also, grants here are used to mean more than just the finance inputs, but refer to the substantive subprojects, hence the use of the term “subproject grants.” 3. Implementation 3.1 The project was approved on June 9, 2005 and became effective December 20, 2005 for a total cost of US$114.5 million with an IDA credit of US$30 million, an IBRD loan of US$50 million, and the remaining US$34.5 million equivalent from the government contribution. All conditions listed for effectiveness, including appointment of head of the 8 implementation unit (DHGE-IU director), were satisfied. The project was rated satisfactory in the Implementation Status Reports (ISRs) throughout most of implementation, except for a brief period. Disbursements were slow initially, picked up after Mid-Term Review (MTR) resolved some procurement issues, and also after some restructuring of component activities. The project was due to close June 30, 2011, but this was extended by 18 months to December 31, 2012. The IDA credit was spent except for US$45,000. US$6.9 million were cancelled from the IBRD loan, mainly due to changes in Component 1. 3.2 Developing a revised legal framework that would enable the evolution of autonomous and accountable institutions was a major activity that culminated in the passage of a new HE Law in December 2009. The law was also intended to harmonize previous laws that sometimes contradicted each other. However, the new law was revoked by the Constitutional Court in March 2010 for a variety of reasons, including the opinion of the court that the law was poorly drafted and unclear and could be interpreted to “commercialize” HE, at variance with the constitutional provision for affordable access. This law was also criticized for being too broad with the possibility that it can be applied to all levels of education. A new law applying only for HE was drafted, addressing the issues raised by the Constitutional Court. It was passed in July 2012 and signed by the President in August 2012. Most of the implementing regulations have been completed and there are indications that this law will allow for more flexibility of institutional autonomy and accountability. However, during the period 2010 – 2012, the reversal of the law presented challenges and necessary adaptations of project activities, especially implementation of grants under Component 2. Given that the evolution of autonomous HEIs was a major objective, that part of the PDO was in jeopardy of not being achieved. 3.3 There were two studies envisioned under subcomponent 1.1 which could not be completed. The comparative study of differing funding mechanisms (line item, block grant and performance base contracts) was not able to be undertaken after the MOF put a stop to block grant funding. The tracer study under the NISHE experienced a number of problems that prevented its taking place before project completion. A data collection blue print was completed by 2010, but the Bank team had doubts about the questionnaire being useful for comparative purposes at the national level. Nonetheless, the NISHE informed the mission that they intended to complete and institutionalize the tracer study soon. Also, the mission was informed that the department for academic affairs within DGHE works directly with the HEIs in completing the data. They expect more cooperation from the HEIs than previously since the requirement to report to NISHE is part of the HE Law 2012. 3.4 Another challenge encountered during implementation was the 2009 decision by the MOF to prohibit block grant funding and require the return to annual line-item funding. This decision came about as a result of the MOF interpreting the financial and budget laws in a way that did not allow for block grants to HEIs, even though the practice had been allowed in the previous Bank supported HE projects. This line-item budgeting was counter to the goal of flexible management and created execution delays due to additional steps in the approval process. The project provided TA and training to the HEI staff to adjust budgets and timelines. 9 3.5 Procurement also proved to be a challenge during implementation. Part of this was due to the inexperience of the HEI staff with Bank procurement requirements and the lack of capacity within the DGHE-IU to review and assist so many of the HEI proposals. Initially, procurement documents were sent to the Bank without sufficient screening, causing back and forth while the documents were being reviewed. Another significant factor was the cumbersome procurement procedures of the Bank and long delays in the Bank’s review process. It was reported to the mission that sometimes the Bank procurement team took months to respond, during which time the bidders said they could not maintain their prices. These issues were analyzed and resolved after the mid-term review and project restructuring. 3.6 As a result of the MTR in September 2009 there was a Level II restructuring of the project to address implementation difficulties, but the PDO was not modified. Three activities were added under component 1: (1) the Global Development Learning Network (GDLN), which aims to connect univerisities through out the world and promote south-south collaboration, received funding to add six new learning centers to the existing network; (2) the OU received funding for more overseas fellowships to faculty members to strengthen its curricula and technical capacity, and (3) more TA, focusing on strengthening financial management, for DGHE . This allowed for more access to HE for remote communities via distance education. Under component 3, additional funding was provided to bolster procurement capacity in both the DGHE-IU and HEIs. The restructuring also addressed the 2009 MOF decision to stop block grants and return to line-item budgets. The extension of the project allowed additional time to make up for implementation delays. Most activities were implemented by project closing. Table 3.1 shows those components where the actual expenditure differs most from the planned. Table 3.1. Appraisal and Actual Costs by Component (US$ million equivalent) Appraisal Percentage of Estimate Actual Appraisal Component 1: HE System Reform 8.8 2.1 24.1% Component 2: Grants 97.9 82.7 84.5% Component 3: Project Management 7.8 7.7 97.8% Total Project Costs 114.5 92.5 80.8% Total Financing Required 114.5 92.5 80.8% 3.7 Fiduciary and Safeguard Issues. The project design called for attention to fiduciary issues such as financial management, procurement and corruption. Bank financial management specialists and procurement specialists prepared detailed provisions for the operation of the project. There was an anti-corruption plan prepared as part of the implementation plan for the project. The project did not trigger environmental or other safeguards. 10 4. Achievement of the Objectives Objective 1 4.1 The first objective was to create an enabling environment for the evolution of autonomous and accountable public higher education institutions. Based upon consideration of the outputs and outcomes as discussed below, Objective 1 is rated as substantial. OUTPUTS 4.2 For the purpose of assessing the M&E design, the output and outcome indicators as listed in the project Logical Framework is given in the M&E section (see Section 6), and they are referenced in the discussion below as needed. A major output for this objective was the HE Law in 2012 and associated implementation regulations along with the support provided by the project in the form background studies, seminars, study tours and special reports. Another output was the training provided in key areas of evolution of autonomy such as financial management and procurement. The transition to institutional accreditation (from the previous academic program based accreditation) has also added to the accountability of institutions in that HEIs will have to account to the public for the resources they bring to the educational enterprise as institutions. More than 50% of the procurement staff in 28 of 29 participating staff were certified in national procurement guidelines and practices and many staff also received training in Bank procurement, especially after MTR when procurement problems were addressed. The development and completion of the revitalization plan for the Open University was also a major output completed as part of subcomponent 1.3. 4.3 Another output was the development of a methodology for conducting a tracer survey of graduates, even though this was not yet fully institutionalized in the NISHE at project closing. The tracer survey was delayed due to difficulties in finding an appropriate firm. The survey was begun approximately one month before project closing. The NISHE informed the mission that it was close to finishing this task. Nonetheless, the mission in its visits found that individual universities used their own tracer information on the employment of their graduates and were more aware of the need to follow labor market conditions as a result of the project supervision efforts emphasizing the importance of labor market information. Another planned study comparing different methods of financing programs—traditional line item, block grants, and performance-based grants—could not be completed. This was due to the cancellation of block grant financing by the MOF. OUTCOMES 4.4 As a result of the HE Law, the legal environment has improved. The seven HEIs that previously attained autonomy through the ministerial decree retained that status and other strong HEIs were then able to see their way to attaining autonomy. Indonesian counterparts and observers described how the project’s technical assistance available helped in the technical background work for the law. The support of the project brought together major actors and stakeholders, which built consensus. This comes after having the first 11 attempt passed in 2009 having been overturned by the Constitutional Court (see implementation section). Nearly all of the regulations and decrees in the 2012 law were completed in the last two years as the momentum for reform carried over into the period after the project closure. To date, 11 universities have received the status of autonomy under the new law and now have a clearer legal environment within which to operate. As other HEIs develop their capacity, they can become eligible for the status of autonomy. Despite the twists and turns of the legal process over the past few years, there appears to be a guarded optimism at present among knowledgeable observers that the 2012 law can withstand potential legal challenges and provide the enabling framework for the evolution of autonomous HEIs. 4.5 Other outcomes that reflect progress in improving accountability. The 29 non- autonomous HEIs whose institutional sub-projects received grants developed a number of capacities, for example, improvements in financial and physical asset management, development of Standard Operating Procedures across the academic departments, development of strategic and business plans for the institution. Out of the seven autonomous HEIs receiving the second type of grant for institutional development sub-projects, six successfully completed them and were eligible to apply for the PBC type sub-projects. The project also granted addition grants (known as performance-based contracts) to five autonomous universities who applied for it. The application process for these grants was more rigorous and universities had to demonstrate their capacity to act autonomously with accountability. The universities that the were awarded grants proposed their own sub-project objectives and submitted their completion reports that documented the achievements of their targets. Although these targets varied, they covered research, student selection, revenue generating activities, employability, and social responsibility (including equity scholarships). Overall, the sub-projects reached 90 percent of the key performance indicators. 4.6 Accreditation was awarded to 52 public and 42 private universities on the basis of their overall institutional capacity and quality. The target for institutional accreditation was 5 percent of public and private HEIs (according to the PAD), which was easily exceeded for the public sector but not for the private sector (target was 150, achievement was 42). As part of accountability and showing transparency, 83 percent of participating universities published the results of their major procurements, compared to a baseline of none and a target of 100 percent. Unqualified audit reports were received in seven universities (versus a baseline of none and target of five). There was also an improvement in procurement, by 2012 86 percent of procurement by participating universities occurred within the bid validity period, compared to a target of 90 percent to be reached in 2010. At the time of the midterm (2008), 42 percent of procurement was carried in the bid validity period; by 2010, this was reported at 82 percent. Given the limited procurement capacity at the start of the project, this last indicator represents significant progress. The achievement of objective 1 is rated as substantial. Objective 2 4.7 The second objective was to develop effective support mechanisms for the improvement of the quality, relevance, efficiency and equity of higher education. Based upon the outputs and outcomes below, Objective 2 is rated as modest. 12 OUTPUTS 4.8 The major outputs aimed at achieving this end were the outputs of various sub- projects supported by grants (four categories) awarded to the HEIs. There were 38 competitive subproject grants for a total of US$42.8 million that were awarded under the category of grants to improve academic programs and equity. This was done in four batches or rounds of competition for academic program grants which put priority on engineering, science and teacher training (private HEIs could apply for teacher training grants). Improving quality of teacher training received the largest share (32 percent) of grants while engineering, agriculture and natural sciences each received 14 percent, and small percentages to other fields. The project also provided 41 institutional improvement grants, including five performance-based grants (the most ambitious type of institutional grant) to autonomous universities. The new accreditation system was an output that also supported improved quality as well as efficiency. Under the new system, the institutions would prepare a self- evaluation and strategic plan while undergoing an external evaluation. 4.9 There were three categories of subproject grants for improving institutional performance. There was one category for non-autonomous public HEIs to improve their finances, human resources, procurement and quality assurance to better prepare them for a move to autonomy. In this category, 29 grants for US$13.5 million were awarded out of 115 proposals submitted over three batches of competition. This type of subproject grant, which was new to the institutions, provided technical assistance for staff development, IT infrastructure and software, and policy studies. A second category of institutional capacity grant was provided to the seven autonomous public universities. Each grant required the university to make a formal proposal that was reviewed by a selection committee. The competitive process helped to improve the quality of the grants, however the government stopped providing competitive grants due to changes in the legal framework as well as changes in government priorities. The subproject grants are summarized in table 4.1. Table 4.1. Distribution of Subproject Grants in Component 2 of Project Amount No. of Subproject Percent of Award Type of Grant Awarded Grants Spent (US$ millions) Academic Programs 38 42.8 85% Institutional (Non-autonomous 29 13.5 86% HEIs) Institutional (Autonomous HEIs) 7 4.6 79% Performance Based Contracts 5 13.9 105% (Autonomous HEIs) Total 79 74.7 88% 4.10 The grants generally supported a wide range of activities, generating different outputs. For example, in the University of Bogor, the project supported four elements in a grant aimed at improving agricultures—the research led to 34 publications completed by 13 2011 with additional output expected. In total, the project supported 1,450 research projects (compared to a target of 1,380). This included support for students in preparing their graduation projects as well as support for research teams. OUTCOMES 4.11 Quality in HE is difficult to measure, even though many HE projects state quality among their objectives. As an indicator of improved academic program quality, as specified in the project results framework, grade point average was used as a proxy. This measure increased by six percent from 2.99 to 3.17 (compared to target of 3.13) between 2009 and 2012. This was done for those students participating in the academic quality grant program (under component 2.1), and there was no control group for comparison. Thus the possibility of other sources of increased GPA, such as grade inflation or the impact on quality of the previous HE projects (like Indonesia-Quality of Undergraduate Education), could not be ruled out. The mission also asked about using trend data about GPAs in the years before the grants, which was possible according to the HEIs that were visited, and which could have helped to spot grade inflation prior to the project. When asked about the improvement in quality due to the project, the response was unanimous that the project grants did contribute in a major way to improved quality. It is plausible to attribute some of the GPA increase to the subproject grants. 4.12 Quality in the HE literature generally involves the teaching and learning environment, research by faculty and students, and knowledge services to the broader society (for example, disseminating research through agricultural extension and consulting with local firms and government). There was general agreement during the visits to universities that quality involved improved teaching, learning and research along with better inputs, such as computers and equipment, and training for teaching staff. Many voiced the opinion that the final year research project, which is required of all students, showed definite improvement as students benefited from the grants to conduct their research. Ideally, some assessment of actual learning should be used, but it is difficult to develop standardized assessments of learning achievement. This problem of measuring learning achievement exists in the high income countries, which are just beginning to experiment with such standard assessments tests, like the OECD’s Assessment of Higher Education Learning Outcomes (AHELO)3. No Middle Income or Lower Income country has successfully done anything similar to AHELO and it would have been almost impossible for Indonesia to do so in such a short time. 4.13 An outcome indicator for the relevance and efficiency objective is time to find employment for graduates, which declined from the baseline 8.5 months to 5.8 months, versus a target of 7.6 months (which was a 10 percent reduction). Again, lack of controls make causality difficult to establish and to disentangle other factors such as economic growth and labor market performance. If tracer studies had been completed, it might have given more information to factor into an assessment of relevance. However, a recent Bank labor market study (World Bank 2014) and mission discussions with employers indicate there is still a serious mismatch in the labor market, giving rise to shortages of high level skilled 3 AHELO imeasures learning achievement in specific fields of study, to measure the quality of teaching and learning. Initially AHELO will focus on economics and engineering as well as assessing baic skills. 14 labor, which could be a factor in the decrease in time to get a job. As an indicator of internal efficiency, average time to graduate decreased from the baseline 56.7 months to 51.6 months versus the target of 54 months (which was a 10 percent reduction). 4.14 Beyond the official indicators, the mission also observed other evidence of improved relevance during its visits to universities and in discussions with other stakeholders. In addition to improving the labor market relevance in terms of priority fields (science, engineering and teacher training) as evidenced by the demand for science and engineering by firms, the larger universities are engaging in more entrepreneurial activities related to their research and new academic fields of study. The Universitas Indonesia used one of its grants to encourage IT business start-ups among its students. Universitas Gadja Mada used one of its grants to set up an IT company that provides services to other universities and businesses and already employs about 100 staff. The Faculty of Pharmacy at Universitas Gadja Mada is also developing herbal products to address a number of health problems, including diabetes and heart disease. While the mission was not able to conduct a large statistical sample, based upon the institutions visited, the evidence does point towards a change in attitude and culture taking place within the HE sector, especially in the stronger and more established HEIs. The grant recipients explained that these results, especially student research, would not be possible without the resources and incentives provided by the project. 4.15 The equity element of Objective 2 was considered a very important one by the both Bank and the Government. Each grant proposal had to indicate how the HEI would respond to the equity objective. Despite a problem in establishing a baseline for the indicator, some progress has been made, since this was clarified at MTR and the project team made efforts to improve. The original idea was to increase the share of low income groups to be 20 percent of enrollment. Available statistical and institutional data does not allow for this to be done year by year. The latest available data from household surveys (2010) showed that only 5 percent of the HE enrollment comes from the lowest income quintile. The HEIs have responded creatively by using outreach to the secondary schools to make low income families more aware of their possibilities. Some HEIs have experimented with the practice of home visits to assess the income status of the families since income tax or other official data do not exist. As evidence of its commitment to this equity objective, the Government has also instituted its own means-tested scholarship scheme based upon this project (Bidik Misi). The Government understands that one of the major constraint is the low share of low- income students who graduate from secondary. As a result of this experience, efforts to expand secondary school to poor, rural areas and distant outer islands are being increased. Overall, the achievement of Objective 2 is rated as being modest. 5. Efficiency 5.1 Efficiency is rated as substantial. 5.2 Overall, the project has produced significant benefits for beneficiaries and the Indonesian economy. Depending on assumptions, the net present value of the benefits to students would vary from US$47 million to US$164 million. This excludes efficiency gains to the universities as well as economic benefits for research and development. 15 External Efficiency 5.3 Investing in education generates individual and social benefits. Individual benefits generally include higher wages and more employment options. A higher education project may increase access, allowing more students to benefit from higher education. Likewise, a project may lead to higher income due to improved education relevance and quality. The social benefits includes the education’s contribution to improving labor productivity and promoting innovation. Education plays a major role in the study of economic growth. 5.4 The PAD included an analysis of the estimated individual benefits. The base estimate assumed that the project would benefit students through higher education quality, leading to a wage increase 15 percent, although several lower alternatives were considered. Based on the assumptions, the PAD estimates that the rate of return varies between 25 to 50 percent. The PAD makes clear assumptions about the value added of increasing quality on wages. 5.5 The ex ante analysis does not include any project contribution to increased enrollment as the result of scholarships targeted at poor students. Likewise, the analysis takes into the entire project cost, even though part of the project focuses on system-wide reform and capacity building or the support to the Open University. Likewise, it assumed it that many students would benefit from the project, despite the fact that the research grants were focused on certain academic departments. 5.6 The ICR calculates the ex post rate of return, based on project data. As with the PAD, this also focuses on the project’s contribution to existing students by improving quality, factoring in the effect on increased wages, reduced unemployment, and faster graduation time. The ICR adjusts many of assumptions made in the PAD, including a reduction of the number of beneficiaries to reflect students in Departments that directly benefited from research grants. The ICR makes clear assumptions of the value added of the beneficiary students; it assumes that students of departments which received a grant will earn the equivalent to the wage differential between a student finishing secondary school and a student finishing tertiary education, equivalent to about US$290 per year, which exceeds the amount estimated in the PAD (US$200). No explanation or evidence base is given for this assumed benefits stream. The ICR uses data showing a decline in education premium. More recent estimates seem to show the wage premium for higher education increasing, which may lead to a modest increase in the grant’s net present value (Purnastuti, Miller, and Salim, 2013). The estimates also updated the number of grants actually given (38 grants were actually awarded, compared to the planned 26 grants). Likewise, the costs were only based on expenditures for B1 grants (about 47 percent of total project expenditures), which reflects the fact that other elements of the project would not directly impact on education quality received by the students. The ICR estimates a discounted benefit from quality to be around US$131 million; if the returns are closer to the estimates in the PAD, this would reduce the discounted to around US$75 million to US$95 million. 5.7 Unlike in the PAD, the ICR also includes the additional income due to the reduced time needed to graduate and the reduced time needed to find a new job that resulted from the project’s interventions. Because the discount rate of 10 percent (same as in the PAD), this makes an important contribution as they are “front-loaded” benefits. 16 5.8 On the other hand, the estimates do not appear to consider the impact of the equity scholarships that were granted to 3,200 students. However internal project estiamtes suggest that only around 40 percent from the lowest income quintile (World Bank, 2013).. These scholarships were only available for new students and not for students already in the university. Assuming that 1,600 (50%) students were able to attend university because of the scholarships and using similar assumptions in the ICR, the net present value of these scholarships are around US$19 million. 5.9 The net present value of the benefits to students varies depending of the assumptions. Using a lower value of the benefits to improved quality and excluding other benefits, gives a net present value of US$47 million to US$67 million, leading to a rate of return of at least 16 percent. Including the assumption about the benefits from increased equity and efficient increase the net present value to US$107 million to US$127 million, with a rate of return of at least 24 percent. Including all of these benefits using the ICR’s estimation of the benefit from increased quality, this leads to US$164 million with a rate of return of 33 percent. 5.10 Finally, neither the PAD nor the ICR attempted to calculate the private or social benefits for investment in research. Studies have shown that the rate of return to research can be quite high, although estimates vary greatly (Hall, Mariesse, and Mohnen, 2009). The project did support research in strategic fields (such as agriculture, technology, and natural sciences) and that subsequently were published, suggesting at least some of the research had a positive impact. Internal Efficiency 5.11 Initially, the project suffered from slow disbursement as it took more time than expected for the universities to learn about how to implement the grant program. The disbursement schedule was updated during the mid-term review (2009) and the project generally disbursed in a timely fashion after the mid-term review. The IDA grant was fully disbursed, while part of the IBRD loan (14%) was cancelled. This was done because the demand for grants was less than expected. This is a prudent practice for uncommitted resources. The project closed with an 18 months delay due to problems in establishing the project with a context of a changing legal framework. 6. Ratings Outcome 6.1 The overall project outcome is rated moderately satisfactory. Based upon the evidence and analysis in Section 2, the relevance of project objectives was high, while the relevance of project design was substantial. Based upon the analysis of outputs and outcomes in Section 4, the achievement of the first objective was substantial and the second one was modest. Based upon the analysis in Section 5 of the project’s estimated benefits and costs, the project’s efficiency, in both external and internal measures, was substantial. 17 Risk to Development Outcome 6.2 At the time of project closing in December 2012, the second HE Law had just been passed. It was not clear how strongly the government might support the reforms. The implementing regulations were slow in being drafted and there was a tendency toward recentralization. However, at the time of the mission almost two years later there is evidence that government support for implementing the autonomy provisions of the law would be forthcoming. The implementing regulations have been completed and more universities (now a total of 11) have been granted autonomous stature. 6.3 The new government elected in July 2014 and inaugurated in October 2014 gave indications that it would support the HELTS over the next five years. Although the DGHE was merged into a new Ministry of Research, Technology, and Higher Education by the new government, there are no indications that HE will suffer from reduced attention. The proponents of this merger argue it will make for better coordination research and HE. Nonetheless, the reform agenda requires considerable financial resources and it is uncertain what resources can be forthcoming. The Bank will continue to provide analytic support and work with other partners. Given the remaining uncertainties in the political process, the risk to development outcomes is significant. Bank Performance 6.4 Overall Bank performance was moderately satisfactory. During preparation and at entry, Bank performance was moderately unsatisfactory, and during supervision it was moderately satisfactory. 6.5 Quality at entry is rated moderately unsatisfactory. The were a number of shortcomings in the preparation of the project. The main shortcoming was the lack of risk mitigation measures for the possibility that the HE Law might not be passed or that might be modified. The Constitutional Court did overturn the law in 2010 and the Ministry of Finance also blocked several aspects of the reform. Both of these risks were discussed in the aide- memoires during preparation but were not incorporated into the project design. 6.6 The Bank team also paid attention to fiduciary aspects by putting together a set of anti-corruption measures that led to an open and transparent procurement and contracting. Due to the decentralized nature of the grants and procurement to the HEIs, the project had strong financial management safeguards in place. The PAD placed emphasis on financial and procurement management in a number of annexes, making sure that publicity of HEI financial affairs and procurement contracts were subject to proper controls and publicized in the local media. At the same time, the project underestimated the need for capacity building at both the national and institution level to implement a complicated grant-based project. 6.7 Generally Bank provided strong technical support in the preparation of the project, including extensive analytic work, constructing a detailed project design that was aligned with the project objectives in most areas. However, one basic assumption that autonomy would lead to better outcomes, had not been tested in Indonesia, although there was some evidence about this in other countries. 18 6.8 Quality of supervision is rated moderately satisfactory. The project was supervised from the Country Office in Jakarta, and there were two co-TTLs during preparation, and the co-TTL based in the country office continued as TTL during the first years of implementation. This provided continuity and close cooperation with the project counterparts, and contributed to solving implementation problems as they arose. The second TTL also was based in the country office and assumed responsibility in 2008. 6.9 As procurement problems arose during the first years of implementation, the Bank team responded to resolve themThe mission found in its site visits that there was almost a universal complaint among the grant recipients about the complexity of Bank procurement rules and procedures. Grant team members reported to the mission that delays in Bank response sometimes meant that quotes from bidders were no longer valid and caused further delays in grant implementation. The Bank recognized the problem, which was partly due to insufficient procurement staff on site, and also due to the inexperience the HEI grant recipients in procurement. The Bank added more procurement staff in the country office and provided more training to DGHE and to the HEIs. This also became a major theme in theme in the mid-term review in 2008, which included capacity building at both the central and university level as well as changes in procurement procedures. DGHE reported to the mission that this made positive difference in helping them to implement the project. Borrower Performance 6.10 Overall the performance of the borrower is rated as moderately satisfactory. The performance of the government is rated moderately unsatisfactory while that of the implementing agency (DGHE-IU) is rated as moderately satisfactory. 6.11 The government gave strong support to the project during preparation and supported the overall HE reform strategy. The MOF and the National Planning Agency played key roles in support of the DGHE in preparing the project. However, after three years into project implementation (2009), MOF issued a decree that stated block grant financing could no longer be used in the project because this violated the Finance Law of 2003. It was explained to the mission that universities were not autonomous (in the view of MOF) but were part of the government according to Indonesian budget law, and one part of the government could not subsidize another part. The decision to apply the letter of the law was taken as part of a recentralization trend according to some observers. Whatever the reason, and despite the fact that the government had agreed to the block grant arrangement, the result was a bottle neck in project implementation that was in the end resolved. Thus the performance of the government is rated as moderately unsatisfactory. 6.12 The DGHE as overall implementing agency played a key role, although it also received support from the implementation unit established within the DGHE and the Higher Education Board (HEB), which is an important advisory body set up by special decree of the MOEC and made up of experienced senior academics. It was reported to the mission that the HEB functioned in some ways as a think tank to provide ideas. The Higher Education Institutions-Implementation Units (HEI-IUs) also were critical during implementation. In its role as coordinator of the overall project the DGHE-IU conducted coordinating meetings twice each year, provided TA to the HEIs as needed, and at times carried out special 19 seminars to develop the implementation capacity within the HEIs. However, procurement capacity in the DGHE was limited in the early years and incomplete procurement requests from the HEIs were passed on to the Bank procurement unit without adequate supervision, leading to procurement requests being sent back to the HEIs for better specifications of equipment and other issues. This, together with the slow response of the Bank procurement process, resulted in the delays during the first half of the project before being resolve at MTR. Thus the performance of the implementing agency is rated as moderately satisfactory. Monitoring and Evaluation 6.13 Monitoring and Evaluation Design. The design of Monitoring and Evaluation (M&E) attempted to capture each of the PDO elements—autonomy, accountability, quality, relevance, efficiency and equity. The key outcome indicators listed in the Results Framework of the PAD are:  The draft law on HE autonomy is to be passed by2010.  Tracer studies HE graduates to be conducted on regular basis by 2010.  Institutional accreditation to be awarded to five percent of all HEIs (public and private) by 2010.  Unqualified opinions by external auditors to be awarded to five public HEIs and 90 percent of procurement awarded within bid validity period by 2010.  A comprehensive evaluation comparing line item finance, competitive grants, and performance based contracts to be completed by 2010.  Unqualified audit opinions for autonomous HEIs The set of intermediate indicators included:  GPAs for students in the academic program grant (Component 2.1)  Reduction of time to graduate  Reduction of time to find a job after graduation  Percentage of poor students in academic grant recipient HEIs to increase.  Official Rector decree establishing internal audit function before signing of Grant agreements (for non-autonomous public HEIs)  Successful completion of PBC grants by those autonomous HEIs. 6.14 The Directorate of Higher Education-Implementation Unit and the HEIs, each of which formed their own implementation units (HEI-IUs), were mainly responsible for collecting the data for indicators, though some special studies were also envisioned, especially the tracer study of graduates. However, the design did not call for control groups of institutions or students nor did it use trend data (versus a single data point as baseline) prior to the project baseline to see how trends were evolving before and after the project. 6.15 While these indicators are important, they do not capture the project’s underlying logical framework. Most of the PDO level indicators focus on intermediate outputs. This was an attempt to measure institutional change. There were no measure of the project’s impact on teaching and research at the PDO level. However the intermediate level indicators did 20 include some measure of the project’s impact on teaching and employability, such as changes in the grade point averages and time to find a job. The results framework did not seem to have any indicator related to the production and input of research. 6.16 Monitoring and Evaluation Implementation. M&E activities were carried out on a regular basis with assistance by Bank missions, which helped in data aggregation and validation. There was some confusion on the part of DGHE and HEIs about the indicator for the equity component of objective 2 concerning the percentage of poor students. The intention was to aim at twenty percent of entering students should come from low-income backgrounds, but DGHE and the HEIs simply reported the percentage receiving scholarships irrespective of their income status. This was clarified and refined at MTR. 6.17 Early on in the project implementation, Bank management requested the Bank team to clarify PDO outcome and intermediate indicators as listed above. Some of the intermediate indicators seemed to be more appropriate as outcome indicators, for example the GPA. Also, indicators for relevance and efficiency could be improved. Thus, at MTR, new indicators were added. For the competitive academic grants (subcomponent 2.1) two new indicators were introduced at MTR--time to graduate and time to get first job, and other indicators, GPA increase and share of poor students, were given more precise specifications than what was given in the PAD. Some modifications of intermediate indicators were required to match available data, but this did not appear to have a major impact on the reported achievements. 6.18 Monitoring and Evaluation Utilization. Although the national tracer study for the NISHE was not completed, at the level of the individual institutions the collection of job experience data was used. It was reported to the mission that the DGHE now has the responsibility of collecting and organizing the tracer data and then delivering to the NISHE. The HEIs visited by the mission knew about the labor market experience of graduates and used the information in career guidance activities. Data on the agreed upon indicators were collected throughout implementation and used by supervision missions to gauge project progress and the need to do some adjustment (Level 2 restructuring) at MTR. The difficulties experienced with defining and collecting data on scholarships for low income led to refinements in that indicator and adoption of equity indicators by HEIs, which now makes visits to the homes to gauge the family situation. 6.19 The fact that comparison groups were not available presented difficulty in attribution of project interventions. It is interesting to note that the indicators for competitive grants (subcomponent 2.1) were used before in the DUE and QUE projects, and involved a comparison group (see the ICRs for these two projects). Using trend data a few years before the project start-up, which the PAD and also the HEIs visited indicated was available from the previous projects, would have been preferable to a single point in time for the base line data. Then it would be possible at least to compare the rate of increase in indicators such as GPA prior to the project compared to that after the project interventions. Thus, taking into account shortcomings cited above, overall the M&E is rated as modest. 21 7. Lessons Epilogue: Developments since Project Closing. 7.1 The HE sector is at an important crossroads. Starting with standard investment projects about 1980, the HE agenda moved onto the reform era of the New Paradigm in the mid-1990s. This led up to far reaching reform proposals as embodied in this IMHERE Project. It appeared that the momentum for reforms began to stall with MOF decisions in 2009 and the Constitutional Court reversing the HE Law in 2010. With passage of the new HE Law in 2012, some observers see the reforms regaining momentum, while others are still skeptical (Suriyadama and Jones, 2012; World Bank (2014). 7.2 Since completion of the project in December 2012, there have been major developments that overall give rise to a guarded optimism for the HE sector, but major challenges remain. The major developments are: the passage of the HE Law (2012): the formulation and issuance of more than 41 implementing regulations required by the HE Law; the elevation of 11 public universities to autonomous status; the improved institutional capacity of these universities, partly as a result of continuing and expanding the institutional developments started by sub-projects in Component 2 of this project; the progress made in the realm of accreditation; the development of a vibrant private HE sector, although quality is uneven. To date, there have been a few challenges to the HE Law by students and some private universities, but the Constitutional Court has upheld the law. 7.3 As has been pointed out, the HE Law covered more than just institutional autonomy (Smith 2012). Its scope was broader and allowed for other positive and needed developments. It considered the issues of quality assurance, access and equity. With respect to equity, and also labor market relevance, the law also establishes Community Colleges, which are intended to serve local and far outer island communities. The law also established the target that 20 percent of HEI students should be from poor backgrounds (the operational definition of this remains to be formulated) and remote disadvantaged places. It also opened up the possibility for establishment of foreign HEIs universities under certain conditions. They must obtain a license from the GOI, be accredited in their home country, cooperate with Indonesian HEI, give priority to Indonesian staff, and support the national interest. A number of foreign universities have expressed interest, as has been done in Malaysia and Singapore. It remains to be seen how the GOI regulates foreign institutions. 7.4 Yet some major challenges must be addressed. The most serious is the mismatch between supply and demand of skills needed for dynamic growth. This has only been partly addressed in the past few years, even with the support of this and other projects. The response of the public universities involved in this project is not large enough or fast enough to match the growth in demand as a result of the new investments anticipated in the next few years. Moreover, the public universities are only one part of a large and complex HE system. To get the balance right in the system as a whole requires a thorough analysis of the economy, labor market and skills development system. Added to this is the challenge of starting up the newly created Ministry of Research, Technology, and Higher Education that can provide the HE system with the resources and direction needed. 22 7.5 As noted in Bank’s 2014 HE-labor market study, finding signs of mismatch in the market is easier than solving them. A similar point was raised by economist Howard Pack in reviewing the experience of Singapore and South Korea—no country has solved the coordination problem between education and the economy (Pack 2007). Singapore industrialized using foreign manpower and later replaced the foreign manpower with local manpower as the labor market and economy signaled what skills and training were needed. South Korea, on the other hand, provided higher education before its industries were established and then suffered graduate unemployment along with the exodus of its highly educated labor. However, it was able to encourage many of them to return to South Korea as industrialization plans took hold. Lessons 7.6 If legislation is essential to the achievement of reform objectives, it also essential to have a risk assessment plan that outlines what measures could be taken if the legislation is not passed. In this project, legislation was essential to project success, which was made clear in early project preparation documents and then mentioned in risk assessment table of the PAD. Many people with whom the mission met thought that the project should not be held responsible for passing legislation since what happens in the legislature is beyond control of the project. While all stakeholders agreed that placing conditionality on the approval legislation was neither desirable nor feasible, the project design could have including more contingency planning. 7.7 In situations where procurement is decentralized, it is never too early to start procurement planning and training to build up decentralized capacity. Procurement became a serious challenge early in the project. The HEIs were not ready to assume responsibility for procurement, especially given complicated Bank guidelines. Despite the fact that low procurement capacity was highlighted in the PAD, the project planned little in the way of capacity building. Delays in procurement can be serious enough to risk achievement of grant objectives and therefore the project objectives. The lack of capacity was one of the leading reasons that the tracer survey could not be carried out during project implementation, which weaked the M&E system. If preparation grants are available, pilot grants and procurements can be undertaken to test out the grant procedures and to harmonize Bank procurement and national procurement guidelines where possible. M&E planning should also begin early in project preparation, even during the analytic phases of sector work to establish what trend data is available as potentially useful as indicators. The indicators chosen for measuring achievement of PDOs, as is often the case, presented difficulty in attributing outcomes to project interventions, although arguments in terms of plausibility can still be made. No control groups were used or attempted. In such situations, which are common for complex projects, it is useful to assemble baseline trend data prior to the project interventions, as opposed to a single data point at the start of the project, such as GPA or time to first job. Also, it is important to assess the capacity of the country to measure important variables.Then resources needed to bolster this capacity could be included in the project. In addition, procurement can play a major role in developing an M&E system. The lack of procurement capacity delayed the implementation of the tracer survey, which is a critical element in higher education. 23 References Berk, David. 2002. “Tertiary Education: Lessons from a Decade of Lending 1990-2000.” IEG Working Paper Series, World Bank, Washington, DC. di Gropello, Emanuela with A. Kruse, and P. Tandon. 2011. Skills for the Labor Market in Indonesia:Trends in Demand, Gaps, and Supply . Directions in Development Series,World Bank, Washington, DC. Employers Association of Indonesia (APINDO). 2013. How Could Indonesia Improve Its Productivity and Competitveness. Annual Report. Hall, Mariesse, and Mohnen, 2009. IEG. 2011. World Bank Support to Education Since 2001. Portfolio Note. Washington, DC: World Bank. Pack, Howard, and Marcus Noland. 2007. The Arab Economies in a Changing World. Peterson Institute for International Economics. Washington, DC. Purnastuti, Miller, and Salim, 2013. Smith, Christopher J. 2012. “The Higher Education Law: a Milestone Achieved but Still a Difficult Road Ahead.” Indonesia Economic Quarterly. World Bank Office Jakarta. October 2012. Suryadarma, Daniel, and Gavin Jones (editors). 2013. Education in Indonesia. Published by Institute for Southeast Asian Studies for Australian National University. Tong, D., and B. Waltermann. 2013. “Tackling Indonesia’s Talent Challenges: Growing Pains, Lasting Advantage.” Boston Consulting Group. Jakarta Office. USAID. 2013. The Impact of US Foreign Direct Investment on the Indonesian Economy. Report by Ernst & Young Indonesia commissioned by USAID. World Bank. 1994. Higher Education: the Lessons of Experience. Development in Practice Series. Washington, D.C. ______. 2001. “Implementation Completion and Results Report: Indonesia University Research for Graduate Education Project.” Washington, D.C. ______. 2003. “Implementation Completion and Results Report: Indonesia Development of Undergraduate Education Project.” Washington, D.C. ______. 2004. “Implementation Completion and Results Report: Indonesia Quality of Undergraduate Education Project.” Washington, D.C. ______. 2005. “Indonesia-Managing Higher Education for Relevance and Efficiency.” Project Appraisal Document. Washington, D.C. ______. 2012. Putting Higher Education to Work: Skills and Research for Growth in East Asia. World Bank Regional Report. Washington, D.C. ______. 2013a. Spendinng More or Spending Better: Improving Education Financing in Indonesia. Jakarta. ______. 2013b. “Implementation Completion and Results Report: Indonesia Managing Higher Education for Relevance and Efficiency Project.” Washington, D.C. ______. 2014a. “Indonesia’s Higher Education System: How Responsive is it to the Labor Market?” Jakarta. ______. 2014b. “Equity, Access and Success in Higher Education.” Jakarta. ______. 2014c. “Tertiary Education in Indonesia: Directions for Policy.” Jakarta. ______. 2014d. Indonesia: Avoiding the Trap. Development Policy Review 2014. Jakarta. ______. 2014e. “Akademi Komunitas: Current Developments and Future Challenges.” Policy Brief. Jakarta. 25 ANNEX A Annex A. Basic Data Sheet INDONESIA MANAGING HIGHER EDUCATION FOR RELEVANCE AND EFFICIENCY PROJECT (IBRD-47890 - IDA-40770) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 114.5 92.5 80.7 Loan amount (IBDR) 50.0 39.7 79.4 Loan amount (IDA) 30.0 30.3 101.0 Cancellation 0.0 6.9 0.0 Cumulative Estimated and Actual Disbursements FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 Appraisal estimate (US$M) 6.7 22.7 44.7 64.7 76.0 80.0 80.0 80.0 Actual (US$M) 3.6 11.2 16.0 26.6 39.3 57.8 73.1 70.0 Actual as % of appraisal 53.7 49.3 35.7 41.1 51.7 72.2 91.3 87.5 Date of final disbursement: 05/23/2013 Project Dates Original Actual Initiating memorandum 02/27/2004 03/11/2004 Negotiations 05/13/2005 05/13/2005 Board approval 11/30/2004 06/09/2005 Signing 08/02/2005 08/02/2005 Effectiveness 12/20/2005 12/20/2005 Closing date 06/30/2011 12/31/2012 ANNEX A 26 Staff Time and Cost Stage of Project Cycle Staff Time and Cost (Bank Budget Only) US$ Thousands (including No. of staff weeks Lending travel and consultants costs) FY04 10.94 69.51 FY05 67.26 333.49 Total: 78.2 403 Supervision/ICR FY06 20.06 132.31 FY07 16.93 79.14 FY08 18.98 76.24 FY09 18.66 67.75 FY10 18.43 80.54 FY11 19.42 57.50 FY12 20.72 78.68 FY13 14.04 93.16 Total: 147.24 665.32 Task Team Members Responsibility/ Names Title Unit Specialty Lending Ratna Kesuma Senior Operations Officer EASHE Novira Kusdarti Asra Sr. Fin. Management Specialist EAPFM Susiana Iskandar Senior Education Specialist EASHE Rizal H. Rivai Senior Procurement Specialist EAPPR Christopher James Smith Consultant EASPR Richard Hopper Education Specialist EASHD Yogana Prasta Sr. Disbursement Officer EASIF Rajiv Sondhi Sr. Fin. Manage. Specialist EAPCO Titie Hadiyati Consultant (Costing) EAPCO Kristian Thorn ET Consultant (Education) LCSHE Steven Burgess Sr. Soc. Devel. Specialist EASSD Joseph Burke Consultant (Performance Funding) External Juan Manuel Moreno Sr. Education Specialist HDNED Robin DePietro-Jurand Consultant (Higher Education) External Maria Jose LeMaitre Consultant (Quality Assurance) External 27 ANNEX A Responsibility/ Names Title Unit Specialty Menno Prassard Pradhan Economist EACIF Anne-Lise Klausen ET Consultant (Civil Service) EACIF Rosita Van Meel Sr. Education Specialist SASHD Shahid Yusuf Economic Advisor DECRG Omporn Regel Sr. Operations Specialist EASHD Elizabeth King Lead Economist EASHD Marlaine Lockheed Consultant (M&E) External Xiomara A. Morel Senior Finance Officer LOAGI Christina Sukmawati Program Assistant EACIF Supervision/ICR Ratna Kesuma Senior Operations Officer EASHE Task Team Mae Chu Chang Lead Education Specialist EASHE Novira Asra Sr. Fin. Manage. Specialist EAPFM Susiana Iskandar Senior Education Specialist EASHE Siwage D. Negara Operations Officer EASHE Rizal H. Rivai Senior Procurement Specialist EAPPR Christopher James Smith Consultant EASPR I Gusti Ngurah Wijaya Consultant EAPFM Kusuma Paulus Bagus Tjahjanto Procurement Specialist EAPPR Anita Buragohain Temporary EASHE William Hardi Consultant External Rimta K. Silangit Liaison Officer Susie Sugiarti Operations Officer EASHE Dhonke Ridhong Kafi Consultant EASFP Santoso Santoso E T Consultant EASHE Muhammad Farman Izhar Operations Officer/Financial EASHE Management Benedicta R. Sembodo Program Assistant EACIF Yvonne Trethewey Consultant – Training EASHE Halsey L. Beemer, Jr. Lead ICR Consultant EASHE ICR Author Chelsea Coffin Consultant – Eco/Fin Analysis EASHE ICR Author ANNEX B 28 Annex B. Timeline of Events-Indonesia Year National Event Education Event World Bank Indonesian Independence from Netherlands after World War 2. 1945 Sukarno, independence leader, becomes first president. 1965 Law allowing private universities. Suharto takes over from President Sukarno after political and military 1966 crisis of 1965. Era of New Order with economic development follows. First University Development project 1980 (first Bank supported HE project) Provision of limited public support 1982 to private universities Second University 1985 Development Project. Beginning of New Paradigm era Bank publishes Higher for Indonesian HE policy Education Policy Paper. development. National Accreditation Board 1994 (BAN-PT) University Research for Graduate Education project. Introduces competitive grants. Board of Higher Education 1996 established by ministerial decree to provide advice to MOEC. Development of Undergraduate Education 1997 project. Satisfactory completion in 2002. Quality of Undergraduate 1998 Education. Satisfactory completion in 2004. Asian Financial Crisis hits 1998 Indonesia hard as well as other countries in region. President Suharto forced to resign 1998 in the aftermath of financial crisis and demonstrations. 29 ANNEX B Year National Event Education Event World Bank Decentralization to regions. Decree giving autonomy status to . 1999 Election of President Abdurrahman six large and well-established Wahid by Parliament. universities. Technology and Professional 2000- Skills project funded by ADB. 2006 Sector loan also promoted competitive grants. President Wahid resigns. Megawati Sukarnoputri (daughter of former 2001 President Sukarno) elected by Parliament. Program Hibah Kompetisi funded 2002 by GOI to promote competitive grants in HEIs. Higher Education Long Term 2003 Strategy lays out principles of New Paradigm. Preparation of IMHERE project 2004 Election of President Susilo Bambang Yudhoyono in first direct popular vote. Re-elected in 2009. Approval of IMHERE 2005 project. MOF rules that block grants no Some observers see a longer allowed in public HE. “recentralization” of HE 2009 management by government. HE Law passed, but over-ruled by Constitutional Court one year later. New version of HE law passed by Parliament and signed by President. 2012 Sound legal foundation for HEI autonomy. Completion of IMHERE Elections result in President Joko Widodo coming to power. DGHE completes implementing regulations required by HE Law 0f 2012. 11 universities given 2014 autonomy. DGHE and HEIs are merged with Ministry of Research and Technology. Aim is to improve applications of university research. ANNEX C 30 Annex C. List of Persons Met Name Title Organization Government and Universities Dadang Sudiyarto Director, IMHERE Project Directorate General of Higher Education-Implementation Unit, IMHERE Project Akhmat Mahmudin Person Responsible for Activity “ Bagyo Y. Expert Staff “ Moeliodihardjo Biemo W. Soemardi Expert Staff “ T. Basarudin Expert Staff “ Abdul Malik Expert Staff “ Deny Kurniawan Head, Division Data Processing DGHE Didit W. Nugroho Secretary Higher Education Board Mr. Nizam Counsel member, former secretary Higher Education Board Sofian Effendi Counsel member “ Usman Chatib Warsa “ “ Johannes Gunawan “ “ Dr. Dwiwahaju Secretary National Accreditation Sasongko Agency for Higher Education (BAN-PT) Mr. Tormarbulang Head, Sub-Directorate of Loans & Ministry of Finance Lumban Tobing Grants Dr. Subandi Sardjoko Director for Education National Development Planning Agency (BAPPENAS) Prof. Dr. Suratman Vice Rector for Research and Universitas Gadja Mada Community Service Prof. Ir.Dwikorita Vice Rector for Cooperation and Alumni “ Karnawati Prof.Ir.Cahyono Agus FacultyForest Soil Science,ex-Dir. “ IMHERE PBC Grant Prof.Dr. Lilik Soetiarso Dean, Faculty Agricultural Technology, “ ex-Dir.IMHERE Grant Institution Capacity Dr. Head of Business Development and “ S.KompiangWirawan Incubation M.Afrizal Hernandar President Director, GMUM Holding “ Company 31 ANNEX C Dr.Eny Faridah Person-in-Charge,Center of Excellence, “ Forestry Dr. Hilda Ismail Person-in-Charge, Center of Excellence, “ Pharmacy Dr.Niken Person-in-Charge, Center of Excellence, “ Vice Dean, Biology Dr. Ria Armunanto Ex-Person-in-Charge, INHERENT “ Prof. Agung S. Vice Dean, Faculty of Pharmacy “ Diana Officer, Directorate of Planning “ Dr. Gatot F. Hertono Director of Academic Development and Universitas Indonesia GDLN Sitaresmi Ismangil Head, Subdirectorate for Coordination “ Academic Program Development, IMHERE Grant Institutional Development Dr. Gandjar Kiswanto Associate Professor, Faculty of “ Engineering, IMHERE Grant Mechanical Engineering Dr. Ade Azurat Lecturer, Faculty of Computer Science, “ IMHERE Grant Computer Science Prof. Dr. Ibnu Hajar Rector Universitas Negeri Medan Prof. Dr. Khairul Ansari Vice Rector I “ Dr. Chairul Azni Vice Rector II “ Prof.Dr. Biher Vice Rector III “ Prof. Dr. Berlin Sibalani Vice Rector IV “ Dr. Kustoro Budiarto Dean, Faculty of Economics “ Prof. Dr. Abd. Hamid K. Dean Faculty of Technology “ Dr. Basyaruddin Daulky Dean, Faculty of Sport Science “ Dr. Lu Ann Person in Charge (PIC) Accounting “ Major Chandra Situmeang “ “ OK Sofyan Hidayat “ “ Muhammad Ridha “ “ Habibi Dr. Selamet Riadi PIC Mechanical Engineering Major “ Zulian Heru PIC Sport Education “ Indra Darma Sitepu “ “ Dr. M. Nustan Hasibuan “ “ Yon Rinaldi PIC Capacity Building “ ANNEX C 32 Ismed Iskandar “ “ Edo Barlian “ “ Prof.Dr. Rochmat Rector Universitas Negeri Wahab Yogyakarta Dr. Wardan Suyanto Vice Chancelor I “ Prof.Dr.Zamzani Dean, Faculty of Language and Art “ Dr. Hartono Dean, Faculty of Math and Natural “ Science Dr. Rumpis Agus Head of BUPK “ Sudarko Dr. Setyo Budi Takarina Head of BAKI “ Dr. Djamilah Bondan Deputy Director IMHERE “ Yosa A. Alzuhdy Secretary IMHERE “ Erwin Setyo PIC Math PJKR “ Endang Listyani PIC Math Education “ Dr.Pangesti Wiedarti PIC PBSI “ Marsudi Waliyono Treasurer, IMHERE “ Sigit Cahyono Staff, IMHERE “ Nikke Yudha Dianita Staff, IMHERE “ Renny Dr. Margana Vice Chancelor, Expert Staff “ Nunik Sugesti Vice Chancelor, Expert Staff “ Armansyah Ginting Vice Chancelor II Universitas Sumatera Utara Erman Munir Secretary IMHERE “ Nasrul H. Bureau Chief. Planning and Cooperation “ Jasmin Kaban Internal Audit “ Rhuslly Siregar Internal Audit “ Sukardi Head of ULP/Procurement “ Rapido Gultom Maintenance Asset Staff “ Zulkahiri Nasution Finance Staff “ Sinta Dahlan Head of Finance “ Nainggolan Ahmad Hatib Bureau Chief, Maintenance and “ Development Asset Edwin Sitorus Vice Chancelor II Expert Staff “ Baihaqi Siregar Head LPSE “ Amar Rinanda Lubis Secretariat Staff IMHERE “ 33 ANNEX C Suci M. Isman Assistant to Rector Universitas Terbuka (Open University) International Agencies and Private Sector Andrew White Managing Director American Chamber of Commerce Sarah Howe Executive Director “ Jerry Strudwick Lead Education Specialist Australian Department of Foreign Affairs and Trade Sofjan Wanandi Chairman Employers Association of Indonesia P. Agung Pambudhi Executive Director “ Makiko Senju Representative Japanese International Cooperation Agency Fitri Diningsih Arifin Program Officer for ODA Loan “ Susiana Iskandar Initial Co-Task Team Leader, IMHERE World Bank Ratna Kesuma Final Task Team Leader, World Bank ManagerIMHERE Project Christopher Smith Consultant, IMHERE Project World Bank Andrew White Managing Director American Chamber of Commerce ANNEX D 34 Annex D. Borrower Comments