IEG Report Number: ICRR14786 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 08/05/2015 Country: Indonesia Project ID: P113341 Appraisal Actual Project Name: Health Professional Project Costs (US$M): 86.72 71.79 Education Quality Project L/C Number: Loan/Credit (US$M): 77.82 67.18 Sector Board: Health, Nutrition and Cofinancing (US$M): Population Cofinanciers: Board Approval Date : 09/24/2009 Closing Date: 12/31/2014 12/31/2014 Sector(s): Tertiary education (50%); Health (50%) Theme(s): Other human development (67%); Health system performance (33%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Judyth L. Twigg Pia Schneider Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Project Appraisal Document (PAD, p. 4) and the Loan Agreement (p. 5), the project development objective was "to strengthen quality assurance policies governing the education of health professionals in Indonesia, through: (1) rationalizing and assuring competency-focused accreditation of public and private health professional training institutions; (2) developing national competency-based standards and testing procedures for certification and licensing of health professionals; and (3) building institutional capacity to employ results-based grants for encouraging the use of accreditation and certification standards in the development of medical school quality. Since the main objective is broadly formulated, this Review will assess the numbered elements of the objective statement, or the specific objectives that are aligned with the specific outcomes intended by the project, while acknowledging the contributions of the specific objectives to the broad aim of strengthening quality assurance policies governing the education of health professionals. While the project's objective was not revised, some key outcome targets were formally revised at a February 2013 restructuring. For this reason, a split rating is performed in this Review. Per new information provided by the Bank team after the ICR was reviewed, at the time of restructuring, US$ 33.23 million, or 49.5% of the actual Bank contribution, had been disbursed. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives/key associated outcome targets? No c. Components: The project initially contained four components: 1. Strengthening policies and procedures for school accreditation (appraisal, US$ 7.18 million; actual, US$ 7.02 million). This component was to focus on improving the accreditation system of medical, dental, nursing, and midwifery schools to make it comparable to internationally recognized systems. An independent body (National Accreditation Agency, NAA) was to be established. The project was to assist in building consensus and developing accreditation policies, standards, processes, and outcomes, and in piloting the new system prior to legalizing it for nation-wide implementation. This was to include development of an accreditation strategic framework, accreditation instruments, assessor system recruitment and training, and a policy mandating publication of accreditation results for easy access by the general public. The project was also to support the Directorate General of Higher Education (DGHE) in synchronizing data collection and analysis for accreditation and for EPSBED, a self-evaluated study program designed by the DGHE to monitor the statistics of higher education institutions. 2. Certification of graduates using national competency-based examinations (appraisal, US$ 12.90 million; actual, US$ 14.61 million). This component was to support the establishment of an independent national evaluation center (National Agency for Competency Examination of Health Professionals, NACEHealthPro) to assure standards of education quality. The project was also to assist NACEHealthPro in establishing computer-based testing (CBT) and objective structured clinical evaluation (OSCE) facilities in about twelve medical schools that would function as regional centers. 3. Results-based financial assistance packages (FAPs) for medical schools (appraisal, US$ 61.4 million; actual, US$ 44.68 million). This component was to allocate FAPs for selected medical schools to meet accreditation standards. The FAPs were to be allocated competitively according to three schemes: one for ten leading medical schools to build their international reputations and strengthen Indonesia's global competitiveness; one to support 13 weak-capacity and seven new medical schools to achieve medical education standards through partnerships with leading medical schools; and one to support ten moderate-capacity medical schools in achieving medical education standards. A Board of Higher Education (BHE) was to be responsible for establishing guidelines for FAP recipient selection and overseeing implementation. FAP resources could be used for improving implementation of the competency-based curriculum, strengthening teaching/training/learning facilities, medical faculty development, or enhancing data management capacity. 4. Project management (appraisal, US$ 5.24 million; actual, US$ 5.48 million). A Central Project Coordination Unit (CPCU) was to be established, with the project financing incremental operating costs, a project management consultant, office equipment and furniture, and project monitoring and evaluation. In February 2013, a Level II restructuring expanded Components 1 and 2 to include three additional professions: pharmacy, nutrition, and public health. It also expanded Component 3 to include two additional universities that had not been successful in the original FAP competition, but that played strategic roles in meeting basic health services needs in Eastern Indonesia (these were known as "affirmative FAPs"). Funds were moderately reallocated across components in light of these changes. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Cost: Planned total project costs were US$ 86.72 million. Actual costs were US$ 71.79 million. Financing: The project was financed by a Specific Investment Loan, originally planned at US$ 77.822 million. US$ 67.183 million was actually disbursed (86.3% of planned), with US$ 10.69 million undisbursed. The final disbursement level was impacted by depreciation of the rupiah and efficiency gains in the procurement of goods (ICR, p. 16). Borrower Contribution: The government's planned contribution was US$ 2.5 million, and higher education institutions were expected to contribute US$ 6.4 million. The actual total contribution of the government and higher education institutions was US$ 4.67 million. Dates:  February 2013: Components were revised and funds reallocated as described above. Indicators and targets were revised.  September 2014: Following a government request, funds were reallocated from Component 3 to Components 1 and 4, in order to support school accreditation and various evaluation and learning activities prior to project closing. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of Objectives is rated High under both the original and restructured project. The project's objectives were highly relevant to country conditions at the time of appraisal. Total availability of health personnel was adequate, but there were issues surrounding their quality, skill mix, and distribution. Over half of the country's 7,000 physicians had graduated from private schools, whose rapid expansion had not been accompanied by improvements in quality standards. The rate of accreditation of medical schools was relatively low, and only half of medical school graduates were passing national examinations. The Medical Practice Act (2004) had established accreditation for medical and dental schools, but there was not a comparable regulatory framework for nursing and midwifery education. The objectives are also highly relevant to Government and Bank strategy, both at appraisal and at closing. Strengthening the quality assurance system of health professional education was one of the priorities of the country's 2010-2014 health sector medium-term development plan and its 2003-2010 Higher Education Long-Term Strategy. The development of a workforce capable of serving Indonesia's remote areas effectively is also critical to the country's Health Long-Term Development Plan (2005-2025). The Bank's 2009-2012 Country Partnership Strategy (CPS) contained health sector objectives aiming to improve quality and coverage of health services, both of which are served by increasing the quality of training for health professionals, as well as a sectoral core engagement component with a focus on strengthening human resource capabilities through medical and health education. The most recent CPS (2013-2015, p. 35) contains an explicit focus on strengthening policies governing the education of health professionals. b. Relevance of Design: Relevance of Design is rated Substantial under both the original and restructured project. The project's planned activities to assist with development of a regulatory framework for institutional accreditation, capacity building for accreditation processes, the development of a competency-based examination system, and provision of financial assistance to education institutions were logically and plausibly related to expected outcomes. The tiering of Financial Assistance Packages (FAP) for medical schools was appropriately intended to provide effective assistance to institutions at all stages of development, and the FAPs were also appropriately intended to garner political support from medical schools for the overall reform program. The ICR states that it was a "major limitation in terms of design" that the FAP grants to improve capacity to meet accreditation standards were to be allocated only to medical schools, even though the project was intended originally to target also dentistry, nursing, and midwifery, and additionally pharmacy, nutrition, and public health after restructuring (ICR, pp. 8, 17). The ICR (p. 8) explains this choice by noting that medical schools already had an incipient system of accreditation prior to the project, so that it was more cost-effective to focus only on them; that strengthening medical schools would incentivize other health professions to follow; that the investments made by medical schools on equipment and testing would also benefit other professions (for example, by sharing mannequins and computer stations); and that the challenges of conducting a competitive grant program for such a large number of schools (around 700 each for nursing and midwifery) were too large. These explanations do not address the fact that the project's broad objective was clearly intended to cover education of a wider array of health professionals. However, the project team later added that, at the time of project preparation, the other professional schools did not have the capacity to absorb FAP grants, and a follow-on project was envisaged that would have covered the other health professions with a block grant scheme; ultimately, this follow-on project was not possible due to changes in the country's borrowing policy that could not have been anticipated by the preparation team. 4. Achievement of Objectives (Efficacy): Rationalize and assure competency -focused accreditation of public and private health professional training institutions is rated Modest under the original targets and Substantial under the revised targets . Outputs:  132 trained assessors are in place for medical schools, surpassing the target of 60.  34 trained assessors are in place for dental schools, surpassing the target of 30.  123 trained assessors are in place for nursing schools, surpassing the target of 105.  70 trained assessors are in place for midwifery schools, not meeting the target of 105.  17 trained assessors are in place for public health schools, not meeting the added target of 40.  16 trained assessors are in place for nutrition schools, not meeting the added target of 40.  41 trained assessors are in place for pharmacy schools, surpassing the added target of 40.  The PAD and ICR provide contradictory information on the target for assessors for medical and dental schools. The project team later clarified the correct targets, which are listed here. Outcomes:  The National Accreditation Agency (NAA) was established and ratified in October 2014, with an adequate budget to conduct accreditation. Accreditation instruments were established and made ready for use for medicine, dentistry, nursing, and midwifery. There had been delays in establishing the necessary legal framework for the NAA, but the government implemented a task force to perform its function until it was officially approved.  21 medical schools went through the accreditation process, not meeting the original target (69) or the revised target (29). At the time of the ICR mission (February 2015), an additional four medical schools were undergoing accreditation.  9 dental schools went through the accreditation process, not meeting the original (24) or revised (10) target. The ICR provides contradictory information on this indicator, but the project team later clarified the correct information.  81 nursing schools went through the accreditation process, not meeting the original target (250), but surpassing the revised target (52). The ICR provides contradictory information on this indicator, but the project team later clarified the correct information.  62 midwifery schools went through the accreditation process, surpassing the original (59) and revised (33) target. The ICR provides contradictory information on this indicator, but the project team later clarified the correct information.  Overall, 173 schools went through the accreditation process, not meeting the original target (402), but surpassing the revised target (124). The ICR frequently refers to these indicators as schools receiving accreditation, but the project team later clarified that the indicator refers to schools going through the accreditation process. The project team later added that, during calendar year 2015, an additional 788 health professional schools are expected to be accredited, indicating substantial progress toward achievement of this objective.  The ICR does not address why accreditation results were less successful for medical schools than for the other health professions, given that the bulk of the project's resources were dedicated to strengthening only medical schools' capacity to meet accreditation standards. The project team later clarified that this was due to the passing of the Medical Education Act in August 2013 that required medical schools to have one license covering both pre-clinical and clinical training (previously, licensing for pre-clinical and clinical training had been separate). License renewal under this new law was more complex and prolonged, resulting in delays for medical schools to start the accreditation process. Develop national competency -based standards and testing procedures for certification and licensing of health professionals is rated Substantial under both the original and revised targets . Outputs: Standards of Competencies and Education were in place for medicine and dentistry prior to the project; these standards were added under the project for nursing, midwifery, pharmacy, and public health, with those for nutrition expected to be completed and trials conducted in 2015. National objective structured clinical evaluation (OSCE) trainers were put in place for medicine (4,950, far exceeding the target of 72) and dentistry (84, exceeding the target of 72). The ICR provides no information for pharmacy, where the target was 72, other than that the target was not met; the project team later stated that 36 trainers were put in place for pharmacy. 221 OSCE item writers were put in place for medicine, exceeding the target of 72; 650 for dentistry, exceeding the target of 72; 48 for nursing, exceeding the target of 36; and 39 for pharmacy, exceeding the target of 36. Targets for OSCE item writers and reviewers were not reached for medicine and dentistry, but were surpassed for pharmacy. 254 national multiple-choice question (MCQ) writers for computer-based testing (CBT) were put in place for medicine, 650 for dentistry, 828 for nursing, and 675 for midwifery, not meeting the target of 1,044 for each of these professions. However, 219 national MCQ writers were put in place for pharmacy, 141 for public health, and 144 for nutrition, far exceeding the target of 36 for each of these professions. Targets for MCQ writers and reviewers were similarly not reached for medicine, dentistry, nursing, or midwifery, but were surpassed for pharmacy, public health, and nutrition. Outcomes: The independent National Agency for Competency Examination of Health Professionals (NACEHealthPro) was legally established in December 2013, though its function as a task force under the CPCU began early in the project period. Objective structured clinical evaluation is now used for medicine and dentistry, and is under preparation for pharmacy, meeting the target. Computer-based testing was established for medicine, dentistry, and nursing, meeting the target, but midwifery still uses paper-based testing. The ICR reports the following achievement on competency tests:  The percentage of medical school graduates passing national competency testing at the first attempt increased from 71.7% in 2009 to 76% in 2014, not reaching the target of 84%.  The percentage of dental school graduates passing national competency testing at the first attempt increased from 81.7% in 2009 to 88% in 2014, exceeding the revised target of 83% but not the original target of 90%. The ICR provides contradictory information on medical and dental school graduates' testing results, but the project team later clarified the correct information.  The percentage of bachelor's degree nursing school graduates passing national competency testing at the first attempt was 57.8% in 2014, not reaching the target of 65%. For diploma nurses (a three-year nursing diploma), 47.8% passed at the first attempt, not meeting the target of 65%. For midwifery school graduates, the percentage passing in 2014 was 64.7%, essentially meeting the target of 65%. There was no baseline for nursing or midwifery school graduates, as national competency testing was newly established by the project. Build institutional capacity to employ results -based grants for encouraging the use of accreditation and certification standards in the development of medical school quality is rated Substantial under both the original and revised targets . Outputs: 43 medical schools received financial support to strengthen their programs, exceeding the target of 42. According to the ICR (p. 13), the two "affirmative" schools had low rates of disbursement and made "no signficant progress" in building capacity to improve education quality. The ICR contains no further information on the specific use of FAP resources and their relationship to resulting outcomes. The project team initially stated only that the FAP-recipient schools used the resources as outlined in the PAD (see Section 2c). Later, the project team also added that the FAPs supported scholarships for medical and biomedical sciences faculty to obtain 176 domestic master's degrees, 5 master's degrees abroad, 45 domestic PhD degrees, and 5 PhD degrees abroad; 23 staff exchanges and 55 student exchanges; nine visiting professorships; 87 participations in international seminars; 337 research grants; 283 teaching grants; and 152 bachelor's degree scholarships for students. The ICR also contains no information about the results-based nature of the grants. The project team later confirmed that FAP resources were released in tranches, based on performance under the previous tranche with technical results assessed by experts mobilized by the Board of Higher Education and Association of Medical Education Institutions. Outcomes: The mean national competency test scores of graduates from FAP-recipient schools increased from 60.61 in 2010 to 67.07 in 2014, a 10.7% increase. Graduates from non-FAP-recipient schools achieved a 11.7% increase (from 57.13 to 63.83) over this same time period. While the scores for FAP-recipient scores remained higher, non-FAP-recipient schools achieved a larger percentage increase, and the levels of improvement were roughly the same in absolute terms. (The ICR, p. 15, provides different information on these test scores, but the project team later clarified the correct data.) The project team later provided additional information on improvements in quality of medical education. The passing grade required for all medical school graduates (from project and non-project schools) taking the national competency test at first attempt was increased from 51 in 2010 to 66 in 2014, indicating a significant raising of the bar for quality standards. The gap in mean test scores between graduates of public and private medical schools supported by the project narrowed by 22% between 2010 and 2014, attributable in part to the project's deliberate support for weaker private schools. Among schools supported by the project, the mean test scores for graduates of leading medical schools improved by 11% between 2010 and 2014, of moderate-capacity schools by 10%, and of weak-capacity schools by 13%, again in part attributable to the project's design feature that partnered weaker with stronger schools. Overall, because the length of medical training (5.5 years) is longer than the project period, the full impact of the FAP grants will take additional years beyond the project's lifetime to be realized. According to the ICR (p. 18), some students reported that their schools focused more on providing short-term training for students to take these competency tests than on implementing meaningful curriculum changes. Also, some schools were reported to be selecting students to take the competency exams based on their likely pass rates. However, the project team stressed that these statements are based on limited anecdotal information, not on a systematic assessment, and are intended only to mark these issues for further monitoring. 5. Efficiency: Efficiency is rated Substantial. The PAD (pp. 13-14, 60-68) used Monte Carlo simulation methods to generate a distribution of anticipated total benefits, at the national level, to compare with costs. This approach employed reasonable, conservative assumptions. It yielded an estimated net present value of US$ 68.9 million over five years, and a gross benefit-cost ratio of 5.53. The ICR (pp. 29-32) employs a similarly conservative approach in terms of impacts and timeline for results, linking improved quality of training to improved quality of care over a practitioner's career and therefore improved health outcomes. It appropriately suggests only modest gains in quality of care, considers only dimensions of health improvement that are relatively easily monetized, and assumes only a five-year time horizon. The discounted total benefits are estimated at US$ 838.3 million, resulting in a benefit-cost ratio of 10.77. Although there were early implementation delays associated with changes in the structure of government ministries, delays in the approval of government budgets, and challenges surrounding the consensus-building and approval processes for the legal frameworks governing accreditation and competency exams, stakeholders remained engaged and the majority of planned project activities were implemented by project closing. The ICR (p. 16) reports that there were also gains from efficient procurement of goods, including computers for CBT and mannequins for OSCE, though it does not provide specific evidence of these gains. It is not clear how these gains occurred simultaneously with the project's overall procurement challenges (see Section 11b). a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Under the original targets: Relevance of Objectives is rated High, with the objectives highly relevant to country conditions, Bank strategy, and Government strategy. Relevance of Design is rated Substantial, with planned activities logically and plausibly linked to intended outcomes. The objective to rationalize and assure competency-focused accreditation of public and private health professional training institutions is rated Modest, as targets for schools undergoing accreditation processes were not met. The objective to develop national competency-based standards and testing procedures for certification and licensing of health professionals is rated Substantial, as OSCE and CBT were put in place for medicine, dentistry, nursing, and midwifery. The objective to build institutional capacity to employ results-based grants for encouraging the use of accreditation and certification standards in the development of medical school quality is rated Substantial, with student test scores indicative of improvements in the quality of medical education and plausible linkages between FAP-financed activities and these outcomes. Efficiency was Substantial, with a benefit-to-cost ratio at closing over twice that at appraisal, and significant gains through cost-effective procurement of computers and other equipment. These ratings are indicative of minor shortcomings in the project's preparation and implementation, and therefore an Outcome rating of Satisfactory. Under the revised targets: Relevance of Objectives, Relevance of Design, and Efficiency are the same as under the original targets. However, the downward revision of targets for institutional accreditation processes renders the achievement of the first objective Substantial, while the ratings for the other two objectives remain also Substantial. These ratings also indicate minor shortcomings, and therefore an Outcome rating of Satisfactory. The final Outcome rating is therefore Satisfactory. a. Outcome Rating: Satisfactory 7. Rationale for Risk to Development Outcome Rating: The legal framework, processes, instruments, manuals, training materials, and modules required to implement accreditation and competency-based examination are firmly in place, and are slated by the Ministry of Health to be applied to additional health professions (ICR, p. 12). The project's information management system will continue to serve the accreditation and examination agencies. The project increased the level and depth of collaboration between the Ministry of Health and the Ministry of Research, Technology, and Higher Education. However, the project provided direct benefits for only a subset of schools even within the seven professions it covered. Scaling up the accreditation and examination effort to cover all health professional schools will require the development of additional accreditation instruments, expansion of training of assessors and validators, measures to ensure the financial and operational sustainability of the accrediting and examining agencies, and monitoring of the impact of the new systems on the labor market for health professionals. Financial risk to the examination process is relatively small, as professional associations and students cover most costs. But funding for the accreditation body is not stable, and according to the ICR (p. 19), government support will be needed beyond 2015. Changes in government may impact political and consequently financial support. Both agencies face challenges in recruiting staff with appropriate qualifications in health education. Finally, there is institutional risk that schools will continue to pursue opportunistic behavior by focusing on training students specifically for competency exams rather than strengthening their overall curricula. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: Project preparation took into account lessons learned from previous higher education projects in the country, including the difficulties in linking financial assistance investments to outcomes, limited utility of financial assistance to support improvements in teaching and learning quality, challenges in maintaining transparency and avoiding conflicts of interest, and the risk of slow disbursements due to weaknesses in management arrangements for financial assistance (PAD, p. 6). Risk assessment was careful and mitigation measures sufficient, with the only high or substantial risk being FAP recipients' lack of familiarity with Bank procurement guidelines (PAD, pp. 11-12). Analytical work included a stakeholder analysis to map key actors, their interests, and potential courses of action (ICR, p. 8); this was particularly important in the Indonesian context, where the private-for-profit sector had been dominant and the process of reforming the accreditation and certification systems highly politically charged. Experts from both the health and education sector were engaged. M&E design was strong (see Section 10a). A Quality Enhancement Review in May 2009 raised issues regarding variance among recipient schools' institutional capacity and incentives to use the FAP grants; these issues were to be addressed through partnering stronger with weaker schools. Also, according to the Borrower's ICR (p. 37), project preparation did not anticipate the need to implement two legislative acts (Higher Education Act No. 12/2012 and Medical Education Act No. 20/2013), which significantly impacted project timing and execution; the project team later stressed that project design could not have anticipated the length and depth of the debates within the Parliament and related cases brought to the Supreme Court. Quality-at-Entry Rating: Satisfactory b. Quality of supervision: There were only two Task Team Leaders across the entire project, and they were based full-time in Jakarta for the project's entire duration. Appropriate specialists were consulted throughout. Challenges were addressed as they surfaced, including the delays in channeling of government resources to the DGHE, the lengthy process for establishing the legal basis for accreditation and examination agencies, and the complexities of managing disparate government and non-governmental entities. The discussion of modification of targets at the mid-term review appropriately recognized the impact of those challenges. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government demonstrated strong commitment and ownership throughout the project, engaging multiple ministries in the process of developing the accreditation and testing systems even across successive administrations. Implementation challenges resulted from the change of the ministry structure from the Ministry of National Education to the Ministry of Education and Culture in 2011; delays in the transfer of resources from the Ministry of Finance to the Ministry of Education and Culture due to delays in approval of the budget; and a lengthy process for building consensus around and finalizing the legal framework for the functioning of the NAA and NACEHealthPro. The government compensated for these difficulties by implementing two task forces to exercise the functions of the two agencies until the necessary legal frameworks were in place, highlighting the level of importance the government assigned to project activities. Government Performance Rating Satisfactory b. Implementing Agency Performance: The Directorate General of Higher Education was the project's main implementing agency. A project steering committee with cross-agency representation was functional throughout implementation. Project Implementation Units (PIUs) were established at each university prior to the signing of a FAP contract to support implementation and administration. Although limited by scarce human resources, the DGHE was able to implement and monitor the project in a satisfactory manner, with particular skill in building consensus and sustaining engagement among disparate entities with often conflicting views. However, the performance of the PIUs varied considerably from school to school, particularly concerning the mandated partnerships between stronger and weaker schools. There were challenges with financial management and procurement; these were effectively addressed by the CPCU (see Section 11b). Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The PAD (pp. 22-28) contained a detailed results framework, with PDO-level and intermediate outcome indicators clearly linked to the project's objectives. An implementation timeline with complete baselines and targets was established. Arrangements for results monitoring were appropriately detailed, with a different agency responsible for collecting and reporting data for each component, and the CPCU in charge of overall M&E. b. M&E Implementation: The CPCU provided consistent and updated information on implementation progress. A Database for Health Higher Education monitored the number of schools going through the accreditation process and the number of students taking and passing CBT. FAP-funded schools recorded progress on their individual websites, and the CPCU regularly sent teams of experts to monitor the schools' technical and managerial progress. c. M&E Utilization: According to the ICR (p. 10), the M&E framework allowed monitoring of identified risks and enabled action to respond to those risks. No specific examples are provided. M&E Quality Rating: Substantial 11. Other Issues a. Safeguards: The project was rated Category C. No safeguard policies were triggered (PAD, p. 16). b. Fiduciary Compliance: Financial management was strong during the early period of the project. After it was determined that FAP recipients varied widely in their capacities for financial and procurement management, the CPCU mapped implementation capacity of the grantees and provided support based on each specific need (ICR, p. 11). Later in the project, moderate financial management challenges were encountered in several areas: challenges in recruiting qualified financial management consultants; delays in submission of financial management reports; delayed audit follow-up; and uncorrected disbursement discrepancies between Bank and project records. Procurement challenges resulted primarily from FAP recipient schools' varying capacities to implement procurement packages, resulting in delays. As a result, the project management consultant was replaced with a procurement consultant, the CPCU mapped grantee capacity, and procurement monitoring was enhanced, improving disbursement rates. c. Unintended Impacts (positive or negative): According to the ICR (p. 18), quality assurance systems developed for higher education are now being considered for use as models for other fields, including engineering. The project helped to establish a network of associations of health professionals, professional schools, government entities, students, and broader civil society in the health field. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome: Bank Performance: Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately The performance of Project Satisfactory Implementation Units varied considerably among medical schools receiving FAP grants. Quality of ICR: Unsatisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (pp. 23-24) provides several lessons, adapted here:  Strong collaboration between government and non-government entities is essential for higher education reform processes. In this case, involvement and commitment of professional associations and associations of professional schools was key to the development and ratification of accreditation and examination standards and processes.  When providing incentive grants, variation in interest and capacity must be taken into account. In this case, there were significant differences in participation, disbursement, and results among FAP-recipient medical schools because these factors were not adequately considered.  A country's legislative environment will strongly govern efforts to introduce new accreditation and examination standards, and legislative approval can be a lengthy and challenging process. Adequate political economy analysis is essential for understanding these processes, and a project's timeline should taken them into account. 14. Assessment Recommended? Yes No Why? To verify data, and to learn lessons from the project's consolidation of stakeholder support for a politically challenging reform and from the transfer of skills from the supported medical schools to other health professional schools. 15. Comments on Quality of ICR: The ICR is concise and clear, and for the most part it follows OPCS guidelines. However, it contains numerous data discrepancies and omissions. In many instances, different values are reported in different places for the same indicator. The ICR is also unclear about the timing and nature of revision of key outcome targets, including the level of disbursements at the time of restructuring. The ICR does not address key analytical issues, most importantly the reasons for the failure of medical schools to meet targets for undergoing accreditation processes, and very little information is provided on the specific use of FAP resources. Lessons are framed as summaries of the project's experience, rather than as lessons potentially applicable to future projects. Most importantly, the project team later added important information that was essential to a fair and accurate assessment of the project; this information was available at the time the ICR was written and should have been included. a.Quality of ICR Rating : Unsatisfactory