IEG Report Number: ICRR14695 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 03/31/2015 Country: Argentina Project ID: P113896 Appraisal Actual Project Name: Ar San Juan Swap Project Costs (US$M): 109.55 108.30 L/C Number: Loan/Credit (US$M): 50.00 48.65 Sector Board: Health, Nutrition and Cofinancing (US$M): Population Cofinanciers: Board Approval Date : 03/16/2010 Closing Date: 03/31/2012 03/31/2014 Sector(s): Health (46%); Primary education (37%); Public administration- Health (11%); Central government administration (5%); Public administration- Education (1%) Theme(s): Health system performance (57%); Education for all (38%); Public expenditure; financial management and procurement (4%); Debt management and fiscal sustainability (1%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Hjalte S. A. Sederlof Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: The Project Development Objective (PDO) in the Loan Agreement (p. 5) is (a) to improve the equity, quality and efficiency in the provision of health services; (b) to improve the equity, quality and efficiency of education services in the first years of schooling; and (c) to strengthen the Borrower’s public sector management. The Project Appraisal Document (PAD, p. 5) had the same description of the PDO. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project supported the provincial government of San Juan by addressing programs in its development strategy, which aimed at providing better outcomes in health and education in the province. To do so, it helped finance salaries for personnel working on six of the province’s expenditure programs – four in public health and two in education – as well as the cost of related technical assistance. The release of Bank funds for these purposes was contingent on the achievement of a set of disbursement-linked performance indicators (DLI). Component 1: Health component (estimated cost at appraisal US$ 94.70 million; actual costs US$ 95.65 million). The component financed personnel expenditures for the implementation of the following expenditure programs in the health sector: a) Personal Health Care Program. This program aimed at enhancing the institutional capacity of service providers to improve quality in healthcare delivery. It sought to implement a quality policy based on modern administrative and managerial technologies in health care provision, new clinical technologies, and continuous personnel training. b) Public Health Prevention and Promotion Program. This program was designed to strengthen the implementation capacity of the Ministry of Health to deliver health prevention and promotion activities. The program covered the following sub-programs: (i) immunization; (ii) safe blood; (iii) tuberculosis; (iv) addiction prevention (alcohol, tobacco, illegal drugs); (v) local health service prevention (supporting healthy’ municipalities) and regulation seeking to ensure adequate health care quality standards; and (vi) health promotion. Activities were related to improving managerial capacities and redesigning programs to achieve better performance. c) Public Health Management Program for Service Providers. This program focused on improving service delivery management in key public sector providers and supporting gradual decentralization and de-concentration of health services. It comprised four sub-programs: (i) support to public health networks; (ii) support to deconcentrated and decentralized health services; (iii) quality control policy; and (iv) personnel training. d) Public Health Management Program for the Ministry of Health. This program focused on strengthening the institutional capacity of the Ministry of Health. It comprised two main sub-programs: (i) quality control of decentralized and de-concentrated units; and (ii) administrative management. Component 2: Education component (estimated cost at appraisal US$ 11.70 million; actual costs US$ 11.60 million). The component financed personnel expenditures for the implementation of the following expenditure programs in the education sector: a) Program for Assisted Promotion. This program sought to ensure that all children who attend the first cycle of primary education (grades 1-3) achieve good learning results without failing any of the early grades. It was to do so by improving applied didactics and efforts to meet the needs of failing students. Participating schools were those serving students from high-risk areas, as well as those with levels of repetition and overage students that were higher than the provincial mean. b) Full Day Program. This program supported schools in urban and remote areas with vulnerable populations. Schools were included on the basis of weak performance and target population characteristics. The program was to increase the time children spend in school by extending the curricular schedule for the first and second cycle of primary school and the first cycle of secondary school. The aim was to improve student participation and performance, as well as parent participation in the schools, and to provide health care and social services. Ultimately, this was to facilitate social inclusion. Component 3: Technical assistance for public sector management (estimated cost at appraisal US$ 3.00 million; actual costs US$ 940,000). This component was to support more effective and efficient public sector management in agencies that were critical for implementing Components 1 and 2. It included:  Strengthening public sector management by focusing on: (i) administrative processes and the internal audit function, coupled with training; (ii) the effectiveness of public procurement; (iii) piloting of program budgeting; (iv) planning, monitoring and evaluation capacity in selected pilot institutions; and (v) internal control functions in selected pilot institutions.  Strengthening management of public health care by: (i) strengthening institutional capacity in the Ministry of Health (MOH) to provide better health care to mothers, children and youth through staff training; (ii) developing a database on health coverage; (iii) developing inter-facility information systems; (iv) strengthening public hospital management; (v) developing a social communications strategy for health promotion and reforms; (vi) developing a quality control strategy; and (vii) developing a medical stock inventory system.  Strengthening managerial capacity at the Ministry of Education (MOE) to: (i) design/redesign, monitor, evaluate and implement education programs; and (ii) expand the full-day program by developing a training of trainers program.  Strengthening project management in the project implementing agency, San Juan Investment Promotion Agency (ASJDI), through technical assistance and training to carry out its functions, including coordinating project implementation, establishing an information management system, managing the technical assistance component, and carrying out sector studies on investment promotion. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project costs. Total project costs were estimated at US$ 109.55 million at appraisal; actual costs at closing were US$ 108.30 million. The amounts included a front-end fee of US$ 125,000. Project financing. The Bank provided US$ 50 million, including the front-end fee. A total of US$ 48.65 million was disbursed. The difference reflects a reduction in the technical assistance component. Borrower contribution. At appraisal, the Borrower contribution for the project period was estimated at US$ 59.55 million; at closing it was US$ 59.65 million. Dates. The project was approved on March 16, 2009 with an original Closing Date of March 31, 2012. The Closing Date was extended twice: a first time from March 31, 2012 to September 30, 2013; and a second time from September 30, 2013 to March 30, 2014, at which time the project closed. The first extension became necessary due to an eighteen month delay in effectiveness, as the Federal Government experienced unforeseen difficulties in passing the decree authorizing the Loan. At the same time, the target date for meeting the disbursement-linked indicators for health was shifted from end-2011 to end-2012. The indicators remained unchanged. This shift allowed technical assistance supporting implementation of the health expenditure program to begin to take effect, as start-up of technical assistance had been delayed awaiting effectiveness. A similar adjustment was not done for the education expenditure program, ostensibly because of slow data reporting procedures that would not allow up-to-date monitoring (ICR, p. 4). The second extension involved the reallocation of US$ 700,000 from the technical assistance component to the health (60 percent) and education (40 percent) expenditure programs for the purchase of vehicles and laptops. The extension was to allow sufficient time to undertake the procurement process. However, the money was not spent, as the procurement did not generate sufficient interest among bidders and was cancelled. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of objectives is rated high. The PDOs focused on improving performance in the public health and education sectors in the province of San Juan. In doing so, it was directly relevant to both federal and provincial development agendas that emphasize promoting social inclusion and poverty reduction. Provincial governments, such as San Juan, play a major role in the provision of public services, including health and education. In San Juan, human development indicators in health (infant and maternal mortality) and education (primary level dropout and repetition rates) were below the national average, and health, education, and public sector management were identified as priority policy areas in the province’s development strategy (PAD, p. 2). The PDOs were also broadly in line with the Bank’s most recent Country Program Partnership Strategy (CPS) for FY15-18. While the Strategy does not explicitly address the need to improve equity, quality or efficiency in the provision of health and education services, these are essential ingredients in any effort to improve human development outcomes and are quite in line with key objectives of the CPS to support improvements in health and education outcomes. The use of a sector wide approach (SWAp) also fits within the objectives of the CPS which calls for multi-sectoral and performance-based operations. The PDOs are also in line with the Strategy’s emphasis on providing direct support to provincial governments in building capacity for public service delivery. b. Relevance of Design: Relevance of design is rated Modest. The project supported the implementation of key health and education expenditure programs listed in Section 2. It used a Sector-Wide Approach (SWAp), with the Bank loan tied to disbursement linked indicators (DLI), and technical assistance to build management capacity in the four participating public agencies. The SWAp was based on comprehensive sector programs. This was reflected in the PDOs focus on equity, quality and efficiency, which were all characteristics that were likely to be affected by implementing the programs. The introduction of DLIs, tied to minimum annual program expenditure levels and annual performance targets in health and education, encouraged maintaining focus on and provided additional incentives to advancing reforms in the two sectors, and it facilitated monitoring. In the health sector, support for the eligible expenditure programs was designed to complement ongoing Bank-supported health projects: Essential Public Health Functions and Programs, Plan Nacer (Loan 7412-AR) and two Provincial Maternal-Child Health APLs (Loan 7225-AR and Loan 7409-AR). The project included a results framework broadly linking the PDOs to outcome indicators and supported by intermediate indicators. The indicators formed the basis for the DLI structure, and meeting the targets was to contribute to better health and education services. However, the framework did not provide a clear causal chain between Bank funding and intended outcomes - the indicators (and their targets) were not defined in terms of what they were intended to measure – whether they related to equity, quality or efficiency objectives. The technical assistance component, in turn, may have been too ambitious and saddled with too many activities (ICR, p. 5) that did not correspond well with the absorptive capacity of the client, and may explain the subsequent underspending on technical assistance. 4. Achievement of Objectives (Efficacy): The PDOs were (a) to improve the equity, quality and efficiency of health services; (b) to improve the equity, quality and efficiency of education services in the first years of schooling; and (c) to strengthen the Borrower’s public sector management. The following assessment of efficacy focuses on the achievement of equity, quality and efficiency objectives, respectively, in each of the two sectors, and on the achievement of stronger public sector management in the Ministries of Health, Education and Finance, and in ASJDI. In the absence of guidance in the project documents (PAD, ICR), the assessment has allocated indicators between equity, quality and efficiency objectives. 1. Improving equity in services – substantial a. Health The project did not include explicit equity-related health indicators; its focus was on improving coverage, quality and efficiency in service provision (PAD, Table 3). However, a number of project outcomes were at such high levels (at or near 100%) that the equity objective was implicitly supported, i.e. if services were extended to the entire population, then services were being extended to the poor, Outputs While a number of outputs under quality and efficiency may have also contributed to improved equity, the project did not draw such links. Outcomes The percentage of deliveries in public hospitals with permanent safe blood availability increased from a baseline of 86 percent in 2008 to 100 percent by end-2012, equal to the target. It has remained at 100 percent. The share of children with a complete immunization schedule at the first year of life increased from a baseline of 32 percent in 2008 to 97 percent by end-2012. It was at 96 percent at end-2013. The number of citizens without social security that were registered in the beneficiary database of the MOH rose from 18,383 in 2008 to 273,420 by end-2012, compared to a target of 160,000. The number was 223,895 at end-2013. b. Education Improved equity was an explicit objective of the education component, and the full day program was (and is) focused on improving schooling of children from low income families. The program for assisted promotion focuses on all students achieving good learning results. In both programs, performance targets were met and even exceeded. However, there is no information on how, specifically, poor students fared, i.e. what happened to equity. There were no specific equity indicators, although improvements in education outcomes in full day programs benefit children from low income families, to whom these programs cater. That still does not give a clear understanding about improved equity, which would have been better illustrated by comparing income groups.. . Outputs No outputs explicitly related to equity in education were recorded. Outcomes Promotion rates in the first cycle of primary education in full day programs increased from a baseline of 81 percent in 2008 to 85 percent by end-2011, compared with a target of 89 percent. By end-2012, the promotion rate had increased to 87 percent. Repetition rates in the first cycle of primary education in full day programs decreased from a baseline of 14 percent to 10 percent by end-2011, compared to a target rate of 12 percent. Desertion rates for the first and second cycle of primary and the second cycle of secondary education in full day programs, which were at 1 percent in 2008, had increased to 5 percent by end-December 2011. A target of 2 percent had been set for that year. By end-December 2012, the desertion rate had declined to 1 percent. 2. Improving quality of services - substantial a. Health Outputs Personal health care programs The share of primary health care facilities using “Mother and Child Health” notebook registers rose from a baseline of 50 percent in 2008 to 100 percent by end-December 2012, compared to a target of 90 percent. It was at 96 percent at end-2013. The number of primary health care facilities with extended open times for assistance had increased from a baseline of 25 in 2008 to 50 by end-2012, equal to the target. It remained at that level for end-2013. The share of primary health care facilities annually audited, controlling use of the “Mother and Child” notebook registers and use of chronological files for mothers and children, rose from a baseline of zero in 2008 to 64 percent by end-2012 compared to a target of 85 percent. By end-2013, it had reached 95 percent. The share of patients with Joint Outpatient Room Strategy (number of JORS per number of deliveries) increased from a baseline of 40 percent in 2008 to 71 percent by end-2012, compared to a target of 70 percent. By end-2013, it was at 68 percent. Public health prevention and promotion program The number of associated healthy municipalities implementing annual plans as “responsible municipality” rose from zero to 9 by end-2012, equal to the target. By end-2013, it had increased to 19. A provincial health promotion strategy for healthy lifestyles, which did not exist in 2008, had been designed by end-2012 and published and operational by end-2013. The target was to have a published strategy under implementation by end-2012. Public health management focusing on service providers A health care quality strategy focused on mother/child health problems and blood safety, which did not exist in 2008, was developed, published, and applied by end-2012, with 60 percent of personnel in targeted areas trained. By end-2013, 70 percent of health personnel in the targeted areas had been trained. 990 MOH health personnel and decentralized hospitals were trained in the quality strategy by end-2012, against a target of 800. By end-2013, 1,015, or about 20 percent of all personnel, had been trained. By end-2012, two annual client satisfaction evaluation surveys were completed and analyzed, equaling the target. None had been undertaken prior to the project. Two surveys were also undertaken in 2013. They indicated recognition of improved quality of services. Outcomes The number of detected diabetes patients under regular care increased from a baseline of 70 percent to 73 percent by end-2012, compared to a target of 78 percent. By end-2013, it had reached 76 percent. The number of patients with accumulated detected/prevalence of diabetes and cardiovascular metabolic diseases increased from a baseline of 22 percent in 2008 to 33 percent by end-2012. By end-2013, it was at 37 percent. The share of 15 to 64 year old women under state coverage with Papanicolau test (Pap smear) done increased from a baseline of 10 percent in 2008 to 48 percent by end-2012, equal to the target. By end-2013, it had increased to 75 percent. The percentage of deliveries with over four ante-natal care consultations increased from a baseline of 21 percent in 2008 to 49 percent in 2012, compared to a target of 40 percent. At end-2013, it was also at 49 percent. b. Education Outputs Program for assisted promotion Promotion rates for the first cycle exceeded the target of a 3 percent increase every year over the project period, rising from a baseline of 77 percent in 2008 to 93 percent by end-2011. Full day program The number of schools participating in the program rose from a baseline of 28 in 2008 to 42 by end-2011, exceeding the target of 32 schools, a 7 percent increase. Promotion rates in the first cycle of primary education in full day programs increased from a baseline of 81 percent in 2008 to 85 percent by end-2011, compared with a target of 89 percent. By end-2012, the promotion rate had increased to 87 percent. Outcomes Program for assisted promotion Under the program for assisted promotion, repetition rates in the first cycle of primary education declined from a baseline of 10.5 percent to 5 percent by end-December 2011, exceeding the target of 8 percent. Under the program for assisted promotion, desertion rates from the first cycle of primary education decreased from a baseline of 3 percent in 2008 to 1 percent by end-2011, exceeding the goal of 2.5 percent. The share of students that achieved passing marks in language in grades 1-3 increased from a baseline of 36 percent in 2008 to 65 percent by end-2011, exceeding the target of 44 percent. The share of students that received passing marks in mathematics in grade 2 increased from a baseline of 26 percent to 64 percent, exceeding the target of 34 percent. Full day program Repetition rates in the first cycle of primary education in full day programs decreased from a baseline of 14 percent to 10 percent by end-2011, compared to a target rate of 12 percent. Desertion rates for the first and second cycle of primary and the second cycle of secondary education in full day programs, which were at 1 percent in 2008, increased to 5 percent by end-December 2011. A target of 2 percent had been set for that year. By end-December 2012, the desertion rate had declined to 1 percent. 3. Improving efficiency of services - substantial a. Health Outputs 1. Personal health care programs The share of primary health care facilities annually audited, controlling use of the “Mother and Child” notebook registers and use of chronological files for mothers and children, rose from a baseline of zero in 2008 to 64 percent by end-2012 compared to a target of 85 percent. By end-2013, it had reached 95 percent. 2. Public health prevention and promotion program The number of associated healthy municipalities implementing annual plans as “responsible municipality” rose from zero to 9 by end-2012, equal to the target. By end-2013, it had increased to 19. 3. Public health management focusing on service providers Integrated management systems were developed and implemented in hospitals of network reference, These did not exist in 2008 but were introduced in 3 hospitals by end-2012, compared to a target of 2 hospitals. The number of accounts of Plan Nacer (maternal and child health) administrators reporting expenditures, or number of open Plan Nacer administrators’ accounts, increased from a baseline of zero in 2008 to 18 (100 percent) by end-2012, equaling the target. It remained at 100 percent at end-2013. 4. Public health management- Ministry of Health Two new monitoring and evaluation systems were set up in 2012 as compared to a target of one, to support deconcentration and decentralization of units in the MOH. The share of field-audited health centers increased from a baseline of zero in 2008 to 100 percent in 2012, compared to a target of 80 percent. It remained at 100 percent in 2013. New budget and inventory systems and a related monitoring system were set up in 2012. Outcomes The project did not include outcome indicators for the efficiency objective in health, i.e. whether the activities were resulting in cost effective choices in the functioning of the health network. The outcomes are, of course, influenced by other, parallel programs, and their effects cannot easily be distinguished; but that should not prevent the introduction of indicators, even if they have to be qualified. Normally, therefore, the absence of outcome indicators for efficiency improvements in health would warrant a lower rating for the efficiency objective, if health was assessed separately from education. But that is not the case here; moreover, the output indicators in health all did exceed their targets, and the initiatives are likely to result in a more efficient functioning of the health network. .b. Education The education component does provide indicators of internal efficiency, i.e. student progression through the system. 1. Program for assisted promotion Under the program for assisted promotion, repetition rates in the first cycle of primary education declined from a baseline of 10.5 percent to 5 percent by end-December 2011, exceeding the target of 8 percent. Under the program for assisted promotion, desertion rates from the first cycle of primary education declined from a baseline of 3 percent in 2008 to 1 percent by end-2011, exceeding the goal of 2.5 percent. 2. Full day program Promotion rates in the first cycle of primary education in full day programs increased from a baseline of 81 percent in 2008 to 85 percent by end-2011, compared with a target of 89 percent. By end-2012, the promotion rate had increased to 87 percent. Repetition rates in the first cycle of primary education in full day programs decreased from a baseline of 14 percent to 10 percent by end-2011, compared to a target rate of 12 percent. Desertion rates for the first and second cycle of primary and the second cycle of secondary education in full day programs, which were at 1 percent in 2008, increased to 5 percent by end-December 2011. A target of 2 percent had been set for that year. By end-December 2012, the desertion rate had declined to 1 percent. 4. Strengthen the Borrower’s public sector management -- modest Outputs A procurement plan was to be prepared and implemented; at project closing, a plan had been prepared, but not implemented. Both the Ministries of Education and Health have introduced program budgeting and improved budget processes, exceeding the target of achieving this in at least one ministry. Other intermediate outcome indicators were not achieved: one relating to the updating of public financial management procedures in accordance with a law enacted in 2000, but still not effective; annual audits of budget resources/institutions; and PEFA indicators, for which neither baselines nor targets were identified during project preparation. Outcomes Indicators for the technical assistance component point to modest results, and many were dropped, including the PDO indicator. Strengthening management processes should be linked to improvements in health and education performance, outcomes to which the technical assistance was to contribute. However, that link was not made, and the achievement of this objective is therefore rated modest. 5. Efficiency: Efficiency is rated modest. The PAD undertook a thorough analysis of the fiscal-financial implications of the project, including a related sensitivity analysis; it did not include an efficiency analysis. The ICR included neither an efficiency or fiscal-financial analysis, nor did it include any explicit qualitative discussion of efficiency. Limited capacity in ASJDI led to several tasks remaining unfinished. However, the fact that there were no problems with financial management or procurement indicates that resources were invested cost-effectively in these areas. In the absence of any ex post efficiency analysis or fiscal analysis, efficiency is rated modest. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of the PDOs is rated high, as the objectives continue to respond to Government and Bank priorities as reflected in their respective current country strategies and the San Jose province strategy. Design is rated modest, due to weaknesses in the results framework and an overly ambitious technical assistance design. Efficacy is rated substantial on improving equity in health and education: while equity indicators had not been explicitly included in health, high outcomes in terms of coverage also point to increased equity; likewise, in education there were no explicit equity indicators, but here, improved performance in full day programs explicitly focused on poor students; less certainty attaches to assisted promotion, where results did not distinguish poor students. Efficacy is also rated substantial for quality in health and education, based on progress on outcomes in both sectors. And it is rated substantial for efficiency in health and education – largely on the strength of the education sector where targets were mostly exceeded; while the health sector included no outcome indicators for efficiency, the activities that were listed are over time likely to have a favorable impact on efficiency. Finally, efficacy is rated modest for improvements in public sector management, in the absence of indicators to draw on to determine achievement of this objective. Efficiency is rated modest.. Taken together, these ratings are indicative of moderate shortcomings in the project's preparation and implementation, and therefore an Outcome rating of Moderately Satisfactory. a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to the development outcome appears modest. The project supported programs that form part of San Juan’s long-term development strategy, and the provincial government can be expected to remain committed to achievements under the project, as well as to future investments into the programs. Risks will mainly be financial and institutional. The former relate both to future trends in recurrent spending and new capital spending, and could include macro-economic risk. However, they are to a large extent tempered by a Federal Fiscal Responsibility Law that sets limits on provincial budget management and debt. Macro-economic shocks would be transmitted through the fiscal-federal system, as provinces are dependent on transfers for a significant share of their budgets. Institutional constraints could be a concern, judging from the amount of technical assistance that was contemplated under the project (some 28 separate activities). However, much of that was never undertaken, and capacity constraints were not signaled as adversely affecting implementation. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project was strategically relevant in its focus on issues -- raising quality and efficiency in health, and equity, quality and efficiency in education -- that were priorities for the province and in the Bank’s country strategy. The project did not, however, directly address equity in health, nevertheless flagged as an objective for health in the PDOs. The basic design, results-based lending tied to DLIs supporting selected expenditure programs of the province, provided strong buy-in by the authorities, as well as a framework for both discipline and incentives for engagement. The risk assessment was thorough, and an extensive technical assistance program was introduced as a key feature for risk mitigation. Monitoring and evaluation arrangements were adequate to monitor the health and education DLIs, but less so in the case of technical assistance, as reflected in subsequently dropped indicators. The more detailed aspects of design could have been stronger. The formulation of the results framework with a large number of indicators not directly related to specific objectives made it difficult to determine individual efficacies; one consequence may have been the absence of equity indicators. The technical assistance would turn out to be over-designed and in some instances too ambitious for the client (ICR p. 5), as only a third of the funds allocated for technical assistance were spent Quality-at-Entry Rating: Moderately Satisfactory b. Quality of supervision: Once the project got underway, implementation proceeded largely on schedule, and DLIs were regularly met for disbursement. DLI schedules were adjusted to reflect the delay in effectiveness. The capacity constraints that were encountered in the technical assistance component were recognized: funds were reallocated to the health expenditure programs and the Closing Date was accordingly extended. Collaboration with the counterparts appears to have been good (ICR p. 7), and unaffected by the changes in TTL. This was confirmed in the Government's own assessment of project implementation. Performance reporting, as reflected in Implementation Supervision Reports (ISR), was concise. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: There was ownership of the project on the part of the Borrower, as it was an integral part of the province’s development program. Significant delays that were initially experienced in project start-up were in large part generated by federal government procedures. These should either have been anticipated or dealt with. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The ASJDI served as the coordinating agency for the project, and it was in charge of the implementation of the technical assistance component. As the coordinating agency, it provided the necessary monitoring in a timely way, facilitating Bank supervision. Its performance on technical assistance was not as strong, and several tasks remained undone (ICR p. 5). Still, the agency had signaled to the Bank that it had limited capacity to manage everything that was expected under the technical assistance component. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Monitoring focused on DLIs, including program expenditure monitoring and monitoring of defined process indicators. The design included formulation of expected results, provision of baselines on outputs and outcomes, and milestones for progress. For collection, analysis and reporting of data, and monitoring of progress, the project relied on the province’s monitoring and evaluation systems for its expenditure programs, while at the same time strengthening the process through project-financed technical assistance. This included in particular the development of manuals on fiduciary oversight. As noted in Section 4, indicators for health and education programs were measurable, but did not always appear to well correspond to objectives. Moreover, there were no explicit equity or efficiency indicators in health. b. M&E Implementation: Implementation of M&E followed the design, and the DLIs were tracked and reported as planned to support disbursement of Bank funds. ASJDI, the implementing agency, consolidated the relevant information and provided evidence of compliance. c. M&E Utilization: M&E data were used to track DLIs and to allow the release of funds. These data also provided the basis for discussions between supervision missions and counterparts on progress and for continued planning purposes. Results formed the basis for adjustments in processes during implementation. M&E Quality Rating: Modest 11. Other Issues a. Safeguards: The project triggered two Bank safeguard policies: Environmental Assessment (OP 4.01) and Indigenous Peoples (OP 4.10). Environmental Assessment . The project classification was Category B. The Environmental Assessment noted that the main potential environmental impact related to the treatment of medical waste in the two main hospitals of the province. The project Environmental Management Plan (EMP) addressed this issue. Indigenous Peoples. Communities of indigenous people were present in the province and participated in programs supported by the project. Drawing on previous projects, the loan included an updated indigenous peoples plan (IPP) for the health and education components, and provided for consultations with affected indigenous people. The project reserved technical assistance funds to support the ASJDI in assisting the Province to implement the EMP and the Indigenous Peoples Plans. b. Fiduciary Compliance: Financial management (FM). A financial management assessment performed in June 2009 concluded that the province’s FM arrangements were satisfactory. Given that this was the first operation of this kind in the province, the FM risk was assessed as substantial and technical assistance to relevant provincial public sector institutions was included to mitigate this risk. The annual financial statements through end 2012 were audited by an independent auditor in a manner satisfactory to the Bank. Procurement. The technical assistance component supported the province’s procurement capacity, and the project was subject to standard Bank procurement requirements. The procurement risk of the operation was assessed as substantial due to the lack of specific experience on Bank procedures. No project specific procurement issues arose during implementation. c. Unintended Impacts (positive or negative): None reported. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Satisfactory Moderately The IEG rating reflects substantial Satisfactory achievement of three of four objectives, and modest ratings for design, achievement of one objective, and efficiency. Risk to Development Moderate Moderate Outcome: Bank Performance: Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately The IEG rating reflects Federal Satisfactory government delays and weaknesses in the implementing agency that affected TA performance. Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The following lessons draw on those presented in the ICR:  Effective technical assistance is appropriately focused . Technical assistance needs to be selective, focused on essential issues, and calibrated to the client’s absorptive capacity. None of these aspects was sufficiently taken into account in this project, resulting in weak performance on technical assistance.  DLIs determine priorities . Reforms and policy measures that are not linked to disbursements receive lower priority during implementation. That was the case with several planned TA activities that were not implemented during this project. DLIs strengthen borrower commitment.  Ownership by authorities is key . As with any results-based lending, the need for a reform champion is key , as this form of lending can be a complex instrument and reforms can be contentious. The content and phasing of reforms would have to be determined by the capacity of reform champions to implement them.  Focused programs stand a better chance of success . It is important to keep the overall program simple and focused on a few critical reforms. An agenda that is too broad can become complex and diffused. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The evidence presented in the ICR and the analysis underpinning it is not sufficient to allow a thorough validation. The narrative is results-oriented, although internal consistency could be stronger; it is not always clear how and why results added up to satisfactory outcomes; in particular, the ICR does not address the causal links in the results framework, nor does it include an efficiency analysis. The presentation of project costs and financing included only the Bank loan. In most other instances, the ICR does follow guidelines. At 21 pages, the main body of the ICR is relatively concise, considering that a three-page outcome and indicator table has (usefully) been included in the text. a.Quality of ICR Rating : Satisfactory