IEG Report Number: ICRR14734 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 06/30/2015 Country: Jordan Project ID: P145865 Appraisal Actual Project Name: Mitigate The Impact Project Costs (US$M): 150.0 149.9 Of Syrian Displacement On Jordan L/C Number: Loan/Credit (US$M): 150.0 149.9 Sector Board: Social Protection Cofinancing (US$M): Cofinanciers: Board Approval Date : 07/18/2013 Closing Date: 07/31/2014 07/31/2014 Sector(s): Other social services (60%); Health (40%) Theme(s): Nutrition and food security (37%); Child health (33%); Other social protection and risk management (16%); Other economic management (7%); Health system performance (7%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Susan Ann Caceres Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Project Appraisal Document (p. 6) and the Loan Agreement (p. 5), the project development objective was "to help the Borrower maintain access to essential healthcare services and basic household needs for the Jordanian population affected by the large and increasing influx of Syrian refugees." b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: There were three components:  Maintaining Access to Essential Healthcare Services (appraisal, US$ 70 million; actual, US$ 69.9 million) was to provide resources to sustain services such as vaccines, drugs, and use of non-Ministry of Health facilities for approximately 2,500 Jordanian patients.  Supporting Household Basic Needs (appraisal, US$ 79.9 million; actual, US$ 79.5 million ) was to co-finance a government subsidy program to ensure the uninterrupted supply of commodities, specifically bread and liquid petroleum gas, to the Jordanian population, as the influx of refugees increased the demand for these goods and services.  Project Monitoring, Auditing and Verification (appraisal, US$ 100,000; actual US$ 100,000) was to finance two audits to provide documentation of retroactive disbursements for vaccines and drugs and verification of the lists of medical bills of patients referred to non-Ministry of Health facilities. For prospective payments, bills were to be tracked to measure against the baseline in the results framework. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost: The total cost of the project was US$ 149.9 million, or 99.9% of appraised value. While component costs were similar to appraisal estimates, funds were shifted from medical treatment to drugs and vaccines within component one, as a delay in the external audit made it difficult for the Government to recover its expenditures for financing of health care services for Jordanians in non-Ministry of Health facilities (ICR p. 22). Financing: The project was financed by an IBRD loan. Borrower Contribution : The borrower made no contributions, nor were any planned. Dates: The operation was prepared within six weeks. The loan fully disbursed within a year, as planned, and closed on schedule. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial With the prolonged conflict in Syria, the Government of Jordan has committed to not closing its borders, and up until December 2014 it provided free health services to refugees. Approximately 619,000 people had crossed into Jordan from September 2014 through project closing. The majority of these refugees are living in Jordanian communities, which puts pressure on public finances. Given high demand for health services and the Government's provision of these services to refugees, drugs and vaccines were being depleted at a rapid rate. At the same time, the country's macroeconomic conditions have become vulnerable (e.g. rising government deficits from gas supply disruptions, deterioration in foreign exchange reserves, and decline in exports with closure of Syrian transit routes). The combination of all of these factors threatened Jordan's human development and poverty reduction achievements, as most costs associated with the refugees living in communities has been borne by the Jordanian government. The majority of international support has been channeled to the needs of the displaced Syrians residing in camps. Providing temporary support for household needs of Jordanians was also intended to mitigate the potential for social instability. The project development objective is substantially relevant to the Bank's current Country Partnership Strategy (FY12-FY15), which aims to build resilience and mitigate risks to internal and external shocks, and to improve the population's living standards. The objective is also consistent with the government's priority to mitigate the effects from the regional conflict on its citizens. b. Relevance of Design: Substantial There was a logical and plausible link between both objectives, goods and services funded, and the outcome and output measures used to demonstrate maintenance of health and household basic services for the Jordanian population. The use of Investment Project Financing under Situations of Urgent Need of Assistance was appropriate. It allowed the operation to avail higher retroactive financing (40%) for payments made before the loan signing date (on or after June 30, 2012) to alleviate fiscal pressure, as gas and bread subsidies were a heavy fiscal burden on the government's budget (1.6% of GDP) (ICR p.15). Disbursements were based on a positive list that was developed during appraisal, but flexibility was permitted to ensure that services and goods continued without a gap. The rationale for supporting universal subsidies was that refugees added fiscal pressures for theses goods and services, which constitute a larger share of poor households' expenditures than that of rich households (PAD p. 18). The government has subsidized wheat and bakeries, which sell subsidized bread at regulated retail prices that have been unchanged since 1996 (ICR p. 8). Existing structures and implementation arrangements were utilized, which facilitated simple processes and rapid implementation. Estimation of financing was based on recent usage data and incremental volumes of bread and gas consumed beyond the level reached prior to the onset of the Syrian conflict to address the temporary need. 4. Achievement of Objectives (Efficacy): Maintain access to essential healthcare services for the Jordanian populatio n: Substantial Outputs:  117 types of drugs and vaccinations were purchased from the rational drug list, exceeding both baseline (90) and target (112). Exceeding the target was due to the reallocation of funding from reimbursement of uninsured patients in non-Ministry health facilities. According to the ICR, the number of beneficiaries of drugs and vaccines slightly increased from its baseline value (6.23 million) to 6.37 million, which exceeded the target of maintaining the baseline value. However, this target represents the entire population of the country. While it is unlikely that nearly all of the population would receive medication or vaccines in a single year, this figure was derived from calculations of the Ministry of Finance based on payments to the Ministry of Health to maintain stock for universal healthcare.  Ministry of Health warehouses with no drug stock-outs declined from 7 to 2 by the end of the project, meeting the target (2). Outcomes:  The immunization rate of children (0-6) remained at 97% throughout the life of the project, which met the target.  The ICR reports that the per capita outpatient utilization rate at primary health centers remained at 0.58% or nearly one visit per person, which met the target. The region clarified that this number was the primary health care utilization rate from the four governorates most affected by the Syrian crisis (i.e. Irbid, Ramtha, El-Mafrak, and El-Zarka). MOH data and the Borrower's ICR (p. 38) also indicate that the outpatient utilization remained at 2 visits per capita per year for the entire population.  The ICR reports that per capita inpatient admission rates at hospitals remained at the baseline value (1.1%), which met the target. The region clarified that this figure represents hospital deliveries (72,611 in 2012 and 74, 188 in 2013). In addition, the MOH Annual Report 2013 shows that a rate of 141 admissions per 1000 persons was maintained during this period. The Borrower's ICR (p. 38) reports a 5.5 % hospital admission rate.  48.6% of beneficiaries of drugs and vaccines were women, which was maintained throughout the project. Maintain access to basic household needs for the Jordanian population : Modest Outputs:  77,773 metric tons of wheat were purchased, which exceeded the target (64,811).  3,196,622 metric ton of liquified petroleum gas was purchased, which met the target (3,196,622). Outcomes:  The number of beneficiaries in safety net programs for bread and fuel was 2.5 million and continued to be maintained at the end of the project, which met the target. While this subsidy is universal, the Bank's financing was to cover a portion of the population.  48.6% of beneficiaries of safety nets were women, which was maintained throughout the project. The ICR does not provide data to demonstrate that access to basic household needs was maintained. 5. Efficiency: The appraisal document stated that financing US$ 80 million of basic goods would prevent the incidence of poverty and its depth from worsening. For example, the wheat subsidies would prevent the price from rising by 35%, pushing about 45,000 people into poverty (representing a 5% increase in the number of poor in Jordan). It further estimated that maintaining the subsidy would prevent 14.4% (or one million people) of the Jordanian population already in poverty from becoming poorer. The appraisal document also noted the negative impacts that could result from disruption in access to medicine and vaccines, including disease outbreaks. Some of these assumptions have not materialized, as the government has continued to subsidize wheat and bakeries, which sell subsidized bread at regulated retail prices that have been unchanged since 1996 (ICR p. 8). The Government has accompanied a subsidy phase out with the implementation of a cash transfer program (PAD p. 23). The ICR (pp. 18, 29-31) repeats the economic analysis from the appraisal document and augments it with other information. The ICR notes that the financing mechanism established was efficient and flexible. However, the final audit has not been completed. While it was estimated that the operation would provide US $20 million to finance services for Jordanians using non-Ministry of Health facilities, as data showed an increasing trend in the numbers and cost of referrals before the influx of refugees, these costs were reallocated due to delays in hiring an audit firm. The operation was implemented within a year, as anticipated, with the provision of retroactive reimbursement. On balance there was modest efficiency. Efficiency: Modest a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The Outcome rating is Moderately Satisfactory based on substantial relevance of objectives and design, substantial achievement of one objective, modest achievement of the other objective, and modest efficiency. Design was simple, with its streamlined financial management approach, 40% retroactive financing, use of a positive list of vaccines and drugs that minimized transaction costs. Data from the MOH demonstrate the maintenance of essential health care services, but information is not provided to demonstrate maintenance of access to household basic needs. a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: While the ICR (p. 20) reports that the risk to the development outcome is low because targets were met, other risks such as the fiscal strain due to the prolonged conflict, a continuing influx of refugees, high rates of poverty and unemployment, and large budget deficits were not discussed. Considering these other factors, the risk to the development outcome is likely higher than indicated in the completion report and is rated as significant. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: A multi-sectoral team prepared the operation in six weeks. Existing structures and systems were utilized to quickly provide resources to the Government given the fiscal pressure from the influx of refugees. Fiduciary assessments previously undertaken were used by the Bank, as well as a simplified review process during preparation (e.g. Directorate of Purchases and Supplies). This was augmented with several ex-post controls and reviews. A positive list of drugs was established to minimize the number of transactions and required documents. Preparation clarified how the Borrower's national anti-corruption rules governed contracts under the operation. Financial management procedures were specified, as well as coordination mechanisms between departments to ensure documentation was provided to the Bank. An accountant from the Ministry of Finance was identified during preparation. Simulations were developed during preparation to estimate the project's impact on poor and lower middle class households, if bread and fuel subsidies were not maintained. There were differences between the lending agreement and appraisal document in terms of eligibility for retroactive financing (ICR p. 21). There were also shortcomings in the selection of indicators thus making it difficult to demonstrate attainment of one of the objectives (See Section 10a). Quality-at-Entry Rating: Satisfactory b. Quality of supervision: Supervision was undertaken by staff in the region and across several units in the Bank (e.g. financial management, procurement, social protection, and social development). Two supervision missions were conducted during the year of implementation. Financial management training was provided by Bank staff for the Implementing Agency. Clear specification of the documents that the Borrower had to provide for disbursements was established. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government of Jordan has committed to not closing its borders, and up until December 2014 it provided free health services to refugees. The King requested Bank involvement in the spring of 2013. The government provided its full support during preparation and implementation. There was a delay in establishing the Designated Account, which was expected to have been done before effectiveness. The project team indicated that funds were transferred into the account on October 3, 2013, indicating a delay of nearly three months. The project team reported that this resulted from lack of understanding of Bank guidelines, but this delay did not have a negative impact on disbursements. While the Government had implemented multiple Bank operations in recent years, the team indicated that this was the first one implemented by the Ministry of Finance. Government Performance Rating Satisfactory b. Implementing Agency Performance: The implementing agency was the Ministry of Finance, which managed the operation through its financial management of bills and verification of expenditures. The Ministry of Finance established baseline data and coordinated project indicators. An accountant from the Ministry was identified during preparation. The Ministry of Finance worked with staff in the Joint Procurement Department, which provided necessary information and documentation. There was a delay in hiring an auditor, which impacted the project's ability to recoup health services costs for 2500 people who used non-Ministry of Health facilities, as required documents were unavailable. Two agencies (Ministry of Planning and International Cooperation and Ministry of Finance) were engaged in the financial management and disbursement arrangements. There were issues with coordination between these two agencies on a few occasions, which caused delay in submission of documents to the Bank (ICR p. 24). There were issues with the final audit submitted to the Bank; the report was initially expected in March 2015 and is now expected on September 15, 2015. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Given the development objective, maintaining and not increasing the baseline level in terms of coverage and access to the Jordanian population in relation to essential household needs (i.e. bread and liquified petroleum gas) and health services was appropriate. However, indicators were obtained only from the financial management and verification system. In the case of health services, this was supplemented with MOH data to demonstrate that health services were maintained. However, there were no indicators to demonstrate maintenance of household needs. The implementing agency had responsibility for data collection, reporting, and analysis. b. M&E Implementation: The financial management and verification system provided the monitoring evaluation data related to universal subsides and indicators generated from the payments and invoices for vaccines, drugs, and medical treatment. c. M&E Utilization: The ICR did not report how data were utilized except in relation to tracking of payments and indicators, as the Government's completion report did not provide information on the monitoring and evaluation aspects of the operation (ICR p. 25). M&E Quality Rating: Modest 11. Other Issues a. Safeguards: No safeguard policies were triggered by the operation (PAD p. 29). b. Fiduciary Compliance: Financial Management: Qualified staff within the Ministry of Finance oversaw the financial management and disbursement functions under the project. An external firm was to conduct audits of vaccines, drugs, and medical bills of patients referred to non-Health Ministry facilities, and subsidies. While the final audit has not been completed, interim financial reports were deemed acceptable by the Bank. The project team indicated that the final audit report initially submitted was rejected by the Bank, and the final audit is expected by September 15, 2015. Procurement: Procurement monitoring was done by the Joint Procurement Department and was evaluated by a visiting agency that interviewed staff and reviewed documents. Procurement of vaccines and drugs followed the procedures of the Joint Procurement Department for contracts up to US$ 5 million. The first audit revealed that payments were made for invoices dated from 2011 and later for retroactive financing, as the legal agreement specified payment date, not invoice date. c. Unintended Impacts (positive or negative): None reported. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Satisfactory Moderately The Outcome rating is Moderately Satisfactory Satisfactory based on substantial relevance of objectives and design, substantial achievement of one objective, modest achievement of the other objective, and modest efficiency. Design was simple, with its streamlined financial management approach, 40% retroactive financing, use of a positive list of vaccines and drugs that minimized transaction costs. Data from the MOH demonstrate the maintenance of essential health care services, but data are not presented to demonstrate maintenance of access to household basic needs. Risk to Development Negligible to Low Significant There are several risks that pose a Outcome: substantial risk to the development outcome such as the fiscal strain due to the prolonged conflict, a continuing influx of refugees, high rates of poverty and unemployment, and large budget deficits, none of which are noted in the ICR. Bank Performance: Highly Satisfactory Satisfactory The ICR substantiates satisfactory performance by the Bank consistent with expectations in preparing and supervising an emergency operation. Borrower Performance : Satisfactory Moderately Based on OPCS Harmonized Criteria, Satisfactory the outcome rating determines the overall Borrower Performance Rating. Quality of ICR: Unsatisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR provides several lessons (ICR pp. 24-25) that have been synthesized by IEG as follows:  Utilizing existing delivery mechanisms and services allows quick preparation , implementation, and response, but technical assistance is also required . In this case, project preparation minimized time-intensive financial management and procurement reviews and relied on country systems already assessed by the Bank. A positive list of drugs and vaccines minimized the number of transactions and documents to be audited, thus reducing transaction costs. Strong ex-post audits ensured compliance with project requirements. Retroactive financing provided immediate fiscal relief. However, technical assistance related to fiduciary management was also needed for successful project implementation.  Temporary time-bound support in emergency situations can help mitigate social instability . The universal subsidies played a role in maintaining social stability in the country, despite being burdened by a large refugee population (with no end in sight).  A strong relationship between the Bank and the government facilitates rapid response . The Bank was able to provide rapid preparation and intensive implementation support drawing on the multi-sectoral staff in the region. This may not have been possible without the working relationship that existed between the Bank and the government. 14. Assessment Recommended? Yes No Why? A field based assessment could bring additional data to assess attainment of the project development objectives. 15. Comments on Quality of ICR: The ICR is thin, with a number of shortcomings. The ICR draws heavily upon the information contained in the appraisal document. The ICR repeats the same information across sections, rather than providing quality evidence. Several of the sections suffer from lack of analysis, contributing to disconnects in ratings between the ICR and ICRR. The ICR does not provide much description and analysis in relation to Government and Implementing Agency Performance and Bank Performance. The ICR provides only a superficial discussion of the risk to the development outcome, rather than examining evident risks. Some information that would be expected is not reported (e.g. compliance with covenants and interim financial reports, assessment of financial management performance and explanation of how beneficiaries of subsidies were verified). The ICR did not report on the external audits. Overall Bank Performance, Quality at Entry, and Quality of Supervision are inconsistently rated in the ICR, making it unclear whether overall Bank Performance is rated as Satisfactory or Highly Satisfactory. Data contained in the ICR and PAD are inconsistent with MOH data and that in the Borrower's ICR (p. 38). a.Quality of ICR Rating : Unsatisfactory