ICRR 14289 Report Number : ICRR14289 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 05/21/2014 Country : Armenia Project ID : P115109 Appraisal Actual Project Name : Access To Finance US$M ): Project Costs (US$M): 50 50 For Small And Medium Enterprises L/C Number : L7657 Loan/ US$M): Loan /Credit (US$M): 50 50 Sector Board : Cofinancing (US$M): US$M ): 0 0 Cofinanciers : Board Approval Date : 02/24/2009 Closing Date : 09/28/2011 12/31/2012 Sector (s): Micro- and SME finance (100%) Theme (s): Micro; Small and Medium Enterprise support (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Stefano Migliorisi Fareed M. A. Hassan Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: The Project Appraisal Document (PAD page 14), states the project development objective (PDO) as “to maintain or increase Armenian small and medium enterprises ’ access to medium-term finance.� The Loan Agreement (page 4) uses an identical wording. This review, as the ICR itself, is based on the above definition of objectives . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project had a single component . Original costs are taken from Annex 5 of the PAD, and actual costs from the ICR. Component 1. Line of Credit (original cost: US$50.00 million, actual cost US$50.00 million) would provide wholesale funding to qualified banks for on -lending to SMEs. The Central Bank of Armenia (CBA) would in turn provide wholesale funding to qualified banks and nonbank financial institutions (NBFIs) for on-lending to SMEs, via the Project Implementation Unit (PIU) – the German Armenian Fund (GAF). The project did not include any contingencies . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Cost Financial project costs at appraisal were US$ 50 million, and included a Credit Line of US$ 49.875 million and a US$125,000 front-end fee. Actual costs at time of closing were US$ 50.0 million. Financing The project was fully funded by IBRD . Borrower Contribution There was no borrower contribution . Dates Approved by the Bank’s Board on February 24, 2009, the project became effective on April 16, 2009. However, the project had a slow start as borrowers preferred to access a credit line provided by Russia with less stringent conditions and reporting requirements than the one funded by the Project, and local banks had no familiarity with Bank requirements. By the time mid-term review took place in November 2010, the project was about fifteen months behind schedule. In July 2011, the project’s closing date was therefore extended from September 28, 2011 to December 31, 2012. The project closed in line with the revised schedule on December 31, 2012. 3. Relevance of Objectives & Design: a. Relevance of Objectives: The objective was to maintain or increase Armenian small and medium enterprises ’ access to medium-term finance. This objective is: consistent with the Government’s Development Strategy for 2012-2025 that includes access to finance, especially for SMEs, as a source of economic growth and job creation . However, the Government strategy focuses on access to credits for SMEs that are start -ups, exporters or led by women – areas that were not targeted in the design of the line of credit consistent with the latest Bank Country Partnership Strategy (2014-2017) whose first pillar (Supporting competitiveness and job creation ) includes measures to remove barriers to credit access for SMEs . The objective has limited ambition as it focuses only on the quantity of credit and not on its quality as measured by its impact on employment, value added and taxes . The relevance of the original objective is therefore rated as substantial . b. Relevance of Design: The design did not present a convincing causal chain between inputs, outputs and outcomes, as they were essentially the same. While the project aimed at maintaining or improving access to credit by Armenian SMEs, the results framework was structured in such a way that full disbursement of the IBRD line of credit through a revolving fund would fully achieve project objectives . As the project did not monitor the aggregate level of SME lending in Armenia, it is not possible to say whether the project achieved its macro objective . The PDO indicator was in fact “cumulative amount of medium term credit granted by the PFIs under the project (including revolving funds).�As such, the indicator fails to capture the essence of the PDO that refers to improving or at least maintaining SME access to credit. Realizing this design issue, the Bank informally introduced an indicator to capture the overall lending to SMEs in its Implementation Supervision Reports . The relevance of design was therefore modest . 4. Achievement of Objectives (Efficacy): The ICR shows that the objective of maintaining or increasing access to medium -term finance by Armenian SMEs was achieved as the outstanding SME Loan portfolio grew from Armenian Dram 155 billion (US$503 million) in June 2008 to Armenian Dram 362 billion (US$900 million) in June 2012. The project disbursed the full US$50 million, and, thanks to the use of a revolving fund, accounted for a growth in SME lending of US$ 98 million or 25 percent of the aggregate growth documented above . As the revolving fund is still operational, the amount is likely to continue growing in the future . Over 7,400 SMEs were supported under the project against an original target of 2,500, mostly in trade and agriculture, 70 percent of the sub-loans were used for working capital, and non-performing loan under the line of credit remained rather low at 1.6 percent against a target of 5 percent. It is instead impossible to evaluate what was the impact of the sub -loans on higher level objectives value added and employment, as the Bank decided not to carry out such an assessment . The PAD (p. 14), after stating the objective and the need to measure intermediate outcome indicators like the ones described above, stated that “credit access, turnover growth and employment outcomes generated by supported SMEs would be measured throughout the project to offer lessons which may be helpful in designing future crisis response schemes for SMEs, in Armenia and elsewhere.� The results framework in the PAD (Annex 3) included aggregate sector indicators to be measured like SME sector contribution to GDP, SME sector employment, and SME contribution to total taxes, and project specific indicators on turnover, jobs and taxes paid by sub -loan recipients, but these were not clearly linked to the project outcome and intermediate outcome indicators, and were never measured . This ICR review concluded that the lack of a clear link between the successful disbursement and management of the line of credit and its impact on value added, employment and taxes makes the relevance of design modest, but does not affect the efficacy of the narrower PDO as defined in the PAD . This review therefore concurs with the ICR in rating the efficacy of the PDO as substantial . 5. Efficiency: Project appraisal did not include a formal cost benefit analysis, while the ICR discusses mostly the project impact on the efficiency of its beneficiaries rather than the efficiency of the project itself . However, it should have been possible to provide information on unit rate norms for management cost or job creation that IEG cannot calculate in the absence of such data . However, several elements in the design seem to indicate that the project ’s efficiency was substantial . The project used an existing project implementation structure within the Central Bank of Armenia, funds were disbursed from the CBA to participating financial institutions quickly as soon as demand from final borrowers picked up, and the quality of credit under the project, as measured by non -performing loans, remained above average for the Armenian banking sector . Finally, CBA’s cost to managed the project were not funded separately and were therefore incorporate in the lending rate that remained competitive, another indication of efficiency. The efficiency of the project is therefore rated as substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of the project’s objectives, its efficiency and its efficacy were all substantial, while the relevance of its design was modest. The combination of these elements leads to an overall outcome rating of "satisfactory.� a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: IEG concurs with the ICR rating of the risk to development outcome as moderate . The major risk is represented by the significant dependence of Armenian banks from official external support for SME term lending, as 70 percent of outstanding loans have been funded by donors or international financial institutions . If such support were to decline, access to term credit by SMEs could suffer . A more sustainable financial market, relying on local pool of funds, would clearly reduce such risks, and Government policies in this area are being supported by the Bank through a DPO series. The other major risk is a further contraction of the Euro area which might affect the demand for Armenian exports in the European Union and Russia, and a possible contraction in remittance flows to the country. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project addressed relevant development challenges . The Bank reacted promptly by preparing this operation to encourage financial institutions to at least maintain lending to SMEs, which were particularly threatened by the crisis. However, it had an inadequate results framework that focused on a set of intermediate outputs rather than tackling the potential impact of the project on value added, employment and taxes. The lack of an economic analysis in the PAD is also difficult to understand as there is enough information from several regions to make sensible assumptions on the possible impact of credit on employment, value added and taxes . As far as the design of the features of the line of credit is concerned, the Bank ’s loan was clearly less attractive for borrowers than competing lines of credit, and the low participation of manufacturers is an indication of a more fundamental issue . As discussed in the ICR, one element is the fact that purchase of used equipment was not allowed under the Bank loan . A second element was the Bank ’s decision to set low thresholds for prior review and competitive bidding, which created unnecessary complications for sub-borrowers. On balance, the Bank’s performance on quality at entry is rated as moderately satisfactory . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The Bank spent substantial resources on supervision, with a total of 7 supervision missions and day -to-day interaction as the last TTL was stationed in the field . All supervision missions included substantial interactions with the Central Bank, participating financial institutions and borrowers, gathering a thorough and frequent feedback on project performance . Several banks improved their safeguards procedure thanks to the project. The Operations Manual was revised several times to help speed up disbursements . However, the Bank did not monitor the impact of the project on higher level objectives like employment, value added and taxes, even though such monitoring had been envisaged in the PAD (Annex 3). Given such moderate shortcomings, the Bank ’s performance on quality of supervision is rated as moderately satisfactory . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government supported several measures to increase SME access to term credit . First of all, it provided a guarantee for the loan to CBA. Second, it has carried or is carrying out policies to improve SME access to credit. In particular, the credit registry is being upgraded, the secured transactions regime is being enhanced, the draft leasing and factoring laws are being prepared, and several structural reforms to deepen Armenian financial markets are underway. Overall, IEG concurs with the ICR rating of Government performance as Satisfactory . Government Performance Rating Satisfactory b. Implementing Agency Performance: Overall, the German Armenian Fund (GAF), part of the CBA, performed effectively its responsibilities of coordination, information management, monitoring, and reporting . It also provided guidance to PFIs and trained their staff on the project ’s procedures. During the ICR mission, PFIs reported a high level of satisfaction with GAF’s professionalism. Overall, IEG concurs with the ICR rating of GAF ’s performance as Satisfactory . Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The original project design did include an integrated M&E framework with one PDO indicator and three intermediate outcome indicators. The baseline for latter were identified during implementation . The overall M&E framework was simple, and it did provide additional indicators to capture the impact of the credit line on value added, jobs and taxes, even though their link to the other indicators was not clearly spelled out . Indicators were redesigned during implementation, but were never formally revised . b. M&E Implementation: All ISRs we reviewed included information on the PDO indicator and the intermediate outcome indicators, while the impact indicators described above were not monitored, except for mid -term review. While a beneficiary assessment was carried out at the end of the project (including an analysis of size, gender, sectoral and geographic distribution of borrowers, types of activities supported, and borrower satisfaction with the loans ), an impact assessment was not undertaken . c. M&E Utilization: Project data were collected systematically and were used in analyzing performance . As noted in the ICR (p. 12), due to careful monitoring, the Operational Manual was amended several times to address implementation impediments. M&E Quality Rating : Substantial 11. Other Issues a. Safeguards: The project was classified as environmental category F – financial intermediary assessment . Sub-lending to beneficiaries that have acquired and /or would acquire land for the needs of activities to be supported with the project proceeds was excluded from eligible expenditures and so was lending for activities that would require physical relocation and/or would be classified as environmental Category A . The ICR (p. 13) found that the borrower followed the established procedures for complying with environmental safeguard policies, and the project did not cause any adverse impacts on the environment, although there were occasional errors in environmental classification of sub -loan applications, and poor quality of some Environmental Management Plans developed for higher risk Category B sub -projects. No resettlement or land acquisition took place . b. Fiduciary Compliance: There was no significant fiduciary issue during the project ’s implementation that could be identified through the documents reviewed by IEG. Most issues were at the initial stage of the project, were minor and related to the absence of invoices and properly documented evaluation reports for some final borrowers . Financial management both at GAF and PFIs was always considered satisfactory by Bank supervision missions . c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately Shortcomings in project design and Satisfactory Bank’s decision not to monitor higher-level indicators or carry out an impact assessment at the end of the project . Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: There are four main lessons that can be drawn from the experience of this project : As noted in the ICR, onerous safeguard and fiduciary requirements slow down project implementation . Designing information requirements to minimize compliance costs, while serving the safeguard objectives, would help ensure successful implementation . Concerning procurement, requirements for competitive bidding should be kept realistic and the eligibility of the purchase of used equipment reconsidered. A weak causal chain in project design can damage the performance of otherwise successful projects . Increasing access to credit by SMEs is a means to an end, not an end in itself . From the limited evidence available, it is highly likely that the project had a positive impact on higher -level objectives, but this could not be proved. The learning potential of projects should always be realized, provided costs are reasonable . The impact of the project on value added, employment and taxes could have been measured and important lessons could have been learned . Project implementation capacity is organization -specific, and should therefore always be built as part of the project unless there is prior experience with Bank project management . GAF had a strong experience in managing KfW projects whose rules are different from the Bank . Capacity building on Bank rules and procedures should have been carried out during project start -up and included in project design . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR was candid and provided useful information to assess project performance . However, the ICR did not draw the necessary conclusions from its criticism of the results framework and how this was implemented by the Bank. a.Quality of ICR Rating : Satisfactory