Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) Report Number: ICRR0022664 1. Project Data Project ID Project Name P120707 PNG - SME Access to Finance Project Country Practice Area(Lead) Papua New Guinea Finance, Competitiveness and Innovation L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-49280 31-Dec-2024 6,553,579.36 Bank Approval Date Closing Date (Actual) 03-May-2011 30-Oct-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 21,910,000.00 0.00 Revised Commitment 8,285,944.18 0.00 Actual 6,553,579.36 0.00 Prepared by Reviewed by ICR Review Coordinator Group Paul Holden J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Pellekaan 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement of June 14, 2011, the objective of the Project was to facilitate access to sustainable credit for small and medium sized enterprises (SMEs) and thus contribute to incremental growth of SME employment and income. The PDO can be deconstructed as: Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) 1. Facilitate access to sustainable credit to SMEs; 2. Contribute to incremental growth of SME employment; 3. Contribute to incremental growth of SME income. These three objectives will be referred to as Objectives 1, 2 and 3 in Section 4 of this review b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Component 1: Risk Sharing Facility: The risk sharing facility would provide 50% partial credit guarantees in local currency to cover losses on the SME portfolios of participating financial intermediaries (PFIs). Appraisal amount: US$11.67 million; disbursed amount US$0.13 million. The PAD states that the total size of the PFI portfolio would by up to PGK 300 million (or US$116.7 million) at the prevailing exchange rates. This component provided a guarantee from the IFC to PFIs that covered 50% of the losses on their portfolio of new SME loans. According to the Project Appraisal Document (PAD), the IFC guarantee would cover up to PGK150 million (US$58.37 million) of losses. This would initially be paid by IFC, but IDA would provide a credit to the government of PNG funds with which to reimburse the IFC for the first 10% of the losses, up to PKG30 million (US$11.67 million), with the balance of PKG120 million (US$46.69 million) of losses being born by the IFC. These outcome targets were revised several times during the project although not officially. For example, the fourth Implementation and Status Report (ISR) reduced the outcome target to PKG 50 million of outstanding loans (US$14.25 million) or less than 10% of the original target. However, the outcome targets were increased in ISR 8 to PKG100 million (US$ 28.5 million) of outstanding loans. The target was increased further in ISR 12 to PKG250 million (US$71.2 million) of committed loans. Component 2: Performance-based technical assistance (TA) to one or more participating financial intermediaries (PFIs) to build and improve their capacity for SME banking operations through the provision of TA. Appraisal amount: PKG3.93 million (US$1.53 million); disbursed amount US$1.15 million. Component 3: Provision of capacity building for participating SMEs. Appraisal amount PKG 13.2 million (US$5.15 million); disbursed amount US$2.93 million. This was to be achieved through: (i) training of SMEs; (ii) mentoring and coaching; (iii) targeted training for women entrepreneurs; (iv) training of provincial governments. Component 4: Capacity building for the Department of Commerce and Industry (DCI): Appraisal amount: PKG9.15 million ($3.56 million); disbursed amount US$2.02 million. This was to be achieved through the provision of TA to the DCI to enhance its capacity to: (i) manage and coordinate the ongoing Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) implementation of the project; (ii) undertake SME surveys to develop baseline performance indicators; (iii) develop and SME policy and Strategy; and (iv) support the operations of the Project Steering Committee. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The project was to be financed through IDA (US$21.91 million), but the 2020 restructuring reduced the cost to IDA to US$8,285,000. Financing. The IFC through a guarantee to the Government of PNG (up to US$46.4 million), the Government of PNG (US$4.6 million) and the PFIs (US$58 million). The actual financing by IDA recorded in the Data Sheet in the ICR was US$6,553,579. The IFC did not pay out under the guarantee. Dates. The Risk Sharing Facility was envisaged to remain in effect for 14 years for both IFC and IDA financing. The World Bank Group Board approved the project on May 3, 2011 and it became effective on September 15, 2011. It was originally scheduled to close on December 31, 2024. The actual closing date was on October 30, 2020, although the Bank South Pacific (BSP) had already withdrawn from the project in September 2017 after which few project activities were implemented. Restructuring. While indicators were dropped and added without formal restructuring (see ICR, paragraph 14 and Table 4), IEG assessed the changes and concluded that there had been no discernible change in the project’s level of ambition and hence a split rating of objectives was not justified 3. Relevance of Objectives Rationale The PDO was aligned with the policy priorities of the government as articulated in its PNG Vision 2050 (2010) and Long-Term Development Strategic Plan 2010-2030 (2010). An important component relevant to each was the need to promote SMEs, which are the largest source of employment in the country. The strategies embodied in these documents were that the SME sector would be a major contributor towards PNG becoming a middle-income country by 2030 and a high-income country by 2050. A key constraint to businesses in PNG was extremely limited access to finance. Numerous pieces of analytical work and surveys confirmed that financing for firms was a binding constraint. In 2010 the ratio of private sector credit to GDP, a key indicator of financial market development was 21%, indicating significant financial repression. At the same time, the capital adequacy ratios of the commercial banks were high, exceeding 30%, with the loan to deposit ratio less than 45% even though bank reserves of the commercial banks were high, indicating a reluctance to lend because of the perceived level of risk on the part of lenders. The World Bank Country Assistance Strategy 2007 (CAS) had as one of its two pillars, “improving the livelihoods and service delivery, especially for rural poor.” The PAD states (para 16) “Support under the second pillar would include, inter alia, support to private sector development. Development of the SME sector will serve to boost income-earning opportunities.” However, the Systematic Country Diagnostic (2018) does not prioritize support to SMEs. Nevertheless, the Country Partnership Framework FY19-FY23 Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) has as a Focus Area Objective 3.1, to improve MSME Competitiveness and Access to Financial Services. The project was therefore largely consistent with the objectives of the CAS at appraisal, and consistent with the Country Partnership Framework at closing. However, the measurability of the extent to which the objectives (“facilitate access to sustainable credit” and “contribute to incremental growth of SME employment and income”) were achieved raises serious questions regarding the way that the PDO indicators were framed. The flow of credit indicator and the loans to businesses owned by women indicator have problems with attribution. This issue is compounded with respect to the other indicators, namely an increase in the number of formal sector SMEs and the incremental employment generated by SMEs (see ICR, Table 1). Attributing changes in either of these variables to the project is not possible since many factors other than credit determine the number of businesses started and the number of employees they hire. A further issue was, as the ICR points out (para 21), that the CPF FY19-FY23 treats the project as closed, since it is not listed as contributing to the achievement of any of the indicators. However, the project appeared in the Completion and Learning Review (CLR) of the previous CPF (FY2012-FY2018). In conclusion, the project’s objective was highly relevant to both government and World Bank development strategies for PNG. However, the relevance of project's objectives is undermined by proposed PDO indicators which measure outcomes that cannot be attributed solely to the project. The relevance of the project's objectives is therefore rated substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Facilitate access to sustainable credit to SMEs Rationale The ICR states (para 4) that “at the time the program was being designed, Risk Sharing Facilities (RSFs) were emerging as a new World Bank Group instrument for catalyzing SME lending, through partnerships between the International Development Association (IDA), the International Finance Corporation (IFC), and national governments.” Partial credit guarantees were seen as an instrument to incentivize commercial banks to lend to riskier clients by guaranteeing to reimburse a portion of any losses that might occur. The model used in PNG was a portfolio guarantee, rather than a guarantee of individual loans. Theory of Change. The results chain for this objective was based on the assumption that the partial credit guarantee program would lead to the participating commercial banks lending more to SMEs. It was seen as a way to overcome what was described as market failures in the credit market. However, whether this was in Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) fact a market failure in PNG is open to question. Firms in PNG, especially smaller ones, had no way of offering collateral to banks, since there was no secured transactions framework that could be used to secure lending. Unsecured lending, especially to SMEs, is risky even in well-developed financial markets. The design of the RSF for this project envisaged participation of the Bank of the South Pacific (BSP), a PNG- owned financial institution, plus two or three of the other commercial banks, which were foreign owned. Under the original design, BSP would be allocated 50% of the RSF, with other PFIs also being allocated 50% between them. In the event, the other financial institutions began to downsize their presence in the country. Although they had initially expressed interest, it turned out that the private commercial banks were ultimately unwilling to participate in the RSF despite a sustained effort on the part of World Bank staff to encourage them to do so - as reported by the Bank project team to IEG. Consequently, only BSP participated in the program. Outputs The original target for the amount of credit to PNG SMEs was PKG550 million, compared with a 2011 baseline of PKG300 million. The cumulative value of SME loans guaranteed by the RSF at the end of December 2017 was PKG114.6 million. The data were specified for this date because at the end of 2017, BSP withdrew from the RSF. Figure 2 in the ICR shows that there was considerable fluctuation in the value of guaranteed loans over the period of the project. However, as Table 8 and Annex 6 in the ICR show, the indicator changed during implementation, from a Committed Loans outcome target in the PAD of PKG550 million, to Outstanding Loans in ISR4 and ISR 8, and then back to Committed Loans in ISR12 and ISR16. The final target in ISR 16 was a target of PKG250 million in committed loans. As noted already, the actual amount disbursed was PGK114.6 million. Outcomes The actual target outcome for the number of loans to women-owned businesses was not specified in the PAD. This was developed during implementation through the ISRs. The outcome target for the number of loans to women was set at 400 (ICR, Annex 1) compared with a baseline of 0, and the target value of loans to women-owned businesses was PKG100 compared with a baseline of 0. Based on ISR 18 in February 2020, the ICR concluded that the final outcome was 262 guaranteed new loans to women-owned businesses out of a total of 1,471 loans (paragraph 51). However, once BSP withdrew from the RSF at the end of December 2017, the guarantees expired. Over the course of the project, the ratio of private sector credit to GDP in PNG declined from 21% to 19%. While many factors contribute to determining this ratio, it is difficult to discern a positive impact on financial market development in PNG as a result of the project. In terms of potential impact on access to credit in PNG, the ICR (para. 28) points out that even if the RSF had fully met its target, it would increase the ratio of private sector credit to GDP by less than 0.5%. “While lending to SMEs would comprise only a portion of overall private sector credit, the scale of the RSF seems unlikely to have moved the needle on credit to SMEs”. While the number of SMEs accessing sustained credit cannot be attributed directly to this project because that access is dependent on many factors other than a source of financing through PFIs, there is no evidence provided in this ICR that the project was able to facilitate access to sustainable credit to SMEs. Therefore the extent to which Objective 1 was achieved is rated modest. Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) Rating Modest OBJECTIVE 2 Objective Contribute to incremental growth of SME employment Rationale Theory of Change. The causal chain leading to this objective was that greater access to finance among the group of SMEs that was receiving credit guarantees would result in growth in the number of employees. The PAD projected a 5% annual rise in employment for the beneficiaries. Although this is not an unreasonable figure, there is no discernable analytical basis for the choice of the projection, nor for attribution. The ICR reports (para 33) that indicators in the ISRs showed that employment rose from a baseline of 500 in 2012 to 2,149 in 2016, which represented a more than fourfold increase. The basis for the increase, however, is unconvincing, since it was obtained from a survey in 2015 of recipients of credit guarantees, who before the loans were utilized were asked how many additional employees they would hire because of the loans they were receiving. Furthermore, the 500-employee baseline was suspect since it was based on an “expert- estimate” during appraisal which was not verified and could not be compared with the sample of borrowers in 2016. In addition, firms hire for many reasons, and for this group their access to the credit guarantees would be only one factor in their hiring decisions. The ICR states (para. 35) that additional information was available from a 2019 survey of 437 SME borrowers from BSP who received RSF guaranteed loans between 2012 and 2015. They reported a 240% increase in employment. However, the ICR noted methodological issue with the survey because it was conducted more than 5 years after the loans were disbursed and recall of historical information would have been a challenge. In addition, there were problems with the responses to the survey because some SMEs declined to be interviewed and others refused to divulge financial information both of which may have biased the results. The employment increases also occurred during the boom that arose from the construction of the US$18 billion natural gas pipeline by Exxon Mobile, which generated economic opportunities for SMEs in PNG. Assessing the counterfactual regarding what would have been the increase in employment without the guarantees was, as the ICR noted, also not straightforward. Because of the lack of evidence attributing higher employment by firms to those receiving credit guarantees provided by the project, the extent to which Objective 2 was achieved is rated modest. Rating Modest OBJECTIVE 3 Objective Contribute to incremental growth of SME income Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) Rationale Theory of Change. The causal chain associated with the relationship between SME income and receiving guaranteed loans is also plausible – the loans could have been used to enhance productivity through the installation of better systems, the purchase of capital equipment and the training of employees. However, obtaining accurate information on income growth was as problematic as obtaining information on employment, and had the same deficiencies. The same survey that asked questions about employment growth, also asked for information on profit growth. The results indicated that firms reported a 5-year growth of 66%, which could certainly be described as “incremental”. However, the ICR reports (para 35) that several firms refused to answer questions relating to financial data, which raised the same potential bias issues as the employment data. Because of the lack of evidence attributing higher incomes generated by firms to those provided with credit guarantees by the project, the extent to which Objective 3 was achieved is rated modest. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale All three objectives are rated modest. The overall efficacy rating is therefore also modest. The credit guarantee scheme failed to achieve its targets, although if BSP had remained in the RSF and the other commercial banks had joined, it is possible that better results could have been achieved. The inability to be able to attribute employment growth and income growth to the RSF resulted in the achievement of the second and third objectives being rated modest. Overall Efficacy Rating Primary Reason Modest Insufficient evidence 5. Efficiency Economic Analysis. The net present value (NPV) of the project was calculated at appraisal, to assess returns to the RSF. The calculation resulted in a financial NPV of US$608.4 million. However, as the ICR points out (para 39), there was a fundamental error in the calculation caused by adding expenses of the project to its revenues. If expenses had been deducted from revenues, the project would have yielded a negative NPV of US$100.8 million. The ICR (paras 40-41) points out that there were additional problems with the calculations of the NPV in the PAD. Assumptions regarding the ratio of the amount of the guarantee relative to the value of guaranteed loans Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) was over optimistic. There were also errors in calculating investment income; the PAD assumed that it would earn income on the total amount of the guarantees rather than only the IDA portion that was placed in a trust fund. The ICR team recomputed the NPV based on actual data from reports submitted to IFC by BSP up to the time when BSP withdrew from the RSF. The calculations show a small loss of US$0.4 million for the RSF component of the project, although the basis for the calculation was not included in the ICR. No NPV was calculated for the other components. Administrative Efficiency. There were several issues associated with implementation. The ICR, (paras 43 onwards) describes issues associated with the Project Implementation Unit (PIU) in hiring and retaining well qualified staff. Since this was the first World Bank project to be implemented by the Department of Commerce and Industry (DCI), the PIU had been expected to offset the lack of experience and capacity in the Department of Commerce and Industry in administering World Bank projects and to provide training and technical expertise. However, the possibility of DCI staff assuming a more substantive role in the latter portion of the 13-year project was not incorporated into project design. Furthermore, the PIU failed to raise efficiency in project implementation over time; while the RSF commenced activities in the first year of the program, the implementation of Components two, three and four were severely delayed. The mentorship portion of Component 3 never occurred because of procurement issues, the survey to establish the Baselines for the project was not completed, nor was the Government Partial Credit Guarantee Scheme feasibility study. The implementation of the RSF came to a halt when BSP withdrew from the scheme. The ICR (footnote 21) reports that motivating factors in BSP’s withdrawal were burdensome reporting requirements and delays in processing claims for loan losses. In summary, the calculations of a potential rate of return at appraisal contained fundamental errors, which if corrected might have resulted in the project not being approved. This ICR estimated a NPV for the project for the BSP activities as implemented which showed a small loss. Finally, administrative costs associated with the project were 14% of total actual costs which is high considering that the PSB undertook most of the burden and cost of implementation. This review therefore has assessed the project's efficiency as negligible. Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) 6. Outcome With a substantial rating for relevance, a modest rating for efficacy and a negligible rating for efficiency, the project had major shortcomings in its efficiency. The project's overall outcome is therefore rated unsatisfactory. a. Outcome Rating Unsatisfactory 7. Risk to Development Outcome The ICR states that “there does not appear to be a material risk that development outcomes…will not be maintained.” (para 84). This judgment is based on BSP continuing to lend to SMEs and the use of RSF type lending to SMEs by the government. However, this conclusion is unconvincing. There are no data in the ICR on loan losses by the government on its RSFs. Furthermore, in arriving at its conclusion on the risk to development outcomes, the ICR does not consider the other components of the project. Before the RSF, the BSP was already lending to SMEs. The conclusion from this review is that significant development outcome risks exist on the grounds that even with external guarantees, lending by BSP fell far short of the target. While BSP indicated that it had learned from the project, there are no grounds to assume that it would significantly expand SME lending beyond levels achieved before RSF in the future. 8. Assessment of Bank Performance a. Quality-at-Entry The design of the RSF portion of the project was based on the need to increase access to finance for PNG’s SMEs, which had been identified in numerous surveys as a binding constraint to SME growth. In the event, only one financial institution participated, which suggests that the Bank team was overly optimistic regarding interest of the two Australian commercial banks in PNG. Supposedly they expressed initial interest but ultimately declined to join the RSF. Although Bank staff consulted with their head offices in Australia (interview with the TTL), in the event, they decided not to participate. The result was that only the locally owned BSP became part of the RSF, which immediately reduced its potential impact by half. When BSP withdrew, it meant, essentially that the RSF portion of the project was closed. In addition, calculations of the NPV of this component contained basic errors, that showed that the project would generate a large positive rate of return. However, if the calculations had been done correctly, they would have showed a large negative return. It is not possible to determine whether, if the calculations had been accurate, the project would not have gone ahead or would have been redesigned. However, the size of the miscalculations suggests a deficient quality at entry for this component. The other portions of the project were aimed at improving the capacity of SMEs, promoting ownership and entrepreneurship, increasing the number of women owned SMEs, and raising the capacity of the Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) Department of Commerce and Industry to implement WBG projects through technical assistance. With all of these, there were significant problems with identifying potential pitfalls because of failing to adequately deal with major capacity issues on the part of PNG counterparts as well as operating in the difficult security environment of the country. The limitations arising from the low capacity of Department of Commerce and Industry (DCI) staff, which constituted a major constraint throughout the project and an ongoing risk to development outcome. This is a factor that should have been identified during project preparation. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision There were regular bi-annual consultations that took place on schedule (ICR para 81). The ICR indicates that although the TTLs were dedicated, changes to the project’s results framework were not always fully documented and the project could have benefitted from a restructuring when the mid-term review was done in 2016. The PNG country managers recommended bi-weekly meetings with BSP and the PIU to bolster the latter’s’ limited capacity and to speed up progress with the project. However, this did not occur. Quality of Supervision Rating Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design A significant drawback in the design of the M&E framework was the complexity of the data that PFIs had to report. For example, BSP was required to collect data on baseline and current revenue and employment of participating borrowers. This raised the transactions costs of the RSF and was apparently one of the motives for BSP withdrawing from the project. A further issue was that as the project progressed, there were numerous changes to the definition of indicators and to targets but which were not formally changed in a restructuring. In most cases ISRs did not contain justifications for the changes. The ICR reports (para 74) that “the changes to project indicators seem to have caused confusion for the task team themselves on which figures to report". Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) Furthermore, the Government of PNG, through the DCI was also expected to conduct surveys, but procurement issues so delayed these surveys that the data collected were not of use in monitoring project progress. b. M&E Implementation Inconsistencies in both data monitoring and evaluation were widespread. There was no regular monitoring of borrowers under the RSF nor of beneficiaries under the other components of the project. According to the ICR, BSP officials noted the gap between the data gathering requirements on the one hand and the failure of the project to provide useful analysis of the data. Furthermore, the Government of PNG, through the DCI, was also expected to conduct surveys but procurement issues so delayed these surveys that the data collected were not of use in monitoring project progress. c. M&E Utilization Data on indicators were collected sporadically and appear not to have been used for project management. The ICR (para 76) indicates that the M&E system of the project was not used to track progress, nor for making changes to the project management structure. M&E Quality Rating Negligible 10. Other Issues a. Safeguards The ICR reports (para 78) that the project triggered three safeguard policies at preparation, namely OP4.01, and environment assessment, OP4.10 indigenous peoples and OP4.12 involuntary resettlement. BSP generated an Environmental and Social Management Framework and a process to screen loans that covered all three of the Ops. Environmental and Social safeguard ratings were moderately satisfactory during implementation and no waivers were requested nor did any compliance issues arise. b. Fiduciary Compliance Initially, following teething issues in the early stages of the project, financial management was satisfactory as the PIU developed capacity and familiarity with both government and World Bank systems. However, the ICR states that once it became clear that the project would not continue, staff qualified in financial management left the PIU. As a result, the earlier high standards of financial management deteriorated. Submission deadlines were missed, and timely documentation was not submitted. Since documentation Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) was not submitted regarding information on the balance in the Designated Account, the project has been classified as a Lapsed Loan. c. Unintended impacts (Positive or Negative) The IFC developed expertise in PNG’s financial sector, including increasing its investment in BSP. d. Other N/A 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment With a substantial relevance, a Moderately modest efficacy, and a negligible Outcome Unsatisfactory Unsatisfactory efficiency the outcome rating is unsatisfactory. Both the quality at entry and the Moderately Bank Performance Unsatisfactory quality of supervision were Unsatisfactory deficient. Quality of M&E Negligible Negligible High quality analysis and Quality of ICR --- High presentation of the evolution of the project 12. Lessons The ICR contains the following lessons which IEG presents with some modification Changes in project indicators need to be made formally to be credible. While project indicators may change over time because of changes in circumstances, these require full documentation so that the logic of why the changes occurred can be analyzed. In the case of this project, inadequate discussion and documentation of the changes in the ISRs made the rationale for changes impossible to follow. Institutions required to provide data deserve feedback. If implementing institutions are required to provide data on their operations, it is important that they receive feedback from the analysis of the data they provide. In the case of this project, failure to provide BSP with a meaningful feedback on Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) the data submitted was at least partly responsible for its withdrawal from the RSF and possibly limited what it might have learned from the project. Transfer of knowledge from consultants to local staff enhances capacity building. For decades it has been agreed that using highly qualified consultants to initially staff PIUs in low- capacity environments is warranted, but over time local capacity requires strengthening for benefits to be sustainable. In the case of this project, little knowledge transfer occurred so that when BSP withdrew from the RSF, the consultants departed, and the project essentially came to a halt because of lack of an understanding of the project on the part of PNG counterparts. When designing a credit guarantee program that also provides training, implementing the training program after SMEs have qualified for loan guarantees reduces the possibility that participants in the training program will not be part of the lending program. In PNG, the opposite sequencing occurred. As a result, only six percent of training beneficiaries managed to access loans. 13. Assessment Recommended? No 14. Comments on Quality of ICR This is a high quality ICR that sets out the issues clearly and undertakes in-depth analysis of the issues and problems with this project. Where there were gaps in project preparation and analysis, they were filled by the ICR. A good example was the identification of errors in calculating the NPV of the project. The ICR calculated what it would have been at project preparation and what it turned out to be at project closing. The discussion is generally clear and thoughtful, and the analysis of important is of high quality. In addition, the ICR follows OPCS guidelines, is concise, internally consistent, results-focused and provides lessons based on analysis in the ICR. A hallmark of this ICR is its presentation of an interesting story line with relevant and informative footnotes. a. Quality of ICR Rating High Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PNG - SME Access to Finance Project (P120707) Page 14 of 14