Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) Report Number: ICRR0024180 1. Project Data Project ID Project Name P125232 MN Third Sustainable Livelihoods Project Country Practice Area(Lead) Mongolia Social Sustainability and Inclusion L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-54870,IDA-65880,TF-A0794 31-Dec-2018 39,595,894.52 Bank Approval Date Closing Date (Actual) 06-Jun-2014 31-Dec-2023 IBRD/IDA (USD) Grants (USD) Original Commitment 24,800,000.00 5,665,000.00 Revised Commitment 42,465,000.00 5,665,000.00 Actual 39,686,137.02 5,665,000.00 Prepared by Reviewed by ICR Review Coordinator Group Alejandra Eguiluz Vibecke Dixon Avjeet Singh IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO), as stated in the Financing Agreement (CN 5487, Schedule I, page 5) and in the Grant Agreement (TF0A0794, Schedule 1, page 5) is “to improve governance and community participation for the planning and delivery of priority investments in rural areas of Mongolia". This description coincides with the one presented in the PAD (PAD385, page 4). The PDO will be parsed, for the purposes of this review, as follows: Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232)  To improve governance for the planning and delivery of priority investments in rural areas of Mongolia  To improve community participation for the planning and delivery of priority investments in rural areas of Mongolia b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components The project was structured around the following three components: Component 1. Capacity Building for Local Governance and Livelihoods (cost at appraisal: US$11 million; revised cost US$5.95 million; actual cost: US$5.95 million). This component sought to build capacity at the local and national levels for the implementation and further development of the Government’s programs and mechanisms to support rural development. It supported efforts to help the districts to effectively manage the significant increases in governance responsibilities related to investment planning and execution that had been delegated through the Integrated Budget Law (IBL); a national law that established a formula-based intergovernmental transfer mechanism for budget decentralization at province and district level. The component would finance training and technical assistance activities to officials at the district level, in areas related to the implementation of the Local Development Fund (LDF), which was established by the IBL as a formula-driven intergovernmental transfer mechanism, providing predictable and sizeable funds to support local capital investment in public infrastructure and services. The training and technical assistance would be in medium term planning, community participation, budget preparation; finance management and procurement; reporting; monitoring and evaluation (M&E) and pastureland planning and management (the pasture land management activities were dropped during implementation). Component 2. Good Governance Performance-Based Support Program (cost at appraisal: US$21.6 million; revised cost US$33.58 million¸ actual cost: US$33.58 million). This component consisted of two subcomponents. Subcomponent 2.1: Performance-based financing (PBF) which funded performance-based grants (PBGs) that were awarded to districts (soums) with funds from the LDF, based on their performance, as measured by an annual review using the Annual Performance Assessment (APA) scoring mechanism. The objective of the PBF would be to enhance good governance in the entire budgeting process (policy planning, budget preparation and adoption, budget execution, accounting, reporting, internal and external control and public scrutiny). Particular importance would be given to citizen participation and to their satisfaction with the LDF process and outcomes, via an alignment of soum budgets with the citizens’ priorities at the rural subdistrict administration units. The APA rated all districts annually, and was applied by independent assessors against a set of agreed performance indicators. Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) Sub-component 2.2: this subcomponent financed the setup and management of the Annual Performance Assessment (APA), and was to be financed by the Swiss Development Cooperation grant. Component 3. Project Management and Monitoring and Evaluation (cost at appraisal: US$3.6 million; revised cost US$2.94 million; actual cost: US$2.94 million). This component financed the staff and operating costs of the Project Implementation Unit (PIU) and the implementation of key functions such as M&E, procurement, Financial Management (FM), and management of environmental and social safeguards. The development of the management information system (MIS) for the LDF; specialized studies; and the project baseline, midterm, and end line surveys. Changes to components During the third restructuring in Sept 2017 (RES28497), at the government’s request, references to the Soum Program were removed from the Financing Agreement, given that budgetary constraints in the country had reduced the funding for that program. The definition of activities under Component 2 was amended to include the provision of financial support from the project to the Local Development Fund. The allocation was of US$ 12 million, and helped cushion the precipitous decline in the Government’s general Soum LDF budget during the recessionary period of 2017 and 2018. The elimination of the Soum Program was an important factor in the decision to drop the pasture land management activities initially planned at appraisal. The components’ costs were revised through the eighth (and last) restructuring in October 2022, (RES52580), to reflect: (a) a reduction in the total value of the Swiss Development Cooperation (SDC) co- financing grant (from US$11.4 million to US$5.67 million), (b) an increase in Component 2 costs related to the LDF ‘top-up’ payments that had been allowed through the third restructuring (RES28497), and (c) reductions in the capacity building and operations costs due to COVID-19 and related budget austerity measures. Thus, Component 1 costs were reduced from US$9.4 million to US$5.95 million, Component 2 costs were increased from US$29.3 million to US$33.58 million, and Component 3 costs were reduced from US$4.10 million to US$2.94 million. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. At appraisal, the total project cost was estimated at US$ 36.2 million. The revised project cost was US$ 42.47 million and the actual project cost was US$ 42.47 million, which represents 117% of the cost at appraisal and 100% of the revised cost. Financing. The project was financed by two IDA credits and a co-financing Trust Fund grant from the Swiss Agency for Development and Cooperation (SDC). IDA-54870-MN was of US$ 24,8 million and the grant (TF-A0794) was of 5,67 million. An additional IDA credit of US$ 12 million was approved in April 2020: IDA- 65880, to support two additional rounds of performance-based grants for the districts, continue with the capacity-building exercise, and to strengthen the project’s M&E. Borrower contribution. There was no borrower contribution in this project. Dates Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) The project was approved in June 2014, made effective in July 2015, and had the MTR in October 2016. The original closing date was in December 2018 and the actual closing date was December 2023, i.e. a five years’ extension in total. The project underwent 8 level-II restructuring processes:  First restructuring (RES20832), in January 2015. The dissolution of the Ministry of Economic Development in Mongolia provoked the need to change the Implementing Agency to the Ministry of Finance. The date regarding adoption of the Project Implementation Manual was pushed back, given the delayed start of project implementation.  Second restructuring, in September 2015, with changes in the disbursement arrangements, as well as in the cut-off date for retroactive financing for the SDC grant, that was changed from January 30, 2015, to May 26, 2015, to reflect the date on which the donor signed the Trust Fund Agreement.  The third restructuring (RES28497) in September 2017, following the MTR, introduced changes in components, costs and loan closing dates (from December 31, 2018, to October 31, 2020), as well as the following changes in the RF: The PDO indicator ‘Percentage of soums who qualify for performance-based financing’ was replaced by the indicator "Average soums’ APA score". This change is not considered substantial, as the meaning of the indicator remains unchanged, because the APA score is the tool applied to qualify the performance of the districts (soums). The % of qualified districts and the APA score are two ways of referring to the same reality. The target also did not change, and this has been reported by the team as a coincidence (conversation of July 26). The PDO indicator "Percentage of soum citizens surveyed agreeing that LDF is meeting local priorities" (disaggregated by gender) was rephrased as ‘Percentage of soums’ citizens surveyed reporting that LDF financed investments reflect their priority needs’ (disaggregated by gender). The new description focuses more on the individual priorities, but both indicators measure the same reality: how citizens assess the match between what they think is needed (locally or individually) and how funds are allocated. The difference in the description does not pose substantial changes to the meaning of the indicator and what it tries to assess. The targets of this indicator were revised down, from 85% (original) to 65%. The PDO indicator “Participation: nationwide household participation rate at rural subdistrict level (bagh) citizens’ public meetings to discuss soum budget priorities, disaggregated by gender” was revised to the following: “Participation: Percentage of bagh households participating in public bagh meetings”. The gender angle was dropped. The new description does not reflect substantial changes in the indicator. Two new PDO indicators were added: “Number of subprojects completed” and “Direct project beneficiaries (Female beneficiaries)”. The definition of activities under Component 2 was amended to include the provision of financial support to the Local Development Fund. This revision allowed for allocating approximately US$12 million to general LDF financing as bridging support to help cushion the precipitous decline in the Government’s general soum LDF budget during a recessionary period.  The Fourth restructuring paper (RES33209), in June 2018, extended the SDC grant to the closing date of June 30, 2019 and increased the financing amount to the project to a total of US$ 5,665 million. Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232)  The Fifth restructuring paper (RES37811), in June 2019, presented an extension of the SDC grant to December 31, 2019, at the request of the government.  The Sixth restructuring paper (RES39881), in December 2019, presented an extension of the SDC grant to June 2020, at the request of the government.  The Seventh restructuring (RES42507), in June 2020, comprised the following changes: Change in Loan Closing Date of the SDC grant to June 30, 2021; Reallocation between disbursement categories; adjustments to the RF (as a consequence of the AF approved in 2020) as follows: the 2 PDO-level indicators that were added at the 3rd restructuring were dropped (number of beneficiaries and number of subprojects), as well as the following intermediate indicator: “Guidelines necessary to improve governance and community participation for the planning and delivery of LDF developed”, which had been achieved in May 2019. Some of the remaining SDC grant funds were allowed to co-finance PBGs, which were previously financed only by the IDA credit.  The Eighth restructuring paper (RES52580), in Oct-2022, introduced the following changes: Component costs were revised to reflect (a) a reduction in the total value of the SDC co- financing grant (from US$11.4 million to US$5.67 million), (b) increase in Component 2 costs related to the LDF "top-up" payments allowed through the third restructuring, and (c) reductions in the capacity building and operations costs due to COVID-19 and related budget austerity measures. Thus, Component 1 costs were reduced from US$9.4 million to US$5.95 million, Component 2 costs were increased from US$29.3 million to US$33.58 million, and Component 3 costs were reduced from US$4.10 million to US$2.94 million. The changes in the RF consisted of: (i) revising the targets of 4 intermediate result indicators: the target of the indicator “Number of Aimag Level Support Team, national government staff, and soums’ government staff trained” was increased from 20,000 to 35,000; the target of the indicator “Percentage of bagh citizens who know the approximate current annual LDF allocation for their soum” was reduced from 25% to 10% (and equally reduced in its gender-disaggregated component); and the target of the indicator “Number of projects completed” was increased from 29,900 to 40,000. A split evaluation is not undertaken, as the changes at the PDO-level indicators that took place in the third and seventh restructurings did not substantially affect the PDO. The changes of the 3rd restructuring were on the description of the indicators but did not change their meaning and what they were set to measure in any substantial way. The dropping of 2 PDO-level indicators in the 7th restructuring is also not considered substantial, because: (i) they were not in the original RF (were introduced at 3rd restructuring); (ii) they are not pitched at outcome level and did not substantially contribute to the measurement of project outcomes. While some intermediate indicator targets were reduced, this does not trigger a split rating as a) they project did not achieve the original nor revised targets; and the more significant indicator targets were increased, and they were achieved. 3. Relevance of Objectives Rationale Context at appraisal Mongolia is a lower-middle-income country in East Asia, with a population of 3.4 million and a territory of 1.6 million km2. It is one of the most sparsely populated countries in the world, with an economy heavily Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) dependent on mining. Agriculture, dominated by livestock breeding, is particularly important for the livelihoods of rural populations who constitute approximately 40 percent of the population (ICR, page 6). The rural livelihoods are, however, increasingly challenged by climate change: increased frequency of severe weather events, exacerbated desertification, land degradation, and natural disasters such as flash floods, fires, and severe winter conditions. Additionally, weak governance and regulation over using common natural resources and a lack of effective cooperation and collective action at scale perpetuate a vicious cycle of declining natural resources. At the time of the appraisal (2014), Mongolia was enjoying a period of rapid economic growth driven by the mining sector, but with rural areas risking falling behind. The country needed to develop efficient and transparent mechanisms for decentralized service delivery, raising the capacity of the provinces (aimag) and the district (soum) officials to manage funds transparently and maintain strong local government performance. The approval of the Integrated Budget Law (IBL) in 2011 (effective from January 2013) was an important development in this regard. The new law established a formula-based intergovernmental transfer mechanism for capital investment resources at the aimag and soum levels. This project was the third and last of an adaptable program loan (APL) series, designed to help soum governments operationalize the rules and regulations that govern the Local Development Fund (LDF), which was created with the introduction of the IBL. The purpose of the overall APL was to reduce vulnerability and achieve secure and sustainable livelihoods for poor and vulnerable near-poor households nationwide. Whereas the first project was a pilot and the second scaled up the interventions, this third project was designed to institutionalize the mechanisms and give sustainability to the results, creating a more effective and transparent system for local governance for sustainable rural investments at a national scale. Consistency with Country Strategy At closure, the project objectives remained highly relevant to Mongolia’s national development objectives. The project directly aligned with two of the nine objectives of Mongolia’s long-term development policy “Vision 2050”, approved in 2020. The PDO aligned with the Governance pillar of this policy by creating an environment of continued learning for civil servants in various technical areas irrespective of time and distance, establishing systems for transparency and accountability in the prioritization of local investments with citizen participation and building trust between citizens and local governments, supporting digitization of government systems by creating an MIS integrated with the national budget mechanisms, and establishing a regular and transparent mechanism for assessing local government performance. Additionally, the PDO contributed to the Regional and Local Development pillar of “Vision 2050” by providing a mechanism to respond to local priorities, as identified by citizens in multiple areas including sustainable agriculture and basic social services (ICR, page 16). Consistency with Bank strategy The PDO continued to be aligned with the new World Bank Country Partnership Framework (CPF) for Mongolia, 2021–2025 (Report No. 132141-MN). It was a key contributor to Focus Area 1 of the CPF: Strengthening Economic Governance, and the CPF envisioned a follow-up operation to support the LDF. The PDO was well aligned to emerging development challenges and priorities in the country program such as climate adaptation and mitigation, food security, and economic diversification. Previous Bank experience Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) The PAD presents how the lessons learned in the previous 2 projects of the APL and in other similar projects nurtured the design of this one, and clear key examples are given on how. The effort in capturing the lessons is also explicitly mentioned regarding the Environmental and Social Management Plan, and mitigation measures that had proven effective under previous projects were brought to this one (PAD, page 17). The novelty in this project was the introduction of the performance-based grants. The lessons from international projects are stated to be drawn to this one, to ensure effectivity (ICR page 26). PDO level. The PDO, “to improve governance and community participation for the planning and delivery of priority investments in rural areas of Mongolia” is coherent with the previous two Sustainable Livelihood Projects (SLPs) of the Adaptable Program Loan. Whereas the first phase was a pilot and the second focused on scaling up interventions at national level, this project aimed at institutionalizing the management of investments for sustainable livelihoods. The PDO is pitched at an intermediary outcome level, as improved governance and increased community participation are both means to arrive at a higher end (improved livelihoods and quality of life for rural communities). Given the alignment of the PDO with the Bank and country strategies, the Bank's previous experience, and the PDO level, the relevance is deemed High. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve governance for the planning and delivery of priority investments in rural areas. Rationale The PAD did not include a ToC as that was not a requirement at the time, but the ICR created a ToC retrospectively, based on the PAD text, both for the Adaptable Loan Program as a whole and for this third project. The ToC outlines that: the activities on training, technical assistance and knowledge exchange for the districts and provinces officials (activities), the development of guidelines and manuals governing the Local Development Fund (activities), the provision of performance-based grants under the LDF (activity) and the development of district level performance monitoring systems (activities) would be expected to lead to having trained staff (output), an improved LDF regulation (output), sub-projects being completed thanks to the grants provided (output) and a measuring system developed for the improved management (Average Performance Assessment, APA) and for the Monitoring Information System (outputs). These outputs in turn were expected to lead to the intermediate outcomes of improved government capacity to manage the LDF, mobilized citizen participation, increased provision of basic rural infrastructure via de performance-based grants (PBGs) and improved transparency and accountability of the district performance. In turn, these outcomes were expected Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) to lead to improved governance and participation for the planning and delivery of priority investment in rural areas (objectives). The logic presented in the ToC is clear and coherent. Although the ICR presents three intermediary outcomes, only 2 of them are captured in the PDO description (the basic infrastructure improvements are not part of the PDO, although they are an important part of the project interventions). The intermediary outcomes that are directly linked to Objective 1 are the following: (i) improved government capacity to manage the LDF, (ii) increased provision of infrastructure via de PBGs and (iii) improved transparency and accountability. The three intermediary outcomes were expected to lead to achievement of Objective 1 of improved governance for the planning and delivery of priority investments in rural areas. Outputs  52.8% of trained province-level staff and national government staff are applying skills learned through the project (planning, procurement, financial management, community participation and M&E), of a target of 80%. Not achieved  73.8% of soums’ government staff trained by province-level support team with improved capacity (applying learned skills in planning, procurement, FM, community participation and M&E), of a target of 85%. Not achieved  52,569 province, national and district level government staff trained, of an original target of 20,000 and revised target of 35,000. Target exceeded  4 LDF-related or other MoF regulations were submitted for MoF review, with SLP3 support, of a target of 4. Target achieved  5.2% of bagh citizens know the approximate current annual LDF allocation for their soum, of an original target of 25% and a revised target of 10%. Not achieved  41,156 sub-projects completed, of an original target of 29,900 and a revised target of 40,000. Target exceeded  1,120,059 direct project beneficiaries, of an original target of 600,000. Target exceeded  571,230 women beneficiaries of an original target of 300,000. Target exceeded Outcomes The Average soums’ Annual Performance Assessment score (previous description of this indicator: “Percentage of soums that qualify for performance-based financing”) achieved 71% at closure, with an original target of 70%. Achieved. The APA score is an index measure of six performance areas, with 34 specific indicators. The performance areas are (a) participation in the planning process; (b) project planning, prioritization, and budgeting; (c) disclosure and transparency; (d) budget execution and procurement; (e) M&E; and (f) asset ownership and management. The average score increased for each of the six APA performance areas, with an especially steep increase in the area of participation. It increased in all provinces. There was a sharp drop in the number of poor performing soums (which did not qualify for PBGs). This can be seen in the decline in number of soums that scored ‘zero’ on one or more areas of the APA (from 41 in 2016 to 14 in 2023). The APA focused on compliance but did not measure results of the subprojects implemented Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) 54.5% of soum citizens surveyed reported that LDF financed investments reflect their priority needs, whereas the original target was 85% (and had gender disaggregation) and the revised target was 65% and was not disaggregated by gender. Not achieved. This indicator was rephrased with the Sep 2017 restructuring from its original wording “Percentage of soum citizens surveyed agreeing that LDF is meeting local priorities”. A new target of 60% was set after the baseline survey. With the Additional Financing in 2020 the target was raised to 65% (email of team from August 4). The ICR reports on the following additional factors that may have had an influence on the low results: (i) The fact that the size of the grants declined substantially between 2017 and 2020, causing some local disappointment; (ii) the fact that unrealistically long lists of subproject proposals were submitted by citizens each year, of which only a small fraction could be approved. The outcome indicators adequately measure results contributing to achievement of Objective 1, as the first one measures the technical improvement of the performance of the districts (APA score), whereas the second seeks validation of this improvement among the citizens. Although one of the 2 outcomes has not been achieved it was the spirit of that outcome (as phrased originally) that the communities perceive that the budget allocations met local priorities. The fact that the outcome was rephrased and the concept of their priority needs was included, added a subjective factor that may have influenced the responses the way the team has explained. This is confirmed by the additional survey data collected. A shortcoming is that the outcomes derived from the subprojects (mostly basic infrastructure) are not captured. The ICR states that the project financed an estimated 6,900 subprojects for education, health, water and sanitation, community lighting, veterinary services, winter fodder, and other community-prioritized services. The estimated population benefitting of these projects is 1.1 million. Having left these important outcomes without measurement is a missed opportunity to measure essential project results. Additional outcomes reported in the ICR not included in the RF, and hence without target values The ICR reports the following results of an end-line survey: 94 % of respondents concluded that the LDF’s efforts were effective in supporting the development of soums and local communities, 71.4 percent believed that the LDF’s investments brought changes that benefitted all citizens and communities, and only 22.7 percent felt that these changes catered solely to a specific group. Despite the mixed results at output level, it is the outcome indicator of the APA scoring system that best captures the achievement of Objective 1, and it has very good results. The efficacy rating of Objective 1 is therefore Substantial. Rating Substantial OBJECTIVE 2 Objective To improve community participation for the planning and delivery of priority investments in rural areas of Mongolia. Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) Rationale ToC The training and technical assistance activities in community participation, the provision of performance- based grants and the development of the performance monitoring systems (activities) were expected to lead to the public officials trained in community participation (output), the completion of subprojects financed with grants (output), and the Monitoring information system and APA assessments carried out to inform budget process (output). These outputs, in turn, were expected to lead to the intermediary outcomes of (i) citizen participation mobilized and to (ii) improved transparency and accountability of the performance at district level. Those intermediary outcomes were expected to lead to the improved participation for the planning and delivery of priority investments in rural areas (objective). The concept of participation was clarified by the team (conversation of July 26) and covers participation in the meetings where the LDF allocation is voted over. Outputs The outputs reported under Objective 1 are also relevant under Objective 2.  52.8% of trained province-level staff and national government staff are applying skills learned through the project (planning, procurement, financial management, community participation and M&E), of a target of 80%. Not achieved  73.8% of soums’ government staff trained by province-level support team with improved capacity (applying learned skills in planning, procurement, FM, community participation and M&E), of a target of 85%. Not achieved  5.2% of bagh citizens know the approximate current annual LDF allocation for their soum, of an original target of 25% and a revised target of 10%. Not achieved  41,156 sub-projects completed, of an original target of 29,900 and a revised target of 40,000. Achieved Outcomes  60.4% of bagh households participated in public bagh meetings, of a target of 60%, Achieved An additional measurement of this outcome is facilitated by the APA score (outcome indicator for PDO1), as it has one criteria that relates to participation, and it registered the highest increase over the course of the project, starting at less than 10% and reaching over 70% in 2023. In order to achieve community participation, this third project of the APL focused on improving the capacity of the government officials. The outcome indicator for PDO2 does not sufficiently reflect this process or the existence/improvement of these mechanisms for community participation through the officials in the government (it is the APA improved scores that give complementary evidence on this). The bagh meetings served to discuss and to vote over the proposals for budget allocation. 47.3% of the respondents in a survey stated they had voted in the selection of LDF projects, and digital voting also became possible as it was introduced during COVID. Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) The outcome indicator does not, on its own, assess achievement of the PDO2. However, when assessed in conjunction with the community participation component of the APA score, it is clear that both indicators were achieved. The rating for Objective 2 is Substantial with minor shortcomings. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The Substantial rating of Objective 1 and the rating of Substantial with minor shortcomings for Objective 2 give an overall efficacy rating of Substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic efficiency Ex-ante economic analysis. The PAD (page 13) states that an ex-ante economic analysis of the LDF-funded activities was not possible, because of the difficulties in quantifying the benefits of fiscal decentralization (as disbursements would not be linked to specific investments on the ground), and because the choice of investments was not known beforehand. Ex-post economic analysis. The ICR presents two economic analyses. One is on the return on the capacity- building activities and the incentive system created by the APA and PBG systems (that would increase the efficiency of the LDF financing) and the other is on the infrastructure sub-projects of the communities financed by the LDF (PBGs). For the first analysis, the methodology used compares the “with project” situation (with the additional LDF financing) to a counterfactual, considering only the government budget for the LDF (without the additional LDF financing). The discount rate is set conservatively at 15 percent, because of the high interest environment. The analysis shows that the project had an NPV of US$34.5 million, and an IRR of 56%. With the sensitivity scenario the project returns remain positive: IRR of 21% and NPV of US$ 4 million. For the analysis on infrastructure subprojects, three types of subprojects have been examined: fodder storage units, wells and street lighting. Subproject costs and benefits are based on surveys of the beneficiary Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) communities and households. The total sample of interviewed representatives from subprojects was 18 for fodder storage units, 20 for street lighting and 19 for the wells. For each subproject, a situation without project (prior to the investment) is compared to a situation with project. The costs and benefits include the cost of the subproject and its maintenance costs and taxes, as well as benefits and time/cost savings of the households in the community. All the assessed investments are viable with positive net present values and internal rates of return. The fodder storage was least profitable (IRR of 16%), but the profits are more tangible as they correspond to cost savings for fodder purchase. Instead, the wells (IRR 199%) and street lighting (IRR 354%) have very high benefits, but these do not necessarily result in additional cash flow for households, as the benefits mostly correspond to time savings or having additional value to time. The annual cost of the APA exercise is considered low, at about 0.8 percent of the average annual LDF allocation to districts, in comparison with similar programs (eight other World Bank-funded projects conducting APA-type assessments spend on average 1.2% of the fund budget on the assessment-ICR, page 21) Other operational and administrative factors affecting efficiency: The project was extended several times, resulting in a total of 5 years extension. The ICR reports on several factors affecting project efficiency:  Factors hampering early implementation: the abolition of the main implementing ministry and a delay in parliamentary ratification of the project financing; the loss of some PIU staff, the need for the government to hire qualified staff to manage the LDF, and challenges with completing the initial APAs in time to inform budget allocations for the following year. These delays necessitated an initial extension of the project closing date and are responsible for the effectiveness date being 1 year later than approval date.  Budget reductions and important component reallocations: (i) A reduction of the SDC grant amount in year 2 that led to a reduction in the budget for capacity-building activities under Component 1; (ii) the reduction of the governments flow of funds to the LDF in 2016 and 2017 due to economic recession and government fiscal constraints encouraged by the IMF. At the Government’s request, the World Bank reallocated US$12 million of the funds earmarked for PBGs to the general district LDF budget.  Delays and poor performance of the project in the first two years of implementation triggered the need to ensure stronger links between the project activities and the general government budgeting processes, causing changes in the PMU and the implementation arrangements to align them to government systems  The Law on Budget Austerity (effective May 2022) and the COVID-19 pandemic significantly restricted travel and face-to-face capacity-building activities. Despite the above, the Technical Audit Report finds that 93 percent of assessed subprojects were implemented on time (ICR, page 47). The economic analysis shows positive results, with the caveat of the sample of the PBGs (58) being too small in comparison with the total number of subprojects (+40.000). Despite the significant delays, the changes in implementation arrangements and funding, the frequent adjustments contributed to realigning the project to ensure its efficiency, and Efficiency is therefore rated Substantial. Efficiency Rating Substantial Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome High Relevance, Substantial Efficacy and Substantial Efficiency result in an overall outcome of Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Political risk. Political interference both in the assessment and in the process of planning and delivering local investments is a risk, especially in periods preceding local elections. If the APA is not completed in a transparent and rigorous manner, trust in the performance-based system may be compromised, which in turn can lower the incentives of local governments to perform well, and for citizens to actively engage in the local investment prioritization and budget implementation process. Therefore, continued political commitment will be essential to maintain a rigorous APA process. Technical risk. There is the risk to the quality of implementation of the APAs. From 2024 onward, the APA process will rely on data from the MIS rather than on independent field-based assessments, as has been done under the project. While the correlation between the field-based APA and the set of MIS indicators chosen to measure performance going forward has been demonstrated, there is a risk that the system could be compromised in its neutrality to artificially inflate ratings. Institutional capacity risk. In the absence of the project, support for capacity-building activities may decline, and governance performance may start to lag behind. Previous APAs have shown a strong correlation between investing in training and capacity of local government officials and an increase in performance results, especially in the area of community participation and engagement of the communities in the decision-making process for budget allocation. Financial risk. The LDF has been financed from different sources of national and local revenue over the course of its implementation since 2013. The sources are varied, the most significant being the royalties from mining, oil, and other large extractive projects, representing approximately 55 percent of the LDF revenues Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) between 2013 and 2023 and currently (as of 2023) 89 percent. Dependence on such revenues that are linked to international commodity prices susceptible to wide fluctuations based on forces outside the Government's control exposes programs such as the LDF to levels of unpredictability. 8. Assessment of Bank Performance a. Quality-at-Entry The World Bank drew on global experience to propose a system of performance-based financing (the novelty in this 3rd project of the series that was not included in the previous ones). The team organized a learning visit for the Government of Mongolia to the Indonesia National Program for Community Empowerment, which had a well-established mechanism for citizen participation in decision-making at the village level. Lessons learned from implementing PBGs internationally and from local governance programs globally were incorporated in the project design, namely the fact that they are effective at bringing about institutional change and that they provide incentives to local government to improve key institutional performance areas and also more broadly in areas of good governance. A challenging design decision involved the local-level implementation arrangements. The project envisioned recruitment of a five-person team (SST) in each province, reflecting a clear need for intensive outreach and training of citizen representatives at the district and sub-district levels. Two main implementation options were considered: i) to co-mingle funds with the LDF and subject the project flow of allocated disbursements to the country systems; or ii) to operate a separate IDA financed “window” for the LDF top up. The team assessed that although the former may carry more risk and may present a greater implementation challenge, lessons showed that this approach was more consistent with the aims of institutionalization that the project wanted to prioritize. Lessons on effective capacity building for local government were also introduced (longer term, frequent and repeated training and with locally-based trainers and an effective back-stopping and oversight structure at the local level). The Environmental and Social Management Plan benefitted from the lessons learned in the previous 2 phases, and incorporated the mitigation measures that had previously worked best. There were minor shortcomings at appraisal: the pastoral risk management activities were not sufficiently well-defined at preparation; and (ii) not including outcome-level indicators to capture the improvements in the livelihoods resulting from the basic infrastructure subprojects. Quality-at-Entry Rating Satisfactory b. Quality of supervision 18 supervision missions were conducted during the life of the project (more than two per year). Several of the 8 restructuring processes show the Bank team’s rigor in applying necessary adjustments to better align the project activities and results to the changing reality. For instance, at the last restructuring (8th), the Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) targets of some intermediary result indicators were revised and increased, to better align them with the actual targets at that time, as they had largely surpassed the original ones. The institutional changes at the level of the main ministerial counterpart during the first 2 years of the project coupled with the lack of definition of the pastoral risk-management elements of the project at appraisal, affected the implementation of the pastoral risk-management, and it was dropped in the 2017 restructuring. Having had zero disbursement in the first year of effectiveness, the disbursements more than tripled within a year following the restructuring of 2017. The approach of delivery mechanisms for capacity building and outreach changed during COVID-19 and the subsequent Law on Austerity, yielding interesting unexpected lessons on how youth and women participation increased with the new electronic mechanisms for participation. Implementation Status and Results Reports were produced regularly. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design At appraisal, the M&E was meant to comprise a set of qualitative and quantitative indicators that would monitor outcome and implementation progress, focusing on the overall LDF performance and responsiveness to local priorities, and citizen’s engagement in the budgetary process. The local government performance would be evaluated annually through Annual Performance Assessments (APAs) carried out by independent experts. The core areas for assessment, in line with IBL requirements, included the following categories: (i) citizen participation (ii) budget preparation, (iii) budget execution (implementation), (iv) reporting (v) financial management and (vi) citizen feedback. Further, the APAs were to be designed to monitor compliance with the environmental and social safeguard policies of the government and the World Bank. To enhance the focus on empowering citizens to monitor the use of public funds, the project was to continue with the Participatory M&E (PM&E) approach already established countrywide in the previous project (phase 2). The aim was to institutionalize the system, supported by unpaid, volunteer Community Supervisors and Evaluators, as an integral part of the official process for implementing the LDFs (PAD, page 41). The PIU had the overall responsibility of monitoring the LDF on behalf of the MOF. At the local level, the Supervision Committee of the elected soum Citizens’ Representative and soum government officers were to provide information and deliver performance monitoring. Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) The M&E system’s focus reflected the priority concern of the MOF, regarding the operationalization and management of the LDF. A shortcoming is that the RF could have included the outcomes of the subprojects on basic infrastructure. (increased quality of life, better access to basic services such as water and electricity, improved productivity of cattle production), since most of the budget allocation was on the subprojects for basic infrastructure services at community level. b. M&E Implementation It was not until 2018 that an M&E specialist was hired in the PIU and original target indicators were established. A baseline took place in 2019. The midterm survey was conducted electronically under COVID-19-related restrictions. With the emphasis on inclusion and women’s participation, the RF included output-level gender-disaggregated measures. The system also allowed for a more detailed look into the performance of different parts of the country and aspects of performance to be able to address specific lagging areas. The RF was substantially revised during the second year of implementation (3rd restructuring in 2017). The introduction of the APA scores in the description of the outcome indicator on improved governance allowed for a comprehensive way to measure overall change in local government performance. The APAs were carried out by independent consultant firms by applying a single standard to rank soums to qualify for PBG. Consulting firms were contracted to carry out baseline, midterm, and end-of-project evaluation surveys. The PIU took the overall responsibility for developing and carrying out the M&E system on the government’s behalf until the end of the project, when it was expected to be taken over by MoF. The M&E specialist in the PIU consistently documented results from administrative data and independent surveys. The PIU prepared and submitted quarterly progress reports and midterm and end line reports to the MOF, World Bank, and SDC in a timely manner. Arising questions on the methodology for measuring different RF indicators were resolved and the PIU was able to gather data to allow for consistent measurement of the indicators through the baseline, midterm, and end line surveys. The targets of some new PDO indicators were only set with the AF once the PIU had recruited an M&E specialist, established a methodology for measuring the new indicators, and established a baseline (Correspondence with Bank team, August 4th, 2024). c. M&E Utilization The project’s M&E results were used to improve implementation performance. For example, the APA performance measures were used to better target the capacity-building activities, allowing to place greater attention on the provinces and districts with lower performance and governance areas that exhibited lower scores. The APA scores were also used to select and pair higher and lower performing districts for peer learning visits. The MTR helped reassess targets and content of the LDF trainings and communications. When the results showed that local government officials placed higher value on in- Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) person versus virtual learning, the project recommended restarting face-to-face trainings where possible, even though the Budget Austerity Law limited the opportunity for travel. The project’s MIS was tested and shown to be a suitable source of proxy information for the APA process in measuring district performance. The comparison exercise between the MIS and the APA motivated a simplification of the APA indicators (reduced from 34 to 12). Recommendations were made to strengthen the ownership and M&E aspects of the APA score. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project complied with the safeguards. The project was assessed as Category B-partial assessment, three safeguards were triggered (OP 4.01, Environmental Assessment, OP 4.04, Natural Habitats, OP 4.10, Indigenous Peoples). OP 4.10, Indigenous peoples, was triggered due the nationwide scope of the project including areas populated by ethnic minorities (when a change to one of the PDO-level indicators took place). An Environmental and Social Management Plan was prepared and disclosed on April 9, 2013. An Indigenous Peoples Plan was prepared and disclosed on April 11, 2013. During the project’s additional financing, both plans were updated and redisclosed on March 1, 2020. The project contributed to strengthening country systems for environmental and social risk management at the district level. The PIU did not have the opportunity, or sufficient capacity, to assess each project-funded subproject individually. The original risk of this situation was originally assessed as low and acceptable, but proved otherwise in reality, as it was reported that there had been use of pesticides that had not been screened by officials (conversation with the Bank team on July 26th, 2024). As a consequence of this the PIU hired a safeguard specialist. When the bank team and the PIU recognized that environmental and social considerations needed to be integrated more systematically in the LDF implementation, the project undertook specialized environmental and social risk management studies, which resulted in the preparation of simplified screening guidelines, which were integrated in the LDF regulation. In its last two years of implementation, the project invested in the development of environmental and social codes of practice and related training materials that can be rolled out at the district level and be used beyond the project. The LDF working was expanded at the district level to include an environmental expert. The project contributed to increasing the LDF benefits for ethnic minority groups. The PIU prepared an Indigenous People (Ethnic Minorities) Framework, with focus on the Kazakh and Duha people living, respectively, in the western and northern parts of the country. Due to their language, customs, and lifestyle, these communities faced a risk of exclusion from the prioritization of the LDF investments. The PIU implemented tailored capacity-building and outreach activities for these communities. As a result, Tsagaannuur district successfully met the APA requirement in 2023 and received a PBG for 2024 LDF financing, which will be used for a mobile kindergarten to benefit both tribes of the Duha people. Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) b. Fiduciary Compliance Procurement The project complied with World Bank procurement policies and procedures. The PIU retained a capable procurement specialist throughout project implementation. Procurement under the LDF followed Government of Mongolia procedures and the PIU conducted regular due diligence reviews to ensure that required goods and small works were procured under project-financed subprojects with PBGs in accordance with the public procurement legislation under local government ownership. The same due diligence was regularly conducted at the province level by the province finance departments.. The previous project (phase 2) had motivated the Public Procurement Law of 2012 that introduced the Community Participatory Method of procurement, which allows citizens to bid on ‘very small value works contracts’ as a community group without being established as a legal entity. The project found that awareness of this possibility among citizens was low, and skills required for such small works, which do not require special licenses, was limited. The project recommended investing in local citizens’ skills for traditional crafts and labor, which would help increase local incomes. Financial management In the early stages of project implementation, FM performance was uneven. Key challenges related to delayed submission of interim financial reports (IFRs), Statements of Expenditures (SOEs), and lack of capacity at the PIU and local levels. A recurring issue was related to poor quality of primary documents and periodic financial reporting by the LPOs to the PIU for accounting of project activities carried out locally. In 2018, the PIU recruited an FM specialist, who was retained through project closing. Given the nature of project activities, many small transactions occurred in rural areas, leading to extensive reconciliation work between local project bank accounts and the main project Designated Accounts and operating accounts. Due to the high workload, the PIU assigned a team assistant to support the project FM specialist which helped significantly in speeding up the reconciliation work and outreach to the LPOs. FM training sessions for the LPOs organized by the PIU also significantly helped in improving the quality of the local FM work. The PIU’s fiduciary performance gradually improved with the continuous support from the World Bank and timely submission of project financial reports became a norm. All annual audited financial statements were submitted on time to the World Bank and had unqualified opinions. c. Unintended impacts (Positive or Negative) The ICR reports climate resilience as a positive impact of some subprojects, according to an assessment of the LDF’S experience in supporting climate and disaster risk resilience at the district level, carried out in 2023. A review of investments in eight soums (across four aimags) indicated that between 7.4 percent and 20.4 percent of the LDF financing is invested in subprojects that support climate resilience (e.g., water storage, fodder reserves, animal health). The assessment also concluded that a more deliberate inclusion of climate resilience indicators, and clarification of eligible activities in this area in the LDF regulation, could further strengthen the impact of the program as a mechanism to support local adaptation. Due to the project experience, the MoF extracted lessons on how to ensure a fairer allocation of transfers across rural areas. With the introduction of the Livestock Head Tax in 2021, the LDF mechanism was Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) chosen to manage and program these tax revenues on behalf of the herder communities from which the taxes were collected. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The following are some relevant lessons presented in the ICR with some language adjusted by IEG: When specific capacity building activities are targeted as identified in needs assessments, this can contribute to effective learning by the government officials. In this project capacity building activities were linked to the results of the APA scoring exercise, with also served to identify needs among officials, so that the training and the technical assistance was targeted for the officials of the districts and sub-districts that most needed them, in specific areas. Predictability and consistency of local transfers may contribute to enhanced commitment of local authorities and citizens. The LDF’s sources of finance, since 2013, are linked to international commodity prices susceptible to wide fluctuations based on forces outside the Government's control, which exposes programs such as the LDF to levels of unpredictability that can undermine the relevance of the fund to local planners and credibility to local citizens. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well structured and clearly presents important information about the project, giving a good picture with plenty of details and insights. The efficiency section is particularly insightful, presenting a detailed analysis. Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MN Third Sustainable Livelihoods Project (P125232) The evidence of results is clearly presented, despite the amount of information and all the changes in the 8 restructurings. The ICR is results-oriented. The discussion on achievements is adequately balanced between reporting on the achievement of outcome indicators and what the project achieved on the ground. It provides good coverage of the implementation experience, presenting the project in its context of being the third phase of a 3-project program (the ToC of the whole program is also presented). The lessons extracted are clear. They appropriately respond to the key issues and are linked to the narrative and ratings presented in the report. The report is consistent and presents a logical linking and integration of its various parts. The ICR is longer than what the guidelines (~15 pages) state. a. Quality of ICR Rating Substantial Page 20 of 20