Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) Report Number: ICRR0023963 1. Program Information Country Practice Area (Lead) Pakistan Education Programmatic DPF Planned Operations Approved Operations 2 2 Operation ID Operation Name P170568 Pakistan Human Capital DPF Series L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IDA-66470 31-May-2021 506,185,262.01 Bank Approval Date Closing Date (Actual) 21-May-2020 31-May-2021 IBRD/IDA (USD) Co-financing (USD) Original Commitment 500,000,000.00 0.00 Revised Commitment 499,999,997.92 0.00 Actual 506,185,262.01 0.00 P172628_TBL Country Practice Area (Lead) Pakistan Education Operation ID Operation Name P172628 SHIFT DPCII ( P172628 ) Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IDA-66470,IDA-69490 30-Jun-2022 397400612.00 Bank Approval Date Closing Date (Actual) 28-Jun-2021 30-Jun-2022 IBRD/IDA (USD) Co-financing (USD) Original Commitment 400,000,000.00 0.00 Revised Commitment 400,000,000.00 0.00 Actual 397,400,612.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group William F. Steel Judyth L. Twigg Susan Ann Caceres IEGHC 2. Program Objectives and Pillars/Policy Areas EVOBJ_TBL a. Objectives The program development objectives (PDOs) of the Pakistan Securing Human Investments to Foster Transformation (SHIFT) Development Policy Financing series (as stated in the ICR, para. 13 and the program document for the second SHIFT [PD2, para. 33]) were to: (i) strengthen CRVS [Civil Registration and Vital Statistics], health, and education systems essential for HC [human capital] accumulation; (ii) recognize the contribution of women to economic productivity; and (iii) improve national safety nets to respond to shocks in a more efficient manner. The PDOs did not change between the first and second development policy finance operation (DPF). For the purpose of this ICRR, the PDOs are taken as:  PDO 1: Strengthen CRVS, health, and education systems essential for HC.  PDO 2: Recognize the contribution of women to economic productivity.  PDO 3: Improve national safety nets to respond to shocks in a more efficient manner. b. Pillars/Policy Areas The Pillars as stated in the PD were identical to the PDOs. For the purpose of this ICRR, the Pillars and Policy Areas (as indicated in the ICR, Section II) are represented as follows:  Pillar 1: Strengthen CRVS, health, and education systems essential for HC: o Improve quality of civil registration and vital statistics; o Improve quality of primary health care and increase the sustainability of national immunization; Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) o Reduce learning poverty among all children.  Pillar 2: Recognize the contribution of women to economic productivity.  Pillar 3: Improve national safety nets to respond to shocks in a more efficient manner: o Improve coordination of national safety nets to respond to the impact of COVID-19 and protect and support HC accumulation; o Update and governance of the National Socio-Economic Registry (NSER); o Improve efficiency of Benazir Income Support Program (BISP) programs. c. Comments on Program Cost, Financing and Dates The World Bank (WB) Executive Board approved US$ 900 million of IDA loans for the series of two operations (US$ 500 million and US$ 400 million, respectively) on May 21, 2020. Actual disbursements upon effectiveness (US$ 506,185,262 on June 19, 2020, and US$ 397,400,512 on 29 June 29, 2021) totaled US$ 903,585,874, due to exchange rate variations. 3. Relevance of Design a. Relevance of Objectives Country Context The series was designed to address deficiencies in Pakistan’s human capital status, with one of the lowest ratings in the region on the WB’s Human Capital Index (ICR, para. 4). Low investment in human capital and gender gaps in education and employment had constrained Pakistan’s economic growth rate and resulted in poor human development outcomes (especially in education and health). Although "Pakistan had a well- established federal safety net program, the Benazir Income Support Program (BISP), the database used for targeting beneficiaries…had not been updated in more than a decade,” and the National Socio-Economic Registry (NSER) needed support to update it and address errors and inefficiencies (ICR, para. 4). The government had also launched a comprehensive poverty alleviation program (Ehsaas) to consolidate safety nets and undertake further investments in HC. It had established a Technical Support Unit in the Ministry of Planning, Development and Special Initiatives to help address “the challenges of harmonizing CRVS services at the local, provincial and federal levels” (PD 1, para. 30). With respect to education, the government was also introducing reforms to address “learning poverty” (the percentage of children not being able to read and understand a short age-appropriate text by age 10), low attendance, and inequity. Relevance of Objectives The objectives of the SHIFT DPF series were highly relevant to addressing the identified constraints on Pakistan’s human capital development, consistent with Pakistan’s strategic objectives, and became all the more important to help mitigate the effects of the COVID-19 pandemic, which was emerging as the PD1 was being finalized (decision review April 9, 2020). PDO3 was especially relevant to enhancing the ability of Pakistan’s national safety nets to mitigate the impact of COVID-19 restrictions on income and hence on HC investment, in line with the Ehsaas poverty alleviation program launched in 2019 (PD1, para. 6; ICR, para. 5). PDO1 was likewise relevant to implementing Ehsaas in some specific areas of reform needed with respect to vital statistics, health care, and basic education, as well as to the National Education Plan and National Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) Education Policy Framework (PD 1, para. 45). PDO2 represented a step toward improving the status and conditions for women to contribute to economic growth, responding to Pakistan’s near-bottom ranking in the Global Gender Gap Report 2020 (World Economic Forum; PD1, para. 49). Although the PD and ICR did not identify specific national policies regarding the status of women, the province of Punjab (with nearly 9 million home-based workers, about half women) had developed a Home-Based Workers Act (PD1, para. 51), and support under the SHIFT series represented a suitable entry point for further strategizing. The objectives were in line with the WB’s Pakistan Country Partnership Strategy (CPS) for FY15-FY19. PDO2 was particularly relevant to Results Area 3 for “enhancing inclusion of women across sectors, increasing economic opportunities, and productively engaging women” (PD1, para. 67). PDO1 and PDO3 supported CPS Results Area 4 regarding Service Delivery, and PDO3 also “complemented the WB’s rapid response to the COVID-19 crisis in Pakistan which initially comprised the Pandemic Response Effectiveness in Pakistan project and eight ongoing projects” (ICR, para. 12). Table 1: Objectives and Prior Actions (PAs) for Securing Human Investments to Foster Transformation Development Policy Financing Series PAs under DPF1 PAs under DPF2 PDO 1: Strengthen CRVS, health and education systems essential for HC accumulation Improve quality of civil registration and vital statistics PA 2: At least three Provincial Cabinets and/or PA 1: The Ministry of Planning, Development Chief Ministers have approved the revised rules and Special Initiatives (MPDSI) has approved and/or by-laws to harmonize birth and death the policy to revamp and reform civil registration registration with the national civil registry at and vital statistics system. NADRA [National Database and Registration Authority]. Improve quality of primary health care and increasing the sustainability of national immunization PA 3: The Ministry of National Health Services, PA 4: The Provincial Finance Departments have Regulations and Coordination has approved the notified the Ministry of Finance on the transfer of essential package of health services at costed essential primary health care vertical community and primary health care center programs from development budget to recurrent levels. budget, effective as of FY22. PA 6: Ministry of Finance and provincial Finance PA 5: The Recipient has enabled the use of Departments approve the inclusion of all national mechanism for pooled procurement of remaining capital costs under the respective vaccines to continue and be reflected under expanded programs on immunization into the recurrent budget from FY20/21 onwards. recurrent budget starting from FY21 to institutionalize national immunization. Reduce learning poverty among all children PA 7: The Cabinet has approved the National PA 8: (i) The Inter Provincial Education Education Plan 2020. Ministerial Conference has endorsed the National Student Assessment Framework (NSAF), and (ii) the Federal cabinet has approved the establishment of the Pakistan Institute of Education, with the mandate to produce, analyze, and disseminate education Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) statistics (revised and expanded from Indicative Trigger). PA 9: Federal Cabinet has approved a policy framework to reduce out-of-school children through comprehensive policy options and approaches. PDO 2: Recognize the contribution of women to economic productivity PA 11: Home-based Workers Bills move in PA 10: The Punjab Cabinet has approved the federal Cabinet and Provincial Cabinets of Punjab Home-Based Workers Act. Khyber, Pakhtunkhwa, and Balochistan. PA 12: Draft amendments on Shops and Establishments Act and Factories Act to provide better and safe working conditions for women including no restriction on working hours, toilets, and childcare facility move in Provincial Cabinets. PDO 3: Improve national safety nets to respond to shocks in a more efficient manner Improve coordination of national safety nets to respond to the impact of COVID-19 and protect and support HC accumulation PA 14: [in PD2] At least three* Provincial PA 13: The Cabinet has approved the cash Cabinets approve a coordination and assistance package under the Ehsaas Program collaboration mechanism for nutrition-sensitive in response to COVID-19 pandemic, which cash transfer described in a Memorandum of includes a framework for improved coordination Agreement between the provincial governments between BISP, NADRA, Pakistan and BISP under the Ehsaas Program. Telecommunications Authority, and provincial governments. *The ICR and PD1 simply state: “Provincial Cabinets approve…” PA 15: The BISP Board has approved the plans for: (i) national expansion of the Waseela-e- Taleem (WeT) Program; and (ii) implementation of nutrition-sensitive conditional cash transfer program at the federal level, as stipulated in the Ehsaas Program. Update and governance of NSER PA 17: BISP Board approved the statutory PA 16: The BISP Board has approved revised regulations for the governance of NSER to allow data exchange protocols and socio-economic its independence from all programs (federal and classification to be applied to the updated NSER provincial), including a governing committee data for ease of targeting by social programs as with federal, provincial, and other core stipulated in the Ehsaas Program. stakeholders’ representations. Improve efficiency of BISP programs PA 19: Cabinet approved a framework allowing PA 18: The BISP Board has approved the (i) vertical expansion of BISP’s Kafalat Program Recertification Strategy, with clear protocols for with updated NSER, and (ii) a mechanism for entry and exit into the kafalat and conditional the periodic update of regular cash transfer cash transfer programs. benefits. Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) b. Relevance of Prior Actions Rationale The two operations had a total of 19 prior actions (PAs; 9 in SHIFT-I and 10 in SHIFT-II), with a total of 22 distinct actions and sub-actions. All but one of the PAs in SHIFT-I had follow-up actions in SHIFT-II (related to implementation of indicative triggers). Fifteen of the PAs involved approval of a measure; only four related directly to implementation of approved policies and measures. All were based on appropriate analytical work and relevant to the stated objectives. PDO 1: Strengthen CRVS, health and education systems essential for HC PA1 was highly relevant to strengthening systems essential for HC by reforming the civil registration and vital statistics system to “provide authentic information on population and birth and death statistics” and thereby facilitate access, monitoring, and evaluation (PD1, para. 72). PA1 and PA2 were based on a number of studies conducted by UNICEF, including the Roadmap for Digital Birth Registration, Assessments of the CRVS system (2013 and 2014), Review of Current Legislation, and Gap Analysis (PD1, Table 7). Consolidation and improved efficiency were expected to “reduce the actual cost and transaction costs for registering life events” and thereby increase uptake and thus access to social services (PD1, para. 73). Rating: PA 1: Highly Satisfactory. PA2 followed through on PA1 by addressing the problems of a “fragmented civil registration system and no clearly defined data sharing protocols across the country” (PD1. para. 39). PD2 tightened up the somewhat vague Indicative Trigger from SHIFT-1 by specifying what was to be approved and the number of Provincial Cabinets expected to approve them (however, the ICR reported the original IT as the PA, indicating “no change”). Rating: PA 2: Satisfactory. PA3 was intended to address inadequacies in the delivery of basic health interventions, especially to the general public, by prioritizing a basic set of interventions that were expected to improve the efficiency of the budgetary process and facilitate reallocation from tertiary to primary health care (ICR, Table 2). PA3 and PA5 drew on several studies by the Ministry of National Health Services Regulation and Coordination, including the Pakistan National Health Vision 2016-2025 and Action Plan 2019-2023, and Minimum Health Services Delivery Package for Primary Health Care Facilities (PD1, Table 7). This represented a first step toward implementing the recommendations of the World Health Organization report: Disease Control Priority – Edition 3 (PD1, para. 40). Rating: PA 3: Satisfactory. PA5 addressed the problems of implementing the Expanded Program on Immunization due to time- consuming and uncoordinated procurement by federal and provincial governments and donor-funded programs (PD1, para. 41). The supported action was expected to lower costs and speed delivery in four provinces through a pooled procurement process. Rating: PA 5: Satisfactory. PA4 and PA6 moved the capital/development budget costs of PHC and immunization programs to recurrent budgets in order to ensure their institutionalization and sustainability. Classifying such programs as capital Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) expenditures had made the budget and procurement process unpredictable and time-consuming (PD1, para. 41). Ratings: PA 4: Satisfactory; PA 6: Satisfactory. PA7 and PA8 were intended to address the government’s reforms to address “learning poverty” among children. They drew on Provincial Education Plans supported under the Global Partnership for Education, the National Education Plan (2020), and the Pakistan National Education Response and Resilience Plan for COVID-19 (PD1, Table 7). However, they were at a level of simply approving a National Education Plan and Pakistan Institute of Education without a clear results chain to measures that would address the causes of learning poverty. While setting goals and gathering/analyzing data were appropriate first steps, expected results were phrased in terms of “more accurate ranking in international comparisons” and “higher accountability” (PD1, paras. 47-48) rather than learning outcomes for children. Ratings: PA 7: Moderately Satisfactory; PA 8: Moderately Satisfactory. PA 9 was added in SHIFT-II to extend beyond the indicative trigger from SHIFT-I by developing policy guidelines more explicitly targeted toward out-of-school children and expansion of the education system (PD2, paras. 50-51). This was motivated by the addition of an estimated 1-2.5 million out-of-school children due to closure during the pandemic, on top of "the approximately 19 million children already out of school" (PD 2, para. 46). Rating: PA 9: Satisfactory. PDO 2: Recognize the contribution of women to economic productivity PA 10 and PA 11 were directly relevant to the objective of giving greater recognition to the economic contribution of informal women workers in the specific area of home-based work by supporting passage of Home-Based Workers Acts in four provinces. The predominance of home-based work among Pakistan’s estimated 17 million female workers reflects the “discriminatory attitudes, workplace barriers and lack of female-friendly workplaces” that have deterred women from more formal employment, and indeed from participating in the labor force, yet home-based work was not legally recognized (PD1, para. 50). To remedy that situation, the province of Punjab had developed a Home-Based Workers Act, and PAs 10 and 11 supported its approval in Punjab and three other provinces, to recognize the contribution of women and lay the groundwork for further support. They drew on a number of relevant studies, including the Punjab Home- Based Workers Survey (2017), Recognizing and Supporting Home-Based Workers (South Asian Regional Consultation), and Labor Force Experiences for Urban, Low-Income Pakistani Women (PD1, Table 7). The Act would establish a Home-Based Workers Council (including representatives of home-based workers and civil society) “to register home-based workers and map home-based workers in different economic sectors and geographical areas…[and] develop a robust information dissemination plan” (PD1, para. 51). The Act would also enable registered home-based workers to “to enter into written contracts with their employers to ensure their access to appropriate wages and other benefits, including employees old age benefits.” Ratings: PA 10: Satisfactory; PA 11: Satisfactory. PA 12 likewise supported PDO2 by amending the provincial Acts governing retail and factory employment to improve working conditions for women. It was assumed that changes in the regulations would be implemented by private retail and factory establishments, resulting in an increase in women working in formal employment in these sectors. Specifying mechanisms and time frames for implementation and enforcement and prospects for further actions and support in future operations would have strengthened the relevance to achievement of the objectives. Rating: PA 12: Satisfactory. PDO 3: Improve national safety nets to respond to shocks in a more efficient manner Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) PA 13 and PA 14 were directly relevant to the objective of improving the ability of national and provincial safety net programs to respond effectively to shocks through improved coordination and data-sharing. They were based on the study underlying Pakistan’s Ehsaas Programme: Shift from Politics of Patronage to Politics of Performance, as well as several ongoing WB studies on NSER Legal Frameworks, Interoperability Frameworks, and Transitioning NSER into a Dynamic Social Registry (PD1, Table 7). The impact of improved coordination had been demonstrated in the COVID-19 response, making “it possible to horizontally expand threefold the beneficiary base for the COVID-19 emergency cash transfer” (PD2, para. 57). As in the case of PA2, PD2 tightened up the Indicative Trigger from SHIFT-1 by specifying the number of Provincial Cabinets expected to approve the mechanism (although the ICR reported the original IT as the PA, indicating “no change”). Ratings: PA 13: Satisfactory; PA 14: Satisfactory. PA 15 was relevant to improving responsiveness to the negative impact of COVID-19 on education by approving expansion of the WeT (Waseela-e-Taleem) program of conditional cash transfer to mothers linked to primary education that had been successfully tested in a third of Pakistan’s 160 districts (PD1, para. 58) and by extending it to conditional cash transfers linked to nutrition. However, implementation of the approved plans was assumed, rather than supported, by follow-up actions in SHIFT-II. Rating: PA 15: Moderately Satisfactory. PA 16 was relevant to improving safety net responsiveness to shocks by approving socio-economic classification (to help pre-identify poor households that could qualify for benefits) and data exchange protocols to protect privacy while sharing data across relevant agencies (PD1, para. 61). PA 17 sought to make the National Socio-Economic Registry (NSER) a national registry independent of BISP (which had initiated it in 2010 to collect data) that could also work with other national and provincial agencies (PD1, paras. 61-62). Classification through a central, unified registry was expected to yield “better targeting, reduced duplication and fragmentation, and ultimately better HC outcomes” (PD1, para. 63). The reforms were supported by WB studies in progress on NSER Legal Frameworks, Interoperability, and “Transitioning NSER into a Dynamic Social Registry” (PD1, Table 7), as well as by complementary technical assistance. Ratings: PA 16: Satisfactory; PA 17: Satisfactory. PA 18 and PA 19 addressed limitations in the BISP program resulting from a decade of failure to update means test criteria and to open new enrollment (PD1, para. 64). Some beneficiaries no longer needed cash transfers, while potentially eligible households whose ability to invest in HC would be enhanced by such transfers were unable to apply. Furthermore, impact evaluations had found that the low value of transfers also limited their effectiveness in supporting HC. The PAs appropriately supported approval of protocols and a framework to address these issues (on the assumption that they would be implemented as approved), drawing on ongoing WB studies on BISP Recertification, Expansion of Safety Nets, and Updating the Universal Cash Transfer Value over Time (PD1, Table 7). Ratings: PA 18: Satisfactory; PA 19: Satisfactory. Rating Satisfactory 4. Relevance of Results Indicators Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) Rationale The series used ten results indicators (RIs) to track implementation and results of eight pairs of PAs (in SHIFT-I with a follow-up in SHIFT-II) and two additional actions (one in SHIFT-I, one in SHIFT-II), with one sub-indicator for female participation. The RIs were oriented toward outcomes that generally represented implementation of the associated PAs, although in some cases they were not necessarily attributable to the PA. However, two RIs did not adequately track implementation of actions in the associated PAs (i.e., three PAs did not have clearly associated RIs), and one of them lacked a suitable data source for monitoring. PDO 1: Strengthen CRVS, health and education systems essential for HC RI 1 directly measured the expected result of PAs 1 and 2 to revamp, harmonize, and reduce costs of civil registration and vital statistics, and an increase in the share of children with a birth certificate could reasonably be attributed to the actions taken (on the implicit assumption that the approved measures were in fact implemented). Definition and measurement were straightforward, and data readily available. Rating: RI 1: Highly Satisfactory. RI 2 related to the expected result of implementing PAs 5-6 to improve the procurement and funding of vaccination programs but served as only a partial proxy for the improvement of essential health services supported by PAs 3-4. Because uptake of vaccinations depends on many factors in addition to availability, notably education and outreach, changes in the indicator could not necessarily be attributed to the associated actions. Definition and data collection were straightforward. Rating: RI 2: Moderately Satisfactory. RI 3 tracked whether the measures approved in PAs 7-9 resulted in a national learning assessment being undertaken. While such an assessment would be an important input into implementing policies to address the problem of learning poverty, it was not necessarily related to establishment of the Pakistan Institute of Education (as approved under PA 8) nor to achievement of the intended result of reducing learning poverty. The ICR (para. 36) appropriately noted the challenge of establishing a results chain from “measuring learning poverty [to] the various mechanisms through which a national assessment could genuinely affect learning poverty.” The indicator was defined simply in terms of the number of national learning assessments conducted. Rating: RI 3: Moderately Unsatisfactory. PDO 2: Recognize the contribution of women to economic productivity RI 4 was directly relevant to measuring the impact of PAs 10-11 on the readiness of home-based workers to register as a result of greater acceptance through Home-Based Workers Acts and associated support mechanisms (including awareness campaigns; PD1, para. 51). Official registration could reasonably serve to indicate greater recognition of the contribution of informal women workers, as well as an important step toward improving their access to social protection. Measurement was straightforward, and appropriately included a sub- indicator for the share of women among those registered. Rating: RI 4: Satisfactory. RI 5 directly related to the expected result of implementing the amendments supported by PA 12 in terms of the readiness of women to seek employment in formal retail establishments and factories, but it did not necessarily track actual implementation of the mandated improvements in working conditions. Although technical assistance for implementation was expected (PD2, para. 55), the PDs did not mention how the new regulations might be enforced, and the time frame for implementation was likely to exceed that of the series, making it difficult to attribute changes in the indicator directly to the action, except insofar as passing the amendments Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) changed perceptions of both employers and women workers. Furthermore, attribution was limited by the many factors other than the legislation that affect demand for and readiness of women to engage in such work. Measurement depended on the biennial Labour Force Survey, which is issued with a substantial time lag (data currently available only through 2021). Rating: RI 5: Moderately Unsatisfactory. PDO 3: Improve national safety nets to respond to shocks in a more efficient manner RI 6 was directly relevant to PAs 13-14 inasmuch as the baseline was nil for these COVID-19 response programs, meaning that an increase in beneficiaries could be attributed to implementation of the approved measures. The existence and reach of these programs directly related to the objective of improving the ability of national safety nets to respond to shocks. Data were collected by BISP/NSER. Rating: RI 6: Satisfactory. RI 7 and RI 8 similarly tracked implementation of approved measures under PA 16 to expand the WeT program nationwide and to implement a nutrition-sensitive conditional cash transfer program. The beneficiaries were mothers, consistent with the objective of overcoming Pakistan’s low HC among women. Data for RI 7 depended on the annual Economic Survey (which is issued with some delay). Data for RI 8 were available from BISP/NSER. Ratings: RI 7: Satisfactory; RI 8: Satisfactory. RI 9 related to a desired result of the revised data exchange protocols and classification approved under PA 16, which would contribute to the objective of more efficient responses to shocks by permitting better targeting by safety net programs (but did not track whether classification data were in fact being applied). It did not relate to PA 17, which would have benefited from a separate indicator of the independence or effectiveness of NSER. Data were available from BISP/NSER. Rating: RI 9: Moderately Satisfactory. RI 10 was relevant to the implementation of PA 18 to the extent that the Recertification Strategy exited BISP beneficiaries who no longer qualified and approved new ones from the bottom two quintiles, thereby meeting the objective of improved efficiency. It did not capture either of the sub-actions under PA 19 regarding expansion of the Kafalat Program and updating of the value of cash transfers, which would have benefited from separate indicators. The data source was a periodic survey that was used for the baseline, but which was not repeated in 2022 as expected (TTL). Hence the ICR (footnote 16) had to undertake a simulation using the 2018-19 Household Integrated Economic Survey in order to estimate results for 2023. Rating: RI 10: Moderately Unsatisfactory. Table 2: Results indicators by Objective and PAs; baseline and target values; status and achievement (Source: ICR, Table 1) Actual as RI Results Associated RI Baseline Target % of Actual Achievement indicator (RI) PAs Relevance (2019) (2023) targeted (2023) rating change PDO 1: Strengthen CRVS, health and education systems essential for HC RI 1: Percentage 36.8% of children 1-2 HS 36% 41% 16% Negligible [est.] under 5 with a birth certificate Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) RI 2: Percentage of children age 12-23 months 3-6 MS 66% 81% 76.4% 69% Substantial who received all basic vaccinations RI 3: Number of national learning 7-9 MU 0 1 1 100% Modest assessments conducted PDO 2: Recognize the contribution of women to economic productivity RI 4: Percentage of home-based 20% of workers Modest which registered 10-11 S 0 0 0% [additional 70% (federal and evidence] female provincial), of which are female RI 5: Percentage of women Substantial working in 12 MU 10% 14% 14%` 100% [additional shops, evidence] establishments and factories PDO 3: Improve national safety nets to respond to shocks in a more efficient manner RI 6: Number of beneficiaries from COVID- 14.8 million 19 safety nets 12 13-14 S 0 (10.55 m. 123% High response million one-off) interventions (federal and provincial) RI 7: Number of beneficiaries 2.8 15 S 4 million 6.52 million 310% High of WeT million program RI 8: Number 15 S 0 500,000 778,641 156% High of beneficiaries of nutrition sensitive CCTs (federal and Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) provincial) RI 9: Number of households classified by 22 welfare status 16-17 MS 0 35 million 159% High million and percentile in the updated NSER RI 10: Percentag e of BISP beneficiary 77.5% 18-19 MU 33% 70% 120% Modest families [simulation] covered from the bottom two quintiles Rating Moderately Satisfactory 5. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Strengthen CRVS, health and education systems essential for HC Rationale RI 1 showed only a slight increase from 36 percent of children under 5 with a birth certificate to 36.8 percent (as estimated by the ICR, Table 7), only a sixth of the targeted increase of 5 percentage points. PD1 (para. 39) indicated that most of the funding for reforming the CRVS would be disbursed in the second and third years of the program, and the ICR (para. 21) stated that several steps to implement the approved policy and mechanism for nationwide registration under PAs 1-2 had in fact been “rolled out.” But (besides the absence of recent data from the Demographic and Health Survey) the targeted impact on issuance of birth certificates showed negligible progress. Achievement Rating: Negligible. RI 2 increased by 10.4 percentage points from a baseline of 66 percent of children aged 12-23 months receiving all basic vaccinations to 76.4 percent, representing 69 percent of the targeted increase of 15 percentage points. This substantial achievement during a situation of decreased utilization of health services during the COVID-19 pandemic was attributable to special mitigation measures such as air-freighting supplies and home-based immunization campaigns (ICR, para. 23). Achievement Rating: Substantial. Although the RI 3 target of one national learning assessment was achieved, this indicator was not sufficient as a measure of achievement of the objective of reducing learning poverty among children. The ICR (para. 26) did note that the Pakistan Institute of Education had become functional, indicating substantial progress in Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) implementing actions approved under PA 8, but it recognized that “institutional actions [only] lay the basis for improved assessment, monitoring, planning, and adjustment of education sector policies and priorities.” The Bank team noted that a number of complementary education projects address learning poverty and enrollment, including: Actions to Strengthen Performance for Inclusive and Responsive Education Program (P173399); Khyber Pakhtunkhwa Human Capital Investment Project (P166309); Punjab Human Capital Investment Project (P164785); and Sindh Early Learning Enhancement Through Classroom Transformation (P172834). An MU rating for the RI triggers a downrating to Modest. Achievement Rating: Modest. Rating Moderately Unsatisfactory OBJECTIVE 2 Objective Recognize the contribution of women to economic productivity Rationale RI 4 was not achieved because registration of home-based workers (HBWs) had not begun, nor was it clear when it might be (ICR, para. 28). Nonetheless, some progress toward implementation of actions under PAs 10-11 was made through passage of the necessary legislative framework and formulation of guidelines in “nearly all provinces.” Further progress, however, would depend on “capacity building and large-scale support.” The Bank team noted that the WB has supported the labor department with HBWs rules development and facilitated the Balochistan labor department and social security institution toward developing a management information system database for HBWs, and that some provincial authorities are making progress with implementation (the Sindh labor department is regularly receiving registration applications from HBWs, and its social security institution has launched an on-line portal where HBWs can register to receive benefits; the Balochistan labor department has started registration of HBWs in two provinces). Based on this additional evidence, the achievement rating is upgraded from Negligible to Modest. Achievement Rating: Modest. RI 5 regarding women working in shops, establishments, and factories was fully achieved, but an MU rating for the RI triggers a downgrade to Modest. Nevertheless, additional evidence provided by the ICR (para. 30) of measures to implement the actions supported by PA 12 with respect to hours, childcare, safe working environment, toilet facilities, etc., warrant an achievement rating of Substantial. Achievement Rating: Substantial. Rating Moderately Satisfactory Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) OBJECTIVE 3 Objective Improve national safety nets to respond to shocks in a more efficient manner Rationale The target for RI 6 was exceeded, with the majority of beneficiaries of COVID-19 response interventions being one-off emergency transfer beneficiaries. This result directly supported the objective of a more efficient and coordinated response to shock via national safety nets, as did approval of a Memorandum of Agreement between BISP and provincial governments (ICR, para. 32). Achievement Rating: High. The target for RI 7 was substantially exceeded, indicating highly effective implementation of PA 15 (i) to expand the WeT Program. Of the beneficiary children, 49 percent were girls (TTL). Achievement Rating: High. Overachievement of the target for RI 8 indicated effective implementation of PA 15 (ii) regarding nutrition- sensitive conditional cash transfer programs and further progress toward the objective of effective national safety nets. Achievement Rating: High. The RI 9 target for classification of households was exceeded, indicating effective implementation of PA 16 to improve the ability to target social programs, consistent with the objective of enhancing efficient responsiveness to shocks. Although the ICR (para. 34), as well as this indicator, did not provide information on implementation of measures under PA 17 regarding the independence and governance of NSER, the TTL has reported that the representative, independent Steering Committee has been formed and held two meetings. Achievement Rating: High. The ICR (footnote 16) attempted to simulate the likely 2023 status of RI 10 using the 2018-19 Household Integrated Survey, yielding an estimate above the target. However, an MU rating for the indicator due to weaknesses in terms of data availability and pertinence to the associated PAs limits the rating to Modest. Achievement Rating: Modest. Rating Satisfactory OVERALL EFF TBL OLD Overall Achievement of Objectives (Efficacy) Rationale Achievement of PDOs 1 was Moderately Unsatisfactory, PDO2 was rated Moderately Satisfactory, and PDO3 was rated Satisfactory, yielding an overall average of Moderately Satisfactory. Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) Overall Efficacy Rating Moderately Satisfactory 6. Outcome Rationale The relevance of PAs was rated as Satisfactory and Efficacy was rated Moderately Satisfactory, yielding a Moderately Satisfactory Outcome. a. Rating Moderately Satisfactory 7. Risk to Development Outcome Two principal, interrelated risks to maintaining progress toward the development objectives supported by this series are macroeconomic prospects and natural disasters. The ICR (para. 51) also noted political, governance, and institutional risks, but did not elaborate on them. Macroeconomic imbalances have created fiscal and inflationary pressures that tend to compromise both personal and public spending on school attendance, nutrition, and social services, thus undermining progress toward improving Pakistan’s low HC status (ICR, para. 52). The COVID-19 pandemic exacerbated macroeconomic risks through “a sharp reduction of domestic economic activity, reduced demand for exports, import shortages, rising prices, higher external financing costs, and fiscal pressures” (PD2, para. 91). Ongoing fiscal and debt challenges could further constrain the budget for social spending and investment in HC. Mitigating measures in the SHIFT series included shifting primary health care and immunization programs to the recurrent side of the budget. In addition, the complementary Resilient Institutions for Sustainable Economy (RISE) DPL “supported better fiscal management and improvements in the digital payments architecture, which was important for resolving circular debt of the power sector, as well as other macro risks” (TTL). Natural disasters pose a recurrent risk to Pakistan’s ability to sustain progress toward HC objectives. The reforms supported by the SHIFT series have substantially improved the capacity of safety nets to respond to shocks such as COVID-19 and the severe floods in 2022, and of federal and provincial governments to coordinate responses. However, costs imposed by future disasters could adversely affect the fiscal capacity to respond to such shocks, undermining both safety net reforms and further investments in HC. Mitigation was supported by the complementary the Program for Affordable and Clean Energy (PACE) DPL, which “supported the reform of electricity subsidies and rationalizing consumer tariffs to better target the poor and vulnerable” (TTL). Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) The political risk of instability has been heightened by the military’s intervention in the recent election and appointment of a Prime Minister. Although the PDs referred broadly to “political risks,” they did not explicitly discuss this risk or how it might be mitigated. PD2 (para. 92) also cited governance, institutional capacity and stakeholder risks as challenges to be addressed in part through the SHIFT DPF series. Several PAs explicitly supported better coordination between federal and provincial governments and programs, especially regarding safety nets. But the involvement of so many actors in a multi-sectoral, multi-level reform program raised the risks of “elite capture, political influence on implementation of reforms, [and] slow decision-making at institutional level.” These risks were mitigated in part by continuing engagement and support programs by the WB and other development partners, as well as by the parallel RISE DPL (see above). The ICR (para. 42) also stated that “the SHIFT series had a strong impact on institutional change and strengthening,” including reforms for better provincial-federal coordination, the merger of two previous institutions into the Pakistan Institute of Education, and independence of NSER, among others. 8. Assessment of Bank Performance a. Bank Performance – Design Rationale Analytical underpinnings and lessons from prior experience The design drew on a wide range of analytical underpinnings in studies by various agencies. PD1 (Table 5) cited three documents as providing a “medium to long-term vision for Pakistan including recommendations on key policies for improved sustainable growth and increased HC accumulation": Pakistan@100; Pakistan Human Capital Policy Note; and Weathering the storm: restoring macroeconomics stability, Pakistan Development Update (June 2019). A number of UNICEF documents provided assessments and guidance for reforms to the CRVS systems, while various World Health Organization, WB, and Ministry of National Health Services Regulation and Coordination reports underpinned reforms to basic health services and immunization. The WB’s Pakistan District Education Management Service Delivery Study highlighted the problem of learning poverty, with reforms guided by the government’s National Education Policy Framework and the Punjab Education Sector Plan 2019 to 2023. The need for and design of measures to improve recognition of and working conditions for women workers (both informal and formal) drew on several studies by the International Labour Organization (ILO), Women in Informal Employment: Globalizing and Organizing, and the Pakistan Bureau of Statistics. Measures to improve safety nets were based on a number of studies by the WB and partners regarding BISP, NSER, and other social protection programs. PD1 (para. 33) reported the following as key lessons from six DPF operations supporting social sectors from 2004 to 2009:  Country ownership and alignment to government priorities is critical for program success. The SHIFT series supported reform actions that were aligned to the federal and provincial governments’ priorities to improve HC outcomes in the country. Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568)  Sequenced reforms supported by a DPF series of two operations provides greater momentum for policy changes. The first SHIFT operation in the series laid out core reform actions that were consolidated by the second operation and supported national reforms that required consensus building between the federal and provincial governments.  Policy lending needs to be complemented by investments that support achievement of results. Investment lending operations and technical assistance in the portfolio and prepared in parallel to the SHIFT series across the federating units financed interventions on a sustained basis to achieve implementation results. Results chain and indicators The PD stated the intentions and expected results for each action, but was less explicit on the underlying rationale, and neither the PDs nor the ICR provided an explicit theory of change or results chain. The RIs generally tracked expected outcomes from implementation of the associated PAs, although in some cases not necessarily attributable to the PA. However, three PAs did not have clearly associated RIs. Identification and mitigation of risks PD1 (paras. 90-91) appropriately assessed the macroeconomic, political, and governance risks as “high,” with low reserves and a high debt ratio aggravated by the COVID-19 pandemic, noting that “Pakistan’s track record on reform implementation has remained mixed, with those benefiting from status quo, elites, and systemic hurdles resisting the reform process both at the federal and provincial levels.” This was to be mitigated by “strong ownership of the HC agenda both at the federal and provincial levels” with support from programs and technical assistance from the WB and other development partners. The SHIFT series included a number of mitigating measures, particularly against the negative effects of the COVID-19 pandemic on the lower-income population and HC, such as actions to harmonize and coordinate social safety nets at the federal and provincial levels, and shifting primary health care and immunization programs to the recurrent side of the budget. Consultation with development partners and stakeholders PD1 (para. 70) stated that the SHIFT series was “prepared in close collaboration with development partners,” in particular: UNICEF for CRVS reforms; World Health Organization, Unkted Kingdom Department for International Development, United States Agency for International Development, and United Nations Population Fund with regard to health care and education; and Japan International Cooperation Agency, UNWomen, and ILO with respect to women workers. The latter was also supported by “rigorous consultations (including tripartite)…in all provinces.” A National Steering Committee led consultations at all levels on CRVS reforms, including the private sector and religious leaders. Rating Moderately Satisfactory b. Bank Performance – Implementation Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) Rationale Monitoring and Adaptation The data required for monitoring implementation were available (as reported in the ICR), with the exception of RI 10. Preparation of the SHIFT series began prior to the advent of the COVID-19 pandemic. The WB then adjusted its program for Pakistan, in line with the WBG Crisis Response Approach Paper (PD2, Annex 1, para. 4), including two human capital projects in some of the poorest districts and a PfoR (Program for Results) operation “to address disruptions to the education sector.” The ICR (para. 47) reported that the DPF series was quickly adjusted on an emergency basis, with the support of “an extensive multisectoral team of specialists…under the guidance of two experienced team leaders, one of whom continued to lead both operations in the series, thus ensuring continuity.” As then designed, the measures under the SHIFT series to reform safety nets and institutions for more efficient responsiveness to shocks were highly relevant to adapting to the COVID-19 crisis, and the ICR (para. 15) said that no further significant changes were needed during implementation.” Rating Satisfactory c. Overall Bank Performance Rationale While design is rated Moderately Satisfactory, generally satisfactory performance in areas other than results chains and indicators, together with Satisfactory implementation weight the overall rating toward Satisfactory. Overall Bank Performance Rating Satisfactory 9. Other Impacts a. Social and Poverty It is too early for evidence to be available on possible social and poverty impacts. b. Environmental It is too early for evidence to be available on improvements in resilience to climate change and other external shocks. Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) c. Gender Data are not yet available on the extent to which measures supported by the SHIFT series might have mitigated the adverse effects of the COVID-19 pandemic on girls’ education. d. Other n.a. 10. Quality of ICR Rationale The ICR provided a coherent narrative of the SHIFT series, both complete (in accordance with guidelines) and concise. It provided adequate context and rationale for WB support; tables explaining the relevance of PAs and the extent of achievement of results; and discussion of outcomes, risks, and lessons. There was no explicit results chain, in part because the PDs lacked specific explanations of the rationale behind each PA (just a discussion of expected results), and its theory of change was at best implicit in the PDOs, PAs, and RIs (which the ICR [para. 47] asserted but did not lay out). The discussion of efficacy (section II.B) was thorough and in some cases (e.g., paras. 21, 23, 26 and 30) provided important supplementary evidence of implementation and supporting measures undertaken. Similar information would have been useful on implementation of PAs 17 and 19, which were not adequately tracked by the RIs. The ICR (para. 56) appropriately noted that the limited time frame of a two-operation series makes it difficult in some cases to monitor outcomes, leading to less satisfactory process-oriented indicators. The ICR (paras. 55-57 and 42) provided a thoughtful discussion of lessons learned and experience with institutional strengthening. More analysis of gender-related impacts of measures regarding education (PAs 7-9) and women workers (PAs 10-12) would have been interesting, although perhaps constrained by the limitations of the indicators and availability of data. a. Rating Substantial 11. Ratings Reason for Ratings ICR IEG Disagreement/Comments Moderately Outcome Moderately Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pakistan Human Capital DPF Series (P170568) Relevance of Results --- Moderately Satisfactory Indicators Quality of ICR --- Substantial 12. Lessons Reforms are more likely to have sustained impact when supported by further actions, technical assistance, and/or complementary investment projects. The ICR (para. 55) observed that PAs supported by further financial and technical assistance were better sustained than those without follow-on projects, which often “suffered from delayed results.” This is especially true when government capacity and resources are constrained. Reforms at the national and provincial levels enable but do not guarantee delivery of human capital development services at the sub-national/local level. The ICR (para. 56) noted that, despite improved coordination between federal and provincial governments and programs, “progress was uneven at the sub- national level.” Motivating sub-national actors poses a challenge for designing reforms and actions to achieve desired results at the local level. Policy and institutional reforms through a DPF series can lay the foundation for continued and more ambitious reforms. The ICR (para. 57) observed that the health financing reforms under SHIFT “allowed the government in Sindh…to push further for getting funds to the district level” and motivated further requests for WB analytical, advisory, and technical assistance. 13. Project Performance Assessment Report (PPAR) Recommended? No Page 20 of 20