93421 Kementrian Koordinator Bidang Kesejahteraan Rakyat BAPPENAS PNPM RURAL IMPACT EVALUATION April 2012 PNPM RURAL IMPACT EVALUATION APRIL 2012 CONTENTS TABLE vi ACKNOWLEDGEMENTS 6 Table 1: Distribution of Block Grant Funding vii ABSTRACT by Type of Activity in 2009 ix LIST OF ABBREVIATIONS 12 Table 2: Social Dynamics and Governance Variables x EXECUTIVE SUMMARY 33 Table 3: 1 I. BACKGROUND Change in Logged Real Per Capita Consumption 5 II. THE PROGRAM NASIONAL 34 Table 4: PEMBERDAYAAN MASYARAKAT Households Moving Out of Poverty RURAL COMPONENT 35 Table 5: 9 III. METHODOLOGY Households Moving Into Poverty 15 IV. RESULTS 36 Table 6: Change in Household Access to Outpatient Care 23 V. DISCUSSION AND CONCLUSIONS 37 Table 7: 28 VI. RECOMMENDATIONS AND Change in Transition Rate from Primary POLICY IMPLICATIONS to Lower Secondary School 29 REFERENCES 38 Table 8: Change in Employment Status 41 ANNEX 1: METHODOLOGY 38 Table 9: 49 ANNEX 2: A NOTE ON POWER CALCULATIONS Change in Social Capital and Governance Indicators 52 ANNEX 3: EXECUTIVE SUMMARY FROM THE 41 Table A1.1: QUALITATIVE STUDY Distribution of Matched Kecamatan by Province Author 44 Table A1.2: John Voss Balancing Tests for Covariates Photography Doc. PNPM Support Facility 46 Table A1.3: Table of Means for Indicators at Baseline Graphic Design Anang Saptoto – Minimi Studio 47 Table A1.4: Published in an edition of 250 exp Rural Provincial Poverty Lines Used to Assign Poverty Status Published by PNPM Support Facility Jakarta, Indonesia, 2013 50 List of Parameters for Cluster Assigned Treatment Print in Jakarta, Indonesia The views expressed in this paper are those of the authors alone and do not represent the views of the PNPM Support Facilty or any of the many individuals or organizations acknowledged here. v ACKNOWLEDGEMENTS ABSTRACT This report was prepared by a team from PNPM Support management of the data collection process as well as initial This paper reports on a quasi–experimental evaluation of with respect to household welfare and poverty reduction, but Facility (PSF) led by John Voss. Natasha Hayward was Task input on the survey instrument and field work methodology. the PNPM–Rural program designed to assess the impact of did see increased access to outpatient health services. With Team Leader and Jan Weetjens provided overall guidance. The Surveymeter team was led by Wayan Suriastini, under the the project on household welfare, poverty, access to services, respect to social dynamics and governance, PNPM created Yulia Herawati and Gregorius K. Endarso supervised the overall guidance of Bondan Siloki. employment, social dynamics and governance. Across 17 positive impacts on measures of social capital and governance data collection and cleaning. Critical support was provided provinces, a panel of 6319 households and 26,811 individuals within the program but these impacts did not spill over by Lina Marliani, Ritwik Sarkar, Juliana Wilson, Lily Hoo, Finally, the team would like to extend its thanks to the from 300 sub–districts were interviewed at baseline in into larger village decision–making processes. Impacts are Arya Gaduh and Christine Panjaitan. The overall conceptual thousands of households from 17 provinces across Indonesia 2007 and again in 2009/2010. A propensity score matching strongest in poor and remote areas where the interests of development of the study was led by Susan Wong as initial who took time from their day to sit down with the survey approach was used to select the sample of sub–districts the poor and the community as a whole are aligned around Task Team Leader (2008–2009). The qualitative inputs to the team and provide the most valuable input to the project, the participating in PNPM beginning in 2007 and a comparison filling critical infrastructure gaps. The project is less effective report were conducted by the SMERU Research Institute, led collected data. The Government of Indonesia and the PNPM sub–district group that has similar characteristics based for less poor and less remote areas where infrastructure by Muhammad Syukri and are published additionally in a Support Facility provided financial support for the project. on data taken from the 2005 national village census that gaps are not as significant, leading to a divergence between separate study. began participation in 2009/2010. Qualitative studies were communities who still seek to utilize funding for additional also conducted in eighteen villages in 3 provinces in 2007 infrastructure and the poor who seek skill training, access The team received critical guidance from Scott Guggenheim and 2010 to enhance understanding of the findings from to capital, and improved access to and quality of health and (AUSAID), Susanne Holste (PSF), Sentot Satria (PSF), Robert the quantitative analysis. The study found that households education services. Wrobel (PSF), Vic Bottini (consultant, TNP2K), Jed Friedman participating in the program experienced positive benefits (World Bank), and Gus Papanek (BIDE). The report benefited with respect to household welfare via increased real per capita from valuable inputs from peer reviewers Vivi Alatas (World consumption and increased chances of escaping poverty. Bank), Asep Suryahadi (SMERU Research Institute), Menno Households participating in the project also saw access to Pradhan (University of Amsterdam), Neil McCullough (AUSAID) outpatient health services increase and increased chances of and Marcus Goldstein (World Bank). We would also like to thank being employed. While these positive impacts were strong BAPPENAS, in particular, Rudy S. Prawiradinata (Bappenas) and amongst poorer households, marginalized groups (including Vivi Yulaswati (Bappenas) for their coordination and support female–headed households and households with lower levels during field work and dissemination; TNP2K, in particular the of education) did not see the same benefits from the project support and guidance of Sudarno Sumarto and Elan Satriawan and the Ministry of Home Affairs (Depdagri). The team would also like to thank Surveymeter, the firm conducting the SEDAP 2007 and 2009/2010 household surveys, for excellent vi vii LIST OF ABBREVIATIONS BAPPENAS : State Ministry of National Developmental Polindes : Village Maternity Center Planning Posyandu : Integrated Health Service Center BLM : Community Block Grant Pustu : Secondary Health Center (Puskesmas) BLT : Direct Cash Transfer RT : Neighborhood unit consisting of several BPR : People’s Credit Bank households BPS : Central Statistics Agency RW : Administrative unit consisting of several RT CDD : Community Driven Development Raskin : Rice for the Poor Depdagri : The Ministry of Home Affairs SPP : Women’s Savings and Loan EA : Enumeration Area SEDAP : PNPM Impact Evaluation Survey FGD : Focus Group Discussions SUSENAS : National Sosioeconomic Survey Jamkesmas : Health insurance for the Poor TNP2K : National Team for Accelerating Poverty KDP : Kecamatan (Sub–District) Development Reduction Program TPK : Program Implementation Team NMC : National Management Consultant PKH : Household Conditional Cash Transfer PKK : Family Welfare Empowerment PNPM : National Program on Community Empowerment PODES : Village Potential Statistics PPK : Kecamatan (Sub–District) Development Program PSF : PNPM Support Facility ix project in less poor areas; fourth, the length of time needed for impacts to develop in the CDD context, and fifth, that the impact of the project on social dynamics and governance has not been assessed using quantitative methods due to the lack of data in the previous KDP evaluation. This paper attempts to address these issues via a set of indicators based on responses to questions from the SUSENAS 2002 survey instrument and a social capital and governance module are constructed to address the following core research questions: • Does PNPM–Rural increase household welfare (measured as real per capita consumption)? • Does PNPM–Rural move households out of poverty? • Do individuals in PNPM–Rural sub–districts experience increased access to education and health care services, and employment opportunities? • What is the impact for these indicators for poor and disadvantaged groups? • Does PNPM–Rural impact social dynamics in the community and the quality of local governance? The research methodology was designed to ensure the impacts found can be attributed to the program. A household panel was constructed from the SUSENAS 2002 national household survey, followed by separate surveys conducted in 2007 (Survei Evaluasi Dampak PNPM or SEDAP 2007) and 2010 (SEDAP 2010) on the same set of households. A propensity score matching approach was used to select sub–districts participating in PNPM beginning in 2007 and a control sub–district group that has similar characteristics based on data taken from the 2005 PODES village census that began participation in 2009/2010. The sample consisted of 6319 households and 26,811 households from 300 sub–districts across 17 provinces. Qualitative studies were also conducted in eighteen villages in 3 provinces in 2007 and 2010 to enhance understanding of the findings from the quantitative analysis. This enabled the evaluation to conduct difference–in–differences estimates of the impact of PNPM on a set of six groups of indicators: EXECUTIVE SUMMARY • Real per capita consumption • Poverty status • Use of outpatient health services • Unemployment rate The past decade has seen governments and multilateral donors The Government of Indonesia has embraced this approach country by 2010. It provides block grants of approximately • Primary and secondary enrollment rates significantly expand their engagement with communities in project decision–making and implementation through as a key part of its poverty reduction strategy by delegating a portion of its poverty portfolio to community–based Rp. 1 billion to 3.5 billion (US$ 111,000 to US$ 365,000) to sub–districts depending upon population size and poverty • Measures of social dynamics and governance Community–Driven Development (CDD) interventions, programs.1 The centerpiece of the community–based portfolio incidence. Villagers engage in a participatory planning and The main results from the study are listed below: which place community members in control of the planning, is the National Community Empowerment Program (PNPM), decision–making process prior to receiving block grants to design, implementation and monitoring of project activities a key component, of which PNPM–Rural, implemented by fund their self–defined development needs and priorities. As a result of participation in the program, real per capita conducted in their communities. The CDD approach comprises the Ministry of Home Affairs, is an expansion of the previous consumption gains were 9.1 percentage points higher not only the enhancement of community welfare, poverty Kecamatan Development Project (KDP). PNPM–Rural currently Previous studies on the predecessor project KDP found among poor households in PNPM areas compared with reduction and access to services of more traditional rural reaches over 60,000 villages in over 5,000 sub–districts, positive impacts on household welfare, poverty and service control households. This represents an overall monthly infrastructure delivery mechanisms, but also the objective including all rural kecamatan in Indonesia. PNPM–Rural has delivery (see Alatas (2005) and Voss (2008)). Building on consumption gain of Rp 39,000 per capita per month in of fostering increased participation in decision–making on scaled–up from an initial 1993 kecamatan (sub–districts) these findings, several issues emerge with respect to the comparison with control areas. The results also point to the part of communities to develop the skills and capacities in 2007 to cover more than 4,000 rural sub–districts in the effectiveness of the project as it has expanded to become a PNPM being most effective at reaching poor households needed to further their own development, and promote national program: First, that marginalized groups do not share and households in poor sub–districts. Households in the better governance by increasing the demand for transparency in the benefits from the program; second, the impact of the lowest predicted 2007 consumption quintile participating and accountability in the local government environment. 1 Community–based programs constitute Cluster 2 of the poverty portfolio scale up on implementation quality; third, effectiveness of the in PNPM saw their real per capita consumption increase by along with Cluster 1 (household–based programs) and Cluster 3 (small and medium enterprise development). x xi 11.8 percentage points more than in control areas. PNPM PNPM is most effective at reducing poverty and impacting to increase household welfare. The program should continue governance practices in the rural space. Further research on households in the poorest quintile sub–districts saw similar poor households when the needs of the poor are aligned given the existing infrastructure gap in rural areas. However, the barriers to adoption of PNPM principles of transparency positive impacts of 12.7 percentage points in comparison with with those of the wider community. The qualitative study these benefits will only be sustained if the infrastructure is and accountability and potential design changes to address control areas. In addition, positive impacts extended to the provided insight into the greater effectiveness of PNPM in of sufficient quality to continue to be utilized effectively. identified barriers are needed. near poor as households in the second and third consumption poor and remote areas. In situations in which there is a gap Future research should focus on the quality of maintenance quintiles also saw their wider consumption increase relative to in basic infrastructure, the needs of the poor are aligned with and overall sustainability of use for infrastructure built by the Continued focus on marginalized groups: The program control households. those of the community with respect to decision–making on program as well as current mechanisms and procedures in should determine whether the program is best–placed to sub–project infrastructure. However, when basic infrastructure place to ensure proper maintenance is conducted. address the needs of marginalized groups and consider The proportion of households moving out of poverty in is in place, communities continue to select additional additional design changes or other development approaches poor sub–districts was 2.1 percent higher in PNPM areas infrastructure sub–projects which have less potential to Targeted approach to Block Grant allocation: As noted to address their needs. compared with control areas. There was no impact on PNPM reduce poverty in contrast to alternative needs expressed by above, the largest gains are made in poor and remote areas. in preventing households from falling into poverty. the poor that center on capacity and skill development, and Block grant amounts should be targeted toward areas with low Renewed focus on strength of participation and inclusion access to capital. levels of existing infrastructure in order to maximize household of the poor and disadvantaged groups in program Impacts on households in less poor sub–districts are welfare impacts. Additional research is needed to understand decision–making: To overcome the “routine” approach to limited. In general, for both real per capita consumption and PNPM is not perceived by communities as a poverty the effectiveness of program in a wider range of contexts program implementation that has developed due to scale movement out of poverty, households in higher consumption reduction program but rather as a program for the entire (poverty, infrastructure, regional) and implementation up and the long period of implementation in many locations, quintiles or households in less poor sub–district yielded community. Communities view PNPM as a program for the procedures (BLM size, length of participation in the program) the program needs a renewed effort to strengthen its core insignificant results. village and select infrastructure sub–projects on the basis of and consideration given to how to customize the block grant approach of community engagement in program activities the broadest impact for the collective community rather than size menu to meet the needs of different local contexts. to ensure that all groups are included and participate fully in Disadvantaged groups, other than the poor, are less likely an opportunity to target the poor. decision–making over the program cycle. to benefit from the program. Disadvantaged groups, such Strategy to address constraints to stronger downward as female–headed households and households with head As PNPM continues its current phase as a national level social accountability from local government: The fact Continued collection of data: Although the expansion of lacking primary education, see insignificant or lesser impacts program, the results above point toward the following that institutions other than PNPM do not yet emulate the PNPM–Rural to cover all rural sub–districts in the country for real per capita consumption and movement out of poverty recommendations for the program and future research: transparency and governance features of the program indicates necessitates the loss of control areas, the panel nature of as compared to control areas. that a key objective of increased social accountability is not the survey can still be valuable in tracking the progress of Continued funding for infrastructure with a focus on yet being met. While PNPM is not the sole vehicle nor primarily key indicators going forward. Subsequent survey rounds in The proportion of individuals gaining access to outpatient maintenance and sustainability: PNPM remains an effective responsible for changes in the local government environment, 2012 and 2014 should be conducted to ensure continued care was 5.1 percentage points higher in PNPM areas means of delivering needed infrastructure to rural communities it is included as one means to introduce and institute good examination of program effectiveness. compared with control areas. Among individuals not seeking outpatient care in 2007, individuals in PNPM areas were 5.1 percentage points more likely to seek outpatient care in 2010 than household heads in the control group. In contrast to the real per capita consumption and poverty status results above, disadvantaged groups also benefit in terms of expansion of access to outpatient care. Among those unemployed in 2007, individuals in PNPM areas were 1.4 percent more likely to be employed in comparison with control areas. PNPM did not have an impact on overall rates of unemployment. PNPM had no impact on school enrollment rates. High rates of existing enrollment at both the primary and junior secondary levels likely reduce the potential effectiveness of PNPM on education utilization rates. PNPM has impacts on measures of social dynamics and governance within the program but these impacts do not spill over into larger village decision–making processes. Key findings from the qualitative study indicate that while the program was effective in creating participation, transparency and accountability for processes within the PNPM program, these impacts did not spill over into general local/village governance as the capacity of communities to impact elite control of decision–making was limited. Contributing factors include a routinized approach to program implementation on the part of the community and the quality of participation. xii xiii I. BACKGROUND xiv xv I. BACKGROUND The past decade has seen governments and multilateral donors program potentially affecting the quality of implementation The paper is organized as follows: significantly expand their engagement with communities in and subsequent effectiveness. Third, the program began project decision–making and implementation. Among several related objectives, participation by communities is expected implementation in areas which on average were less poor than the more poverty–targeted selection process for KDP, • Section 2 presents background information on the PNPM–Rural program. to allow local information to impact planning, develop the skills and capacities of communities to further their own creating uncertainty over the effectiveness over the project in different contexts.3 Fourth, the length of time needed for • Section 3 describes the methodology used to select the sample and the data gathered. development, create a greater sense of ownership on the part impacts to develop in the CDD context: the evaluation of KDP • Section 4 presents the main results. of communities to reduce corruption and better maintain project–built infrastructure, and promote better governance was over a five year timeframe (2002–2007) whereas existing PNPM–Rural locations have had the project for a much shorter • Section 5 discusses the findings and offers conclusions on key issues for the program going forward by increasing the demand for transparency and accountability period (from 1–4 years). Finally, the impact of the project on presented above. in the local government environment. In a standard approach, Community–Driven Development (CDD) interventions seek to social dynamics and governance has not been assessed using quantitative methods due to the lack of data in the previous • Section 6 provides policy implications. recommendations and achieve this by placing community members in control of the KDP evaluation. planning, design, implementation and monitoring of project activities conducted in their communities. In addition to The research design for the PNPM–Rural evaluation attempts these objectives, which differentiate the CDD approach from to address these concerns by utilizing a household panel more traditional means of project delivery, CDD approaches generated from the SUSENAS 2002 national household also claim to realize development objectives, frequently survey, and separate surveys conducted in 2007 (Survei associated with traditional approaches, which seek to enhance Evaluasi Dampak PNPM or SEDAP 2007) and 2010 (SEDAP community member welfare: increased access to services, 2010) collected from the same set of households.4 A set of poverty alleviation, employment and consumption. indicators based on responses to questions from the SUSENAS 2002 survey instrument and a social capital and governance The Government of Indonesia has embraced this approach module are constructed to address the following core as a key part of its poverty reduction strategy by delegating research questions: a portion of its poverty portfolio to community–based programs.2 The centerpiece of the community–based portfolio is the National Community Empowerment Program (PNPM), a • Does PNPM–Rural increase household welfare (measured as real per capita consumption)? key component of which, PNPM–Rural is an expansion of the • Does PNPM–Rural move households out of poverty? previous Kecamatan Development Project (KDP). PNPM–Rural now reaches over 60,000 villages in over 5,000 sub–districts, • Do individuals in PNPM–Rural sub–districts experience increased access to education and health care services, including all rural kecamatan in Indonesia. Previous studies and employment opportunities? on the predecessor project KDP found positive impacts on household welfare, poverty and service delivery. Alatas (2005), • What is the impact for these indicators for poor and disadvantaged groups? in a study of KDP Phase 1, found that KDP had a significant impact on per capita consumption in comparison with a • Does PNPM–Rural impact social dynamics in the community and the quality of local governance? control group, and that the longer communities participated in the program, benefits increased. Voss (2008) also found Qualitative studies were also conducted in eighteen significant gains in consumption, access to outpatient care villages in 3 provinces at baseline in 2007 and at endline in and employment for households participating in the second 2010 to enhance understanding of the findings from the phase of the project (KDP2). quantitative analysis.5 Building on these findings, several areas of concern emerged on the effectiveness of the project going forward. First, despite positive gains in household welfare among the poor under KDP, marginalized groups did not share in the 3 A study examining the EIRR for KDP infrastructure sub–projects concluded benefits from the program. Second, as PNPM–Rural scaled that the largest gains were found in poor and remote areas with a low base up to cover every rural sub–district in the country, capacity of existing infrastructure. was stressed to a greater extent than under the smaller KDP 4 The sample was selected from the 2002 SUSENAS in order to satisfy the needs of the KDP2 impact evaluation. For that evaluation, the SEDAP07 was used as the post–project survey. 5 This study will utilize key findings from the qualitative study to enhance 2 Community–based programs constitute Cluster 2 of the poverty portfolio the understanding of results from the quantitative analysis. For a full along with Cluster 1 (household–based programs) and Cluster 3 (small and discussion of the findings presented from the qualitative study, see SMERU medium enterprise development). (2010). 1 2 II. THE PROGRAM NASIONAL PEMBERDAYAAN MASYARAKAT PNPM – RURAL COMPONENT 3 4 II. THE PROGRAM NASIONAL PEMBERDAYAAN MASYARAKAT PNPM – RURAL COMPONENT Since the 1997 economic crisis, the Government of Indonesia fund self–defined development needs and priorities. Village and the World Bank have increased their engagement with proposals (one of which must come from a women’s group) communities in development projects through the use of the are sent to a sub–district forum where village representatives Community–Driven Development (CDD) approach to project evaluate proposals based on predetermined poverty criteria design. In September 2006, the government decided to launch a and allocate funding for individual proposals. new program utilizing the CDD approach to accelerate poverty reduction and increase employment opportunities in order to The project cycle generally takes 12–14 months and is achieve the targets set in the Mid–Term National Development described in brief below:6 Plan (2005–2009) and the Millennium Development Goals. Existing community–based poverty reduction programs Information dissemination and socialization: Workshops were consolidated into a National Program on Community are held at the provincial, district, sub–district and village level Empowerment (PNPM–MANDIRI). The program is described as to disseminate information and popularize the program. a national movement of stakeholders to reduce poverty and generate employment by increasing community capacity and Participatory planning: Villagers elect village facilitators self–help to achieve a better standard of community welfare. (one man and one woman) to assist with the socialization and planning process. The facilitators hold group meetings, The rural component of PNPM–MANDIRI, PNPM Rural is the including separate women’s meetings, to discuss the needs successor to the Kecamatan Development Program (KDP). of the village and their development priorities. Social and KDP was initiated in 1998 and continued over three phases technical consultants at the sub–district and district level assist through 2007 in approximately 2500 sub–districts. The first with socialization, planning, and implementation. Villagers year of PNPM–Rural was comprised of 1993 sub–districts then create proposals and come together in a village–level as a continuation from participation in KDP. The program forum to decide which proposals will be sent to a subsequent then expanded in 2008 and 2009 to cover almost all rural sub–district–level meeting. Each village can submit up to two Table 1: sub–districts in the country. By 2009, 4,871 sub–districts in proposals to this forum with the requirement that the second Distribution of Block Grant Funding by Type of Activity in 2009 Indonesia were participating in the program. The overall proposal must come from a women’s group. objective of the program is to improve the welfare of poor communities. Specific objectives include: Project selection: Communities then meet at the village Activity Public Infrastructure (Roads, Education Health Micro–credit and sub–district levels to decide which proposals should • Increased Bridges, Irrigation) participation of community members not fully be funded. Meetings are open to all community members. involved in the development process including the poor, An inter–village forum composed of elected village Percentage of Block Grant 65.97 12.71 4.31 17.12 women, and indigenous communities. representatives makes the final decisions on project funding. • Improved capacity of locally based community institutions. Project menus are open to all productive investments except Funding • Improved local government capacity to provide public services through the development of pro–poor programs, for those on a short negative list. policies and budgets. Implementation: PNPM–Rural community forums select • Increased synergy between communities, local government and other pro–poor stakeholders. members to be part of an implementation team to manage the projects. Technical facilitators help the village implementation Block grants can be used to fund any public infrastructure, and include a cash–for–work component during construction • Enhanced capacity and capability of the community and local government in reducing poverty. team with infrastructure design, project budgeting, quality verification, and supervision. Workers are hired primarily from training or capacity building project, subject to a short negative list, along with up to 25 percent of funds used for which provide temporary employment; second, roads and new public service infrastructure such as schools and health clinics the beneficiary village. micro–credit activities with project–created women’s savings will allow greater access to services by reducing transportation PNPM–Rural utilizes a Community–Driven Development groups. During 20097, the breakdown of block grant funding time and cost; third, increased community engagement with approach by involving all community members in planning, Accountability and reporting maintenance. by type of approved sub–project was as above in table 1. government, enhancement of community skills and capacity implementing and monitoring of community activities funded During implementation, the implementation team reports and increased willingness to hold government accountable by the program, with a special emphasis on marginalized on progress twice at an open village meeting. At the final PNPM is designed to achieve its objective through the is expected to result in better local governance, resulting in groups (including women and the poor). The project provides meeting, the implementation team hands over the project following three primary mechanisms: first, new infrastructure decision–making that sees greater benefits for the community. block grants of between Rp 1 billion to Rp 3.5 billion to to the village and a designated village operations and projects, including roads, bridges, irrigation are designed to sub–districts depending upon population size and poverty maintenance committee. increase production and market access in the local economy incidence. Villagers engage in a participatory planning and decision–making process prior to receiving block grants to 6 Taken from the PNPM project website. For a more detailed description see: 7 These percentages are consistent with 2007 and 2008. www.ppk.or.id. 5 6 III. METHODOLOGY 7 8 III. METHODOLOGY In this section we develop the methods used in sampling, both PNPM–Rural 2007 participation and outcome indicators. Box 1: Data Sources identification of future impacts, and data issues. See Annex 1 From this process, a set of 150 pairs of matched treatment and for a more detailed description. control sub–districts were selected for the sample. Tests to compare the effectiveness of the propensity score matching The SUSENAS is an annual household survey administered by the Central Statistics Agency (BPS) designed to assess A. Identification procedure demonstrate that for all of the observed covariates household welfare conditions on a national scale. Currently interviewing over 200,000 households in every district in there is no significant difference based on participation in Indonesia, the survey covers such topics as household consumption, housing conditions, health care, pre natal care, The approach of the research design is to use the most PNPM–Rural 2007. Thus the covariates are “well–balanced” education, employment and income. Specialized modules dealing with specific topics such as housing, health, culture and rigorous viable methodology to select a sample that is between treatment and control groups indicate a high education are administered to a subset on a rotating basis. The data is representative of both a national and district level. able to attribute impacts on indicators to PNPM–Rural degree of similar for the variables which were included in the after the 2010 follow up survey. matching process. While the methodology represents the The PODES is a national village census, also administered by BPS, and conducted three times per decade in all villages The primary problem in program evaluation is that we wish to best opportunity given the data available to properly identify across Indonesia. The data are a complete enumeration of every village in Indonesia, recording information on compare the experience of those participating in the project impacts, there are some caveats. The methodology described characteristics (such as land size, population, water supply) and available infrastructure (number of schools, hospitals, with the counterfactual, or experience without the project. above does not account for factors which are not included doctors, markets, transportation and financial institutions). The survey used in this study is the 2005 version, including Unfortunately, it is not possible to observe the counterfactual in the matching process and which have the potential to data on 68,819 villages. outcome of no project in areas where the project is assigned. introduce bias into the results. However, this is mitigated, to Instead, a control group must be created which represents some extent, by the fact that the methods used to estimate the counterfactual scenario comprised of sub–districts similar impacts eliminate factors which do not vary over time.8 to those receiving PNPM–Rural. To solve this problem, the research design takes advantage of the phased approach B. Data Household Level: 2007 Rupiah measure for 2010 consumption per capita.11 The to the program’s implementation to create a control group totals for each year were then logged and differenced. The using sub–districts which began participation in PNPM–Rural in late 2009. Due to measurable similarities across a range Primary data sources include the 2002 SUSENAS, the 2005 PODES village census, and Survei Evaluasi Dampak • SUSENAS 2002 Section VI: dwelling characteristics, sanitation and access to drinking water; advantage of using logs in this fashion is that estimates can be interpreted as the percentage point difference in growth of observable characteristics the control group represents outcomes that would have occurred had the project not taken PNPM–Rural (SEDAP 2007) 2007 survey and the 2010 SEDAP 2010 survey. The evaluation utilizes a household panel • SUSENAS 2002 Section VII: Household food and non–food consumption; rates of real per capita consumption between treatment and control groups. place. The treatment group consists of sub–districts beginning participation in PNPM–Rural in late 2007 while the control with data collected from the SEDAP 2007 survey conducted from August to September 2007. The household sample was • Social Dynamics and Governance Module: community participation in village meetings and activities, trust in Poverty status is assigned based on the 2007 and 2010 groups consists of sub–districts beginning participation selected from households participating in the 2002 SUSENAS. community members and government officials, collective BPS provincial poverty lines. Households are assigned as in late 2009/early 2010. The analysis below compares how A second survey of the same households was conducted in action, access to information, access to services and “poor” or “non–poor” using their 2007 and 2010 BPS real per the experience of areas which participated in the program early 2010 (SEDAP 2010) to create a panel. The overall sample self–assessed poverty. capita consumption measures and the 2007 and 2010 BPS differs from changes observed in the control group. The includes 6319 households from 300 sub–districts with 26,811 Rural poverty line. Households are then placed into one of difference between the magnitude of the respective changes individuals for the 2007 survey round and 6139 households Individual Level: four Poverty Status categories: 1) Remained Poor, 2) Never in the treatment (PNPM–Rural 2007 Kecamatan) and control from the 2009/2010 survey round indicating an attrition Poor, 3) Out of Poverty, 4) Into Poverty. (PNPM–Rural 2009) groups for outcome indicators is the rate of less than 3 percent. Data used for the sub–district • SUSENAS 2002 Section Va: Health impact attributable to the program. level propensity score matching were taken from the 2005 • SUSENAS 2002 Section Vc: Education Access to health indicators are also constructed using a A propensity score matching methodology was used PODES census of villages conducted by BPS, including a range of variables (see Annex 1) describing the infrastructure, • SUSENAS 2002 Section Vd: Employment “change in status” categorical variable. The sample for access to health indicators consists of individuals that were to construct the counterfactual. The ideal method for economic and demographic conditions of all sub–districts Consumption is measured as the change in the logged real sick in both 2007 and 2010. For incidence of outpatient care generating the counterfactual is a randomized selection of in the sampling frame. Demographic variables were derived per capita consumption between 2007 and 2010. Measures conditional on being sick, individuals are assigned into one sub–districts for participation in the program. However, entry through aggregation from yearly SUSENAS household surveys. for consumption per capita in 2007 and 2010 are taken directly of four categories: 1) Always sought outpatient care, 2) Never into the program was not assigned randomly and although from the 2007 SEDAP I and 2010 SEDAP II surveys,9 using the sought outpatient care, 3) Newly seeking outpatient care in the program sought to target the poorest locations, other The survey instrument is comprised of questions from the 2002 SUSENAS instrument.10 The 2010 data are then adjusted 2007, 4) Previously sought outpatient care and not seeking considerations that were taken into account in assigning 2002 SUSENAS national household survey and a separate using a set of regional price deflators to arrive at a constant in 2007. participation render the use of poverty mapping and other social capital and governance module. Due the demands of objective criteria problematic to the extent that it is not the research design, sections of the instrument available for Unemployment status is calculated via two methods. possible to formulate a systematic method for selection of analysis are limited to a subset of questions taken from the 9 Food expenditure is defined as the sum of all weekly food categories Following Suyadarma, Suryahadi and Sumarto (2005), we sub–districts into the 2007 or 2009 phases of the program. 2002 SUSENAS core instrument and a separate social capital multiplied by 30/7. Non–food expenditure is defined as the sum of yearly construct two different measures for unemployment. The first expenditure divided by 12. The total expenditure is calculated as the sum Lacking randomization or clearly specified and systematic and governance module. Specifically, from the 2002 SUSENAS of food and non–food totals. measure excludes discouraged workers and includes an active selection criteria, the evaluation employed a propensity score core instrument: 10 The fact that the 2007 survey was conducted in August/September and labor force population of adults aged 18–55 consisting of matching technique in which a set of variables or covariates are not in January may have impacted the data collection for the consumption employed (both at work and not at work but still employed), selected based on their availability and likely correlation with measure. Because this time period was heading into the fasting month, self–employed and unemployed. The second measures we might expect estimates to be slightly higher than normal. Seasonal differences may also impact estimates. adds discouraged workers to the labor force population 8 The discussion of the matching and estimation methods are deliberately 11 The Farmers’ Terms of Trade Index, which reflect changes in rural consumer kept brief in the main text. For a detailed discussion see Annex 1. and producer prices by province, were used as the deflator. 9 10 and considers both declared unemployed and discouraged levels as PNPM–Rural.14 Kecamatan which participated in these E. Qualitative Methodology17 workers as unemployed. Discouraged workers are defined programs or in any phase of KDP between 2002 and 2007 were as those not working or declared unemployed that either not included in the sampling frame. In addition, areas which The qualitative component visited 18 villages in 9 indicate difficulty in finding a job or have no other valid were under sampled in the 2002 SUSENAS, including Aceh, sub–districts in West Sumatra, East Java and Southeast reason for lack of employment (school attendance, retirement, Maluku, North Maluku and Papua are not included in the Sulawesi during the period April–June 2010. The sample of household duties). sampling frame. The remaining sub–districts from the 2002 villages was selected based upon length of participation SUSENAS not excluded from the previous participation in in the program (including a control group from Education access is measured using transition rates for similar CDD program or under sampled in the 2002 SUSENAS sub–districts beginning participation in PNPM in 2009) appropriate age cohorts between primary and lower were then pooled and matched using the methods described as well as by poverty level. Study teams conducted the secondary school. Net school enrollment is defined as the above. The sampling was not stratified by region in order to following activities: (1) 8 key informant interviews, including number of children enrolled in the appropriate age group ensure the largest pool of control sub–district available for facilitators, village officials, and community leaders, (2) divided by the number of children in the appropriate age matching to each treatment sub–district. For the geographical 4 village informants, consisting of 1 male poor and 1 group in the population. Age groups are defined as 7–12 years distribution of sub–district by province, see Table A1.1. male non–poor resident, and 1 female poor and 1 female for primary school, and 13–18 years for secondary school. non–poor resident; (3) 5 focus groups discussions including Transition rates are percentage of each age cohort enrolled in For each selected sub–districts, twenty–two households village officials, male poor and non–poor residents, and primary school in 2007 that is also enrolled in lower secondary are sampled from the 2002 SUSENAS. From each female poor and non–poor residents. Profiles on poverty, school in 2010. sub–district, two enumeration areas (EA’s), a sampling unit of infrastructure, demographics and other characteristics sixteen households defined by geographic proximity and used were constructed for each village in the sample. Social Dynamics and Governance variables reported on by the BPS for SUSENAS sampling procedures, were selected. below are described in Table 2. These are a representative At the household level, eleven of the sixteen households were subset of a large set of variables included in the social sampled in the 2007 survey. Selection was based on the order 17 For a detailed discussion of the methods used in the qualitative dynamics and governance instrument.12 of households listed in the 2002 SUSENAS with replacements component see SMERU (2010) (households numbered 12–16) used when it was found that C. Sampling members of the first eleven on the list were no longer in the village where the EA was located. Table 2: Social Dynamics and Governance Variables Sample size was determined using power calculations.13 The sample size was calculated taking into account the Attrition for the 2007–2009/2010 period was approximately multi–stage sampling design. The required sample size is 2.8 percent. From the total number of households sampled Incidence of Collective Percentage of population 2250 households and 150 sub–districts (15 households per in the 2007 survey, 6143 were interviewed in 2009/2010. The Action participating in joint activities sub–district for both the treatment and control groups based survey attempted to follow all households leaving their original to benefit the community on an estimated treatment effect size of .14. An additional 50 2007 location from the 2007 sample within or to existing percent was added to the sample to account for expected SEDAP provinces, or Jakarta. Households which could not be Trust in Percentage of population attrition between 2002 and the final round survey in interviewed either moved out of the country or to non–SEDAP Village Government indicating “strong” or 2009/2010. provinces (excluding Jakarta), or saw all household members “somewhat” agree with the pass away during the period under evaluation. Households statement: “Village officials can The sampling frame is constructed from households migrating out of the sub–district but which could not be be trusted”. included in the 2002 SUSENAS. Due to the dual purpose of tracked represented only 8 percent of the sample. the 2007 SEDAP survey: 1) an endline survey for the evaluation Petitioning of Percentage of households of KDP2 (see Voss, 2008) and 2) a baseline for the planned D. Estimation15 Local Government joining a community effort to PMPM–Rural evaluation, households were selected from the petition village government to 2002 SUSENAS national household survey. It is important Estimation was conducted using a address a need or concern to note that the sample selection is taken from that dataset difference–in–differences approach. While the specific and not from all sub–districts and households in Indonesia. methods vary depending upon the specific variable16, a The sampling frame from which sample sub–districts and difference–in–differences approach is used to generate Participation in Percentage of households households were selected consists only of sub–districts and estimates of program impact. The change in control areas, Village Meetings attending most recent households which were surveyed in the 2002 SUSENAS. which represent the counterfactual of changes in indicators village–level meeting if the program had not been run yet, is compared with In addition, some sub–districts from the 2002 SUSENAS are changes in indicators in the treatment areas. The difference Perception of Local Percentage of households excluded from the sampling frame due to participation in in these changes is the impact attributable to the project. It Government Addressing indicating “strong” or similar CDD programs, location in conflict or tsunami affect is important to note that impacts are representative at the Community Needs “somewhat” agreement with areas, or due to limited coverage in the 2002 SUSENAS. The sub–district level for all households and do not represent the following statement: “the evaluation identified five programs using similar approaches impacts specifically limited to villages where sub–projects government takes my needs in terms of implementation and per village disbursement are constructed. into account” Access to Information Percentage of households Concerning Development indicating they have access 12 Due to findings indicating a lack of impact (See Section 4 below) not all 15 For a detailed discussion of the methods used in the qualitative component Funds to information concerning variable are reported on. These six are a representative subset. Full results see SMERU (2010) the use of funds for village available on request. 16 For a detailed discussion of the econometric methods used to generate development 13 See Annex 2. estimates of impacts, see Annex 1. 14 See Annex 1, Section A.1 for list of programs. 11 12 IV. RESULTS 13 14 PNPM has a significant impact on consumption. differences estimation approach which yielded the results The results suggest that PNPM has a significant impact on discussed above, we can eliminate heterogeneous factors IV. RESULTS changes in logged real per capita consumption. Looking which are fixed over time. We correct for this problem via a at the full sample, households receiving PNPM saw their second household level matching using household level consumption per capita increase by 9.1 percentage points variables from the 2007 SEDAP data and generate the same more than in control areas over the period 2007–2010. estimates as above using the matched household sample.22 This finding is in contrast to the previous evaluation of the Looking at the column “Household–level matched sample”, predecessor project KDP2 which lacked consistent and robust the results shown in Table 3 show a similar pattern to those evidence for impact from the project on the full sample.20 observed with the first differencing approach discussed above. Positive impacts for the full sample, 1st quintiles of PNPM has a stronger impact on poor households. 2007 predicted per capita consumption, and 2005 sub–district For the first quintile of households ordered by 2007 per capita poverty score are significant at 5.3, 11.2 and 9.5 percentage consumption, there is an 11.8 percentage point difference points respectively. In addition, there is a positive impact in the growth rate of real per capita consumption between of 8.6 percentage points for the 3rd quintile of predicted PNPM, and control households. For households in relatively per capita consumption also consistent with the first wealthier quintiles, PNPM appears to be less effective. At differencing approach.23 the top end of the consumption distribution, there is no significant impact for households in the 4th and 5th quintiles. For real per capita consumption, the distribution of PNPM There is stronger evidence for impacts in the 3rd and, to some benefits does not extend to traditionally disadvantaged extent, 2nd quintiles: for the former, consumption growth was groups. The impacts attributed to PNPM for poor households 15.6 percentage points higher. For the 2nd quintile, the impact are not realized by female headed households or households was 8.4 percentage points higher, although significant only at with heads lacking primary education. Given the results for the 10% level. real per capita consumption above, we might expect similar results for these groups. However, looking at impacts on female PNPM impacts extend to the near poor. As might be headed households and households stratified by education of expected given the significant finding for the full sample, the household head, the same pattern does not emerge as we we find evidence for significant impacts for the 2nd and 3rd find no significant positive impacts for PNPM. This is perhaps quintiles for PNPM in contrast with impacts confined to somewhat surprising given the emphasis PNPM places on the first quintile for the previous evaluation of KDP2. While incorporating women into the project process. Separate the 2nd and 3rd quintiles do not represent poor households, women’s meetings are conducted as part of facilitation given the relatively concentrated consumption distribution activities and one of the proposals from each village must in Indonesia, they do represent households which are “near come from women’s groups. Evidence from a recent study poor”, given that over half of all Indonesian households are on PNPM and Marginalized groups, as well as the PNPM Rural clustered around the national poverty as of early 2010.21 Evaluation Qualitative component, supports these findings. Despite procedures within the program to incorporate women PNPM has a stronger impact on households in poor and the poor, the project still has difficulty reaching various sub–districts. In addition to quintiles based on predicted pockets of highly vulnerable groups, including female–headed This section discusses the main results from the analysis, estimator with a household matched sample.18 Effects are per capita consumption, we also generate quintiles based on households and household heads with no primary education. including both the quantitative and qualitative components. presented for the full sample and samples stratified by a 2005 BAPPENAS generated poverty score of sub–districts. Decision–making is still concentrated among elites and Section 4.1 addresses household welfare as measured by real predicted 2007 consumption quintiles19, sub–district poverty The poverty score is based on a range of factors, including activists in the village who tend to have strong influence over per capita consumption. Section 4.2 considers the impact of quintiles, and the education level and gender of the household education, health, demographic and poverty data. The results not only overall project decision–making, but also within changes in household welfare on changes in poverty status. head. The results are shown in Table 2. are similar with a positive 12.7 percentage point impact in sub–groups such as women’s groups designed to generate Section 4.3 presents evidence on expanding access to health the log growth rate of real per capita consumption on PNPM proposals for use of project funds.24 Program managers also care. Section 4.4 addresses impacts on access to education, households in the first (poorest) quintile. cite PNPM project facilitators’ focus on reducing elite capture specifically transition from primary to lower secondary school. vis–a–vis the majority in the village and the fact that less Section 4.5 looks at employment. Section 4.6 discusses Estimates from matched households demonstrate attention and effort have been paid to include the hard–to– findings on social dynamics and governance. References to 18 See Annex 1 for a detailed discussion of the estimation approaches. consistent and robust results. A further concern for the reach population segments. Facilitators, tasked with inclusion significant results refer to the 5 percent level, unless otherwise 19 A primary concern for the validity of the results demonstrated in Table validity of the results presented above is heterogeneity at of marginalized groups are less than effective, due in part to a noted. See Annex 3 for a summary of key findings from the 2 is the potential for bias due to measurement error. Households which the household level. Although matching at the sub–district large administrative burden which creates time constraints, as were measured too low or too high in 2007 and then properly measured Qualitative Component. in 2010 (or vice versa), will see large changes which do not represent the level ensures that households from the same sub–districts well as lack of proper training. The result is that marginalized true change in consumption. This effect has a tendency of convergence experience the same sub–district level conditions in terms groups are typically not included in the decision–making A. Household Welfare within the consumption distribution: poorer households see large gains of the economic, social and other environments, significant process and that sub–projects funded by PNPM block grants relative to richer households. Moreover, using the 2007 real per capita household level heterogeneity for variables that could impact are not typically those which are perceived by marginalized consumption measures to generate quintiles could lead to biased As described in Section 3.2, the measure of the change in and inconsistent results as mismeasured households are not assigned consumption (and the indicators considered below) could households as bringing the largest benefits. household welfare is the difference in logged real per capita to their true quintiles. For example, non–poor households that were remain. Such heterogeneity could introduce bias if correlated 22 Variables include ownership of durable assets, household income, housing consumption between 2007 and 2010. We compare the under measured relative to their true consumption would populate the with PNPM treatment assignment. Through the difference–in– conditions and demographic characteristics of the household, including changes in consumption between treatment and control sample for the first quintile rendering it a poor representation of true age and education. See Annex 1. first quintile households measured without error. To address this concern 23 The matched household sample also shows a significant positive impact households using a first differences approach with the full we create predicted per capita consumption quintiles referenced above 20 See Voss (2008) p 26. for the 4th quintile of kecamatan poverty score however this is not reflected sample and then using a difference–in–differences matching using household level asset and demographic variables from the 2007 21 World Bank (2010). US$2 per day poverty headcount ratio was 50.6% in in the first differencing model. SEDAP survey. 2010 and 58% in rural areas. 24 AKATIGA (2010), pp. 3–4 15 16 areas, large returns can result in consumption impacts which on the household–matched conditional comparison of means changed their status with respect to use of outpatient services Consumption gains represent a significant return are beneficial to the poor. When infrastructure is already in model poor households in 2007 in poor sub–districts in PNPM by moving into outpatient care in 2010 after not seeking it on project investment. In 2009, the approximate block place, the marginal impact on household welfare for the locations were 16.7 percentage points more likely to escape in 2007. Estimates represent the percentage point difference grant amount per capita was Rp 67,000 for the 2009 cycle. poor is small given the lack of impact on the local economy poverty than control areas. There is limited support for this between treatment and control individuals that were sick Considering the 9.1 percentage point differential between and consequently consumption gains are not significant. finding using the full sample multinomial logit model at 3.2 and did not seek outpatient care in 2007, but were sick and PNPM and control household per capita consumption growth percentage points more likely to escape poverty, at the 10 accessed outpatient care in 2007.31 The results are shown in rates, the amount per month generated by the project is B. Poverty Status percent level. In addition, there is a strong significant impact Table 6. approximately Rp 39,000 in 2010 Rupiah at the per capita of 22.5 percentage points for poor households in the least consumption average, indicating that the yearly impact on In this section, we employ two models to obtain estimates poor sub–districts using the household–matched comparison PNPM community members are more likely to access average per household from the project is Rp 384,000 or 5.7 on changes in poverty status. We use a multinomial logit of means model. However, this finding is not repeated using outpatient care as a result of the program. As shown times the amount invested in 2009. Considering the more model on the full household sample and then a conditional the multinomial logit model where there is no significant in Table 6, the results from the multinomial logit and the conservative estimate of 5.3 percentage points from the comparison of means test using the matched household impact specification. matched household conditional models demonstrate that matched household sample, the year impact is approximately sample constructed for the per capita consumption analysis PNPM expands access to outpatient care. Among individuals Rp 221,000 or 3.3 times the block grant invested in 2009. above. Households are placed into four categories based on Female–headed households and households with heads that did not seek outpatient care in 2007, PNPM community However, as we have noted above, these benefits are not poverty status in 2007 and 2010: 1) never poor; 2) moved out lacking primary education do not see positive changes in members were 5.1 and 4.5 percentage points more likely to homogeneously distributed. of poverty; 3) moved into poverty; and 4) stayed poor. Poverty poverty status due to PNPM. Consistent with the findings use outpatient services than control households in 2010 for lines are taken directly from BPS provincial rural poverty lines. on per capita consumption, the lack of consumption gains the multinomial logit and conditional comparison of means PNPM is most effective at reaching poor households and The multinomial logit model for the full sample allows us to because PNPM is not creating positive changes in poverty models, respectively. There is also some evidence of similar households in poor areas. The results discussed above consider the probability of inclusion into the four categories status for marginalized groups. Female–headed households impacts among poor households. For the first quintile of for household welfare point to PNPM being most effective simultaneously, whereas the conditional comparison of and households with low household head education follow 2007 per capita consumption the likelihood of moving into at reaching poor households and households in poor means model on the match–household sample considers a similar pattern to consumption with insignificant impacts outpatient care was 6.2 and 5.7 percentage points higher sub–districts. Previous studies on KDP support this conclusion only households which were (1) poor in 2007 and moved out from the program. for PNPM2 households but only at a 10 percent level of by demonstrating the advantages of the PNPM approach in of poverty; and (2) not poor in 2007 and moved into poverty.26 significance. poor and remote areas. Torrens (2005) and Dent (2001) in PNPM is not regarded as a poverty reduction program analyzing the return to subproject investments showed that Given comparable 2007 poverty rates for treatment and control by community members. Considering the discussion with Community members with relatively less education see the largest gains for KDP2 participants were in areas where households27, the categories of greatest interest are 2) and 3): respect to alignment of needs identified by the poor and gains in access to outpatient services due to PNPM. In potential production was suppressed due to barriers to market households moving out of poverty and households falling into sub–projects proposed and funded by communities, the contrast to the consumption and poverty results above, access. New roads, irrigation infrastructure and water projects poverty. In the conditional comparison of means model, using qualitative component provides additional findings on household heads with no primary education benefit created access to markets that were previously inaccessible the household matched sample, we restrict the sample to how communities perceive PNPM which may contribute to significantly in terms of expanding access to outpatient or not viable due to high transportation costs, allowed more those households which were poor in 2007 for category 3) and the findings from the quantitative survey. While a primary services. Considering both models, we see a 4.3 and 7.5 than one crop planting per year, or greatly reduced the time to those not poor in 2007 for category 4. Coefficients indicate PNPM objective is improved household welfare and poverty percentage point difference for PNPM household heads with devoted to water collection. One of the primary reasons for the the percentage point difference in households moving out reduction, communities themselves do not regard PNPM as no primary education for the multinomial logit and conditional lack of proper infrastructure is the high cost of construction in of or into poverty in treatment households relative to control a poverty reduction program. Instead, the program is viewed comparison of means models respectively. Female household poor and remote areas. Torrens (2005) finds that KDP is able households. Results refer to Tables 4–5. as for the community as a whole, rather than targeted toward heads do not see the same benefits. to build local infrastructure at a lower cost than comparison the poor. In some cases, PNPM was perceived as a direct estimates for standard government contractors due to locally While there is some evidence that PNPM moves counterbalance to the household level poverty–targeted The distribution of health benefits is more favorable to sourced materials and community contributions; this would households out of poverty, PNPM is not effective at program: community members expressed the view that PNPM disadvantaged groups and less concentrated in poor be even more advantageous from a cost perspective in remote preventing households from moving into poverty. Looking should not be pro–poor targeted given the existence of other sub–districts. Aside from female headed household, the areas where the potential for consumption gains are large. at the full sample, we find that poor PNPM households are prominent programs for the poor. In the majority of villages, poor, and less educated households show consistent benefits 2.1 percentage points more likely to move out of poverty poverty criteria with respect to proposal planning were not from the program. This is despite infrastructure activities for Impact are largest when needs of the poor are aligned than control households using the multinomial logit included in the decision–making process and poor household health comprising just 2.4 percent of all funds disbursement.32 with needs of the community. Evidence from the qualitative model, and 7.9 percentage points more likely employing members were not specifically targeted for inclusion Moreover, given the small percentage of funds used for component provides some additional insight into the the household–matched conditional comparison of means on temporary employment lists for PNPM infrastructure construction of health facilities, the biggest factors may come relative effectiveness of the project in poorer and more model. This is somewhat consistent with the findings from the sub–project construction.29 from new roads reducing transportation and time costs and remote areas. In poor villages with low levels of existing previous evaluation of KDP2.28 However, results are significant consumption gains noted above which allow for greater infrastructure, the needs identified by the poor were aligned only at the 10 percent level. In contrast to findings from the C. Access to Health Care spending on health care, rather than new health infrastructure. proposed sub–projects proposed and subsequently funded evaluation of KDP2, we do not find any impact from PNPM on The fact that impacts are widespread but not found in poor by communities, focusing on irrigation, roads, agricultural preventing households from moving into poverty. This section utilizes a similar approach to section 4.2 by sub–districts may indicate a preference in poor sub–districts inputs and training. Where existing infrastructure was in place, considering changes in usage of outpatient facilities by in favor of roads, irrigation or other projects which have a typically in less poor villages, the needs identified by the poor Impacts on poverty status are largest for the poor in poor household heads using a multilevel logit model on the full more direct impact on production, but which given their more were not aligned with the projects funded by communities. areas. The findings for changes in poverty status are generally sample of individuals and a conditional comparison of means remote status, may not reduce the transportation and time In these cases, communities continued to fund infrastructure consistent with the results for per capita consumption. Based model on the matched household sample.30 As described in costs enough to see increased access to health care. In other projects, such as roads, bridges and irrigation, whereas the section 3.2, household members that were sick in 2007 and sub–districts, communities may find it easier to prioritize poor identified capital, skill training, jobs, education and health 2010 were divided into 4 categories based on 2007 and 2010 improving health facilities. Findings from the qualitative as primary needs. Under PNPM over the course of the period 25 See SMERU (2010) for background discussion. Further evidence is based usage of outpatient services. Here we focus on individuals that component confirm this view in that access was considered under evaluation, 66 percent of all project funds were spent on field notes from the Qualitative Study and consultation with Qualitative sufficient in most communities.33 on infrastructure in comparison with 17 percent on health and Study authors. 26 For a detailed discussion see Annex 1, Section 1.3. 29 See SMERU (2010), pp. 40–41. 31 The sample size was not adequate to compare household heads that education and 17 percent on microfinance activities.25 As noted 27 See Voss (2008) p 11 for 2007 SEDAP baseline indicators by treatment 30 The lack of a full individual level panel precludes considering the entire sought care in 2002 but did not seek care in 2007. above via the Torrens and Dent studies, in infrastructure–poor group. sample. 32 NMC (2007) 28 See Voss (2008), pp. 27–28. 33 See SMERU (2010), Section 5.4. 17 18 Third, communities indicate that gender is disappearing as a D. Access to Education factor under consideration for schooling through the lower secondary level. Similar to the consideration of access to health care services above, the amount of funds spent on education projects under E. Access to Employment PNPM is relatively low. The primary means of PNPM impacting access to education is likely to arise from consumption gains PNPM has limited impact on a long–term employment as well as reduced time and cost of access. As noted above, status in participating sub–districts. One of the key features in contrast to the previous evaluation of KDP2, the current of the PNPM approach is the employment of community dataset contains an individual level panel which allows for members in the construction of village projects. Given that an estimation of the impact of PNPM on individual cohort this employment is temporary for the purpose of project enrollment. We employ the multinomial logit model on construction, it might be expected that such employment cohorts which over the course of the evaluation period gains would disappear once the project reached completion. would graduate from primary to school to lower secondary However, as Papanek (2007) argues, the majority of school34 creating three categories: (1) not in primary school, employment gains due to PNPM–RURAL are likely to result (2) in primary school at baseline but not transitioning to from indirect sources due to increased economic activity lower secondary school in 2010 and (3) in primary school at rather than direct employment through the program. The baseline and transitioning to lower secondary school. The results shown in Table 8 support the view that PNPM has had conditional logit model with fixed effects is also considered to a limited impact on employment: adults aged 18–55 who check robustness. We also evaluate the impact of the project were unemployed in 2007 had a 1.35 percent chance of being on enrollment rates for primary and lower secondary school employed in 2010. When discouraged workers are added to using household level cohort panel.35 the labor force, these impacts disappear, indicating PNPM is less effective in assisting individuals who have stopped Project data indicates that key indicators are strong community is not perceived as influencing decision–making PNPN does not impact transition rates from primary to looking to work due to difficulties with finding employment. within the project but are not replicated for general in non–PNPM village affairs.39 Two key barriers are cited. First, lower secondary school. As shown in Table 7, there is no village governance. Data gathered from the MIS system has the short time frame for evaluation of two project cycles is significant impact from the project on transition from primary F. Social Dynamics and Governance demonstrated that participation, access to information and likely to be less than sufficient given the long incumbency of to lower secondary school. This result extends not only to satisfaction of beneficiaries, particularly for women and the traditional and religious power structures. Second, both groups the full sample but also to marginalized groups as well as As stated above, a separate module was added to the survey poor is strong and meets project targets. Women participate do not consider that incentives are strong enough to warrant by gender. In addition, no significant differences emerge in instrument addressing key indicators of social dynamics in meetings at a rate of 48 percent and the poor at 60 percent undertaking PNPM approaches to other aspects of village primary and lower secondary enrollment rates.36 This result is and governance. Here we employ a conditional logit model based on project data from the 2007–2009 period. In addition, governance. Communities do not view their participation similar in part to results found in the previous KDP2 evaluation. to identify the impact of PNPM on changes in proportions questions looking at access to information about PNPM and and input into village affairs as having the potential to impact The existing high rate of enrollment in primary and, to some given the binary nature of the indicators and also consider satisfaction of beneficiaries demonstrate rates of 60 percent decision–making. With respect to local government village extent, secondary schools (at approximately 95 percent and a matched household logit model to test robustness. For and 68 percent respectively. When we look at these indicators officials are primarily responsive to regulatory and legal 85 percent respectively) indicate that access may not be a the purposes of this paper, we limit the discussion to the from the standpoint of the larger arena of village affairs requirements, for which most other activities do not include significant barrier in most communities given the relatively following variables: incidence of collective action, communal incorporating all meetings/projects, rates of participation, key PNPM principles such as participation and transparency. low amount of PNPM funds spend on education sub–projects. trust in village government, collective action to petition local access to information and satisfaction are significantly lower Findings from the qualitative component confirm this view.37 government, participation in village meetings, perception in comparison with PNPM at approximately 24 percent and 29 Facilitation effectiveness and “routine” implementation is First, existing primary and lower secondary infrastructure of local government addressing needs of the community percent. This suggests that while the project is successful with a contributing factor. Also from the qualitative component, is typically available in most villages in the sample. Due to and access to information on village development funds. regard to inclusion and governance within PNPM, the factors a contributing factor to the lack of effectiveness in the increased interest in pre–primary education the majority of Questions from the module discuss general village affairs and contributing to PNPM attendance do not spill over into wider social and community capacity built by PNPM in influencing education sub–projects built under PNPM and its predecessor are not specific to PNPM or any particular project aside from village affairs. non–PNPM affairs is the approach to project implementation. KDP have focused on pre–school and kindergarten facilities. questions concerning village government. Results refer to Frequently, the project process and procedures are viewed as Second, poverty was not a significant factor in access to primary Table 9.38 Findings from the qualitative component indicate routine or “mechanistic” in their implementation in the sense and lower secondary schooling. Although approximately that community capacity to influence elite control that procedures are followed in order to satisfy requirements 25 percent of the quantitative sample stated that they had PNPM demonstrates no significant impact on social of decision–making outside the program is limited. rather than to build the capacity of community members. difficulty accessing education, the primary obstacle was dynamics and governance. The primary finding for social The qualitative component offers some insight into the One factor cited is the tendency of “requirement satisfaction” the lack of resources to send children to upper secondary dynamics and governance is that there is no significant reasons behind the lack of spill over for social dynamics among village officials in implementation of other project schooling which is typically located in the district center. pattern of impacts which emerge, either for the full sample and governance. The primary factor cited is continued elite activities that can lead community members in PNPM cases or subgroups including the poor, poor sub–districts and domination of decision–making and control over access to to simply follow PNPM procedures but not embrace the marginalized groups. As shown in Table 9, regression results information and participation. The study found that traditional principles behind them. A second factor, also cited in other 34 These cohorts are children aged 11 and 12 in classes 5 and 6 of the typical from the conditional logit model as well as the matched power structures, both religious and customary have not been studies on PNPM40 is the quality of facilitation, which is six year primary school course. household logit model demonstrate no significant coefficients. impacted significantly by the project. While in many cases, impacted by administrative burden, lack of sufficient training 35 Given that the location of upper secondary schools is typically in district Also included are baseline and endline means for treatment PNPM is able to operate with significantly less influence in and lack of quality candidates. Facilitators frequently either and sometimes sub–district centers, requiring students to live away from home, it is difficult to accurately assess enrollment rates at this level. Doing and control groups which indicate little movement over the comparison with regular village government affairs and other have too many administrative tasks to devote enough time so would have required tracking of individuals which was not feasible period under evaluation. development projects, capacity built within the community to to community empowerment and/or do not have the skills or given the scope and budget of the study. successfully implement PNPM according to community needs training to be effective.41 36 Results for net enrollment rates are only given for the full sample in the and with full participation from different segments of the tables. However, additional estimates on sub–groups also demonstrated lack of impact. These results are available on request. 39 SMERU (2010), p 14–15, 29, 64–67. 37 See SMERU (2011), Section 5.3. 38 Results from regression results not shown in Table 8 pertaining to social 40 See AKATIGA (2010) dynamics and governance are available upon request. 41 SMERU (2010) p 64–67. 19 20 V. DISCUSSION AND CONCLUSIONS 21 22 V. DISCUSSION AND CONCLUSIONS In this section we consider the main findings in the light of and access to credit. Because communities still tend to view the key issues for PNPM going forward presented in Section 4. PNPM as for the wider community, they focus on the same roads, bridges and irrigation that tend to get funded in poor Summary of findings. The findings above indicate that PNPM and remote areas. In contrast to the first case, additional created positive impacts on household welfare, poverty status infrastructure on top of sufficient existing infrastructure is less and access to health services for households in sub–districts likely to have a large economic benefit and thus the selection receiving funds over the period of 2007–2009. Aside from of projects is not aligned with the needs and interests of access to health care, these positive impacts did not extend the poor. This suggests that PNPM could be more effective to marginalized and disadvantaged groups, defined as by targeting block grants for infrastructure in areas where female–headed households and households with heads, who economic impacts are largest while focusing on facilitation has not completed primary education. PNPM did not impact and capacity–building to better target the poor in areas with enrollment rates for primary or lower secondary schools, existing sufficient infrastructure where additional typical including transition rates from primary to lower secondary PNPM infrastructure projects are less likely to have a large school. For social dynamics and governance, there was no economic impact. impact found for the program over a range of indicators, including communal trust, collective action, participation and Impact on marginalized groups. The results above confirm access to information. for PNPM a key finding from the previous evaluation of KDP2 that aside from access to health services, the project did not Extent to which PNPM has sustained impacts on have a significant impact on marginalized groups, particularly household welfare and poverty reduction. PNPM remains with respect to household welfare and poverty reduction. The a cost–effective means of providing needed infrastructure findings from the social dynamics and governance indicators, to raise household consumption and move households out as well as the qualitative component, identify key factors which of poverty. Despite a shorter period of evaluation and the underlie this result. First, given the lack of influence on existing extension of the project to every rural sub–district in the power structures and the frequent lack of effectiveness in country, PNPM has been able to sustain the positive impacts building community capacity due to a routinized approach on household welfare and poverty created in previous phases to project implementation and lack of effective facilitation, of the program under predecessor KDP. While this evaluation the influence of disadvantaged groups on decision–making does not directly address the quality of implementation, is likely to remain marginal. Second, since communities do similar findings in comparison to evaluations of previous not regard PNPM as a poverty reduction program, but rather project phases suggests that the scale up has not adversely in some cases as a counterbalance to household targeted affected project outcomes with respect to household welfare programs, decision–making within PNPM is less likely to be and poverty. In addition, the results suggest that time frame oriented toward the needs of disadvantaged groups. Due to needed to achieve impacts is less than the five year period these factors, the project may not be well–placed to address for the previous evaluation of KDP2. Although the number the problems of disadvantaged groups directly. of project cycles under evaluation for PNPM was limited to 2 in comparison with 3–4 for KDP2, impacts for household Social Dynamics and Governance. PNPM faces significant welfare and poverty status were largely consistent, if smaller challenges in translating social accountability/transparency due to the shorter evaluation period. Moreover, for household the qualitative study, the relative effectiveness of the project gains developed within the program into influence on consumption, the results for the full sample which reflect in poor and remote contexts is partly driven by the level development planning and activities outside the program. The to a greater extent the overall impact of the project were of existing infrastructure and the extent of alignment of lack of significant impacts on social and governance indicators significantly larger under PNPM in comparison with the KDP2 community interests with the needs of the poor. As presented points to the need for a sustained period of facilitation and evaluation where impacts were concentrated among poor above, over 70 percent of project funds go to infrastructure a greater emphasis on the skills and institutions of the households and households in poor sub–districts. projects, primarily roads, bridges and irrigation. In poor and community themselves, to build up community capacity remote areas, where existing infrastructure is likely to be for more effective collective action and demand for better Effectiveness of PNPM in different contexts. A second less developed, these projects tend to have strong positive governance. If facilitation is to continue, quality will need to theme that emerges is the contextual variability of impacts economic benefits (as seen in Torrens (2004) and Dent (2001)) improve in order to have an impact, particularly with respect to on household welfare and poverty. Although impacts were which impact the welfare of the poor; in these cases, the the perception by community members of community–based found for the sample as a whole, they were concentrated interests of the poor in obtaining basic roads, bridges and program such as PNPM as not primarily poverty reducing. In among poor households with the largest gains in the first irrigation are aligned with the wider community. However, in addition, changes in project design should be considered quintile of household per capita consumption, and among areas where existing infrastructure is already well–developed, which to address social accountability constraints both within the poorest quintile of sub–districts. Based on findings from the poor tend to cite other needs, including job/skills training PNPM and outside the project. 23 24 VI. RECOMMENDATIONS AND POLICY IMPLICATIONS 25 26 VI. RECOMMENDATIONS AND POLICY IMPLICATIONS Overall, the results indicate that the primary mechanism for Continued focus on marginalized groups: The project PNPM to create positive benefits for participating communities should determine whether the project is best–placed to is the impact of sub–project infrastructure built through the address the needs of marginalized groups and consider program in reducing poverty and increasing household welfare additional design changes or other development approaches and access to health care, particularly in poor and remote to address their needs. areas; the project has not yet been effective in extending these benefits to disadvantaged groups or in influencing social Renewed focus on strength of participation and inclusion dynamics, governance and decision–making for development of the poor and disadvantaged groups in project activities outside the program. The results highlight some decision–making: To overcome the “routine” approach to considerations going forward for the PNPM–Rural program project implementation that has developed due to scale up and future research: and the long period of implementation in many locations, the project needs a renewed effort to strengthen its core Continued funding for infrastructure with a focus on approach of community engagement in project activities to maintenance and sustainability of infrastructure: PNPM ensure that all groups are included and participate fully in remains an effective means of delivering needed infrastructure decision–making over the project cycle. to rural communities which improves household welfare. The project should continue given the existing infrastructure gap Continued collection of data: Although the expansion of in rural areas. However, these benefits will only be sustained PNPM–Rural to cover all rural sub–districts in the country if the infrastructure is of sufficient quality to continue to necessitates the loss of control areas, the panel nature of be utilized effectively. Future research should focus on the the survey can still be valuable in tracking the progress of quality of maintenance and overall sustainability of use key indicators going forward. Subsequent survey rounds in for infrastructure built by the projects as well as current 2012 and 2014 should be conducted to ensure continued mechanisms and procedures in place to ensure proper examination of project effectiveness. maintenance is conducted. Targeted approach to Block Grant allocation: As noted above, the largest gains are made in poor and remote areas. Block grant amounts should be targeted toward areas with low levels of existing infrastructure in order to maximize household welfare impacts. Additional research is needed to understand the effectiveness of the project in a wider range of contexts (poverty, infrastructure, regional) and implementation procedures (BLM size, length of participation in the project) and consideration given to how to customize the block grant size menu to meet the needs of different local contexts. Strategy to address constraints to stronger downward social accountability from local government: The fact that institutions other than PNPM do not yet emulate the transparency and governance features of the program indicates that a key objective of increased social accountability is not being met. While PNPM is not the sole vehicle nor primarily responsible for changes in the local government environment, it is included as one means to introduce and institute good governance practices in the rural space. Further research on the barriers to adoption of PNPM principles of transparency and accountability and potential design changes to address identified barriers are needed. 27 28 REFERENCES Abadie, Alberto and Guido W. Imbens, 2002, “Simple and Bias—Corrected Matching Estimators for Average Treatment Effects1”, NBER Technical Kano, Shigeki, 2003, “Japanese Wage Curve: A Pseudo Panel Study”, University of Tsukuba, Japan, May 2003. Working Paper, No. 0283, Massachusetts: National Bureau of Economic Research, October 2002. Labonne, Julien and Rob Chase, 2007, “Who’s at the Wheel when Communities Drive Development? The Case of the KALAHI–CIDSS in the Abadie, Alberto, David Drukker, Jane Leber Herr and Guido W. Imbens1, 2001, ”Implementing Matching Estimators for Average Treatment Effects Philippines”, Social Development Papers, Paper No. 107, The World Bank, September 2007. in Stata”, The Stata Journal, Vol. 1, Number 1, pp. 1–18. Lawson, David, Andy McKay and John Okidi, 2003, “Poverty Persistence and Transitions in Uganda: A Combined Qualitative and Quantitative Alatas, Vivi, 2005, “An Evaluation of Kecamatan Development Project”, June, Jakarta: The World Bank, June 2005. Analysis”, Kampala: Economic Policy Research Centre, December 2003. AKATIGA, 2010, “Marginalized Groups in PNPM,” Jakarta, Indonesia: The World Bank, PNPM Support Facility. Lawson, David, Andy McKay and John Okidi, 2003, “Poverty Persistence and Transitions in Uganda: A Combined Qualitative and Quantitative Analysis”, Kampala: Economic Policy Research Centre, December 2003. Becker, Sascha O. and Marco Caliendo, 2007, “mhbounds – Sensitivity Analysis for Average Treatment Effects, German Institute of Economic Research, Working Paper, Berlin: DIW, January 2007. Mansuri, Ghazala and Rao, Vijayendra, 2004. “Community Based (and Driven) Development: A Critical Review”. World Bank Policy Research Working Paper No. 3209. Becker, Sascha O. and Andrea Ichino, 2002, “Estimation of average treatment effects based on propensity scores”, Centre for Employment Studies Working Paper, Laboratorio R. Revelli. McCulloch, Neil. and Baluch, B, 1999. “Distinguishing the Chronically From Transitory Poor– Evidence from Pakistan”, Working Paper No. 97, Institute of Development Studies, University of Sussex, UK. Chantala, Kim, Dan Blanchette and C. M. Suchindran, 2006, “Software to Compute Sampling Weights for Multilevel Analysis” Carolina Population Center, UNC at Chapel Hill, January 2006. McKenzie, David, John Gibson and Steven Stillman, 2006, “How Important is Selection? Experimental Vs Non–experimental Measures of the Income Gains from Migration1”, Policy Research Working Paper Series, No. 3906, The World Bank, May 2006. Chowhan, James and Neil J. Buckley, 2005, “Using Mean Bootstrap Weights in Stata: A BSWREG Revision”, The Research Data Centres Information and Technical Bulletin, Vol. 2 No. 1, pp. 23–37. McLaughlin Kerry, Satu Adam, and Hoppe, Michael 2007, “Kecamatan Development Program Qualitative Impact Evaluation.” Jakarta: The World Bank. Cook, Thomas D., William R. Shadish, Jr., and Vivian C. Wong, 2005, “Within Study Comparisons of Experiments and Non–Experiments: Can they help decide on Evaluation Policy?”, Paper presented at the French Econometric Society Meeting on Program Evaluation, Paris, France, December 2005 . Morgan, Stephen L and David J. Harding, 2006, “Matching Estimators of Causal Effects: From Stratification and Weighting to Practical Data Analysis Routines”, Sociological Methods & Research, Vol. 35, No. 1, 3–60. Deaton, Angus, 2000. The Analysis of Household Surveys: A Microeconometric Approach to Development Policy. Baltimore: The Johns Hopkins University. National Management Consultants 2005, “Kecamatan Development Program Phase Two: Fourth Annual Report 2005” : Jakarta, National Demographic Institute Faculty of Economic University of Indonesia, 2002, “Evaluating KDP Impacts on Community Organization and Household Management Consultant for KDP National Secretariat, Directorate General Community and Rural Development, Ministry of Home Affairs Welfare”, Depok: Faculty of Economics University of Indonesia, July 2002. National Management Consultants 2007, “Kecamatan Development Program Phase Two: Final Report”: Jakarta, National Management Consultant Dehejia, Rajeev H. and Sadek Wahba, 2002, “Propensity Score–Matching Methods for Nonexperimental Causal Studies”, The Review of Economics for KDP National Secretariat, Directorate General Community and Rural Development, Ministry of Home Affairs and Statistics, Vol. 84, No. 1, pp. 151–161, February 2002. Olken, Ben. 2005. “Power Calculations for Community CCT Project”, Jakarta: The World Bank, mimeo. Dent, Geoffrey, 2001 “Ex–Post Evaluation of Kecamatan Development Program (KDP1) Infrastructure Projects.”, Jakarta: The World Bank. Papanek, Gustav F., 2007, “Employment and the PNPM–RURAL Program”, Jakarta: The World Bank, December 2007. Dragoset, Lisa and Fields, Gary, 2006 “US Earnings Mobility: Comparing Survey–Based and Administrative–Based Estimates.” Society for the Study of Economic Inequality Working Paper, WP2006–55, ECINEQ. Pritchett, Lant, Asep Suryahadi, Sudarno Sumarto, 2000, “Quantifying Vulnerability to Poverty: A Proposed Measure, with Application to Indonesia”, SMERU Working Paper, May 2002. Fields, Gary et. Al., 2001 “Household Income Dynamics: A Four Country Study.” Paper prepared for the NBER Conference on Labor and the Global Economy, Cambridge, MA, 2001. Raudenbush et. al. 2006. “Optimal Design for Logitudinal and Multilevel Research.”, National Institutes of Health, mimeo. Handa, Sudhanshu and John A. Maluccio, 2007, “Matching the gold standard: Evidence from a social experiment in Nicaragua”, Population and Rubin, Donald B. and Neal Thomas, 2000, “Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates”, Journal Health InfoShare, No. 07–100, August 2007. of the American Statistical Association, Vol. 95, No. 450, pp. 573–585, June 2000. Heckman, James J., Hidehiko Ichimura and Petra Todd, 1998, “Matching As An Econometric Evaluation Estimator”, Review of Economic Studies, Rubin, Donald B., 1979, “Using Multivariate Matched Sampling and Regression Adjustment to Control Bias in Observational Studies”, Journal of the Volume: 65, Issue: 2, April, pp. 261–94. American Statistical Association, Vol. 74, No. 366, pp. 318–328, June 1979. Institute for Social and Economic Research, Education and Information (LP3ES), 2008, “PNPM–RURAL Component Qualitative Baseline Report”, Rosenbaum P. R and Rubin, Donald, 1983, “The Central Role of the Propensity Score in Observational Studies for Causal Effects,” Biometrika, 70, pp Jakarta: The World Bank. 41–55. Jalan, Jyotsna and Martin Ravallion, 2001, “Does Piped Water Reduce Children’s Health Improves on Average as A Result of Policy Diarrhea for SMERU, 2010, “A Qualitative Study on the Impact of the 2010 PNPM–Rural in East Java, West Sumatra, and Southeast Sulawesi,” Jakarta, Indonesia: Children Interventions that Expand in Rural India?”, Policy Research Working Paper, The World Bank Development Research Group Poverty, August The World Bank, PNPM Support Facility. 2001. 29 30 Smith, Jeffrey and Todd, Petra, 2003. “Does Matching Overcome Lalonde’s Critique of Nonexperimental Estimators?,” University of Western Ontario, CIBC Human Capital and Productivity Project Working Papers No. 20035, University of Western Ontario, CIBC Human Capital and Productivity Project. Suryadarma, Daniel, Asep Suryahadi and Sudarno Sumarto, 2007, “Reducing Unemployment in Indonesia: Results from a Growth–Employment Elasticity Model”, SMERU Working Paper, Jakarta: SMERU Research Institute. January 2007. Suryadarma, Daniel Asep Suryahadi and Sudarno Sumarto, 2005, “The Measurement and Trends of Unemployment in Indonesia: The Issue of Discouraged Workers”, SMERU Working Paper, Jakarta: SMERU Research Institute, July 2005. Suryahadi, Asep and Sudarno Sumarto, 1999, “Update on the Impact on the Indonesian Crisis on the Consumption Expenditure and Poverty Incidence: Results from the December 1998 Round of 100 Village Survey”, SMERU Working Paper, Jakarta: Social Monitoring and Early Response Unit, SMERU, August 1999. Torrens, Anthony, 2005, “Economic Impact Analysis of Kecamatan Development Program Infrastructure Projects”, Jakarta: The World Bank. Voss, John, 2008. “Impact Evaluation of the Second Phase of the Kecamatan Development Program”, Jakarta, The World Bank. Wassenich, Paul and Katherine Whiteside. 2003. “CDD Impact Assessment Study: Optimizing Evaluation Design Under Constraints”, World Bank CDD and Social Capital Anchor, Mimeo. Wiehler, Stephan A., 2006, “Bias Reducing Estimation of Treatment Effects in the Presence of Partially Distorted Data”, Berlin: Swiss Institute for International Economics and Applied Research, University of St. Gallen, Switzer–land, January 2006. World Bank, 2005, “CDD and Social Capital Impact Designing a Baseline Survey in the Philippines”, Washington, D.C.: The International Bank for Reconstruction and Development, The World Bank, May 2005. 31 32 TABLE 3: TABLE 4: CHANGE IN LOGGED REAL PER CAPITA CONSUMPTION HOUSEHOLDS MOVING OUT OF POVERTY Sample First Differences Household Matched Comparison Sample Multinomial Logit Model Household Matched Comparison of Means   of Means Model     Model Coefficient Std. Error Obs Coefficient Std. Error Obs Coefficient Std. Error Obs Coefficient Std. Error Obs Full Sample   0.091** 0.026 6143 0.053** 0.016 6142 Full Sample Full 0.021* 0.013 6143 0.079* 0.043 532 Predicted Quintile 1 0.118** 0.048 1229 0.112** 0.030 1227 Kecamatan Quintile 1 0.032* 0.2 1208 0.167** 0.083 172 Consumption Poverty Score   Quintile 2 0.084* 0.051 1229 0.039 0.030 1226   Quintile 2 0.027 0.01992 1230 –0.0093 0.0116 137 Quintile 3 0.156** 0.046 1228 0.086** 0.033 1229 Quintile 3 0.0001 0.0001 1229 –0.0059 –0.012 104 Quintile 4 0.015 0.046 1229 0.008 0.034 1228 Quintile 4 0.0315 0.022 1229 0.029 0.096 93 Quintile 5 0.056 0.057 1228 0.021 0.033 1226 Quintile 5 0.006 0.213 1246 0.225** 0.096 90 Kecamatan Quintile 1 0.127** 0.066 1208 0.095** 0.034 1206 Disadvantaged No Primary 0.0139 0.0165 1907 0.109 0.064 248 Poverty Score Groups   Quintile 2 0.070 0.069 1230 0.055* 0.029 1226   Primary 0.0038 0.01748 1925 0.033 0.076 189 Quintile 3 0.073 0.083 1229 –0.023 0.038 1228 Female Head 0.022 0.026 873 0.002 0.108 69 Quintile 4 0.124* 0.073 1229 0.134** 0.035 1228 Quintile 5 0.014 0.064 1246 0.020 0.030 1244 Disadvantaged No Primary –0.012 0.025 6143 –0.025 0.037 6142 Groups Primary –0.028 0.028 6143 0.027 0.033 6142 Female Head –0.096 0.157 6143 0.019 0.027 6142 Note: *denotes significance at the 10 percent level, **at the 5 percent level. Coefficients represent the percentage point difference Note: *denotes significance at the 10 percent level, **at the 5 percent level. Coefficients represent the percentage point difference in the log growth rate of real per capita consumption between PNPM and control households. The first set of estimates use a first in the percentage of households moving out of poverty between PNPM and control households. The first set of estimates is differencing approach on the full sample, including regression adjustment The second set of estimates uses an Epanechnikov kernel marginal effects calculated at the mean derived via a multinomial logit model on the full sample. The second set of estimates uses to create a sample matched at the household level and conduct comparison of means tests. an Epanechnikov kernel to create a sample matched at the household level and conduct conditional comparison of means tests. 33 34 TABLE 5: TABLE 6: HOUSEHOLDS MOVING INTO POVERTY CHANGE IN HOUSEHOLD ACCESS TO OUTPATIENT CARE Sample Multinomial Logit Model Household Matched Comparison of Means Sample Multinomial Logit Model Household Matched Comparison of   Model Means Model Coefficient Std. Error Obs Coefficient Std. Error Obs Coefficient Std. Error Obs Coefficient Std. Error Obs Full Sample Full 0.0014 0.0088 6143 0.009 0.008 532 Full Sample   0.051** 0.0157 4811 0.045** 0.017 5451 Kecamatan Quintile 1 0.0067 0.2123 1208 0.069 0.05 172 Predicted Quintile 1 0.062** 0.028 2483 0.057** 0.028 1562 Poverty Score Consumption   Quintile 2 –0.03 0.02 1230 –0.023 0.015 137   Quintile 2 0.002 0.021 2209 0.044 0.034 1322 Quintile 3 0.0003 0.0002 1229 –0.005 0.019 104 Quintile 3 0.021 0.023 1804 0.03 0.024 1064 Quintile 4 0.009 0.022 1229 0.006 0.019 93 Quintile 4 0.034 0.023 1750 0.043 0.033 905 Quintile 5 0.0001 0.0005 1246 0.007 0.014 90 Quintile 5 0.024 0.026 1307 –0.005 0.036 698 Disadvantaged No Primary –0.005 0.015 1907 0.012 0.019 248 Kecamatan Poverty Quintile 1 –0.032 0.028 1871 0.046 0.039 1115 Groups Score   Primary 0.023 0.016 1925 0.002 0.015 189   Quintile 2 0.025 0.021 1939 0.053* 0.029 1057 Female Head 0.003 0.028 873 0.021 0.014 69 Quintile 3 –0.004 0.032 1884 0.013 0.037 1003 Quintile 4 0.02 0.028 1953 0.029 0.038 1196 Quintile 5 0.011 0.027 1906 0.22** 0.03 1070 Disadvantaged No Primary 0.043** 0.021 3152 0.075** 0.027 1755 Groups   Primary 0.0006 0.02 3060 0.034 0.034 1679 Female Head –0.065 0.049 1036 –0.053 0.047 570 Note: *denotes significance at the 10 percent level, **at the 5 percent level. Coefficients represent the percentage point difference Note: *denotes significance at the 10 percent level, **at the 5 percent level. Coefficients represent the percentage point difference in the percentage of individuals newly seeking outpatient care in 2010 (after not seeking outpatient care in 2007) between PNPM in the percentage of households moving out of poverty between PNPM and control households. The first set of estimates is and control households. The first set of estimates is marginal effects calculated at the mean derived via a multinomial logit model marginal effects calculated at the mean derived via a multinomial logit model on the full sample. The second set of estimates uses on the full sample. The second set of estimates uses an Epanechnikov kernel to create a sample matched at the household level and an Epanechnikov kernel to create a sample matched at the household level and conduct conditional comparison of means tests. conduct conditional comparison of means tests. 35 36 TABLE 7: TABLE 8: CHANGE IN TRANSITION RATE FROM PRIMARY TO LOWER CHANGE IN EMPLOYMENT STATUS SECONDARY SCHOOL Sample Multinomial Logit Model Household Matched Comparison of Means  Sample Multinomial Logit Model Household Matched Comparison of Means Model   Model Coefficient Std. Error Obs Coefficient Std. Error Obs Coefficient Std. Error Obs Coefficient Std. Error Obs Full Sample   0.031 0.021 1042 0.046 0.054 1038 Full Sample .0135** 0.006 5241 0.017 0.011 4238 Predicted Quintile 1 –0.002 0.015 373 0.031 0.046 362 Consumption Note: *denotes significance at the 10 percent level, **at the 5 percent level. Coefficients represent the percentage point difference in   Quintile 2 0.012 0.018 280 –0.004 0.037 261 the percentage of individuals employed in 2010 (after being unemployed in 2007) between PNPM and control households. The first set of estimates is marginal effects calculated at the mean derived via a multinomial logit model on the full sample. The second set Quintile 3 0.035 0.033 229 0.013 0.028 220 of estimates uses an Epanechnikov kernel to create a sample matched at the household level and conduct conditional comparison Quintile 4 0.01 0.008 142 0.021 0.019 135 of means tests. Quintile 5 0.082 0.056 114 0.037 0.052 101 Kecamatan Poverty Score Quintile 1 0.023 0.017 185 0.028 0.036 183 TABLE 9:   Quintile 2 –0.018 0.025 227 0.0005 0.029 218 CHANGE IN SOCIAL CAPITAL AND GOVERNANCE INDICATORS Quintile 3 0.043 0.039 186 –0.034 0.018 183 Quintile 4 0.027 0.018 232 0.013 0.016 225 Sample Conditional Logit Model Household Matched Comparison of Means Model Quintile 5 0.005 0.011 222 0.023 0.019 211 Coefficient Std. Error Obs Coefficient Std. Error Obs Disadvantaged No Primary 0.052 0.041 340 0.045 0.037 321 Groups Incidence of Collective 0.011 0.015 6137 0.009 0.017 5982 Primary Female 0.011 0.023 331 0.009 0.011 308 Action Head^ Trust in Village Government –0.005 0.009 6137 –0.003 0.011 5843 Enrollment Rates Petitioning Local 0.013 0.024 6137 0.018 0.026 6012 Full Sample Primary School 0.003 0.008 3589 –0.008 0.013 2973 Government   Lower Secondary 0.034 0.027 1216 0.028 0.021 1008 Participation in Village 0.008 0.015 6137 0.01 0.021 5941 School Meetings Perception of Local 0.043 0.034 6137 0.039 0.029 6041 Government Addressing Community Needs Access to Information 0.056 0.038 6137 0.041 0.034 6019 Concerning Development Note: *denotes significance at the 10 percent level, **at the 5 percent level. Coefficients represent the percentage point difference Funds in the percentage of children enrollment in primary school and lower secondary school in both 2007 and 2010 respectively, for the 2007 11–12 year old age cohort between PNPM and control households. The first set of estimates is marginal effects calculated at the mean derived via a multinomial logit model on the full sample. The second set of estimates uses an Epanechnikov kernel to Note: all are full sample. Coefficients represent the percentage point difference in change in each binary outcome variable between create a sample matched at the household level and conduct conditional comparison of means tests. PNPM and control households. The first set of estimate utilizes a conditional logit model while the second a household matched ^: Sample size too small conditional comparison of means model. 37 38 ANNEXES 39 40 Within each EA, eleven households from the sixteen were Lacking randomization for PNPM participation, a common ANNEX 1: METHODOLOGY sampled based on their household identifying number in the approach is to estimate the probability of D using a propensity 2002 SUSENAS. The first eleven were initially targeted and score matching approach to choose a comparable control surveyed unless the household head in 2007 had left the village, group by conditioning selection on a set of observable could not be located or refused to be interviewed, in which characteristics. A set of observable covariates X are selected case the survey teams would target the next household from such that the distribution of all covariates in X is the same A.1.1 Sampling the list of sixteen. In cases of households splitting or moving between selected treatment and control groups, satisfying within the village, the household of the household head from the condition that conditional on X, outcomes measures for Kecamatan Level. The sub–district sampling frame is Table A1.1: Distribution of Matched the 2007 SUSENAS was considered to be the 2010 location. the treatment and control groups are independent of the comprised of all rural sub–districts participating in PNPM Kecamatan by Province Since the EA is a geographical designation, it is not expected treatment assignment D: in 2007 as candidates for the treatment group and all that ordering of the households by household identifier sub–districts not participating PNPM until late 2009 and number is correlated with outcome variables. Therefore, the (3) Pr (D = 1/X, ycij) = Pr (D = 1/X) PNPM–like programs during the period of 2002–2007 for the Bali 10 sampling process at the EA level is not likely to bias results. control group. The PNPM–like programs were identified based A further source of potential bias is rates of attrition being As Rosenbaum and Rubin (2003) show, if the true propensity on their similarity in approach with regard to community Banten 14 correlated with the treatment variable: only households not score Pr (D = 1/X) is known for each observation, the condition organization, community–led decision–making and amount migrating were included in the sample as resource constraints in (3) is satisfied. In practice, we must estimate Pr (D = 1/X). The disbursed per village or sub–district. Five programs met The Special District of limited following households outside the village. However, standard method is to regress the selected covariates on the the criteria: Yogyakarta 2 the number of households which could not be tracked was treatment indicator variable using a standard probit or logit only 9 percent of the overall sample and comparison of model and then use a matching process to select observations • Community Empowerment for Rural Development (Asian Development Bank) Jambi 15 means tests demonstrate no significant differences in attrition rates between treatment and control households and the for the treatment and control groups that best satisfy the condition in (3). • Community and Local Governance Support Project (Asian Development Bank) West Java 34 percentage of.43 Within households, households (but not individuals) were tracked if they remained in SEDAP provinces Kecamatan level matching. Since the treatment for PNPM • Urban Poverty Project (World Bank) or if they relocated to Jakarta. was assigned at the sub–district level and the sampling • Program Pengembangan Prasarana Desa (Japan Bank for International Cooperation) Central Java 34 Sampling weights are composite two–stage weights calculated strategy dictated choosing households within kecamatan, we first conducted propensity score matching at the sub–district • Australian Community Development and Civil Society Strengthening Scheme (AUSAID) East Java 64 using PWIGLS in STATA and take into account sampling at both the sub–district and EA level. to level to select the overall sample. A group of sixty observable covariates were selected from the 2005 PODES census of South Kalimantan 27 villages and 2002 SUSENAS conducted by BPS. The covariates In addition, provinces under sampled or not sampled in the Lampung 28 A.1.2 Identification consist of sub–district level indicators on infrastructure, 2002 SUSENAS survey were not included in the sub–districts demography, economic and geographic conditions, and sampling frame, including Maluku, North Maluku, Papua The identification problem in program evaluation. The poverty and education and health index variables constructed West Nusa Tenggara 4 and Aceh. Due to resource constraints, some provinces with evaluation seeks to identify the impact of PNPM on the from a poverty mapping exercise by BAPPENAS in 2005 as sub–districts in remote areas such as West Kalimantan were changes in a set of outcome indicators. Let yij be the change in part PNPM Kecamatan selection. For the sample of remaining Riau 21 excluded when it was determined that only a small number the outcome indicator of interest for household i in sub–district sub–districts surveyed in the 2002 SUSENAS (see Section of sub–districts had the chance to be included in the final j. If we could observe changes in the treated and untreated A.1 above), we then regress the covariates on the treatment South Sulawesi 61 sample. Selection was conducted using the propensity score states we could simply compare the difference in the mean indicator using a logit model. From this regression, we then matching methodology described below resulting in 300 total change for both states to estimate the impact of the program: predict the probability of participation in PNPM, an estimate Southeast Sulawesi 12 sub–districts comprised of 150 pairs of matched treatment of Pr (D =1/X). Due to the limited number of sub–districts and control sub–districts. In order to ensure the best possible (1) E (yij / D = 1) = E (yij / D = 1) – E (yij / D = 0) available for the control group and the need to meet sample results for the matching procedure, the sample was not North Sulawesi 13 size requirements, we conducted the matching using the stratified by region; matched pairs were selected from the where D = 1 if the treatment is received and D = 0 if the nearest neighbor without replacement method to select 150 entire pool of sub–districts in the sampling frame.42 Ultimately, West Sumatra 31 treatment is not received. The standard evaluation problem is pairs of matched treatment and control sub–district. Use of 17 provinces were included in the sample: that E (yij/ D = 0) is not observed. Instead, we seek to construct this method can be problematic in that poor matches can South Sumatra 21 the counterfactual state of what would have happened in results. However, as Rubin (2000) notes, this is a not problem Household Level. Within each sub–district, two enumeration PNPM locations had the project not occurred. If we can find a as long as matched covariates have equivalent or balanced areas (EA) were selected randomly for the household level North Sumatra 65 control group of sub–districts ycj with identical characteristics distribution between treatment and control groups. All sample from a sampling frame comprised of households to our treatment group yj, where c indicates the control covariates were tested using simple comparison means tests surveyed in the 2002 SUSENAS core module. EA’s are a group, we can replicate the unobserved state E (yij / D = 0) by and Kolmogorov–Smirnov and Hotelling tests of equality of sampling unit of sixteen households used by BPS in selecting substituting E (ycij / D = 0) so that: distributions and found no significant differences for all tests the sample for SUSENAS and other surveys. Because EA’s are among all covariates indicating that the sub–district sample is selected directly from the district level, sub–districts can differ (2) E (yij / D = 0) = E (ycij / D = 0) well–balanced and satisfies the condition in (3) that treatment in the number of households sampled in SUSENAS surveys assignment is independent of outcomes conditioned on 42 The limited number of kecamatan available for matching due to the exclusion of participation in other PNPM–like programs rendered attempts although there is a minimum of two for the 2002 SUSENAS. In practice, finding a control group with identical properties selected covariates. The results of these tests are shown in to stratify matching within districts and/or provinces infeasible as the In cases where there were more than EA’s sampled, two EA’s is impossible. A standard solution would be to randomize Table A1.2 below. In addition, tests for equality of distribution quality of matching on covariates would not be strong enough to credibly were selected randomly. In some cases, due to problems of assignment of D, which would ensure that (2) is satisfied given for the change in real per capita consumption and poverty claim balance between the treatment and control groups. Although there remoteness or difficulty in access, EA’s were replaced with the adequate sample size. incidence were conducted using the 2002 SUSENAS data and are concerns with respect to covariates not included in the matching with respect to differences across districts and provinces, this is mitigated to approval of the World Bank evaluation team. the 2007 SEDAP baseline survey data and demonstrated a some extent by the fact that fixed covariates will be addressed through the similar lack of significant difference, indicating that there was difference–in–difference approach. 43 Results available upon request. 41 42 no significant difference in the overall time trend between model. Balancing tests confirm that balance was achieved for 2002 and 2007 for the treatment and control group, lending all covariates with a large range of common support producing Table A1.2: Balancing Tests for Covariates further support to the balancing characteristics of the sample. a Household–level matched sample of 6,142 households.46 Satisfying the condition in (3) indicates that our matching A.1.3 Estimation Variable name Comparison of Kolmogorov– Share of villages with 0.552 0.787 was successful for the covariates selected but unfortunately Means Tests Smirnov test Scouting Movement it is unlikely that the covariates we selected are the only The analysis utilizes several different models to conduct for equality of factors that are correlated with both outcome indicators and comparison of means teats on samples and samples stratified distribution Share of villages with 0.104 0.974 treatment assignment. There are likely unobserved factors by predicted 2007 per capita consumption quintiles and 2005 youth clubs that are not balanced between our selected treatment and sub–district poverty score quintiles, household head education p–value p–value control sub–district that could bias results. These can be and household head gender. All estimates use standard errors Share of elderly 0.915 0.974 classified into two categories. The first are time invariant. that take into account clustering at the sub–district level and Poverty score (BAPPENAS 0.160 0.120 households Because we are using panel data, these fixed factors will be sample weights constructed as described in section A.1.1 Index) eliminated using the difference–in–differences approach for except when otherwise noted. The following section describes Share of land with access 0.500 0.653 estimation. The second category, unobserved factors that the models used for each indicator. Education and health score 0.291 0.181 to a main road vary over time are the most difficult to resolve as they cannot (BAPPENAS Index) be addressed directly. However, the literature comparing experimental and non–experimental evaluations emphasizes Share of villages with 0.497 0.942 that non–experiments using approaches, such as propensity Number of villages 0.692 0.513 motorcycles score matching perform better when three criteria are met44: 45 Covariates include: housing conditions, access to electricity, age of the household head, gender of the household head, agricultural occupation Total number of 0.958 0.583 Share of villages with bus 0.985 0.899 • There is a rich set of data available from which to choose observed covariates. of household head, household size and province dummies. 46 Results of the logit regression and balancing tests are available upon households terminal • The request. treatment and comparison groups are sampled using Total surface area 0.608 0.029 Share of villages with 0.713 0.787 the same instruments. shopping cluster • The treatment and comparison groups come from similar geographic areas. Share of villages with self– supporting 0.454 0.446 Share of villages with 0.872 0.324 permanent market The design will meet two of these criteria: both treatment Share of coastal Area 0.835 0.942 and comparison groups will be sampled with the same Share of villages with 0.882 0.583 instruments, and these instruments, the PODES census of Share of hill or mountain 0.647 0.653 access to small business villages and the SUSENAS household survey, provide a rich area loans set of variables on which to condition. Geographic proximity is a criteria unlikely to be met by the research design, but it is Share of poor households 0.158 0.787 Share of villages with 0.421 0.899 expected that this will be mitigated to some extent through access to credit facilities the use of the difference–in–differences matching estimator Share of income from 0.174 0.993 to correct for any unobservable factors that are time–invariant. agriculture Share of villages with 0.252 0.942 As Smith and Todd (2005) demonstrate, this difference–in– farming credits differences matching estimator is the least biased estimator Share of income from 0.748 0.993 in studies comparing the effectiveness of different estimators manufacturing Share of villages with 0.724 0.271 at replicating randomized results. As noted above, the lack of village head university– significant difference in time trend between 2002 and 2007 Share of income from 0.079 0.583 educated also provide support for the success of the matching with services respect to factors impacting key indicators. Share of households with 0.246 0.383 Share of households with 0.146 0.146 fixed phone line Household level matching. The sampling strategy electricity necessitates that we select households from within each Share of land available for 0.321 0.324 selected sub–districts or the final household level sample. Share of households using 0.707 0.974 agriculture in Sub–district Despite households from the same sub–districts experiencing firewood the same sub–district level conditions in terms of the economic, Number of primary 0.974 0.053 social and other environments, significant heterogeneity can Share of households with 0.859 0.583 schools per household still exist across households within sub–districts. To correct for clean drinking water this problem we conduct a second household level matching Number of middle schools 0.147 0.021 process using the full sample of households and a propensity Share of households with 0.440 0.583 per household score matching approach. In choosing covariates, we select clean washing water household and individual characteristics from the 2007 SEDAP Number of high schools 0.572 0.324 survey.45 We then estimate the propensity scores using a logit Share of villages with 0.484 0.383 per household PDAM 44 Smith and Todd (2003), Diamond and Sekhon (2005) and others. 43 44 Real Per Capita Consumption. Comparison of means tests are used to produce estimates for the full and matched Table A1.3: Table of Means for Indicators at Baseline household samples. The first is a simple comparison of means tests on the full sample using regression adjustment following Rubin (2000) and Heckman (1998). Covariates from the Number of doctors per 0.528 0.324 Percent of adult females 0.472 0.446 Variable Name Mean in 2007 sub–district level matching process are simply included with capita with primary education the treatment indicator variable in an OLS regression on the Control Treatment outcome indicator in the following specification: Number of commercial 0.537 0.181 Percent of adult males 0.395 0.115 banks per capita with secondary education Per Capita Consumption (Rp) 365426 331898 (1) Dyij = aC + dTij + bXj + uij Number of BPR banks per 0.999 0.583 Percent of adult females 0.896 0.223 Poverty Rate– BPS 12.7 12.9 Where Dy is the change in real per capita consumption for capita with secondary education household i in sub–district j, d and b are coefficients to be Access to Outpatient Care 37.3 35.1 estimated, C is a constant, T is the treatment effect, X contains Number of savings 0.957 0.053 Percent of adult males 0.367 0.513 the covariates used in the sub–district level matching and cooperative per capita with university education Enrollment Rate– Primary 96.4 95.2 u is the usual error term. As Rubin (2000) shows, regression adjustment using this method can lead to significant bias Number of hospitals per 0.448 0.446 Percent of adult females 0.769 0.446 Enrollment Rate– Secondary 80.8 77.1 reduction in comparison with un–adjusted models. capita with university education Unemployment Rate 6.6 6.1 Second, to address bias generated through heterogeneity of Number of puskesmas per 0.696 0.383 Percent children 0.453 0.271 factors at the household level, we also provide a treatment capita Unemployment Rate with 8.2 7.6 effect through comparison of means estimates using an Percent adults 0.491 0.383 Discouraged Workers Epanechnikov kernel matching procedure on the propensity Number of pustus per 0.863 0.271 scores generated in the household level matching process capita Percent elderly 0.776 0.383 Incidence of Collective Action 72.9 75.2 to estimate treatment effects. In kernel matching, for each treatment household the control is constructed from a Number of private doctors 0.267 0.115 Trust in Village Government 72.5 73.2 weighted sample of control households so that control per capita households with the closest propensity score to the treatment Petitioning Local Government 28.6 34.5 household are given greatest weight. Bootstrapped standard Number of pharmacies per 0.792 0.583 Note: Results show p–values for comparison of means tests errors are calculated for all estimates using a set of 100 capita and Kolmogorov–Smirnov tests for equality of distribution Participation in Village Meetings 78.1 73.9 replications. The matched panel sample is also used to between treatment and control groups on the 2005 PODES generate estimates for the indicators discussed below again Number of midwives per 0.877 0.899 and 2002 SUSENAS covariates. For all covariates, there are Access to Information Concerning 14.2 14.8 using simple comparison of means tests. capita no significant differences between treatment and control Development Funds sub–district at the 10 percent level or less. Measurement Error in Real Per Capita Consumption. Amount of expenditure on 0.834 0.223 Measurement error is a concern when using consumption development per capita as a measure of household welfare, even more so given the two–period panel. Since consumption is a dependent variable Amount of village 0.887 0.446 for the analysis, the impact of measurement error is to decrease government income per the precision of estimates but does not bias results, assuming capita the measurement error is not systematically correlated with the treatment effect. Given that SEDAP survey methodology Average number of years 0.675 0.653 was used in both surveys, this is not likely to be case. However, of education household a problem arises when attempting to measure effects by head quintiles using 2007 per capita consumption as the baseline. Since mismeasurement of consumption can place households Average number of years 0.535 0.446 in quintiles that do not represent their true consumption, of education spouse the resulting samples for each quintile can generate biased estimates of the true population quintiles. This problem is Percent of adult males 0.898 0.721 particularly acute in the first and fifth quintiles as under or over with no schooling measured households whose true consumption might place them in the middle of the consumption distribution populate Percent of adult females 0.761 0.653 the tails and push truly poor or truly wealthy households out of with no schooling the sample 1st and 5th quintiles. In this situation, the literature on income and consumption mobility has shown a tendency Percent of adult males 0.377 0.942 of convergence toward the mean with households in low with primary education quintiles demonstrating large gains with small or negative gains for wealthy households (see Dragoset and Fields (2006) and Fields et. al. (2001) for a thorough review). 45 46 To avoid this problem, we construct measures of household cases: poor, transitory, not poor. However, given that as a result Employment. Similar to the above individuals are placed in Table A1.4: Rural Provincial Poverty Lines Used to As- welfare that are not directly correlated with the baseline 2007 of the intent of the matching process poverty rates are nearly one of four categories: real per capita consumption but are generally correlated identical in 2007, breaking down the transitory group into sign Poverty Status with household welfare. The first measure is the 2005 movement into and out of poverty is of greater interest. In this • Unemployed both years BAPPENAS sub–district poverty score that utilizes a range of case, Lawson, McKay and Okidi (2003) in a study on changes • Employed in both years demographic, education, health care and poverty indicators in poverty status in Uganda argue that a multinomial logit • Unemployed in 2007; employed in 2010 Province 2007 2010 to construct a poverty score index at the sub–district level. Second, we construct a predicted consumption measure approach is more appropriate when considering components of poverty transition. We follow that approach and generate • Employed in 2007; unemployed in 2010 Bali 147 963 188,071 using household level variables from the SUSENAS with the treatment effects for the probability of moving out and School Transition. Children aged 11–12 in 2007 are placed in following specification using OLS: moving into poverty using the following specification: one of the following categories: Banten 140 885 188,741 (2) Dyij = aC + dHHij + bXij +gPRij + uij (3) DPSij = aC + dTij + bXij +gPRij + uij • Not in school in both years The special district of Where y is 2007 per capita consumption for household i in Where DPS is the change in poverty status according to the • In primary school in 2007 but not in lower secondary school in 2010 Yogyakarta 156 349 195,406 sub–district j; d, a, g and X are coefficients to be estimated; C is a constant; HH is a matrix of household level variables; X is a four categories listed above for household in sub–district j; d, a, g, and b are coefficients to be estimated; C is a constant; T • In primary school in 2007 and in lower secondary school in 2010 Jambi 152 019 193,834 matrix of sub–district covariates; PR is a vector of province level is the treatment effect; X is a list of household level control dummies and u is the usual error term.47 Consumption is then variables;48 PR is a vector of province level dummies and u is Treatment effects are estimated as with change in poverty West Java 144 204 185,335 predicted using the estimated coefficients for each household the usual error term. Marginal effects at the mean are used to status using both multinomial logit and conditional and then used to create a set of 2007 predicted per capita calculate the treatment effect as the change in probability of comparison of means models. The specification is similar to (3) Central Java 140 803 179,982 consumption quintiles. The threat of bias in measuring effects being included in a particular category due to participation in with the dependent variable the change in use of outpatient by quintile is avoided because both predicted consumption the program. care, employment status and school transition using the East Java 140 322 185,879 and sub–district poverty index are not systematically categories listed above. We also estimate a conditional correlated with the measurement error from the baseline 2007 The multinomial logit model can be quite restrictive as it comparison of means model using the household matched South Kalimantan 144 647 196,753 per capita consumption. makes the somewhat strong “independence of irrelevant sample as discussed for Changes in Poverty Status above. alternatives” assumption. This implies that introducing other Riau Islands 145 634 265,258 Change in poverty status: In addition to estimating impact alternative categories or reducing the number of categories on continuous changes in economic welfare discussed above, would not change estimated probabilities due to a lack of Lampung 130 867 189,954 we also consider discrete changes for households with regard correlation in the error term across categories. To mitigate to poverty lines using a multinomial logit on the full sample potential problems with this approach and test robustness, we West Nusa Tenggara 194 019 176,283 and a conditional comparison of means model using the also employ a conditional comparison of means model using household–level matched sample. the matched household sample. To estimate the treatment South Sulawesi 115 788 151,879 effect for moving out of poverty, we consider households We begin by assigning households into poverty status that were poor in 2007 and then compare the probability of Southeast Sulawesi 127 197 161,451 categories of “poor” or “not poor” for both 2007 and 2010 using escaping poverty between PNPM and control households the BPS provincial rural poverty lines for the respective years. using the Household–level matched sample. Similarly, for North Sulawesi 149 440 188,096 Table A1.4 shows the poverty lines used in current Rupiah. moving into poverty, we restrict the sample to households not Next, household are placed into one of four categories based poor in 2007 and estimate the probability of becoming poor West Sumatera 163 301 214,458 on their poverty status for PNPM and control households. South Sumatera 161 205 198,572 • Stayed poor Changes in Use of Outpatient Services, Employment and • Never poor Transition from Primary to Lower Secondary Schools. North Sumatera 154 827 201,810 • Moving out of poverty (Poor in 2007, Not Poor in 2010) Outpatient Services. Similar to the approach used for changes • Moving into poverty (Not Poor in 2007, Poor in 2010) in poverty status above, we consider the impact of PNPM on household heads that did not seek outpatient care in 2010. Source: BPS; all figures indicate per capita current Rupiah We then use a multinomial logit model to measure treatment The sample is restricted to household heads sick in both 2007 Note: All figures are in current Rupiah. Source: World Bank, impact on the full household sample. The multinomial logit and 2010 which are then assigned to categories based on BPS. model has the advantage of being able to consider multiple upon the pattern of use of outpatient care. cases for a single categorical variable when there is no logical or meaningful ordering of the categories. Previous attempts • No use of outpatient services in either year such as McCulloch and Baluch (1999) have argued an ordered • Use of outpatient services in both years logit approach using three categories where the moving into and out of cases listed above are combined into one “transitory” • No use of outpatient services in 2007; use of outpatient services in 2010 • Use of outpatient services in 2007; no use of outpatient services in 2010 47 Regression results are available on request. Household level variables include: housing conditions, access to electricity, ownership of durable goods, age of the household head, gender of the household head, agricultural occupation of household head, ownership of farmland, 48 Variables include: age, gender and years of education of the household household size and province dummies head, access to electricity, housing conditions, land ownership, ownership of durable assets. 47 48 ANNEX 2: A NOTE ON POWER CALCULATIONS List of Parameters for Cluster Assigned Treatment with Repeated Measuresclusters will be determined through the power calculations. In This note describes power calculations conducted for the expenditure that are so small as to be somewhat negligible, addition, the frequency, duration and number of measurements are easily initial KDP impact evaluation. However, given that sample and would require a far greater number of sub–districts to be defined. size is determined by the inherent statistical properties of sampled in the survey. the indicators of interest in the population, the results are applicable to the PNPM evaluation described above. Unlike a typical single–outcome measurement study, the Parameter Symbol Value Source research design employs a panel dataset with sampling at Non–experimental Research Design baseline (2002 SUSENAS) and follow up (2007 WB implemented Cluster Size n Determined w/Calculations The project will utilize a difference–in–differences matching survey). Introducing repeated measures of the same estimator to determine program impact. The 2002 SUSENAS, household necessitates accounting for correlation over time # of Clusters J Determined w/Calculations approximately 200,000 observations, will be used as the in the calculations. Simply using the difference in household sampling frame to select treatment and comparison groups expenditure per capita as an indicator and conducting the Intra–class correlation p .15 SUSENAS Panel from PNPM and non–KDP households using matching power calculations using the standard approach noted above techniques. These same households will be surveyed again in would lead to a biased estimation of the required sample size. Type I Error A 5 percent Standard 2007 to create a panel. The primary indicator variable will be As a result, additional parameters must be estimated that total monthly household expenditure per capita, calculated correct for time sensitivity: the within–person variance49, and Power 80 percent Standard from total monthly household expenditure (SUSENAS survey variance in growth rates at the individual and cluster level50. Instrument: Section VII, Q29), divided by the number of In addition, frequency, duration, and number of measures, Effect Size d .14 Determined w/Calculations household members. and the functional form of the expected growth path must be specified.51 Variance within person Sigma 1.0 SUSENAS Panel The sampling methodology will consider two treatments defined by their history of participation in Community Driven However, the remaining parameters concerning intra–class Variance in growth rates Tau 1.0 SUSENAS Panel Development (CDD) projects between 1998 and 2007: correlation, within person variance, between person and cluster growth rate variance and the effect size must be estimated. • Treatment 1: Households located in the sub–districts Treatment effect (d): the study Frequency F .20 .4 per year participating in PNPM • Treatment effect (d): the study will be able to detect a • Treatment 2: Households located in sub–districts participating in PNPM–Rural 2007. As KDP treatment was treatment effect size of .14, determined from a minimum benchmark increase in rural per capita monthly expenditure. Duration D 5 5 years assigned at the sub–district level, with all households located in the sub–districts participating in the project, • Intra–class correlation (p): is estimated from the 2002–2004 SUSENAS Panel. Clustering will be done at the Sub–district Measurements M 2 2 at baseline, 1followup households located in a sub–district participating in PNPM level, as that was the unit of treatment assignment for Function form of growth path c Linear SUSENAS Panel will be considered the treatment group listed above. the program. Households not located within a sub–district participating in a CDD project are considered candidates for the • Within–person variance (sigma) and Variance in growth rates (tau): the SUSENAS panel of household sampled comparison group. annually will be used to estimate the variance for the indicator across a single household52. Power Calculations for Clustered Sample Repeated Measures • The study will assume a linear growth path for the indicator variable. variance (sigma) and variance in growth rates (tau) where The primary tool of analysis is the “Optimal Design” software, Standard power calculations will estimate three statistical a cluster is defined as a sub–district, the unit of treatment. developed and described in Raudenbush et al. (2006). properties for each indicator: mean, variance, and Statistical Properties of Target Indicators The 2002–2004 SUSENAS Panel is used to estimate these within–cluster correlation, and then calculate the sample parameters for the rural households.53 The results imply a sub–district sample size of 450, 150 for each size required to detect a pre–determined treatment effect As noted above, we first estimate the statistical properties treatment and 150 for the comparison group. Within each for a given statistical size and statistical power. Usually the of the target indicators – in particular, the mean, standard Power Calculations Strategy sub–district, fifteen households will be randomly sampled from treatment effect size is based on previous studies or the deviation, and within–cluster correlation (r), within–person the 2002 SUSENAS for sub–districts participating in PNPM. The expectations of those involved in implementing the program. Initial calculations demonstrate that sample size is not total number of respondents per treatment is thus estimated For the PNPM–Rural case, we take a slightly different approach. sensitive to changes in the parameters for variance over to be 2,250. Because it is expected that approximately 20 The effect size is based on a minimum amount of change in time or the functional form of the expected growth path. percent of households will be lost due to attrition, the project per capita expenditure that the study would deem worthwhile 49 The variance of measurements of an indicator for the same household will over sample by 20 percent in each sub–district, increasing across time. to detect, in this case 1–1.5 percent per annum increase in 50 This is essentially the variance in the change in income between the two the required sample size by 450 households. In addition, 675 rural per capita monthly expenditure. Power calculations time periods surveyed. The overall variance in growth rates is represented households will be added to each treatment group to assure are conducted using this effect size in order to estimate by tau, which can be broken down into the sum of the between person 53 Note we likely overestimate r from the SUSENAS. SUSENAS does not an equivalent large sample size of poor households. The total the required sample size of households and sub–district. variance in growth rates and the between cluster variance in growth rates. conduct a random sample from each sub–district. Instead, it samples households to be sampled for each treatment group is 3,375. 51 See Raudenbush, et. al. (2006), Sections 10–11 for background on all several census blocks within sub–district. If there is geographic clustering Smaller effect sizes would correlate to change in per capita additional parameters needed for power calculations using a panel. within the sub–district, the within–cluster correlation estimated form the 52 Note that the SUSENAS Panel, while providing parameter estimates for the SUSENAS may be higher than the true within–cluster correlation. See also study, is too small to consider as the primary data source. Olken (2006). 49 50 ANNEX 3: EXECUTIVE SUMMARY Rural Households FROM THE QUALITATIVE STUDY Indicator Mean S.d. (s) r Monthly Expenditure per capita 165287 87408 0.14 Introduction (motorcycle taxi) service providers, construction workers, or migrant workers. In the last two years, many villagers The National Program for Community Empowerment in Kabupaten Bombana and Kabupaten Konawe Selatan (PNPM) Mandiri is a poverty reduction program launched (Southeast Sulawesi) have worked at public gold mining sites by the Government of Indonesia in 2007. One of the biggest both as miners and as providers of goods and services for Calculations based on repeated measures: components of the program was the empowerment of village the miners. community. Following the format of its predecessor, the Parameter Symbol Value Source Kecamatan (Sub–district) Development Program (PPK), PNPM The condition of road infrastructures in the sample Mandiri encouraged community participation in every stage of villages relatively varied. In East Java and West Sumatra, Cluster Size n 15 Determined w/Calculations the program. Based on what was agreed by the community, the most of the village and dusun55 roads are in a good condition, village administration then submitted development proposals whereas in Southeast Sulawesi, many parts of the districts # of Clusters J 150 Determined w/Calculations to the sub–district. The program required that the block grants or even province roads that pass the sample villages are in be allocated for distribution at the sub–district level; the villages very bad condition. During the past three years, most of the Intra–class correlation p .14 SUSENAS 2002 must compete to prove that they deserve the grant based on damaged roads have been repaired, partly funded by PNPM. the principals of urgency and significance for the community. However, inadequate public transport facilities remain an Type I Error A 5 percent Standard unsolved problem and the villagers generally rely on ojek To evaluate the impact of the program, the SMERU for transportation. Power 80 percent Standard Research Institute in cooperation with the PNPM Support Facility (PSF) conducted a qualitative evaluation study. In the sectors of basic education and health, most of the Effect Size d .14 Determined w/Calculations This study compared the sample villages’ recent condition with sample villages are already equipped with adequate their condition prior to the program implementation, the data facilities. However, primary schools are not available in Variance within person Sigma 1.0 SUSENAS Panel of which had been collected through a baseline study in 2007. some villages in Southeast Sulawesi so the children have to The data collection was done through focus group discussions attend schools at the not–so–near neighboring villages. Other Variance in growth rates Tau 1.0 SUSENAS Panel (FGDs), in–depth interviews, and observation of the PNPM education facilities such as preschools, junior high schools, Rural activities. The study was conducted in 18 villages in and senior high schools are generally unavailable in villages Frequency F .20 .4 per year nine kecamatan (districts) in three provinces, namely East other than kecamatan capitals. Health facilities in the sample Java, West Sumatra, and Southeast Sulawesi. Following the villages in Southeast Sulawesi are still lacking in numbers. In Duration D 5 5 years sampling method of the baseline study, the locations of the other sample villages, health facilities such as Polindes (village study were divided into three categories: (i) villages/nagari54 maternity polyclinics), Pustu (secondary Puskesmas56), and Measurements M 2 2 at baseline, 1followup that had participated in PPK phase two (PPK–II) since 2002 Posyandu (integrated health service posts) are available but and were recipients of the PNPM 2007 (hereafter referred to their conditions are in need of improvement. Function form of growth path c Linear SUSENAS Panel as K1); (ii) villages/nagari that had not participated in PPK–II but were recipients of the PNPM 2007 (hereafter referred to In terms of clean water supply, most of the villagers in as K2); and (iii) villages/nagari that had not participated in sample areas do not consider it a major issue. However, PPK–II nor the PNPM 2007 but were recipients of the PNPM some villagers from certain dusun or RT57 still find it difficult 2009 (hereafter referred to as K3) when the government to access clean water supply. Economic facilities, such as proved their commitment to include every sub–district in the traditional markets are accessible for the sample villagers program implementation. The study was conducted between in general. March and October 2010. Characteristics of Study Areas All sample villages are rural areas that mainly depend on farming. Some of the villages are located in coastal areas, but the majority of the villagers live from farming and keeping livestock. In addition to farming, the villagers work in small trading sector as kiosk owners and in service sector as ojek 55 A dusun is an administrative area within a village, consisting of a number of RT (neighborhood units). 56 Puskesmas stands for pusat kesehatan masyarakat (community health center). 54 A nagari is a village–level administrative unit in the West Sumatra Province. 57 An RT, or neighborhood unit, is the smallest unit of local administration consisting of a number of households. 51 52 Main Findings them, this misuse of name is justifiable as long as there is no willing to give up their lands for the program when requested. problems more than the villagers themselves. The village elites case of non–performing loan. The Bank is taking this seriously At the locals’ villages, the roads were narrow and the villagers also thought that not all the decisions made and information 1. Implementation of PNPM Rural and is now conducting mission to improve the SPP practice. were not willing to give away their small lands, so they did not gathered from the village meetings should be disseminated However, in one village in East Java, the community leaders get the PNPM infrastructure projects. to the community, especially if money was involved. The Almost all sample villages chose infrastructure projects deliberately refused to get the SPP fund for fear that they fact that the villagers very rarely asked their leaders about for the open menu program category. Only one village would not be able to repay the loan; consequently, they did The sub–district facilitators considered that the any information, decisions, and activities at the village level (in Dharmasraya) submitted proposal of non–infrastructure not get the open menu program. workloads given were not evenly spread within the added to the problem. Information, if any, was usually given activity—training on developing home industries. available resources. Some facilitators had working areas to the villagers during informal meetings, such as arisan59 and Infrastructure projects constructed were roads, bridges, Participation is still high in PNPM forum, however, the covering 10 villages, while some others had to facilitate more religious gatherings. irrigation systems, waterways, school buildings, and posyandu. villagers’ participation in the decision making of the open than 50 villages, just like what happened in one sub–district The villages chose infrastructure projects because (i) there menu program and the SPP was often instrumental, only (not a sample sub–district) in Agam District, West Sumatra. The villagers were generally passive when it came to is not adequate number of infrastructures at the study area; to fulfill the program’s formal requirement. The increasing Moreover, the facilitators thought that the technical and information on development, except for that on direct aid (ii) PNPM was regarded as a program for general public; (iii) number of villagers present at PNPM meetings did not fully administrative works, such as monthly report writing, have such as Raskin and BLT. At the village or dusun level, such they wanted to cushion the perceived negative impact of alter the village elites domination in the decision making taken most of their time so they could not make the most of information was usually given orally from the head of the targeted programs such as the Direct Cash Transfer (BLT), Rice process. The villagers in general, particularly the poor, were their job as facilitators. village to the head of dusun/RW60/RT and then from the head for the Poor (Raskin), Household Conditional Cash Transfer still passive participants. The condition was due to some of dusun/RW/RT to the villagers. The information was generally (PKH), and Health Insurance for the Poor (Jamkesmas); (iv) the factors: (i) kinship, (ii) patronage system, (iii) the village elite’s The sub–district facilitators also believed that there on the program’s activities and implementation. Information bias towards elite and nonpoor villagers opinions during the reluctance to live by the principles of democracy, and (iv) the should have been special facilitators who were assigned to regarding activity funds or budget was seldom given to the decision making process. elite’s feeling of superiority over their fellow villagers. These empower the SPP recipients, because they already had heavy public. Moreover, the village administration staff generally factors caused inequality during the decision making process. workload and because not all facilitators at the sub–district gave information which was instructive and mobilizing, such The Female Savings and Loan (SPP) program is considered had the skills related to microcredit empowerment. There as the information on community work. greatly beneficial because it has (i) developed the Female participation is high in PNPM planning and were microcredit facilitators at the district level but they were recipients’ businesses, (ii) improved households’ financial implementation process, however, the increasing female actually more needed at the sub–district level. When there were problems or unsatisfactory results, capacity, and (iii) replaced loan sharks. The recipients used participation still did not reduce male dominance. the villagers generally did not voice their complaints or the program fund to develop their old business and to create Although male dominance was less noticeable in a special 2. Governance, Participation, and Representation in dissatisfaction to the village administration. They only new business. The program implementers required that forum for female, a meeting held to generate females’ ideas Decision Making talked about the problems among themselves or with the recipients already have their own business. A small portion which would propose one suggestion for the SPP and another community leaders. Only a few villagers were willing and had of the program fund was used for households’ urgent needs. for the open menu program, it was still critical in influencing In most of the sample villages, although villagers the courage to tell the village administration. This condition Especially in Ngawi, SPP has reduced the villagers’ dependency the results of the special forum. In some sample villages, the participate in the decision making forum, the decision was due to some factors, such as the villagers’ reluctance, fear on loan sharks since the program offered competitive interest final decisions regarding the suggestions from this forum making process generally involved only the village elites— of the village apparatus, and apathy (because of previous rates and simple procedures for those who have already had were made at the village level where the decision makers were the village apparatus and the community leaders. These unattended complaints). their own business. the village elites, which are dominantly males. people believed that they already represented the whole community. Other members of the community were usually In general, the participation model set out by the PNPM There are cases where PNPM implementers limited the poor’s In sample villages, no serious conflicts have happened passive participants when they were present, only listening did not have any significant impact on the changes in access to the SPP program by imposing strict requirements for during PNPM implementation. However, in a small number to and agreeing to what the elites decided. Some villagers, the governance system (participation, transparency, and fear that the poor would not be able to repay the loan. There of villages, PNPM implementation could and had led to especially the poor, did not attend the meetings because they accountability) at the village level. This was apparent from were also cases in which certain people cheated to get the conflicts, such as conflicts of interests between jorong58/ felt inferior. Other reasons for the villagers’ absence in the all villages, regardless of whether they had been beneficiaries fund by including names of the poor on the list of people who dusun, conflicts between the village administrations and the meetings were apathy, unfavorable time of the meetings, and of the PNPM since 2002, 2007, or 2009. Only one village submitted the program proposal without their consent. The program implementation team (TPK), between the TPK and not getting any invitation. claimed to feel the impact of PNPM on other activities. In other fund would then be used by the cheating non–poor for their the community, between the locals and the nonlocals (those villages, participation and transparency applied during the own benefit. who migrated to the villages from other places), and conflicts During the decision making process at the village level, PNPM implementation were regarded as the program’s special regarding supplies of goods and services. In addition to lack the females were often represented by formal institutions features that did not have to be applied on other programs. The SPP fund distribution was often considered by most of knowledge about the program – most likely due to lack of regarded to speak for women, such as the Family Welfare of the program implementers in the villages and the program socialization, lack of coordination with or involvement and Empowerment (PKK) or Bundo Kanduang in West The fact that PNPM did not have any significant impact on village apparatus as a pre–requirement to get the open of relevant people in the program implementation also caused Sumatra. Consequently, women were less represented than the government system in general was caused by some menu program. Therefore, a lot of community members did the conflict. In West Sumatra, the area unit for the program men. However, compared to the past (i.e. pre–PPK/PNPM era), factors, namely: (i) the elites dominance and the villagers’ lack whatever they could to get the fund, including by means of implementation was jorong. The nagari leaders felt they more women have attended the decision making process at of initiative, thus preserving the status quo; (ii) there is no deceptions. For example, many business groups applying were not involved so no one could facilitate communication the village level although it did not really change the fact that guarantee (incentive) for the village apparatus and the villagers for the SPP loan were instantly established only to get the between jorong. Lack of coordination also created conflicts men still dominated the process. In addition, most villagers that if they applied the PNPM mechanisms on other programs, fund. In many areas, names of the poor were falsely included between the village administration and the TPK, while still believe that men are leaders so they, instead of women, they would be given something in return, such as a project; on the list of the fund recipients; but when the money was conflicts between the locals and the nonlocals were triggered should make the decisions. and (iii) the village apparatus and the villagers’ tendency to disbursed, it was distributed among the non–poor. This is due by jealousy over economic gap—the villages where the live by the existing norms. If a program or an activity did not to misinterpretation of SPP as prerequisite to get open menu nonlocals live are more developed than those where the The system of representation did not function properly, require participation, transparency, and accountability, they program. Hence program implementers and villagers “collude” locals live—just like what happened in Dharmasraya. With the clearly seen from the absence of mechanisms at the would not impose those requirements. to make their village entitled to SPP in order to ensure they PNPM requiring the community self–sufficiency, the program RT/dusun level to get the villagers’ aspirations or to get the open menu fund. They gave the loan to non–poor funds were often granted to the nonlocals’ villages; the roads disseminate results of the village meetings. No meeting out of fear that the poor cannot pay back the loan and thus there are wider and the villagers had more lands so they were was held to absorb the villagers’ aspirations at RT/dusun level 59 An arisan is a social gathering in which the participants operate a savings and loans mechanism. jeoperdize the village’s chance to get open menu fund.For because the village elites claimed to understand the villagers’ 60 An RW is a unit of local administration consisting of several RT (neighborhood units) within a kelurahan (a village–level administrative 58 A jorong is a dusun–level administrative unit in the West Sumatra Province. area located in an urban center). 53 54 3. Poverty and Its Dynamics to develop or felt satisfied easily, unproductiveness due to 5. Dynamics of Needs and Fulfillment old age, being economically dependent (widows who do not In the majority of the sample villages, the number of poor have job), and the increasing prices of daily needs. In almost all sample villages, the poor’s main needs were people was declining although the rates of decline differed job opportunities, additional capital, and skills upgrade. among the villages. Factors that have brought about the Poverty alleviation programs, especially those with special After that came scholarships, free health services, and decline were, among others, (i) new job opportunities such as targets, such as BLT, Raskin, and Jamkesmas, contributed infrastructures to support their main livelihood (such as an the gold mine exploitation in Kabupaten Bombana, Southeast significantly in preventing the poor from getting poorer. irrigation system and farm roads). The government and some Sulawesi; (ii) opportunities to become migrant workers; The BLT fund, Jamkesmas cards, and Raskin were considered groups of villagers have made efforts to fulfill the needs but to (iii) regional segregation, creating new economic centers; to have helped the poor with their main needs of emergency no real avail. This was because (i) the existing programs were and (iv) new factories/plantations at the neighborhood. In cash, health expenses, and food. The three programs have also not adequate to fulfill the villagers’ needs, (ii) sociocultural addition, the decline was also caused by the rising prices of been regarded as the top three government programs which conditions in the villages, such as social envy, the elites’ or dry land commodities in West Sumatra and sea commodities are considered most beneficial for the poor. Unfortunately, nonpoor’s bias, and (iii) corruption or ineffectiveness during in Southeast Sulawesi. Other factors were the better rural the program targeting was in the hands of the officials/ the program implementation that reduced the impact on the roads infrastructures, the increase of farming productivity, village elites with no room for the poor to participate or for villagers’ needs fulfillment. and government aids. PNPM contributed by facilitating the transparency in targeting. construction of infrastructures such as roads and bridges for PNPM–Rural was seldom used to fulfill the poor villagers’ the general sectors and irrigation systems and farm roads for PNPM implementers did not think of PNPM as a poverty fundamental needs. The PNPM open menu program was the farming sector. reduction program. Hence, PNPM was regarded as not often used to build general infrastructures that did not having significant direct role in reducing poverty. They directly cater for the poor’s needs. This is due to the view that In contrast, two villages in East Java saw an increase in the regarded it as merely a regular village development program. PNPM is a program for all villagers, not for poor people. The number of poor people. This was due to the decrease in sea Consequently, they did not put the poor as top priority. This SPP could partly fulfill the needs for capital loans, but it was commodities productivity caused by environment degradation was apparent from the fact that they made project plans difficult for the poor to gain access to the program due to (sea pollution by the industrial wastes) and the decline in without considering the benefits for the poor; they did not its strict condition that required borrower to have business labor participation caused by industrial mechanization. In the specifically include the poor on the list of workforce for the before applying for loan. two villages, they did not make use of PNPM as an instrument PNPM construction projects; and they did not exempt the to solve the problems. poor from providing cash contribution for project activities. PNPM has not been successful in empowering the village community because of some factors: (i) the improper Villagers usually only connected poverty characteristics 4. Dynamics of Access and Quality of Public Service structure of power in the villages where the elites were with the aspects of asset ownership, daily needs (including dominant, marginalizing the poor; (ii) the mechanistic education and health) fulfillment, and type of occupation. In general, the sample villages already had public service model of empowerment of the program, in which facilitators For example, having minimum assets and no fulltime job facilities in education, health, water supply, and economy were only told to make sure that the program stages were was considered as the main reason for someone being poor. (market). This has contributed to the increase of the villagers’ properly conducted and not to improve villagers’ awareness This belief has not changed significantly during the period access to public services during the past three to eight years or capacity with regard to program objectives to encourage of PNPM implementation in the villages (three to eight years (of PNPM implementation). In this matter, PNPM was thought the creation of good governance (participation, transparency, depend of village category). to be quite beneficial since it helped to provide additional and accountability) and the improvement of the villagers facilities or to improve the condition of the existing facilities, economy capacity based on self–sufficiency; and (iii) the Poverty dynamics were determined by factors, such as including road repairs. Besides, the road infrastructures cases of mismatch between the program mechanisms and the economy, social, public and government institutions, improvement facilitated by the program was also considered local cultural characteristics, in which the PNPM encouraged government’s support, and programs’ targeting. Economy helpful in improving the villagers’ economy. Nonetheless, in individual participation in the program implementation or factors, such as the rise and fall of prices of farm/sea some sample villages, public facilities were still difficult to in the village/nagari administration matters, while the local commodities and prices of daily needs as well as government access by some villagers. This was caused by, among others, (i) culture (such as in West Sumatra) urged stronger communal aids, played the biggest role in most of the cases of poor the limited number of facilities, (ii) the unavailability of public representation by restoring the nagari administration households whose economy condition has fluctuated for the transport to reach the facilities, and (iii) the absence of quality tradition; (iv) the ineffective work of the facilitators due to past eight years. service especially in health. heavy loads of technical and administrative work; and (v) poor quality and inexperienced facilitators and the frequent Groups of poor people that remained poor were generally The villagers thought that the quality of public services rotation as well as high facilitators turnover. caused by the absence of skills and capitals to improve needs to be improved. In health, for example, Jamkesmas their livelihood. It was specifically because (i) there was card holders felt that they are not treated as well as other limited number of alternative job opportunities in addition to patients. In some sample villages, civil administration services, their main field work, which is in farming sector; (ii) the poor especially the obtainment of ID card and family card, were were generally junior high school graduates and had only considered complicated since the villagers had to go all the traditional skills (as farmers, fishers, or construction workers); way to the kabupaten. (iii) they did not have enough capital, especially cash. Although there had been some aids offering credit for capital, what they really need was financial aid, like direct cash transfer, that they did not have to repay. Other significant factors according to the villagers were: mental attitude that did not feel the need 55 56