@~~~~~~~~~~~~~~~~ m~ 3~~~~~~~~~~~~~~~~~~~- b2~ ;! E : > i i b § U 3 ~~~~~~~~~i:. X 11*0>| , 0 4' S r I ' t, Ihdone'Ra Keca tn IDeveloment Prngamo Building a Monitoring and Evaluation System For a Large-Scale Community-Driven Development Program Susan1 Wong Mla)y 200(3 Environnent and Social Dcvelopnment lJnit East Asia and Pacific Rcgion Discussioni Paper I his publication was developed ainid pr oduced 1by the l\nv'ironnici-ir and Social D)evelopient lJnit (EASEIS), East Asia and Pacific Region of tlhe World Bank. 'I'hC Environment, Rural Development, and Social DevCloptimenIt itrC part o tliC Environncntally and SociUIly Sustainable Development (ESSD) Network. Papers in the EASES Discussion lPaper Scries are nlot forima] putblicatioins of the World Bank. ''lley are pLub- lislh(I and cirCulaItCed to C11COurage (lisussion and coiniincilt vvithin the development CoImImIuILith.ih f tillndillns, inltcrprctaitionis, judg,miients, and conclusions expressed in this paper allC those of-the autilor(s) and siotiuld ilot bc attributed to the \Vorld B3ank, to its afililated organizations, or to members of thc Board of Executive Directors or the governilicIts they repi-escit. Copies of this paper arc ;available trom: Stisian Wong, EASES, Roomil MC8-10() Swoig1 ( Dworldbank.org ABSTRACT The \'Vorld Bank has invested heavily in community-driven development (CDD) programs througlhout the world since the mid-1990s. As with other development programs, these CDD investments require strong monitoring and evaluation systems to assess progress and evaluate program impact. The Kecamnatan Development Program (KDP) in Indonesia is a large-scale CDD program that developed an extensive monitoring and evalluation system to track its progress and evaluate its impact. The system consisted of quantitative and qualitative tools, including internal monitoring from program stakeholders and external independent monitoring by Indonesian non-governmenital organizations (NGOs), journalists, and audit teams. The Program commissionied several impact and technical studies to measure its impact on poverty alleviation and local governance. Because of the strong emphasis the World Bank and the government of Indonesia placed on oversight, monitoring and evaluation became integral components of overall program management. Findings were discussed regularly with managers and used to improve the design and operations of KDP This paper describes the monitoring and evaluation system developed for KDP. It includes a detailed description of its components, how they were constructed and operationalized, and the challenges encountered while incorporating these multi- faceted activities into a large CDD program. lt is hoped that this case study will aid practitioners in the development community who seek innovative and effective tools to monitor and evaluate their programs. I II,, palreI- al(l il(Ice(l thlC overall dxeclopcllcilt of the K D) P N oIlitoi-iI- and Ev-alatioln Sy'stemII, hntbeditc.I (rreatly troil the \'ision, suloort, and guidance of Scott (Gurenlilmini and Victor lBottilli of th \Vorld Bank. I)avid TI imhcrmini also provi(le(l tf-esh insight and Coiiillcllts ito improve this paper. Ihe KIDP mlonitoringr and evaluatioL tCeamll membIeIrIl'S (ICSCIer e acknowledgemCient ,ndl spci.lal a)precilItion tor implementing and refiningh thie systemlls CoiMp)olicilts over the vcair,. Ihlcsc team menibers arc: Sandra Usilnam; 111v Basnri, Yulli I;stadewi, 'csi N Mlavani, iAri Sirega,l; Pinkan I Itaii \V, lu(ii I ladi lotok, I. Ainini, Azlil Fitra, Viraxvan Nugrahadli, KuikUIi K(urnil, I leiL Gunawan, loko Suhartanto, I lotman 1) S, 'onct lRizal, Kn0ioro l'risti, Inde Nut-dartilalbl, andlkl ; lk \VuWllalati. 'I'hc manyv accoilnlishhnicnlts of1 DI)l'5 NMlonitoring atid lvaluation (onl)upllelit wvould not have beeni possible without the comiml1itimient and support otcollcagmes associated with the Miinistry o01 toili Atlairs, KDP)l) National Secretariat, National ANiIIlangemilent Consultants, ldiek conmsuiltani is, coliulitIllt l members, nid tlte ulimllrous !\(,()s and1 journlinsts who joined collectivelh' in this massive nit)nitorin- nind evaluation ettort. Thev aIrc too niumillcrotIs to Imentiol hlere by nlamile, bLUt thicir inyput and supp 0ort atrc ratcfiilxv ai,pprecilated. TABLE OF CONTENTS L.ist of' AC ro rlv'm ls ......................................................................................................... V i 1. ntlrOdIuctioil ........................................................................................................1 2. lrief Plrogram; Descriptioll ............................................................1 3. Development of KDI's M onitorin and EVAlUation S'StCeIl ...........................................................................3 4. D evelopm ent of 'thic N i]& 1 ..........................................................................................................n .. 5. Components of the Monitoring Sv llsten.. ...........................................................................................................9 5.1 Field Monitoringaldl Reporting by Government Officials iiand Constiltanits ..............................................9 5.2 Community Participatory Monitoring.1. 1 5~~~~~~~~~~~~~~~~~~.2 Oiiiilt'};ltCplO) \l)ltIilr......................... 1 1 1 , 1 1 ,........................... - 1 1 .......................... 5.3 Case Stut(lies aniCd DoCUMCttion of LCssons I.earned ............................................................................ 12 5.4 NMIC 1i nancial SUpervision and(I';rining ULnit .............................................................................. 14 5.5 Handling Complaints and Grievance P'roceIures ............................................ 15 5.6 NGO Independent Provincc-Based M onitoring(r .......................................................................... 18 5.7 In1dCepedleCt M\lollitol-ilng by VJour,-na.ists ............................................... 20 6. Components of teEVAlualtiOn Sy'StCIl .................................................................................... 21 6.1 Im pact' Evaluation Sttud(l ................................................................................... 21 6.2 Infrastructure and Ecoinomilc Loan E'valUation0|s ................................................................................... 23 6.3 Audits and Financial RcviCws ................................................................................... 24 6.4 \WVorld Bank Supervision iMissionsand Financial Reviews ............................................... 25 7. Costs Of tile Monitoring_ ai( n I EVlAualtion SN'StelIl ................................................................................... 25 S. Difficulties EncoutnIterIed ................................................................................... 25 9. Conclusionis and [utrc ImprlOVemlnCltS unLrci K.D' Phase Two ......................................... 26 Bibliography ................................................................................... 29 Annexes: A . Brief Description ot K DP ......31 B. KDP Project Design SumllIlLary ........... 40 C. KDP Monitoring and Evaluation Framework .......................................................... 42 D. 'ITermis of Reteirenice tor NGO Inidependc1eint Monitoring of KMDP . ......................................................... 46 E. Sample \WVorkshop Schiedule for NGOs Involvedl in InldepenTident MonitoriTIg ............................................ 53 I Complalints/CGrievances Database Fiekis ..........................................................eld. 55 G. 'ITerimis of Referenice fir Financial Suipervision and Trailling Unit .......................................................... 56 1-1. Terimis of Reference tfor KDP Financial Atidit .......................................................... 59 AJI (Aliansi Jurnalis Independen) Mlusbangdes 111: meeting to report upon Association of Independent Journalists results of UDKPII meeting, to organize 3appeda (Badan Perencanaan Pembangunan the community Implementation Team Daerah), Regional Development to oversee project implementation, and Planning Agency at the province or dis to decide upon the village technical trict level assistance. Bappenas (Badan Perencanaan Pembangunan Miisbangdes Pertanggungjawaban: Nasional), National Development meeting to report upon project Planning Agency progress and finances halfway through Planning Agency ~~~~~~implemenetation. BPD (Badan Pemwakilan Desa) Democratically elected village represen INGO Non-Governmental Organization tative council. This body was intro- NiV3C National Management Consultants, duced under Regional Giovernance Law located at central level 22 of 1999. On\m Operations and Maintenance BPKP (Badan Pemeriksa Keuangan dan PjAG( (Penanggungjawab Administrasi Pembangunan) Finance and Kegiatan) Project Administrator for KDP Development Supervision Agency at the kecamatan level BRI (Bank Rakyat Indonesia) Indonesian Pj$X (Penanggungjawab Operasional People's Bank Kegiatan) Project Manager for KDP at CDD Community Driven Development the kecamatan level. The PjOK is the DPRD (Dewan Perwakilan Rakyat Daerah) section head of PMD at the kecamatan Local Parliament. DPRDI is the parlia- level ment at the province level. DPRDII is PiNSD (Pembangunan Masyalcakt Desa) the parliament at the district level. Community Development Agency FD (Fasilitator Desa), Villago Facilitator under the Ministry of Home Affairs. PMD is the government implementing FX (Fasilitator Kecamatan), ,Cecamatan agency for KDP Facilitator Podes (Potensi Desa) Annual village survey KDP Kecamatan Developmenit Program Susenas (Survei Sosial Ekonomi Nasional) Kecamatan Subdistrict containing on average 20 National Expenditure Survey of to 25 villages with as many as 60,000 Households persons TK - Kab (Tim Koordinasi Kabupaten) KMKab (Konsultan Manajemen Kabupaten) Coordination Team at district level con- District Engineer Consultant sisting of various government min- KMViProp (Konsultan Manajemen Propinsi), istries and chaired by Bappeda Provincial Consultant tX-Frrop (Tim Koordinasi Propins,) XPXIM (Kantor Perbendaharaan dan Kas Coordination Team at provincial level Negara) Treasury State Office consisting of various government min- LD (Lembaga Demografi) Demographic istries and chaired by Bappeda Institute, University of Indonesia UDXP (Unit Daerah Kerja Pembangunan) LWiMD (Lembaga Ketahanan Masyarakat Desa) Kecamatan Development Forum, com- Village Community Resilience Board posed of village representatives and key local officials LP3ES (Lembaga Penelitian, Pendidikan dan UDKP I: kecamatan - level meeting to Penerangan Ekonomi dan Sosiall discuss what KDP is and which villages Institute for Social and Economic will participate in the program for that Research, Education, and Information year. M&E Monitoring and Evaluation UDKPII: kecamatan - level meeting to MIS Management Information System decide which village proposals will be Musbangdes (Musyawarah Pembangunan Desa), accepted and elect members o' the Village Development Meeting UPK Musbangdes /: meeting to provide KDP tIPX (Unit Pengelola Keuangan) Financial orientation at the village level and to Unit elected by the UDKPII forum usual- select village facilitators. ly composed of two or three persons who handle the kecamatan and village Musbangdes I1: meeting to decide finances for KDP which ideas the villages will put forward as proposals. 1. INTRODUCTION Dletailed iftormationi rcgul-(lin, the coimponenrts iS iilclli(lc(l iln thev text, wlilc anlnexcs scerve as a refer- TI'he WCorldl Batnki hlas invested1 healvilyr in "oilltcli- ence ior those dCveloping their own llM&E, systems. tyl-(dInveii (developmelt" (CDD) programs and a1pproacelis sille the id(1-1990s. CDD)1) prolects are broadly (lehiled( as tho se that olivc collti-ol Oldf,ichl s)ion and resources tioSs co ut gi\ -oups. i\siCDi) 2. BRIEF KDP PROGRAM slas;t( tcst-ts )-oilllili) -tls s lD) DESCRIPTION ivestm-ents grow ralpidlYl it is important tllit lloni-l- tOlil(t and evaldation (N'l&L) s\'stems be established Thle iKccaimatam Development Program (IKDI) is at to assess the progranms progress as vell as evilluate its goverllnlrient of I ndonesila programill itii(l aIt allevialt- impacts. All too ofteln, devClopmllCnt programs do lolt in poverty in niral communities and improving incorporate adequate or ri(rorous inionitorertv i rrl ci-i.iivi. tcis an ) impo.l HICO'l)altC oide(Itil;tC of, ri(ror-ous illollitolm-> aInld local govcirance. The finst pilhaset f KD) bega evaluation mechanisms to measure rloCss 1998 and ended in 2002. Ihe second1 l will run CeVaIluatc the impact of interveintions.' 'licsc svstcis fr-olll 2002 to 2006. K.DP provides block grants of' are crIcial to provi(ie timely reliable inforimtion alnd Rp35() million to 1 billioon ($40,000 to SI 14,000) alloXv prog1ram managlelrs to adjust activities as need- dircctly to kecamatmn2 and villages for smnll-scale e(i aindi to assess deveclopment effCctivClless infr-alstructure, social, anud e(ol(1mom icaCtiVitiCs. Thc Kecaumitan Dcvclopmeicsnt program (mv )l) iK Kl)l' begoan in 1998 aIt a timle of tremendcious politi- Indonesia is one of the llirgest \WVorld Bank-financed cal Uphealvall and econiomic crisis. ThC econiomiic cri- CDD projects, witlh a budget of'somiec $700 mnillion sis reversed vears ofl progress in poverty rediuctio for its first two phases (1998 to 2006). During its and plunged millions of rural poor below the poverty first phase (1998-2002), IKDlP devClopCd an CxtC1n- linie. Econ1oml1ic gr]oVwtlh (dcclineldC to legaltive 13 per- sive AII&E system to track the programs progress cent in 1998, in marked contract to the 8 percenit andl eValua,1te its impact. Thie system colisiste(d of both a11111nlll griowtth rates of carlier years. Iublic dIemI1on1- quantitative anld( qualitative tools, incililludn internal stratiolls aind( riots shIook thIc countrY anid led to the mloniitorilngr fi'olni program stalkelholders and iiide- downlftli of PresiCelt SuLharto after 32 eawrs ill pelnd(enit external mlonitorinig by I ndonesian ill 1011- piower Invdonesia entered into a period of political crovemillCiltalt oriraniz;ltiolS (NGOs), jourralists alnd tl'1SitiOn1 lea;din inl*lju 1'999 to the co ftrys first aud(iit team1s. I he program;lill commissiiSionedti several free general electi'ons in 44 years. impact and technical studies to measure cost effec- tiveess iand the impact of the program onl poverty I'he chai-gc in leadership, loweveI; did l1ot solve the alleviationi and local governante B1ecause oftlte couitry's deeper problemis. Despite higlh rates of stronr emiiphalsis the World Bank ad tIe govern- poverty redIuctioll, vulnerability remaianed high and imenit placed on0 M&E fiom tIlc outset, it beCCnl.eC ;1ll manY poor did( n10t benetfit adlequLtly ftrom the integral comiiponienlt of ovcraLll programi manr(rement coultryVs cCilonMic grrowtll. Three decades of 'autlhor- Findiiings were discussed( regularly witlh managers and itariani rule ha1(d also uld(ecriniled local capacity and Used to imnprove KDil desigil an1d operations. placed heavy restriction1s oil local C0111111omunity organii- izing. Locdl level governillen1t waIs gererally unre- Thlis pilper describes the M&E system developed for spoinsive and uiaccountable to the nieeds of its the first plhse of K.IDP It aimilS to provide develop- citizens. Corniptioln was rampant, and the misill11- nllent practitiollers witlh a case Studv' Of how 1moilitor- agmen-ent ol'goverilmient projects and fLtinds led to a ilg and evaluationi approaches anid tools can iie generl;d disillusionmli1eIlt VVith pLublic programs. incorporated into hu-,ge-scale CDDD proraulns. 2 Kccamatman are Ind onesian sub-districts conrtainimg on averagc 2t) to 25 villhiges wvith a pioipulartiotn ofso ine 60,00(0 pcrsons. Fir a I flihs ic the broiiad (lefi nition it CDD ais dicfinc(l in tuc Wotrkl fidlcr descripnition otikccaratrin and ndt Ilinesi. n admilnistranVc Bank's Pnxverr Reduction Stratcgy Sotrcebook, Nai 2001. levels, sce AinnecŽ A. Indonesia Kecamatan Developmetnt Program 1 (o\CrnllnllCllt-spuonIsoe(d Coifllmllit\ (lvC\o)lnllClt llskev principles arc: programs ill lndonesia, aIs in 11manllv otherl colilntrics, ° omI;nil/?ta/-iciP/(/O/ d(/t'/17POiVT'/77'/11 a//)0poo WerC VUlnerabice to politiciil rnainipulation and captuire 1711!! -COM/fl/flI?l a11s - Ci in- munitiCs should halVC I- local clites. \While new decentralization lalvs wclr cont-ol over (decisionis and r-esotirces that iffict passed ill 1999 to give districts greabtter les them. Coniiimnities take ownership of all aspects and aithloritv; thils did nl)t mlleall that power would of pi-(ojects, toim planning and dec'ision mking to automaltically shitft to colnlilluillitics or ineed( ice imnplementation. Participation shoul(i be broad( democratic. blased aimid inCludie poor- people and other mnarrgin- alized groups. KDl)l was implemented in the mid(st of Iind(onesia's I groues p li. lf 7,71prrny- 1 KhlP emhai;Szes tralnspatrency m -p pltil transiti n period and decentralz-throrout the proet tion process. D)espite a volatile political and econom- cycle. Inforiation is pokvcr. Decision making and ic backdrop, K D)P evecntually became it corc part of f th gvriiietssraey lldcetalzain-n financicail manag;trlicnint sliotl}(i bzc open andz( sha;red tegoverm srtgyo with the citire comimunity. Evcryone shotil(d have improving local goverianLce. Nationali; and local (g,o'- acess to key programil information. criiiiieilts l;ltvc "bLIo-l£lt ill" tO the KDP)l app5rOilCh lc ;s eruents have "bought ii to the [)approach s o !Sl/sabilili//y - Activities should be sustainable, evidene(i b v increasing budgetarv contributiolls to b o c s evi -1~~~~~~~~~~~~~~~~~~ 1[1]l(1,]r l z ztitx self retbanice ;in(i vilfl.l£ the prograim nand the replicattion of Kl)l mechai- g ll;lil;lagelenll t ofall acxtivities. nisiims for other puiblic programs. Current govern- - KDP strives to keep projects simple. iiciit policy proino(tes rlil KL)P (ics(ril its it nhere shotild be no comniplex nrles or procedures; foundation of its national poverty redluctioll strateo. i O-nly simplle strate 'ies and( meitliodis sliouil(i be KDP's golis are to: Used. o Alleviate poverty 1b)y raising rural incoimes and (o'/A o titioh. fir 'inl(s There should be open, ding ihealthy conipetition between villages for KDP provuin£ 1 increasedl cconiomiic Oppyortunities at the vill;ige level in the poorest kefaatasn. finds. KDP encourages villagers to select projects o Strengthen local government and community based on their individual merits. inistitritioins bAl empowering thielmi to -nallil-ge increasedl findling and( becoming accountable (Geogra c c¢verzge f or it. o Bulild public infrastmucture through labor-intensive Base(d upon a nationatl pooverty ranking of kecanmatan, methodlsI the goverrnienit selected the poorest kecamnatan to participalte in KDP.'he program began in 501 By pushin,g (ecision making dowin to the lowest lev- kecaniatan in 20 provinces throughiout the couLitry. eIs, KDP aiims to allow villagers to participate By its thir(d prroject vear (2001-02), it had scaled up and directly manage developim eint resources. The almost twofold to cover 984 kIecam-atan with a total progralm, in essence, seeks to empower the rural poor pop ulation of approximatelv 35 million. KDP is and encoura.ge more democratic and participatory operational in almost one out of every four villages forms of local goveranice. All IKDP activities aimi1 at in Indonesia. ('Iable I). allowing villagers to maiike their own choices about the kindis of projects that they need and(i wtant. The prograni gives power to coimimunliltcs by placing ftinds and the planning and decisioni makiln 'I'he overall management of KDP is the responsibili- processes directly in the hands ofvillagers. ty of the l\linistry of I lome Aff'airs Communlity Di)evelopmnent Agency (PiNmI)). PMD deals with the Scc First Kccamratan Dcvclinipment Projcct Pr ;cct Appraisal programis day-to-day operations at the nationil level. Doct)imcnt, April 21, 1998. Government coordination teams, representing van - 2 Indonesia Kecamatan Development Program Table 1: KDP Geographic Coverage, 1998-2002 Year One of KDP Year Two Year Three Total in Indonesia (1998-1999) (1999-2000) (2001-2002) * 20 provinces - 20 provinces - 22 provinces 32 provinces *105 districts * 116 districts * 130 districts *341 districts * 501 kecamatan - 727 kecamatan * 984 kecamatan 4,048 kecamatan * 3,524 villages * 1,325 villages * 15,481 villages * 69,168 villages otis miniistries, cxist fiom the national (down to thc Mtld cvalu1atC its CffCctivClncss anld im1palct over tinm. clistric t lc'cls to over-secK llDP. 'I'hc uIniiqteniess and imagnitude oft \'Di-' in the Indonesian coitcxt, especially' in strcssing thc im1por- I'l\) h ired consultant teamis to providc tcch1nical tance Of cOmmli1luInity particil)ptioi, transparencv, and assistanlce to the prograim. 'I'le oftice of the National instituitioll stregtheing, dcmandcd that a solid Mianagenment Consultants (NMC) based in lakarta NM/l&c'E systemii be in platce to docuLimcit the 1KDPl) lcd tle treams of consultants and provided(Jctocll experienc and to distill lessons.ln In man ways, KD h , and to f;,~ ~ ~ ~ ~~~~~epcrci c ad t distl cso.- Inia wv D dncidCe ad supervision to (lois5iltitt 5011112 is a consultanr to vri t SoOuIlIilS inl 1,660 consultants tioim 16 private comipanics at the lIndolesia IIand oftlts a potential model fol fUtIuec national, provincial, (listrict, and kecamatan levels democratization and dCvelopIment cfforts.th 'erCforc, givc technical assistance auid guidance to K.DP findings trom K DI) had to he fully nitore(l docu- ) nael~~~ite(1iisfotiKD had( cv; to e nontrd,dCI For a morc detailed description of KDi, see mented, and ealUated. Ann1ex A. 'I'le system ;also nee(c(i to provide effective clhainiiels of colillullication ti'oill the ficldl to the national level-and back-to inforimi mnana;gemncnt decisionls 3. DEVELOPMENT OF KDP'S and enstirc that corrective actioni was taken if neces- MONITORING AND EVALUATION sarv Reliable and timelv information had to be in SYSTEM the hands of those who could a,ct UpOnI it and resolve The Context issues an(l p )hlems expeditiousl. Monitorinig is the reguilar checking of project Challenges in developing the progress throughi the routine and systematic collcc- M&E system tion of informiation.T his information, whcin verified and analvyzed, highgights prog-css or problemis that 'i'here were sevenrl challenges in developing the allow project mallanarers to adnust plans and imple- MS&E System. melntationi as nceeddl. Evaluation is thle assessment of e AmbitiOus O/?/CtOiVes and widle i-vare/v O/Jc.liviies- the mcrit of the project by mneasuln1g1- it aga-inst the Amlonig the most difficuilt childleges in develop- achilevemnent of its objectivcs. EvaluaWtion answers thie ing an M&E systcn were the program's amibitiotis questions: hals the pro>ect met its objectives and what objectivcs and wi(ie scope of activities. While ftactors lecl to these outcomles. What were the pro- KI)'Ps overall goal was poverty alleviation, for ject's imnpacts? mnany stakeholders issucs of process and dcnocra- tization were of equal, or greater, importance. Telc The KDP MI&E System was designedl to collect sys- progranm's emphasis on citizen involvemeit, inClI- tenlatically informal;ltioln about the programll's progress SiOI, and opCeI and transparent discourse on devel- Indonesia Kecamatan Development Program 3 o(pinrit priorities pro-)iVed the fililedilln Cal bUild- responlld (ptiickl to complainits.4 ing(, blocks tor (dicnocratic practices iII a socictV po olitical cIhanIge arid olflict - I ;astlv, K1)I branded with thrcc (decades of authoritarian ruic, Pllhasc (OIIC opCeratcd in thc midlst of a tuiiulluous top-down planning-, an(l corrupt practiccs. political ali econiomilc backdrop thl-olughoLIt its Capturing the breadth of that process rc(uirc(i the project period: a prolongcd drought; scvcre finan- uisc of coiplelliclitary qualitative anid quant1ittive cial clisis; a political colul followe(d ly thlc first frce Nl&E instrLIMcts als wcHl as coniccrte(d eff'orts to g'enlral cicctionis hcld iIn 44 VC'a-s; tour presi(ieits svsteinaticaUvN (doc-tiiuinet the process. and niumilicrotis cabinet r-CsIhlufflCs inou tol'1- Wars; 'I'hIC opCIe meInIu an,Uld mul.ti-sectoral nature ot arned insurgenciCICs; cthlilc riots and repeated ter- KDI' also 111Can1t thla,t at V!lietV' oftplul7LItS ald to r-orist attacks ariontd(i the archiliaigo, and the be monitored and evaluated for quality anrd cost October 2002 Bali bombinrg. Over onc-thiirdi of clfectiveness. Ihis nicant designring arid imiplc- Inrdonesials provinces had rece-nt expcriciccs with meictiniig conitrol systCeis for roads, water, an(l a coliflict or ure currently in thc mli(ist of civil varietyx of infrastructure projccts, as well as eco- unlrcst. While KDl)' conitiniuied rclitivelv unscathed IIOIIIic atis.lCtIVItICS LCII as WOmIIIIs SalIVgIs Inlld 1;ii from thcse political events, project operationis were scicilies. suspended in ccrtai n troubledi areas for varin,g (;eog,niphic/tscope andz log,listi(s - The slheer nia-gii- stretchics ol'tilime. Insecurity in provinces sutch as tude ariid geogrioaphical scope of KD)' also niadc Aceh, Nlialuikui, Central Kaliniantan, arid Central mlonlitorinig and( reporting, a chiallenge. Indoncliasia is Sulawesi atfictcd the pro(gramn's ability to inmple- ;an arcipliC o got ofvC- 1 3,00() islhinds arid loglistics imeit aid(l iiorlitor activitics. ;ird comlliuilillcati(ons posed challenges. Reachinrg pmovinlcia;l Calpitals wIS eIaSV ciL(enough, blut Visitilln rcmote villages and islands was somlictimliCs dliffi- Opportnifies cult, espccially in the easternl islands andl the I)Despite these challenges, Indonesia had i niiimerous Malu;kus. Also, with over 1,660 con1sultants ' ' ;l(i~~~~~~~avanitagers thalt mladei It coiidICIVC to n1plrCInlciltIllbr nationwide, obtaining standard, timnlxh an(d isefutl the M&L syste reports wats riot always casy. Simply maniagi riga ol - i - ' ' °"In N,sl.,. e, elnoo-acl/ lil( /ItO mon open/ polilcta/ cli/}Z{Iit, - data and reports from ovcr 15,00() villages natioll- A t o o P u ' t ~~~~~After- thc dtowvnf;il of p residenlt SUllalrto In 1998, widce presented enorimiouis data entry and 1luity- KVI' benefited from the more open political cnvi- control issUes for the marianrenienit Information - ronillelit i several ways. F irst, while itimilld(atioll s 'stcml o anld rcpressiion wcre, and still arc, all too prevalent ( ocap/^tisshr oro-/tion ls-ightingS corrrtiOil wasmin parrts of the coulintr there was a grrowing trend( also 11iYgI1 oni thc wi2l(tcnd. lin(loisltlc Ihas traditional - also high on the agenda. Indonesia has traditional- for citizenls and community groups to speak out lv been ranked as onc of the most corrupt corin- anid vo'ce tile'l oicn This a rcater oticniliess tries in the worln and tough measuries to detect, tacilitated the documnentation and reporting of revenlt and de al with cor Crpt plractices hadstakeolder feedback oo the programs progress. bzuilt into the M&L. system. Corrut practi:es Grozth of ciSicit 0lga71iZations - The bur- werc endemic, particularly in government devel- opmlenit programs, and it wo geoning democratization process also paved the wav for a rapid expanision of media and civil socie- anti-corniption strategv and effort to couniter these tendnciccs. KIN) Implemented a tvree- I orLgalilzations. After Prcsident Suiliarto's (lown- pronged strategy that gulded clemenits of project fall, media organitions flourished and the number of provincial NGOs grew exponentially. design as weHl as the N'l&E svstem: (i) eliminate complexity in project dlesignl features andl hind Fo 4 For hirtticr dertaik ln anti-corruption measures, see Secomd flows; (ii) cnstire transparency, of all project infor- grcam:ir Dcvclo n ,pl, nE. Pr 'ject Appraisal Docomnnt, maltion including financial transactions; and (iii) Nlav 23, 2001, Annex 11. 4 Indoniesia Kecamatan Development Program \While (LICstio0S r'Cilmlinl o(Ver the quaIity ;'d I-nd egIl- Armeecent coItained a S2 mzlillin budg(et for lationl of somie of these grorips, hre is littie doubt specific M&E activitics and special studies. that tIlcv have gr-owil ill prominence aidI illtLieC1cC 1 998. Ih'is gi-owth heledr triendotisly in 4. DEVELOPMENT OF THE forning partnerships with civil societyr gr-olips to M&E PLAN mndertake iid(lepliei(lcit nollitoliii(r of 1K )D * ilA'/il/)oo/o /lCo/slI//(l/s-I 1(dlicsi;eiIa Cj)OyS a Dc\veloping the 0\vea-Lll inlon1itoning(r and cvtilution high nationa;ll adiult literacy rate of 87 percenit plaln fotr suchi ani extensive an(id iulti-t'fceted progra-n andl KDP could IrcCrit edIucatted con1sultanltS ad was not an easy task-nor was; it at one-offexercise. facilitators, usIalIly with university degrees. here IL1KDP evolved, and to the program mana-gers' cr-edit were problems ill SOnIC prIovinces, stclh als Paipua resuilts fr-on0 mon0itoring reports ald evaluationls and the ouiter islaintis, xvithi recrulitirno educated influenced projct iml ementation, the aniinta. arid experiencedl consUltailnts, hut iln the majoritv redesign of the progrrams procedures, tralililnig pllns, of provinlces hiriigr suitable staff lid not pose sig- anid the next K DPl)2 (2002-2006) phase. For cxam- niticait problems. That bleiig said, K DlP still hadtd pIe, if fiel llonitolilig or audit reports revealed to invest in staff (Ievelopilient, especially f'or problems withi elite iinterirenice or incorr-ect imple- conimunitv tacili tation anrd medi ation skills, meintatioi of'procedLures, Imnallglecrs contactcd local techicaIICll supervisionl, and reporting. Obtaining'r a statf to investi,galte and initiate remedlial actiols. In good gender baliace among Consultan ts, espe- seveIal provilices, extremlie cases ledi to the temporaryY ciallv for technical staff, also posed anl enorimiouis SuSpeIlSiOnl of aCtivities. challenge.lhr *Rda(tI{I/i-vJ cIIX' O/fcOImI/n/iu aO;;- Another adlvan- 'I'lie starting point for development of the Ml&E, tage was llnoiesia;ls generall ef-licienit coliilililli- plan was under-standirIg a proect's backgr -otiund andici cationls newvork. Telephones, ftxes, post anid design, including its context objectives, oiitpLltt5 e-mail couldI be used down to the district level it' inputs, anl activities. What were the key ob ectives not lower. By Year 3, over 80 percent of district anld outputs of' KDP? I-low did the steps in the proj- facilitators, and(i all provinicial coordirators, could ect cycle work, and what critical information had to use e-ma 'il to send( their reports andl commiaii(:;Cte be collected a,nd at what stages? What research quies- withi thez national level. The widespread knowl- tions or hypothieses must the NV&E7 system address? edge anld uise of a singie nrational laguage across 'To aniswer these questiols, the K.DP M1\4E team most of thle archipelalgo siiplificd reportinig ia-id undertook a thorough review of KDPs background com1111muiciailtions l)etwecrl the central and fiteld documllelIts and L-oan Agreeieint. This docuLilenit offices, review was comilplemenited with (iiscussionis with * /'il gn oveirnne/nle (4e r/odd Lank sn/)upor/ and corni- key K.DP actors in the governmient and WVorld in//it'n/ lo develo/nintr a S/r0on,' M'&Esvst'm- Bank, and provincial stakeholders, to ascertaill TIhe establishmenit of the KDP M&E svsten whhat they felt were the "heart and spirit" of KDP wouldl not have been possible if the VWorld Bank and what thiey hoped to see as a reSult of thc thrce- an(d the Indoniesian govenilIenit hid niot cotiliit- year program. ted financial and technical resouli-ces toward this Wllat everItually (levelopeci ;ItS the batsic M&kE plati effort. /I&E activities suchi as jourinlist iloilitor- or framlework is listed In Annex C. Based on diSCuS- ing, special stLdies, complaint-handling proce- sions with stakebolclers key research questions and dures, and impact monitoring were incorporated i 1-1 ~~~~~intioriiiatioii nieedls emqgcd,( suicli as the followinl(Y into the Project Appraisal Documilenit and Loan Agreemeit. The program establishiedc a 22-personi 1. Poverty and Socio-Economiic Impact M&E Unit and Complalilts -lanidling Office to a. What is the socio-econiomic impact of KDP on develop and impilieimet this system. The Loan houiselhold wetfitre? rindonesia Kecamatan Development Program 5 1b. What are tilc social and c-ollolic benrefits of Ptar/iC;pio,11/;ol: the proc.t's activitics? a. Arc villagers actively participatillg in all sta VUIIIYkg d.. 4J',XL6VY LLF'1J Y Mu tU IL U V U'JUUU(4 UW Target set for lt l L oan Agreement IActu.a! in Yr 3 for Yr 3 Difference for Yr 3/ Comments I. Inputs Number of project kecamatan 725 986 +261 kecamatan +304, if East Timor is included, KDP has worked in 1,029 kecamatan since 1998 Number of villages with subprojects 6,000 15,000 +9,000 Grants and subloans disbursed Rps. 450 Rps 1,600 + Rps 1,150 billion (Rp billion) billionbi!o iI. Outputs Number of subproject agreements >6,000 115,000 +9,000 Percentage of agreed wvork completed | > 80% 95% I +15 Percentage of sites visited by >10% 100% 1+90%, All villages are visited by supervision staff government and project staff Recommenclations frnm studies receivpI Yes Yes Studies and evaluations completed in 2001-2002. Recommendations incorporated into program design. III. Impacts 1. Sub-project impacts Benefits/returns ,>270 6O0% +40%'Xo/V, VVieichted averacie taken from sample of KDP infrastructure Number of beneficiaries 3 million Est. + 32 million 35 million Sub-loans; % repaid by schedule >80% 45% 35% 2. Poverty assessments using X Yes Power and sample size feasibility Susenas household data study completed in 2001. Economic impact assessment to be completed mid-2003 after last round of Susenas data becomes available 3. Governance l l l % of infrastructure subprojects with O&M >50% 86% +36% committees formed In kecamatan with sub-loans l % of UPKs operative >50% I UU100 +50% % of revolving funds with increased capital >80% Survey underway at beginning of 2003 to ascertain final amount for KDP1. Audits (kecamatan sample) 2.5% |30% +27.5% Independent monitoring 1 l 1 Number of locations providing 20 240 +220 (distributed through NGOs) information on kecamatan list Number of trips by journalists to monitor villages 1 20 >250 >+230 Agreed study recommendations I Yes [ Yes 1 Studies completed Source: Schedule 6 of KDP Loan Agreement and KDP Phase One: Final Report Note: Figures for actuals and differences were reported in KDP Phase One: Final Report 8 Indonesia Kecamatan Development Program Box 1. Steps in developing the monitoring and evaluation framework and plan 1. Review all the project documentation. Understand fully the project background and design, especially its objectives, outputs, and project cycle. 2. Meet with key stakeholders such as the government, implementing agencies, fieldworkers, and World Bank, to find out what important information they need from the program and what research questions are interesting to them. This can be done through key informant interviews, focus groups, or workshops. 3. Decide on the key research topics and hypotheses the project wants answered by the end of the project. Determine the key questions and issues that the M&E system should answer. 4. Determine the indicators for the project based on the inputs, outputs, objectives, and research questions mentioned above. Check that all the indicators are measurable, relevant and gender-disaggregated when possible. 5. Determine means of verification or tools to collect the information for the indicators. Ensure that these tools are realistic given the country context, budget, and human resource capaci- ties. 6. Decide who is responsible for collecting that information. 7. Determine the approximate time frame for collecting the information. 8. Develop rough budget approximations for each activity. 9. Ensure that there are ways for monitoring and evaluation results to feed into management channels to improve the program. Link the M&E information to decision-making forums and develop regular reporting flows. 10. Review periodically the M&E plan, perhaps every six months or year, to assess what is rele- vant or if additional research areas need to be added or taken out. One of the most common mistakes made is when managers ask to collect a large volume of information but it's not clear why that information is needed or what purpose it would serve. If the information is not used, it may be wiser to stop collecting the information. * Induependenlt journalists monitorin 5. COMPONENTS OF THE MONITORING SYSTEM l vatv/ aonionud Iimpac t evaluation stud 5.1 Field monitoring and reporting by * Tcclhnical infrastructure and economic activities government officials and consultants evaluations * Au(dits andc tinancial reviews KDP is a grovernment of Inidoniesia program. KDP * \Vorlcl Bank supCrvisioll mI issiOIns funds are uLbflic fi nds, and government authoritics have a responsibility to cnsure that the program pro- ceels according to its principles and procedurcs and that finds are used appropriately and cffectivchW Government officials at all administrative levcls- natio(al, provincial, district, kccamatan, andc village- are responsible for monitoring KDI' One weakneiss of thc reporting system, however, has been that Indonesia Kecamatan Development Program 9 M&E System Principles and Functions Mechanisms To check regularly project progress through Examples: the routine and systematic collection of Consultant & government field reports/MIS information. Financial supervision Monthly independent reporting from NGOs & journalists Complaint handling mechanism Case studies To assess the impact of the program e 4,600 household impact survey Case stLidies Audits Bi-annual World Bank supervision missions To gain a variety of perspectives from different Monitoring by implementers, such as consultants, sources communities. o Monitoring by independent actors such as NGOs, journalists o Participation of other donors and NGOs in World Bank Supervision Missions To use quantitative and qualitative methods Consultant reports, MIS and household surveys provided to complement research findings quantitative data o The quantitative reports were complemented by qualitative work in the form of case studies and descriptive reports from NGOs, journalists, complaints & grievances reportinig to the national lcvcl occurre(d only sporadi- intenitioni was to reiducC form11s and rpcporting Ro the ca1ll\ usually if thcrc was a probleimi in the field or mlillilnlimum amounit needed so as not to overburden specific cases rcqLuir-ed follow up action.T his lack- thle field staff adaisical reporting5 was a programn weakness that nceds to be corrected in the next phase. For KPl) 1 monthly reporting fOrmatS varied depending Phiase 2 there: will he a grea1ter institutionallizedl mon- upon the feldc( position, but they generally covered itoring role for (district parliaments, g(ovenirnet the following topic: Activities comnpleted( during the past montil and coordination tecamns, an(d local village councils. Thesep ' ' ' ~~~~~~~~~plans for the followingr mioiitli groups wil reccivc specialized training in monitoring . r 1n1 c 1rtnr I Status of activities in the liro .cct cycle and reporting -I'scipioIo I ODescription of infrastructure, economic andi social Most of the reporting for the programi was donie activities througih the monitily reporting ot KDI' Con1sultants o Financial information at the national, provincial, district, and kecamratan L Level of commurity participation in activitics levels. The consultants were rcsponsible for writing . Informnation dissemination and transparency monthily rcports. Using the M&E plan and key miidi- Traning and activities with vilage councils, keca- cators as a rcterence, the M&E UJnit of the NMC matan financial units (UPKs), and other local devcloped( standard reporting forns and tield tested institutions thcmii in several provinces before finalization. The o Coordiiniationi workl with government coordiniation formis incorporated critical performanice and(n ma- teams, parlialments, and other government a1gen- agcmcint indicators (Anncx C, Kl)P I&E cies Framework, Means of Verification). The program ° Coordination with NG(Os, journalists, or other originally started with som01e 90 formis but by Year 3 civil society groups the formis were revised and redclued to 64, incluIding o Problims enicouLitered and proposed solutiolis adiniilistrative, financial, and narrative formats. The o Any othel issues 10 Indonesia Kecamatan Development Program iAttrachmnents: reporting firnis, photos, coliplaints lciils So tlc reClilmailnd Silellt. Falcilitators SO(liC- received, etc. ttiies expicrienced physical i ntiilndittiOli in the lic( l toil)l villil- c litcs ()I- o)ticl- aIctorls. IN'lorlCo)ver, Field stafl senlt tilr ntllolthly' r-cplo-ts directly to thilrl' l I :If ' ' SfOilC conlstiltallts fcarc( that IiinliCIrtartioit ctipallis for forw'ard(lilln to the Ministry of I lile r)leinS would be seen as a lcti( of thetfiro Affairs (PMi 1)), with copies to tllcil- local ficld suIpeCr- own lpoor wo)rk pcr-formialicc, andl(i tr-ic(l to hlie(l v'isorws and the NMIC in jakarta, whlicll waIs esl'pOllSi- pr-Ob)lCemS. BCetter COillninu,iCaltmon0S bMetwCeen the lble for tcchrical supervision of* ficld consultants. TI na;tinllel and1 teiel helped to improve thiiS r-C(rional coordinators aIt the NIVIC rcvicwed the *> llUt~~~~~~~~~~~~~~~eriltlivC C11\'II'0111lilt SomiC\}ewht 13)v Y;1r 3.. reports and quantitative dlata was cntcircd iiito( ;1 aowver, plroect manacrers ncd( to ensure more \CC1sESS f\I;lamiagecmnenlt Ilnfitrmna;ltionl SV'Stemn inl o911C1 le l replortino andi(i colmlmiuiicationt channels (idur- Jaikarta. The NNMiC and PMID tisedl the monthhl I I ingr the ncxt phiase. reports to assess progrcss in the piroject cicle, iolli- Eliteril'ng project (fatal centrally ilnto the AlI IS svs- toI- financial disbursemenlts, and keep track of prob- tem in akta has meant tihr contl over data lemis arisinor in thie field. 'I'he NMIC also encourae a u pd *7, ge qu' llityl bLut pILCC(i l ltlCA',V bUrdenI Oll MIS stafft at field Staff to rCviCw theil- progress relrolts at tilc th {> ' ' ' ~~~~~~tiec ceiitl-;L l Ivel to c leall the (l;ltil andI' lltowl\' Up1 mllonlthly' pRVllC;I aIndl diEstrict level coordlina;ltionl Ilmissing rclports and(1 incomplete informationl. Centralized datta nianir-cmeit has also iimeanit that Any reporting systeim dealing witi oveI- 1,600 conI- provincial and district COnlSUltailntS lack direct Sultant reports each Illonlthi is daunting uilr thle resi()nsililit for managing the data (they rceivedl best of circuiimstainces. W'hide the r-cportilln sVstCeIl copies of the reports) aind have ilot il-acde optlimal fiinictioine(d anl(l the reports p)rovidedl very uisciil use of it br their ow I project anllgcIlem t Under (lu;llait;la in fnlormailtioon p s (14)J'2 thie MIS hiinction will be decentralized to (]luanltti tI ativ d 1i(L c1ualt'itVCitn f'e plt illrogress, the regionls. there were also several (liff;cultics: Consultant reportinlg was solietimiles sporadic, *< KDl had great dificullty controlling tihe economic incomilplete or late, especiall'y fiomil rclimote loani portion of the portfolio. Th hroughlout thie pro- provilnces. Also, writing skills were ofteil lacking, gain, repa;'nlent rates were low, averalilg about acld SOille rei-ts were dlifficult to undlerstand. 40 percelt. AccouLitinig systenils for the loans were 'I'le central oflice MIS unit spent consi(derable rclinentar ald depended for the most part ol tille at the eid of each iioilitIl or project ve'ar col- tie experienlce local groups ailready had ill manag- lecting outstail(iilig reports troil fieldi con.sultants. ingr ilcrofina;lnces. Financial Uiilits at the keca- 1y mid-pr-oject KlP had inlstitutcd policies tying matail levcl (Ul'Ks) fotuin(d it difficuilt at timlies to salaries to re-porting,c wlichi resulted( in a 30 to 40 coillplte forms tor ll reportiig. Ille nicrofi- perceilt illCSC;ISC ill l'Cpol'tillg overall (witll Wilde fl;ailcc p)ortiol, including tIle reportilg formis, has p' iCi;lcvrease miolis) rBportiml cndlot'thc)progrr;l beein coiilpletelv redesigoiedc for KDP2. provincial variations). By, tile cild of tile programl, aIplroximilatelty 90 p crcemlt of field conisultanilts were It is anticipated that witlh thi clecentralization of the reportinig rcgularly, b1ut tlis required substantial MIS for - DPs ilext phase, iiiore effective r-eiiiforceiiiit atndl followv-tmp, r(iilorceileiltailcIfbllo-u1 and stronger supervisioln a;nd enltorcellent Illeasures, Althoughl the overall political clialtc was illore reportilg should improve openl and f-eccdomi of speechi had gecierally inicreased siince 1998, it was sonietifiecs clifficult to 5 C P encourag;e whistle blowinig aild ulrgC Col1suIltanlIts oX 52 Commun ty Participatory COil,illumlity illeinbers to raise corlrplaiilts or prob- Icms openly Indlonesia l aid a iolgr history of TIhe best type of nioiltoriilg is i(lloiltorilig coilduct- intimindatioin and retribuition amld soile ed cbv coillmnumiltles themiselvcs. Communiiiiiltics take lndonesianls felt that it was fitile reportinlg prob- ownlrsill otIte process wil the are respolsible Indonesia Kecamatan Development Program 11 or monlitoring' pl'OjCCt alCtiVitieS. is promotes paIr- nd East ava N (0s involved in province-batsed ticipant lcarning aibouit the programn and its performii- mlionitoring(r helped at number of villages, special anlcl, aiid CelIllhaCs Undl-rstanding ot othicr- 7moiups, and tcears conldlict collinillity particic - stakeholdcrs' points otfvicxw. It also increases the like- tory moniitoritig.r I he! hlcipcd villatgcrs (lcci(de lilod00 tilhat evaltl;ltioll infortlilatioll will bc used to What (ItiestiOlls Were imnportant to thelli about imiprove procct piertoriniliince. KDP, how to coliect dtata to answer those qeics- tionls, and helped villagcrs analyze corn iLinitiv i\'lonitoringt57r andt e'valualEltionl b! outsidle parties of'tenl Bnding(trs. NGO l(lfcilitation has beenI a sUccess toCuIS on issues imiportant to doniors aInd imple- storv in several locations, and1C the activitv will be nicitcrs; bLut colliluitielils iav have other issues that they valLie and wish to imionitor. Comimnitiyv partici- patoriy mlonitorinig allows tile colilmilnillities to become the (Iqlestion-makcrs, the collectors o tifor- Docasentatio ,. iation, and, ulltilllmately the enid uisers. Over the vears, KDI encouraged differenit kincds of BecauSe OfK1)'S large scale and ambitious goals for improving local goverinance, it was Important that *orililill.11tvt partlicipatory mlonitoring:r p iio m n- the pro7rami research and ca-refilil' (lociiulent field Momitorimig by village councils (lBPDs): I .aws for i the ciection of village counlcis were passed in 'rt e fN I t staf mem e 1999. rior to thi s, village chief. gencrallv responisil Ior coniducting researcih and writing case fl of5ro \ a ~sthl(iiCs oll 1()) P, ssistedI bv otlher N Mo St;ltt f(or appointed the villages assembliCs (LVIIDs) anld vtl- lag-e comuiiitty resiilicnce boards (I NKM Ds) from larger or more technical case stuidis.TIhc case stutiies village elites. Biy the midldle of KtDl' however, vil- were important nitoring and comication lagers were democratically electing their represen tools because thcy provided ricih (cscrip tive inforiima- ticnlim )e s ntio ancd analyses on multiplC themiies related to KDP have resposibi for oitorig KDP activities implementation. The cse sd wor provided ual- a;t Al stages: socialization, planning, implernenti-t itative in-(lepth infonination to help explain some of tio, and maintenance. The BPDs select members th inis from th utitative survevs an t imonthilv reporting data. to ioiiltor eachi phalse. Results of thlis moniitoring are theln disciussed at counicil meetings or fed into Case stuLdics bean with NMC M&E staff mcetilngs larger village Imieetings. to discuss possible topics or themi es, whichli usually Monitoring by special community groups or arose from field reports, suipervision missiois, or geni- teamns: KDP encouraged commllunlit' monitoringt eral feedback from KDP staff The case study staff groups in each village durinig Years 2 and 3. compiled a list of topics and (developed rougth action Comllillities were encouraged to form special planis and sched(uiles, then conitacted regional mnan- teamns or groups at the comanduni level during agers and provincial coordinators to brainstorm pos- the vilge meetings to monitor KDP. Tbhese com- sible locations to research the topics. In general, the munihty monitoring groups w.ere indrependent ot case stulies covered at least three different provinces. the village implementation teams. The community Once locationis were deci(led, staff developed general teiuin members shared responsihilities for check-inig research plills, including question guides pertaining financiallCI; accounts, monlitorinig banik transactionis and list of persons to interview in the field. In the and material purchiases or rentals, visiting stippliers field, they' used a combination of methodologies to to confirimi the costs of goo(ds, and monitoring collect information, including: (hirect observation; suLbprocct activities, includingr infrastnictuirc CO- fOCUS gRoUp discussions; oey' informant interviews; mini-surveys, and rapid appraisals. The staff adjusteCd ° Community participatory monitoring facilitated research plans and question gides as needed. Case by NGOs: in several provinces, including Achil studies averaged approximately' two to three months 12 Indonesia Kecamatan Development Program ofl icl(lwork follo%{'cd hby a m; othi of anazlying data I -agiuagc was kcpt sliipilc so that v'illage c 1ommun111i- ;1a(1 reporting tiIl(liiigs. tics COtOI(I uLndCrstan1d thc stories. Lt xccpt r ra imio c lcigtlhy study on(1) j)' imp1dact Kindc 1 1DP , the roga published and distrib- On locl govcrnancc, rti case: studies were puirp()scl" uttr(e CIe,ilt C:sC Stti(hiCS oi,: hbncf andlimI iiltcdi to ei,ht pgrcs so that thc' \vcrc niot Lessons fromll K)D) Year 1 too 6hcavx" olr the 1X1)1 audicncc. 1hc idea was to P articipation of \oMei and11an thc Poor captire lessonis fi-oml iifcrcnt project areas aind( share Conltlict Rcsolutioll that information \vidclh \with pro jcct stakcholdecrs. Information Dissemination and tlc Rolc ot thic Vildlagc Failitator Box 2. Steps in Conducting Participatory Community Monitoring NMC training for NGOs assisting with independent monitoring included a module on community participatory monitoring and the steps to begin the process in KDP communities: 1. Call a general meeting to ask who would be interested in participating in the monitoring exer- cise. It is important to check if villagers actually have the time to participate in this activity. 2. Once the community decides who wishes to join the participatory monitoring, the facilitator meets with the group at a time convenient to them. 3. The facilitator builds consensus on the purpose of the exercise, clarifies his/her role, expecta- tions, and reviews the schedule. 4. Community members then identify the monitoring and evaluation questions they want answered related to the KDP activity. Questions come solely from the community; there are no predetermined questions or forms from the facilitators. The group then brainstorms on ways to collect the data and who is responsible for data collection. 5. Once the data is gathered, the group meets again to analyze it and reach consensus on find- ings, conclusions, and recommendations. 6. The group then reports to the larger community and KDP field consultants and together they prepare a plan of action to improve performance if needed. In Aceh Province the PUGAR NGO provided independent KDP monitoring and facilitated com- munity participatory monitoring in four villages. In Lok Nga Village, for example, PUGAR asked the KDP Kecamatan facilitator and village leaders to call a general meeting where PUGAR explained its purpose and the benefits of monitoring. At the meeting, the villagers appointed seven members for the community participatory monitoring team representing women, youth, the poor, and community leaders. The team decided on each member's responsibilities and then chose their leader, a woman named lbu Mardiana. PUGAR met with the community monitoring team each month to review progress. The monitoring group reported back to the village regular- ly on project progress. "I encouraged women to take part in the community participatory monitoring and now they real- ize how useful it is," lbu Mardiana says. "To get a good quality road, I myself sat on top of the roller machine and watched the workers work." Another female member of the group adds: "We have a very good quality KDP road and it is very cost effective as well. We controlled the quality of the materials by checking it when it was delivered by the suppliers and we did this every time." Indonesia Kecamatan Developmnent Program 13 o Conainuitv Participatorv iNloniltoin(r communnilitY financial Ullits aIS wcl] as supcrvising aindc o VoIl1Cil I ,cad;Cs in IKl)P adllditing KD DP financcs at the local Icvel. (For Trmis ° I%lnfStlu`tIrC 13est Pralctices uinde(r K D)P of Relerencc tor this Unit, seC Anncx G) o K D)P's Impal act on I ocal (overnancc in Ninc Villagres Ihc unit was nOt estalished as oly an intrl auldlit unilit becasIISC pojct pr i fil imCt kcit thiat it Itach case stl(lv was sent to 3,200 consultants, g'ov- would hbe more lcneticial for UPKIKs, I 1K< Nl)s, a ndl cniilenlt officials, local parliaets, journalists and economiIc loan groups to be mCntored in financial NGOs att the national lcvcl and in the fic(i. Bv' read- skills. Atiditors woul(d have focutsed chicfly on poinlt- ing thcsc case studics, it Was hoped that prograill ing out problems rather thaln conlcenltr-atinig onl wavs stakeholders, CspeCiCIla. ConlSUltant;s, woul(d lcarn fromil to ilimprlove skills. Thus, the unit was ConlCCiVC(i as a the cxperiences in othelcI) KDP areas. group ot 'roving accountants who woll d I-evIC\Vi nfinancial records, biut also provide muchI-lineeded(ic( Results of a Julne 200()1 survey luesti airrequest- financial trainingZ an;d i-tor**i*g. inlg fccdback on1 the case studics iniilctcd that rcad- Crs fotiuid the stu(dies to be verv informnative and L)cspitc initial delays in recniitilig staff iaind finalizing useful rcference guides especially for providing infor- the Ulnit's scope of work, it acconiplished a great deal gines otherprovinces arvdnd h liowrhyi-itl eradS Illmltmoll onl problems ia oticr provinccs and liow tle) in a little ovcr a Ycar and strengrthened KDl's fina;n- were solved. Thc case stuid(ies also gavc thieCmI a i-ichier cial ovCIesight fIllnctioln. The lnilt auditCd and trallicd understanding of howv nilcs and procedlires wcre ptut tinancial and programn staff in 13 of the 22 KDP ilnto practice in the field. Respondents also fouiii(d the provinces and 140 kccamiatan. It visited; an average of sililelC Ilangru'agC of the case Studies Casy to uil(lCr- 36 percenit of thc KD)P kcciainutain in each province, staild. IKI)P will conltiniuC tiesC caIse stut(lies in the sampling two or threc villages inl each kecaniatall. nlext phase. Tlhc teami also playcd ani iiimportanlt supportive IolC ill fornmal World Bank supervision inmissions. 5.4 NMC Finacia Sunvison* and *rhTe ulilt's findini£gs varied frioim provincc to province. Ta~inngUnit In provinices sucih as Central and Eastjava, financial Financial Imonlitorinlg was an imilportant part of the management WaS on the wle, tairly sound, with prOtrRlillS overal IllOllitOrll 'r ICtiVitiCS.e(JiVen rc,gular ConlsultE1ant suplervision and solid bookkcclincr Programs mvrl ioniitoring( activities. Gi1ven II Indonesia's endeic corruption, it was importan-t to by the UJlPKs and I KMI)s. In other provinces, sucI suLpervise financial transactions and procuLrement as North Sumatra an South Sulawesi there were closcly, especiallv at the kecainatan and village levels. systenmatic problemils related to mlisuse of in(ds, lack Coin iu;ity participatory mon iotori ng played a crL- of financial oversight from district and kecailatail cial role in overseeing village finances and procuire- consultants, and lack of documnetation for revolving mnent but there was also a nee(d for the progrram to fiinds. After eachi trip, the unit presented its findings provide its own financial supervisi(in ail(l dlevelo to local consultants an(d government autlhorities, and community capacitY for finai cial nagement the national level. Problems uLn(covered were sent to the complaints handling unit's consultants an(l gov- The NMC established a five-person Financial eminiient for follow uip as part of overall KDP pro- Supervisioni andlFraiiiig JiUit in May 2001 to 0rami managemnent. For example, in North Surnatra improve the financial skills of the kecaniatan finan- and South Sullwesi, the unit's report led to the tem- cial UnlitS (UPKs), village boards (l(K\'I Ds), and eco- pomarv suspeisioni of project finanlicincg unitil the iden- nomic loan groLups, andi check their financill and tified p1roblems were corrected.'The unit also adcinillistrative records at the kecainatiai and( village provi(le(d follow-up financial training in problem Ievels. 'The Unlit served a dual fiun:Ction of ploviding provin(ces. miuch need1ed(i oii-the-job training and i menatorilng to 14 Indonesia Kecarnatan Developnment Program 'I'lte NMC also carried out two confie(lntial sulrveys inappropriate intervention fi-oifl governililenit officiais USilrg ticld consultants to ascertain li consultants or 1KDI consultants, and unfireseeable events su:ch wCIeC bcing paid according to tleil- contracts so that as natural disasters (foircc mnajcure) caises (Tiable 4). thic programill ;inalgferl could take a.iction1 (o1 1n(on- 1K DI I-eceivC(l the l;g-eCst nuilmiler fioilo 1K)' field( peftormling or corrupt consultant companies. Tle conisultants oi- Nl\IC statf'who visited the field, 'hlie survcvs foilu(I tailt several companies were holding reg"ulalr mintlv reports from fieI(l coisiltaiits andI baick on transportation and(i othcr allowances to field reports fr-oml NAIC stafftlvsits ItpoXi(le(l useChll consultants. INMD notified( and watrined( the compa- intormatioin on problIems in the field( and w'hat rmiced- nies but eventually fbOur oftthe consulting firmils wci-er cd to be followed( up by the proper parties. Letters terminate(l. were also sent in by communit meminbers, NGOs, ald ano1NinouislS the uIilt also tracked the ncwspa- 5.5 Handling Complaints and per reports froml AlI reporters (see sectioin on Grievance Procedures An1/1 S journalist moniitor-ing) and ensuire d that problems (lescribed in those replorts were comiiiinuni- KDI) establishied( a complaints resolutio(n process to calted to field staft and governiment officials for tol- ensure thiat comilmunlities had a foninl or outilet to air low-up. Regular supervZisionI missionis by the World complaints, and resolve problems. Complaints and Bank, NMVIC, and PNl\D were also uselil soLirces of questiois about KDP could be dilrected to any' KDI' informalation on field problems. tlie Handling fieId consultItant or government official Conoplaints Complaints lJni,t witil assistan;e fiom IVI IS staftf could also be sent to a special post office box in enteredl complaint and grievance inforinoationi into a Jakarta and the KlD)' national p)roIect secretariat central ACCI'SS database in Jakarta. (See Annex F oftice. 1ro'ect brochures and posters includ(c(d infor- for Complaints and Grievances database fields). mation on where to sencd coiimplainits. Five staff at the NMC, the Hlandling Conmplaints Unit, were respon- As of Jul 2002, 1,900 problems or complaints had sible t'or ensuring thatt complaints and uestions were been reported-an average of 45 per month-of alinswered promptly and investigated fuirtlher in the whicI 49 percent ha been resolved ail 51 percelit fiel(. 'I'he unit coordinated withi KDIP fiel(i minagers wvere still in process (Itble 4) The most common and locl gvenmeto ialtenureprmptf- problems or complaints related to violations of KDI' aliici locawl grovernmilent otfficials to CensLire promipt fol- low-uip action in responise to comilplalints. KDI' coml- priniciples or procedures (52 percenit) followed by plaints procedures are illustrated in Figure 1. mishandling of funds (38 percent) and force majeure (5 percent). Force majIeure cases were tracked in case Thlere wvere four general categories of complaints or follow upl actions Were necessarv or finds must be probleiios from the field: those regarding the KDP recordled. Some 77 perccilt of the problems or cases process an(l procedLures; misallocation of KDP fli uds; wer-c reported by consultants followedt by NGOs (7 Table 4. Summary of Complaints Received under KDP1 (January 1999-July 2002) Types of Problem or Complaints Number % Resolved Still in Process Violation of Principles or Procedures 987 52% 499 488 Misuse of Funds 724 38% 304 420 Intervention 75 4% 51 24 Unforeseeable events (Force Majeure) 102 5% 58 44 Other 21 1% 17 4 TOTAL 1,909 100% 929 980 Indonesia Kecamatan Development Program 15 tFiure 7. KDP rievance nd Redress Proedus This diagram was adapted and revised from an original complaints procedures diagram drafted by Enurlaela Hasanah and Sadwanto, NMC in 1998. ( Complaints to P.O. Box ( Complaints to consulants Complaints to (local or national level) Government Officials Handling complainits (HCU) staff open or receive complaints Register and enter into database Solve through ec v Solve through consultant channels Dgovernment channels Crosscheck/collect supporting data; consult affected communities Compose/draft Suggest type Prepare official official letter of action field trip Implement action If problem solved, Receive feedbac If problem not solved, record final outcome from field suggest another In database course of action and close casem 16 Inidonesia Kecamatan Development Program Table 5. Actors/Sources of Information for Complaints Source of Information Number of Complaints % of Total Complaints Consultants 1,468 77 % Monitoring NGOs 140 7 % Community members 99 5 % Mass Media (journalists) 41 2 % Other 161 8 % TOTAL 1,909 100% Sotirce: KDP NMC Complaints Handlinig Ulit database perl-ccnt), corr muinitn' lcttecrs or reports (S percent) cases. Only five KID corruption cases went to andc AlI or other joturnifists (2 per-cenit). While court. lhese cases werc in Lampung, Central Java, NGOs and media accounted for onlv 180 caises or 9 North Sumatra, and two in Nusa 'Inggara 'I'imor percent of the total number of complaints receivel, (N'l'). As of June 2002, threc caises in Central these civil societ groups usually plalyed an nport;lit Java, Lampung, and N'lI were finishedi and the role in t-ollowing uip and tnonitor-itlg caises. lor examlz- ctilpirits were prosecuted suCcesstlitl Two cases, in pIe, onlcC grievances sticI as mistisc of fiincis were N'1 and North Sumatra, were still in trial. rieported to KDP/l Al journalists followedl the stories While these couirt cases were ftew and far between, andt reported1 upon thlemll as the caIses unfoldled( the "victories" were widely pLublicized and sent a stronig signal to the public thalt corruption wotild As'l'able 4 shows, it was muchl more dlifficult to be punlishled in KL)P' resolve cases related to the mIisuIse offtilt(ds andC Cor- niption than those involvintg procedural violations or Under KDP 2, there will be more trainlin, for coin- inter-vention by otitside parties. Anong the cliffictil- sultants on dlispute resolution. Also, the Hlandling ties ftaced in resolving problemes in genel-al were: Complaints fiinction will be decentralized so that ° Consultants were sometimies not creative enougih there will be a complaints officer in each region, in seekingi alternative solutions or persistent in which should itliprove CornmmuniCations with provin- solving problems. Better training and clearly cial and district government coordination teamis anid definied( procedures arneetded; resolve grievances nore expcddtiously * COn1SUltants had (ilffer-itig perceptions of problems In response to several coniption and legal disputes or how to resolve them. arisin(, in KDl; areas, the p rograrn initiateil to the eni(i of the trid. I' rated into the desigil of projects so thilat rnanilagers cai euse the results to adjust or redesign prograil coilpo- Mveaknwesses.: nenits. KDP's orilgina.l program desigil inllctided * IJlic abdlltics otf'ourmillsts wcere Imilltcd desprite the o r t r impact studies, includiillg the collection of baseline fact thilat the KIDI' conitratct pald for orientation data, aid technieal evauatioins to assess progari anid refreshler trainiilg for the jourilnalists. llere illla tt were problems witli clarity in xvriting, accuir-ac:y, balalnced reportiilg, cross cleckilng soUIrces, aild u,ilfailliliar'iy' witi K.DP procedures. 6.1 Impact Evaluaion Study * Soillc K-DP villaifges are located in reillote atreas, so In 1999 KDP coitracted tie Deillograpilc Iiistitute it W;I1S CifittCLilt for the 'ourna;lsts to cover thlicil of the University of Indonesia (LD) to coIldtLiCt a111 * .lourmltlists didlnot gset out to tile fied as often ias imiipact stuid lookling at two ilajor issues: KDP's thev slhoLild lhave. t ti-ey Sil ld Ilave. ilpipct on louiscliold Welfare, and its ililpaict oil coi0- * Investigative reportilng by the jourlnalists xWas lull- -l11;.1l1 oqandzatione ea p Develotions. Indonesia Kecamatan Development Program 21 I[or the hlouseholdo( wcl fare comIpolient, 1.1) co )dtlCt- to participate in the thir(d and tinal vear. I 1) use(d the ed an initial fiaitsi'ilit-v study to dIcterminie if it WaS Ci'ea 1 villageS withi the villages in the siaime kecit- actualkl' possible to evaluate quantitatively the imilpact mattall that dlidl lnot participatc thart vcir as contr-ols. of K l)l on povCrty, ilcOiics, and livings standards to th-Ie sv as tike questionls related( to: zr,CIl the av'ailable datal Ih11e sample had th be suttl- nt-iC large] - C to (listi 1gu ishl relaltively smal;l difthrenices I lo)isciholdi socio-econioiimic characteristics iil per Capitia iiicomiie growth rates bhetween those ° Coin ii participation in the arious phases of areas benefiting fromil the program andi a set ol coni- K[1' (planning, implementation, maintenance, trol areas. lurthcrmorc, the control arlcas had to be cost-sing) as well as other d'elopment pro- slufficicntly comparable to ensuire that the maeasured rcts di ttfre-niccs were really duie to the programi itscif The The role ofldifferent groups in the projcct prOCeSS study rccoiii ndiciledc the appropriate samiple sizc and (villasge coulicil, womenil poor) Information shin lr, andtasaec the luSC oT ofticiall sociO-econIom1ic statistics (Susellias) "I tt ansaecy dlatascts for 1998-99 anid 2001-02 to colmpftre daitat 0 Impact of the subproject interventions P Problems cncounltere(d and( handling of complinlts before and aIfter K )P 1 intervenItioIns, and inl DR)- p o-rall1 and(1 conitrol areals. The W\/orldl Banko statt in~ o General level of satisfaction with the projCt jakarta xvill comiplete the study of the Susenias pancl In some enumerationl areas, field researchers followed daltalsets hIw midl-2003.Sl up the (qu.anititativc survey (questions with inore prob- or the conunit orgnizto cop t, ulittiv techniues and uestios etailed conducted a baselinle 4,60)0-househ1oldl surVey in pro- fieid notes fromii the researchers were very helpfil in gramn and conitrol areias and followcd by another exlaining SOme of the household survey results and I s , - - . describinu, the details of soimec probleimi issues. pianecl survey looking at impact after one prolect cyclc was conplete. 1The survey followed a quasi-experi- Results of the houselhold survey were mixed. KDI' mental (lesign using a double differenice metlhodolo- infrastnicture investimienits were very popular an(d gy comparing the groups before and after the beneficiaries expressed satistaction over the results. interventioni. LD selected 74 pairs of KD)P and nion- TIhc perceive(d benefits oftthe infi-astnicture develop- KDPI enum1ieractioni areas and helid interviews with ment were lairly equitably distributed to p eope 2,600 househlol(d memilbers andc 2,050 local leaders in a:cross iicoIlie groups. Compare(l to colitrol areas, the seven provinces of North Sumatra, South community leaders in the KDP areas had signifi- Suma1witra, \Vest Java, Central Java, East Nusa cantl)' more positive perceptions regarding the Tenggara, Central Kalimantania , and Central iipact of K I)lP infrast' iture projects on household SuLIIiWCS'i. incoelic, access to markets, schools and health fiicili- The identification strategy ofseleing programn and ties. VVomen and tile p oor in KI)P areas were also control areas tookl advanitage of KD) P's stagr(rered more involved in project activities comilpared to coni- designl in villag:e selection within kecalmaltanil. trol areas. Lastld KDIP beneficiaries had greater Accordng, to KI)'1 procedures, for kecamatanu coi- Ikowledge of the role of village councils, kecarnatan taminillg more thanil ten villa