Paper 44794 5 System of Monitoring and Evaluation The World Bank Policy Paper extracted from the World Bank Study on Review of Effectiveness of Rural Water Supply Schemes in India, June 2008 June 2008 Paper 5 System of Monitoring and Evaluation M onitoring and evaluation (M&E) are Bank at the request of the Government of India. important performance management Data was collected from representative schemes tools. They help in ascertaining and beneficiary households spread over 10 states whether the policies and programs are meeting and a set of indicators were developed to judge the objectives. They can provide useful feedback the effectiveness of schemes. information for taking corrective actions in the design of policies and programs. They are also Indices of Effectiveness the basis for accountability in the use of program funds. The study used 17 key indicators of water supply service, on the basis of which four indices of The monitoring and evaluation of the rural water effectiveness were constructed relating to supply programs in India is being done mainly reliability and adequacy, affordability, through periodic physical and financial progress environmental sustainability and financial reports, visits of area officers and other sustainability. The list of indicators used is government officials, and evaluation studies and given below: sample surveys conducted from time to time. The main purpose of the monitoring and Reliability and Adequacy evaluation activities is to find out whether programs are being implemented as per their Liters per capita per day (lpcd) from the objectives, and to identify bottlenecks, if any, in piped water scheme in summer the implementation of the programs. Lpcd from the piped water scheme in other seasons The present M&E system needs much Proportion of household requirement of water improvement because it has a focus on outlays met by water from the scheme in summer and infrastructure building and does not capture Proportion of household requirement of adequately the outcome of the expenditures water met by water from the scheme in being incurred and the impact on rural people. other seasons The latter aspects of program performance have Number of days of water supply each week been captured through the indicators developed in summer in the 10-state study on the Effectiveness of Rural Number of days of water supply each week in Water Supply Schemes undertaken by the World other seasons 2 Paper 5 and System Evaluation of Monitoring Hours of supply each week other than Monitoring and evaluation are important in summer performance management tools. They Time taken to fill a 10 liter bucket help in ascertaining whether the policies Time spent on water collection each day and programs are meeting the objective. in summer They can provide useful feedback Time spent on water collection each day in information for taking corrective actions other seasons Incidence of supply system getting affected by frequent breakdowns Household assessment of water quality based on the following criteria: The O&M cost per household served as a ratio to the average income of standpost users · Percent of households that consider the supplied water to be bacteriologically Environmental Sustainability contaminated · Percent of households that consider the Incidence of source drying out during summer supplied water to have chemical problems such as salinity, excessive iron, and so on Financial Sustainability Affordability Extent of O&M cost recovery (the O&M contribution made by beneficiary households The operation and maintenance (O&M) cost divided by the O&M cost) per household served as a ratio to the average Proportion of private connection users income of piped connection users regularly paying water charges 3 Paper 5 and System Figure1 Indices of Effectiveness, State-Wise Comparison Evaluation of Monitoring AP = Andhra Pradesh; KAR = Karnataka; KER = Kerala; MAH = Maharashtra; ORSS = Orissa; PUN = Punjab; TN = Tamil Nadu; UP = Uttar Pradesh; UTTK = Uttarakhand; WB = West Bengal. As liberal criteria have been used to assign the terms of another index. It is, therefore, not top score for the various indicators, it is expected possible to come up with a general ranking of that the value of indices for the schemes would states in terms of effectiveness of water supply be high (8 or higher in a scale of 1 to 10). schemes. The best that can be done is to rank the However, the actual values of the indices are states in terms of each of the four indices of found to be generally lower than expected. effectiveness. The index of reliability and adequacy does not show much variation across A state-wise comparison of the indices of states. The value of the index for Andhra effectiveness (Figure 1) reveals that the ranks of Pradesh, Maharashtra, and Kerala, exceeds the states differ from one index to another. Thus, average across states. On the other hand, the some states are doing well in terms of one index, index values for Punjab, Karnataka, and Uttar but their performance may not be as good in Pradesh are the lowest among the states, 4 4 Paper 5 Paper 5 and System Evaluation of Monitoring indicating that these states have a relatively The present monitoring and evaluation worse performance. mechanism needs to be revamped to independently monitor the inputs, The indices of affordability, and environmental processes, outputs, and outcomes of the and financial sustainability exhibit greater inter- rural water supply services and disclose state variation than the index of reliability and its findings to the public adequacy. In terms of affordability, Maharashtra, Andhra Pradesh, Orissa, West Bengal, and Uttar Pradesh have a relatively better performance, while Kerala, Punjab, and Uttarakhand are at the bottom. The performance of Uttar Pradesh, case. Thus, Maharashtra may be regarded as the Maharashtra, and Kerala is better than the best performing state among the 10 states all-state average in terms of environmental sustainability, while Punjab, Andhra Pradesh, Benchmarking Rural Water Orissa, and Tamil Nadu are the worst performers Service Performance among the states covered in the study. Andhra Pradesh, Orissa, and Tamil Nadu perform poorly As mentioned earlier, the present M&E also in terms of financial sustainability. Two other mechanism needs to be revamped to states with relatively poor performance in terms independently monitor the inputs, processes, of financial sustainability are Uttar Pradesh and outputs, and outcomes of the rural water supply Uttarakhand. West Bengal has the worst services and disclose its findings to the public. performance in terms of financial sustainability Performance improvement targets should be set among the 10 states studied. By contrast, and monitored by states. An incentive scheme financial sustainability of the water supply could be introduced to reward the good schemes is relatively much better in Punjab, performing Gram Panchayats on scheme Maharashtra, and Kerala. For all the four performance and service delivery targets. A list of indices, the performance of Maharashtra is indicators1 for benchmarking the performance of better than the average across states. Indeed, 1 Maharashtra is among the top two states in each This is a list of `outcome'-related indicators on the performance of schemes. A similar lit on the `impact'-related indicators can be based on household surveys. 5 5 rural water supply service across states is given Performance improvement targets below. The data for these indicators could be should be set and monitored by obtained through a random sampling of states. An incentive scheme could representative schemes across the state. Separate be introduced to reward the good information needs to be collected for single and performing Gram Panchayats on multi village schemes. It is recommended that an scheme performance and service annual survey of schemes is undertaken using the delivery targets list of indicators given above. In addition, there could be a large survey covering both schemes and households, once in five years. List of Indicators (Separate data required for single village and multi village schemes) Reliability and Adequacy Percent schemes supplying water as per design norms (design norm of 40 lpcd or more than 40 lpcd) Percent schemes supplying daily (at least four hours of regular daily supply) Percent schemes with households spending less than 30 minutes per day in collecting water Percent schemes with no major breakdown in the past six months (major breakdown defined as `more than two days of disruption in water supply') Percent schemes with good water quality--no bacteriological contamination and no chemical problems of arsenic, fluoride, salinity Financial Sustainability Percent schemes with more than 80 percent O&M cost recovery Percent schemes with more than 80 percent collection efficiency This Report has been prepared by Smita Misra (Sr. Economist, SASDU, Affordability World Bank), the Task Manager of this study. Water tariff for household connections as a ratio of rural per capita income The study was carried out under the overall guidance of Sonia Hammam, Water tariff for standpost users (shared connections) as a ratio of Sector Manager, Water and Urban, SASSD, World Bank. Data analysis has been undertaken by Professor B.N. Goldar and his research team at the rural per capita income Institute of Economic Growth, Delhi and the consumer survey was carried Percent schemes with more than 50 percent household connections out by the ORG Centre for Social Research (a division of A.C. Nielsen Environmental Sustainability ORG MARG Pvt Ltd). Comments and inputs at various stages of Percent schemes with source providing more than 80 percent yield preparation from the following World Bank persons are gratefully acknowledged: Michael Carter, Rachid Benmessaoud, Clive G. Harris, (as per design norms) Alain R. Locussol, Francis Ato Brown, Alexander E. Bakalian, Oscar E. Alvarado, G.V. Abhyankar, R.R. Mohan, S. Satish, N.V.V. Raghava, and Policy Papers Catherine J. Revels (WSP-SA). Special thanks are due to the Department of Economic Affairs, Ministry of Finance, the Department of Drinking Water This is one of the six policy papers that have been prepared on the basis of the Supply, Ministry of Rural Development, and the Rajiv Gandhi National World Bank study on Review of Effectiveness of Rural Water Supply Schemes in Drinking Water Mission for their interest and collaboration in the study. India (June 2008). These policy papers, published along with the Report, are on the Comments and data inputs during the preparation of the Report are following themes: gratefully acknowledged from R.P. Singh and M. Nagaraju (DEA), Paper 1: Willingness of Households to Pay for Improved Services and Affordability Bharat Lal and R.K. Sinha (RGNDWM) and their team, and the Paper 2: Inefficiency of Rural Water Supply Schemes in India respective State Government officials. Paper 3: Multi Village Water Supply Schemes in India Paper 4: Operation and Maintenance Expenditure and Cost Recovery The Report has been discussed with the Government of India but does not Paper 5: System of Monitoring and Evaluation necessarily bear their approval for all its contents, especially where the Paper 6: Norms for Rural Water Supply in India Bank has stated its judgements/opinions/policy recommendations. Author and Task Manager: Smita Misra (Sr. Economist, SASDU, World Bank The World Bank Pictures by: Guy Stubbs/Water and Sanitation Program­South Asia June 2008 Created by: Write Media Printed at: PS Press Services Pvt. Ltd. The World Bank, New Delhi Office, 70 Lodi Estate, New Delhi 110 003, India Tel: (91-11) 24617241, 24619491