POLICY RESEARCH WORKING PAPER 2O9$ Reconsidering the Evidence The sensitivity of empirical results to potential data errors on Returns to T&V and model misspecification Extension in Kenya can yield misleading policy implications and investment signals. A widely disseminated Madhur Gautam study of the impact of the Jock R Anderson training and visit (T&Vj system of management for extension services in Kenya is a striking example of how innocuous data errors and alternative specifications lead to strikingly different results. The World Bank Operations Evaluation Department Sector and Thematic Evaluation Division April 1999 PC y RESEARCH WORKING PAPER 2098 Summary findings Gautam and Anderson revisit the widely disseminated highly sensitive to regional effects. When region-specific results of a study (Bindlish and Evenson 1993, 1997) of effects are included, a positive return to extension cannot the impact of the training and visit (T&V) system of be established, using Bindlish and Evenson's data set and management for public extension services in Kenya. cross-sectional model specifications. T&V was introduced in Kenya by the World Bank and After testing the robustness of results using a number has since been supported through two successive of tests, Gautam and Anderson could not definitively projects. The impact of the projects continues to be the establish the factors underlying strong regional effects, subject of much debate. largely because of the limitations imposed by the cross- Gautam and Anderson's paper suggests the need for sectional framework. Household panel data methods greater vigilance in empirical analysis, especially about would have allowed greater control for regional effects the quality of data used to support Bank policy and the and would have yielded better insight into the impact of need to validate potentially influential findings. extension. Using household data from 1990, Bindlish and The impact on agricultural productivity in Kenya Evenson found the returns from extension to be very expected from T&V extension services is not discernible high. But Gautam and Anderson find that the returns from the available data, and the impact may vary across estimated by Bindlish and Evenson suffer from data districts. The hypothesis that T&V had no impact in errors, and limitations imposed by cross-sectional data. Kenya between 1982 and 1990 cannot be rejected. The After correcting for several data processing and sample data fail to support a positive rate of return on measurement errors, the authors show the results to be the investment in T&V. less robust than reported by Bindlish and Evenson and This paper - a product of the Sector and Thematic Evaluation Division, Operations Evaluation Department - is part of a larger exploration by the department of the effects of the investment in agriculttiral extension in Kenya. Copies of the paper are available free from the World Bank, 1818 H Street NW, Washington, DC 20433. Please contact Madhur Gautam, room G7-026, telephone 202-473-0442, fax 202-522-3123. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/html/dec/Publications/Workpapers,/home.html. The authors may be contacted at mgautam@worldbank.org or janderson@worldbank.org. April 1999. (17 pages) The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent. Produced by the Policy Research Dissemination Center Reconsidering the Evidence on the Returns to T&V Extension in Kenya Madhur Gautam and Jock R. Anderson' Operations Evaluations Department World Bank l The authors wish to thank Gem Argwings-Kodhek, Hans Binswanger, Sushma Ganguly, Robert Evenson, Gershon Feder and Roger Slade for their comments. Reconsidering the Evidence on the Returns to T&V Extension in Kenya Introduction The effectiveness of the public agricultural extension service in Kenya has been, and continues to be, a controversial issue. Two successive IDA-funded projects, the National Extension Projects I and II (NEP-I and II) have supported agricultural extension since 1982, at which time the World Bank introduced the Training and Visit (T&V) system of management. The objective of the projects was to make the Kenyan extension service more effective and efficient. Towards the end of NEP-I in 1990, the Africa Technical Department of the World Bank undertook a study to evaluate the impact of the agricultural extension projects it had supported in Kenya and Burkina Faso. The Kenya study (Bindlish and Evenson, 1993) estimated the returns to extension at 350% (marginal internal rate of return), with a "lower bound" estimate of 160%. The returns to extension in Burkina Faso (Bindlish, Evenson and Gbetibouo, 1993) were estimated in a similar fashion at 91%. These studies and their findings received wide attention in the Bank and elsewhere. At a time when many borrower countries were becoming disenchanted with the T&V approach because of its perceived high cost, as well as increasing concern within the Bank about "results on the ground" from the extension portfolio, the high estimated returns were greeted with skepticism in some quarters (World Bank, 1994, subsequently published in Purcell and Anderson, 1997). Nevertheless, sincp the estimates were arrived at by using household survey data, collected by an independent agency (the Central Bureau of Statistics) rather than the Ministry of Agriculture, and were based on formal statistical methods, the high estimated returns lent credibility to the claims made by the supporters of T&V. The findings vindicated the Bank's stated policy of using extension as a major plank in the overall rural development strategy for Africa (Cleaver, 1993). Hence, the Bank speeded up its already rapid pace of introducing the T&V system in Africa to the extent that, at the end of 1997, 22 countries had a national extension program using the T&V system of management, with active Bank projects supporting a total investment of over $700 million. The present note is part of an impact evaluation study by the Operations Evaluation Department (OED) of the investment in agricultural extension in Kenya supported by the two World Bank projects, NEP-I and II. The purpose of this particular note is to test the robustness of the widely disseminated assessment of the economic returns to agricultural extension in Kenya estimated by Bindlish and Evenson (1993, revised in 1997). The matter is examined within the limits of the available cross-sectional data and the findings of this enquiry should, therefore, be considered as preliminary. The fuller OED study when complete will use panel data to overcome the limitations imposed by cross-sectional analysis and is expected to provide more robust results. Nevertheless, the findings in this paper do highlight the shortcomings of cross-sectional data to inform policy decisions. Specifically, three issues that could potentially have an important bearing on the results of Bindlish and Evenson (hereafter B&E) are addressed. The first, related to the use of cross- sectional data by B&E, is the sensitivity of their results to possible omitted factors, particularly region-specific effects due to natural productivity potential or other factors. The second is the sensitivity of the estimated returns to the functional form used for modeling agricultural production. The third concerns data-related problems. All three are, in the final analysis, empirical issues, although the first two follow from well-established theoretical considerations in dealing with cross-sectional data and technology specification. As the following discussion explains, the results are sensitive to omitted regional effects and data considerations, but are seemingly robust with respect to the functional form for the production function. A brief recap of the B&E study The estimate by B&E of returns to the Kenyan T&V system was based on estimates from an empirical "meta-production" function. The study was designed to overcome a key limitation of most studies that have attempted to establish the impact of extension in a production-function framework, namely the specification of an appropriately exogenous variable for extension supply (Birkhaeuser, Evenson, and Feder, 1991). B&E use a variable based on front-line extension staff2, specified as a ratio of extension workers to the number of farmers in each location (a sub- district administrative unit).3 Since the allocation of staff to each location is determined by the central government, the variable is exogenous to the household decision making process and hence is a valid regressor for farm-level production function estimation. The actual data on staffing were, however, available only for 1990. For previous years, B&E had no choice but to construct staff-farm ratios somewhat arbitrarily, based on the number of years each staff member had been in the same location. The weighted staff-farm variable, constructed to capture the lagged impact of extension on productivity and the key variable in calculating the returns to extension, is thus measured with error (B&E,1993, page 26, first and second paragraphs). The original design of the B&E study was to obtain panel data by revisiting some 700 households that were interviewed in 1982 for the Kenya Rural Household Budget Survey. While the 1990 survey was able to collect the information needed to estimate a production function for about 670 households, comparable information for 1982 was lacking, preventing the estimation of a fixed or random effects model (B&E, 1993, p. 25). Given the circumstances, B&E were forced to estimate a cross-sectional production function using the 1990 data, supplemented by the information available from the 1982 survey in an attempt to control for household- and area- specific factors affecting production. Household-specific effects were controlled by including total production and area cultivated by the household in 1982, and the 1990 age, education and sex of the head of the household. To control for area-specific characteristics, dummy variables for the "production potential" for the zones in which each cluster was located were included (for medium- and low-potential zones), as were the topography indicators (for hilly and undulating regions).4 To control for economic and infrastructure conditions characterizing the area, the regression included 1982 cluster-level means for variables representing access to roads, transport facilities, farm size, farm-level cropped area, farm-level livestock capital, value of cash crops produced, value of other crops produced and household non-farm income. 2B&E tried other extension variables, including supervisors and subject matter specialists, but the results were not significantly different. 3On average, a district has 6 divisions, a division has 6 locations, and a location has 6 sub-locations. 4A "cluster" is a census enumeration unit used by the CBS for the 1982 RHBS sampling frame. In the sample, each cluster belongs to a distinct sub-location. 3 The results from this exercise were reported by B&E (1993). Using a Cobb-Douglas functional form to model crop production, they found the extension variable to be highly statistically significant and estimated the marginal rate of return on investment in extension at about 350% (using the 415 observations that had all the necessary data). Subsequently, a data processing error in the creation of the extension variable was discovered, and after correcting for the error, B&E (1997) noted that the key qualitative results did not change.5 Issues The main concerns relate to model specification and data-related problems. The data issues are dealt with first to allow subsequent comparisons across different model specifications meaningful. On specification, the functional form issue is discussed first followed by a discussion on the omitted region-specific factors. Data Problems As part of the ongoing OED impact evaluation of agricultural extension projects in Kenya, an attempt was made to match up the 1990 farm-level output with the output from the 1982 RHBS data. For this purpose, both the 1990 B&E survey data and the 1982 RHBS data were obtained.6 A household- and cluster-level comparison across the years, however, revealed some differences and inconsistencies. The most significant issue is the method of aggregation across crops to arrive at the farm- level output. A comparison of farm-level output across the years shows a significant drop in farm output (after accounting for farm area) from 1982 to 199O.' These changes were surprising and did not have a ready explanation. Further examination of the plot- and crop-level data from the 1990 survey, however, revealed that output quantities from different crops had been aggregated to estimate the farm-level output (and used as the dependent variable in B&E production function estimation). Also, in the 1990 survey, output data were collected only for 5 major crops (maize, beans, soghum, millet and potatoes). Needless to say, the production-function estimates reported by B&E (1993 and 1997) are less than appropriate. Nevertheless, it should be noted that the 5 crops, on average, constitute about 70% of farm output value from the 1982 sample data and about 68% from the 1997 sample 5Quantitatively, a re-estimation yields a statistically significant coefficient on extension of 0.25 instead of the 0.29 reported in B&E, 1993. The corresponding estimate of the returns to extension in Kenya is 278% (based on 418 "complete" observations). However, as discussed below, further measurement errors have been discovered and the revised estimate is no longer valid. 6The generous assistance of Robert Evenson and Vishva Bindlish in making available the 1990 data set, and of Steven Block for the 1982 data, is warmly acknowledged. 7The significance of the differences first came to light during an ongoing DEA analysis of technical and efficiency changes from 1982 to 1990 and 1990 to 1997. The initial results indicated significant technological regression from 1982 to1990 and significant technological progress from 1990 to 1997 (details forthcoming in Gautam and Alevy, 1999). Neither could be satisfactorily explained by the events that have taken place in Kenya since 1982, and prompted a closer look at the data. 4 data. It is thus reasonable to assume that a high percentage of farm output was captured during the 1990 survey. To proceed with the analysis, since the 1990 survey data do not contain information on output prices, national prices have been used to estimate the value of farm-level output for the 1990 sample.8 The benefit of access to the raw data from the 1982 RHBS survey also allowed a re- creation of the cluster- and household-level variables used by B&E (as contained in the processed set of 1990 data). A comparison of these variables also revealed some differences: (a) in the 1982 cluster- and farm-level control variables and (b) in the agroecological control indicators. As with the 1990 survey data, the 1982 cluster- and farm-level production variables used in the B&E analysis were also inappropriately aggregated and had to be reconstructed.9 Other problematic variables are the cluster transportation and (road) infrastructure variables. It is unclear what the coded values of these variables represent.10 In fact, alternative infrastructure variables are available in the 1982 data. These are the distances from the farm to the various types of road and public transportation reported by households. A comparison of the agro-ecological zone (AEZ) classification used for the B&E analysis with the AEZ classification variables available in the 1982 data also shows differences. In addition to the AEZ classification, B&E used indicators for the regional productivity potential (RPP) as a further control for differences in land quality. Since the 1982 data did not have comparable RPP indicators, these data were independently compiled." The RPP indicators are also substantially different from the indicators used by B&E. To determine which of the alternative sets of AEZ classification and RPP indicators to use, farm output was regressed on each set of indicators. The results show that the 1982 AEZ classification and the new RPP indicators have a higher explanatory power (R2) for both the 1982 and 1990 farm-level production. Finally, to maintain the focus on methodological comparability for the purpose of the present discussion, aggregate data on district agricultural output and extension costs as well as the related assumptions used to calculate the rate of return to extension are the same as used by B&E (1993). 8 The prices were obtained from the FAO data base and cross-checked for consistency, where feasible, with the Economic Surveys for various years published by the Central Bureau of Statistics, Government of Kenya. 9 The quantities of crops produced, instead of their values, appear to have been summed and divided by an arbitrary factor of 12 to arrive at farm-level production used to control for household productivity differentials and for the cluster-mean values of cash and other crop production in the B&E study. 10 For example, in some sub-locations, all infrastructure variables (including those not included in the analysis) take the value 9, while most others take 1 or 2. l l Given the differences observed in the AEZ indicators, a verification of the accuracy of the RPP indicators was sought as part of the ongoing OED impact study of extension projects in Kenya. The help of the Tegemeo Institute of Egerton University, Kenya, is gratefully acknowledged for the compilation of the RPP indicators for the study areas. The new indicators are based on information gathered from the respective district annual reports and farm management handbooks for Kenya. 5 Functionalform An overly restrictive functional form can provide misleading signals about the structure of the production process, and may yield estimates of coefficients that are not informative for policy formulation. While the Cobb-Douglas form is convenient in its parsimony and ease of estimation, it imposes a priori restrictions on the coefficients that may not be valid."2 Of course, every functional form implies some restrictive features, so it is a matter of judgment as to what is feasible and reasonable to choose among a large number of possible forms, whilst seeking not to impose restrictions that inhibit genuine insight. The Kenya situation poses a particular difficulty. On the one hand, parsimony is highly desirable given the cross-sectional nature of the data and the need to identify the effect of extension on productivity using a variable with modest degrees of freedom (there are 71 locations for which the staff-farm ratio is available). Thus, a fully flexible functional form such as the complete translog is not feasible."3 Nevertheless, it is desirable to test whether the functional form makes a significant difference to the coefficient of greatest interest, viz. the extension supply variable. Since a complete characterization of the underlying production structure is not of immediate interest, the strategy is to allow some flexibility in the functional form with respect to variable inputs and the extension supply variable. The other, control variables, are still treated as intercept shifters. While not fully satisfactory, this strategy allows a check on the stability of the elasticity of production with respect to the extension supply variable. Among flexible functional forms, three alternative specifications were tried, namely the translog, the generalized Leontief, and the square-root quadratic. The models were applied to the original data used by B&E, using the appropriately aggregated farm production as the dependent variable but prior to introducing any of the other modifications. The translog performed the best in terms of goodness-of-fit (using the adjusted R2 statistic) of the estimated relationship, although the square-root quadratic performed better in terms of the coefficient of determination of the untransformed dependent variable. The full results from the alternative specifications are not reported in order to save space. The estimate for the focus partial elasticity of production from the translog is 0.22, which is higher than the Cobb-Douglas estimate of 0.14 (detailed results for the Cobb-Douglas are presented below), but the elasticity from the square-root quadratic is lower at 0.06. Both estimates are, however, just over one standard error apart from the Cobb-Douglas estimate. These results thus suggest that, at least in this application, the Cobb-Douglas specification may not be too restrictive. To focus on the other issues, and to be brief, the rest of discussion is based on results using the Cobb-Douglas functional form. 2 For example, it forces the elasticities of substitution to be unity between inputs, and the partial elasticities of production to be constant across varying input intensities. 13 B&E specified the production function to include 5 "conventional" inputs (land, hired labor, proxy for family labor, "cash inputs" and fixed capital), 1 extension variable, 5 variables to control for household effects, four to control for weather effects, 13 agro-ecological variables, and 9 regional and infrastructure control variables. 6 Specification The main concerns in dealing with cross-sectional data include the handling of household- and region-specific effects. B&E attempt to control for these effects by including locational characteristics and household human-capital variables. These data are from the 1982 survey as well as data collected for the 1990 study. The specification issue, however, remains because of the possibility that agro-ecological zones (AEZs) in different parts of the country (e.g., in different districts) are likely to have different production patterns and productivity. Further, it is also possible that deployment of extension staff, the key variable used to measure the supply of extension, is correlated with the productivity potential of each area. If the deployment is independent, the regression coefficient on the extension variable can be taken to represent the impact of extension on agricultural productivity, with the variation across locations providing the opportunity to measure an unbiased and efficient estimate of the impact of the marginal extension staff deployed. If, on the other hand, staff deployment is influenced by the regional productivity effects, (for example, if higher potential areas receive priority in staff assignments or receive a relatively larger number of staff, as was the plan for NEP-I), the extension staff variable would implicitly also capture regional productivity differentials in addition to the impact of extension. Even if the intended initial deployment by the center (Ministry headquarters in Nairobi) was neutral with respect to regional productive potential, the problem may re-emerge if staff are able to transfer out of relatively undesirable locations into more desirable ones. This behavior would be understandable with staff moving to locations with better amenities and access to infrastructure within each district, or moving to more progressive and dynamic districts. Subsequently, if that dynamism translates into higher value-added production, then the staff-farm ratio would again be "spuriously" positively correlated with the value of production. From an empirical standpoint, the issue reduces to whether or not the cluster-level and other variables included in the B&E regression adequately control for region-specific effects. If not, then the regression will suffer from an "omitted variables" problem and will give biased and inconsistent estimates. In such an event, the extension staff variable would erroneously attribute to extension the effect of "natural" productivity differentials. On the other hand, if staff deployment follows a standard rule of thumb, say a pre-determined staff-farm ratio as is typically prescribed in T&V implementation plans, or if the turnover of staff is not region-specific, the correlation between the regional productivity potential and the staff-farm ratio would probably be about zero and the estimated coefficient would remain an unbiased estimate. B&E recognized this problem and attempted to control for the "staff deployment effect" by treating it as an "endogenous" variable (page 120). However, the specification bias in this case is one of omitted variables; the extension supply variable is correlated with the error term because it is correlated with omitted regional productivity effects. B&E's instrumenting equation (page 126) validates this concern.'4 The appropriate solution would thus be to effectively control 14 For example, mean locational education level of household heads, the availability of extension under the previous system (which was reportedly biased towards more productive and progressive areas), and hilly terrain have a positive and significant effect in explaining the variation in the extension supply variable, while low and medium potential indicators have a negative sign. The number of farms and the mean household size in a location have an unexpected negative influence on staff allocation. 7 for productivity effects."5 In the event that this is not possible, an instrumental variables approach could be used to overcome the problem. Unfortunately, valid instruments (i.e., determinants of staff-allocation that are not correlated with the regional productivity differentials) are not available and in general would be rarely available. By instrumenting the extension supply variable on regional indicators, B&E exacerbate the problem, providing an explanation for the increase in magnitude of the estimated coefficient reported by B&E.16 Whether or not the estimated model suffers from specification errors can be tested. The procedure adopted is to include the district fixed-effects to verify the robustness of the estimated coefficient on the staff-farm ratio."7 The households sampled in the survey were from 84 clusters, each from a sub-location belonging to 71 distinct locations in seven districts. Among the observations available for estimation, the information on the extension variable is available for the 71 locations. Thus, one the one hand, there appears to be sufficient variation in the key extension variable-the staff-farm ratio-to allow estimation of the impact of extension on productivity even after controlling for district fixed effects. On the other, the limited variation poses a dilemma in that, given the already extensive use of dummy variables by B&E to control for the agro-ecological zone and regional productivity potential effects, including the district fixed-effects might eliminate any meaningful variation in the extension-supply variable. With the data in hand, the analytical solution to this problem is not obvious. However, its potential empirical significance is dealt with through a series of sequential tests to check of the sensitivity of the staff-farm ratio coefficient. The details are discussed in the results section below. Another potential problem with a district fixed-effects estimation with a single cross- section is the possibility that the efficiency of the extension service varies by district. With the basic unit of organization of extension being the district, this could result in district fixed effects being indistinguishable from extension impact. This possibility, however, goes against the basic premise of the T&V "system of management", introduced by NEP-I, and the intent of establishing a national system with a unified and consistent organizational structure, in contrast to the previous extension system, which was disparately organized and said to be inefficient. Whether or not this is the case is an empirical question and can be statistically tested by including cross terms for extension and district effects, in addition to pure district effects, to identify district-specific extension impact. If the districts are differentially efficient, the 5 If the problem is due to omitted regional productivity effects, which creates the correlation between the staff-farm ratio and the error term, including the regional fixed effects explicitly in the regression would eliminate the source of the problem. Factors other than regional productivity potential that influence the allocation of staff to different regions are unlikely to be correlated with errors associated with the farm-level production function. Hence, the staff-farm ratio will be independent of the production error term and OLS would be appropriate. 16 Given the positive correlation between the staff-farm ratio and productivity, as suggested by B&E's instrumenting equation, and assuming that natural productive potential will have a positive effect on output, it can be shown that the bias in the coefficient of staff-farm ratio should be biased upwards using the standard "omitted variables" formula (Greene, 1990). Based on their own findings, however, B&E conclude that the estimate is biased downwards. 17 This is a rather weak test since within each district there will be variation across locations (and sub- locations) in natural productivity potentials and hence, possibly, in staff-allocation. Nevertheless, given the limitations imposed by available data, a weak test is to be preferred to none. 8 coefficcieins for each district will be significantly different from each other. Further, if horm5geteity of extension effectiveness across districts cannot be rejected, then two further tests can Jie performed with a view to establish the overall impact of extension, as measured by returns to extension expenditure. One test is the significance of a simple sum of the district-specific extension coefficients (implicitly assigning equal weight to each district in the sample). The second test is the significance of a weighted sum of district-specific extension effects, where the weights are the sample shares of each district in the total value of production. Results As a result of the aggregation error for the farm production variable noted earlier, the results reported by B&E (1993, 1997) are not valid. The corrected estimates of the farm meta- production function (i.e., using the reconstructed value of farm production) are presented as model I in table 1. Note that, with the exception of the correction made for the data-processing errors in the extension variable as noted by B&E, 1997, the other independent (or explanatory variables) and model specification are exactly the same as were used in the original B&E (1993) analysis (the original estimate for the coefficient on the extension variable was 0.29 and rate of return on extension investment was estimated to be 350%, with a lower bound"' of 160%). The results for model I indicate a lower, but qualitatively similar, result; the extension coefficient is estimated at 0.14. However, it is barely significant at the 5% level. The marginal internal rate of return to T&V implied by this estimate is 161%; the lower bound estimate, however, is now negative. Qualitatively, thus, it could be argued that the results, albeit not as robust, are still positive and along the lines reported by B&E (1997). For the rest of the analysis, the results of model I of table 1 are considered as the base case, or the "B&E" model, to which subsequent comparisons will be made. The remaining results are presented as follows. Using the same data for the explanatory variables as the "B&E" (1997) model (without any other modifications), extending the specification to allow for district fixed-effects, the column labeled model II in table 1 gives the results of including district dummy variables. In model III, the extension variable is replaced by a series of cross-terms for district dummy variables interacted with extension to test for differential efficiency of extension across districts. Incorporating the data modifications to the other explanatory variables, as discussed above, table 2 presents an alternative set of results. Models I, II and III in table 2 present the results corresponding to the respective models in table 1 but using the updated data. The results in table I demonstrate the sensitivity of the coefficient on the extension variable to district fixed effects, even without making any other changes. The coefficient on extension goes from 0.14 and significant at 5% level, to -0.003 and not significant at standard levels of significance.'9 An F test for the joint significance of the district fixed effects is highly significant, rejecting the hypothesis that the included location-specific and other variables 18 The lower bound corresponds to the rate of return associated with the lower limit of the 95% confidence interval for the estimated coefficient on the extension supply variable. 1 To verify whether any specification bias persists in the model with district fixed-effects, due to correlation between the staff-farm ratio and the error termn, the instrumental variables (IV) technique was applied to model II of table 1. As anticipated, the IV estimnate of the staff-farn ratio coefficient was insignificant (+0.04). Hence, the remaining models are estimated using OLS. 9 adequately control for regional effects. In addition, the agro-ecological zone variables continue to be jointly significant in the presence of district dummy variables, suggesting that one set of variables is not a substitute for the other; that is, neither is dispensable. The two RPP indicators, however, become jointly insignificant in the presence of the district fixed-effects. As discussed above, it is possible that by including too many fixed-effects (for AEZs, RPPs and districts), the effect of any meaningful or genuine variation in the extension supply variable may be reduced to the point of making it redundant. The large drop in the magnitude of the extension coefficient and its significance, thus, warrant some further investigation. To determine whether the base-case result represents the genuine impact of extension or a spurious result due to omitted factors, a series of tests were performed. These included dropping the AEZ and RPP indicators to test the sensitivity of the coefficient on the extension supply variable. As noted above, including the district fixed-effects makes the RPP indicators jointly insignificant. Hence, it is not surprising that, when the RPP indicators are dropped, all coefficients and their standard errors remain virtually unchanged relative to the full fixed-effects model (model II) in table 1. Next, the set of AEZ indicators was dropped (despite their joint significance noted above). Again, there were no substantive (or significant) changes in the standard production parameters or in the extension coefficient. Further, the overall model fit is better with district fixed-effects and without AEZs (adjusted R2 of 0.70) compared to the base-case model (adjusted R2 of 0.65). Finally, dropping all the RPP and the AEZ indicators, the model fit continues to be superior (adjusted R2 of 0.70), while the estimated coefficients are not substantively different from the full fixed-effects model. These tests reveal the importance of the district-specific effects in explaining the productivity differences across households. They also provide confidence that, while it is theoretically possible that too many fixed-effects can be more of a nuisance than an aid in statistical inference, in this empirical application such is not the case. To test the hypothesis that extension may be differentially efficient across districts, and hence that the district effects are obscuring the extension impact, model III in table 1 presents the results of the regression including cross-terms between district effects and extension. Two of the districts (Mlachakos and Muranga) have positive and statistically significant coefficients, while one (Kisumu) has a statistically significant negative coefficient. All of the remaining districts have an insignificant coefficient but the sign is negative. While the signs on the individual district effects are possibly interesting in themselves, from the evaluation point of view, the overall impact of extension is of greatest interest. Towards this end, the two tests mentioned above are performed. One is the sum of the coefficients on the seven cross-terms, which turns out to be small (0.037) and insignificant. A more appropriate test is the significance of the weighted sum of district-specific extension effects, where the weights are the sample shares of each district in total output. The results of including cross-terms between district effects and extension indicate a negative overall extension impact (-0.246), which is statistically insignificant. As sketched above, a number of modifications need to be made to the variables used in the analyses reported in table 1. These include: the new indicators (dummy variables) for medium and low potential zones; the AEZ classification available in the 1982 RHBS data set; re- specification of household head's education status to dummy variables for primary and higher levels of education; updated cluster-level means for non-farm income, production, farm holding 10 size and livestock values. Also, the cluster-level infrastructure variables can be replaced by the corresponding means for distance to all-weather road (which includes tarmac and all-weather gravel roads); distance to dirt roads and distance to nearest bus/matatu routes. The results of including these modifications are presented in table 2 models I-III, corresponding to those of table 1. In model I, even the specification without district effects shows that with updated variables, the extension variable is now no longer significant at any conventional level, and in fact has a negative coefficient (-0.08).20 The traditional input variables have coefficients similar to those for the base model (model I) of table 1. The test for district- specific effects, model II, shows that district effects continue to be jointly significant. The extension coefficient continues to be statistically insignificant and negative.21 Finally, the district-specific extension efficiency is tested in model III, where now only the coefficient for Machakos is positive and significant, while the coefficient for Kericho is negative and significant; the others are all negative but insignificant. The test results for model III also show that the simple sum of district-specific extension effects is negative (-0.9), although it is insignificant. The weighted sum of the impact of extension across districts is also negative (- 0.15) and insignificant. The marginal rate of return to extension implied by the estimated coefficients from all models in table 2 is negative (for none of the models in table 2 does the extension coefficient have a positive sign). Summary and Conclusions This note is part of a larger exploration by OED of the effects of the investment in agricultural extension in Kenya. The investment was supported by two World Bank projects starting in 1982, which introduced the T&V system of management for public extension services. The impact of the projects has been the subject of much debate. The particular purpose of this note is to revisit the earlier widely disseminated results on the impact of T&V extension in Kenya by Bindlish and Evenson (1993, 1997). Using household data from 1990, B&E found the returns to extension to be very high. The findings presented in this note, however, show that the returns estimated by B&E suffer from data errors and limitations imposed by cross-sectional data. Correcting for several data processing and measurement errors, the results are shown to be less robust than reported by B&E. The second problem with the B&E results is that they are highly sensitive to regional effects. For the same data set and cross-sectional model specification as used by B&E, with the inclusion of region-specific effects, a positive return to extension cannot be established. The sensitivity to 20 Given the significant impact of the new variables on the extension coefficient, a sensitivity analysis was undertaken to determine which variables had the biggest impact. Results (not reported) show that the new RPP indicators have the largest impact (reducing the extension coefficient to 0.04 and insignificant), followed by the 1982 cluster-means for infrastructure variables, area, production, livestock assets and non-farm income (reducing the extension coefficient to 0.11 and insignificant). 21 The sensitivity tests show that the extension coefficient fails to attain significance or substantively change in magnitude even when the AEZ and RPP indicators are excluded in the presence of district fixed effects. 11 possible omitted factors is underscored by the non-positive rate of return to extension implied by an alternative set of regional productivity indicators, even without regional fixed effects. An important caveat is necessary with these results. Although a number of tests are undertaken to test the robustness of the results in the presence of regional (district) fixed effects, it is not possible to definitively establish the factors underlying the strong regional effects. This is largely because of the limitations imposed by the cross-sectional framework. A more appropriate analysis would be to use panel data methods, which would allow a better control for the fixed regional effects and provide better insight into the impact of extension. The main conclusion from this analysis is that the anticipated impact of T&V extension on agricultural productivity in Kenya is not discernible from the available data. Preliminary results also suggest a differential impact across districts. A value-weighted sum of district- specific impacts is used to determine the aggregate impact and to test if it is significantly different from zero (although it is counter-intuitive that extension could reduce the productivity of farmers as implied by the negative coefficients for some districts, it may be possible at the whole-farm level, say, by advising inappropriate crop-mixes). The test shows that the hypothesis that T&V had no impact in Kenya between 1982 and 1990 cannot be rejected, implying that the sample data fail to support a positive rate of return on the investment in T&V. The findings highlight issues pertinent to empirical analyses intended to inform Bank policy. First is the sensitivity of empirical results to potential data errors and model mis- specification, errors that can yield misleading policy implications and investment signals. This is particularly important when dealing with cross-sectional and imperfect data, often all that are available in many countries. The B&E study provides a pertinent example where seemingly innocuous data errors and alternative specifications lead to strikingly different results. An important lesson emerging for the Bank and other concerned agencies from this analysis is the need for great vigilance in empirical analyses, especially with regard to the quality of data, used to support Bank policy, including the need to validate potentially influential empirical findings. As is the case in such empirical studies, it must be reiterated that the results reported here are situation-specific and are subject to the limitations imposed by the available data. To firmly establish the achievement of concrete results and to draw broad policy implications, the need to rigorously establish impact and to validate the empirical findings in other settings, with the use of appropriate data, cannot be overstated. Household panel data would have been particularly helpful in overcoming some of the problems that appear to have a significant bearing on B&E's results on the returns to extension in Kenya. The earlier-mentioned OED impact evaluation work is using such data and the results are expected to be available in early 1999. 12 Table 1: Production functions using B&E data. Variable Model I Model n Model HII Intercept 5.098** 4.252** 6.341** (0.557) (0.600) (0.768) Log of Area 0.458** 0.440** 0.426** (0.045) (0.041) (0.041) Log of family size 0.276** 0.198** 0.174* (0.077) (0.070) (0.068) Log of hired labor 0.032** 0.021** 0.020** (0.008) (0.008) (0.007) Log of cash inputs 0.035* 0.001 -0.009 (0.018) (0.017) (0.017) Log of fixed farm capital -0.028 0.030 0.021 (0.057) (0.053) (0.052) Log of extension staff-farm 0.142* -0.003 ratio (Sfratio) (0.071) (0.083) Sfratio x Dummy variable 0.717** for Machakos (0.217) Sfratio x Dummy variable 1.009** for Murang'a (0.256) Sfratio x Dummy variable -0.340* for Kisumu (0.138) Sfratio x Dummy variable -0.267 for Kericho (0.241) Sfratio x Dummy variable -0.242 for Taita Taveta (0.248) Sfratio x Dummy variable -0.629 for Trans Nzoia (0.385) Sfratio x Dummy variable -0.211 for Bungoma (0.197) Farm output, 1982 0.989** 1.055** 1.255** (000 Ksh) (0.230) (0.218) (0.217) Farm area, 1982 -0.029 -0.007 0.001 (acres) (0.024) (0.022) (0.022) Age of head -0.003 -0.003 -0.003 (years) (0.003) (0.003) (0.003) Sex of head 0.001 0.091 0.043 (Female=1) (0.092) (0.084) (0.081) Education of head -0.029 -0.017 -0.008 (above primary=1) (0.044) (0.041) (0.040) Normal crop 0.440** 0.528** 0.557** (Normal=l) (0.145) (0.140) (0.143) Failed crop 0.184 0.667** 0.538* (Failed=1) (0.244) (0.238) (0.238) 1990 Cluster mean for 0.298* 0.356** 0.340** normal crop (0.129) (0.121) (0.117) 1990 Cluster mean for -0.536 -0.726** -0.805** failed crop (0.292) (0.265) (0.258) Continued Table 1: Production functions using B&E data (continued). 1982 Cluster mean for -0.009 -0.022 -0.032 access to roads (0.027) (0.030) (0.031) 1982 Cluster mean for 0.011 0.029 0.015 access to transport (0.039) (0.044) (0.046) 1982 Cluster mean for -0.144** -0.146** -0.103* cropped area (0.040) (0.043) (0.045) 1982 Cluster mean for 0.091** 0.026 0.016 farm size (0.020) (0.020) (0.020) 1982 Cluster mean for 0.023 0.336 0.144 livestock value (000 Ksh) (0.168) (0.182) (0.197) 1982 Cluster mean for -0.029 0.015 0.014 cash crop value (000 Ksh) (0.019) (0.019) (0.020) 1982 Cluster mean for -0.011 0.018 0.020 other crop value (000 Ksh) (0.015) (0.015) (0.015) 1982 Cluster mean for -0.127 -0.015 -0.017 non-farm income (000 Ksh) (0.092) (0.088) (0.086) Medium Potential Zone -0.181 0.026 -0.108 Dummy variable (0.117) (0.109) (0.113) Low Potential Zone -0.372* 0.248 -0.016 Dummy variable (0.153) (0.158) (0.170) Hilly area -0.506** 0.068 -0.027 Dummy variable (0.160) (0.174) (0.174) Undulating area 0.070 0.202 0.068 Dummy variable (0.120) (0.139) (0.153) AEZ (range) -0.94 to 0.55 -0.65 to 0.25 -0.99 to -0. 15 Jt. Signif. (F value) (7.374)** (3.033)** (5.195)** N 418 418 418 Adj. R2 0.646 0.713 0.735 Test for Joint significance of District effects 15.843** 4.121** (F value) Test for the sum of Sfratio x district effects 0.003 (F value) Test for the weighted sum of sfratio x district effects 2.609 (F value) Notes: Dependent variable is log of the reconstructed 1990 value of farm production. The extension variable is the revised variable used by B&E, 1997. Standard errors reported in parentheses. * and ** indicate significance at 5 and 1% levels, respectively. District effects, with Machakos as the base case, range from -0.23 to 1.48 for model II and from -1.63 to -0. 19 for model III. 14 Table 2: Production functions using modified data. Variable Model I Model H Model Inl Intercept 3.207** 2.953** 3.947** (0.717) (0.790) (0.868) Log of Area 0.384** 0.411** 0.422** (0.041) (0.040) (0.040) Log of family size 0.226** 0.172** 0.161 * (0.068) (0.064) (0.064) Log of hired labor 0.018* 0.014* 0.013 (0.007) (0.007) (0.007) Log of cash inputs 0.024 0.001 0.008 (0.016) (0.016) (0.016) Log of fixed farm capital 0.013 0.089 0.068 (0.048) (0.047) (0.047) Log of extension staff-farm -0.077 -0.110 ratio (Sfratio) (0.069) (0.086) Sfratio x Dummy variable 0.432* for Machakos (0.191) Sfratio x Dummy variable -0.201 for Murang'a (0.309) Sfratio x Dummy variable -0.261 for Kisumu (0.184) Sfratio x Dummy variable -0.493** for Kericho (0.188) Sfratio x Dummy variable -0.106 for Taita Taveta (0.263) Sfratio x Dummy variable -0.235 for Trans Nzoia (0.349) Sfratio x Dummy variable -0.036 for Bungoma (0.181) Farm output, 1982 0.040** 0.037** 0.038** (000 Ksh) (0.009) (0.008) (0.008) Farm area, 1982 0.005 0.004 0.004 (acres) (0.013) (0.013) (0.012) Age of head -0.001 -0.002 -0.003 (years) (0.003) (0.002) (0.002) Sex of head 0.097 0.134 0.107 (Female=1) (0.079) (0.076) (0.075) Education of head 0.064 -0.041 -0.033 (above primary=1) (0.141) (0.135) (0.135) Normal crop 0.631** 0.562** 0.516** (Normal=1) (0.126) (0.127) (0.135) Failed crop 0.521** 0.510** 0.656** (Failed=1) (0.201) (0.194) (0.202) 1990 Cluster mean for 0.331** 0.413** 0.418** normal crop (0.111) (0.108) (0.108) 1990 Cluster mean for -0.611* -0.625** -0.617** failed crop (0.243) (0.230) (0.230) Continued 15 Table 2: Production functions using modified data (continued). 1982 Cluster mean for -0.064** -0.035 -0.020 distance to AW roads (0.022) (0.023) (0.026) 1982 Cluster mean for 0.035 0.045 0.021 distance to dirt roads (0.047) (0.051) (0.060) 1982 Cluster mean for 0.104** 0.094** 0.108** distance to transport (0.024) (0.025) (0.029) 1982 Cluster mean for -0.076* -0.123** -0.075 cropped area (0.036) (0.038) (0.043) 1982 Cluster mean for 0.037** 0.032* 0.028 farm size (0.014) (0.014) (0.015) 1982 Cluster mean for 0.013 0.015 0.001 livestock value (000 Ksh) (0.018) (0.018) (0.024) 1982 Cluster mean for -0.438** -0.227** -0.228** cash crop value (000 Ksh) (0.060) (0.068) (0.082) 1982 Cluster mean for 0.040* 0.013 0.010 other crop value (000 Ksh) (0.018) (0.020) (0.021) 1982 Cluster mean for -0.018 -0.020 -0.009 non-farm income (000 Ksh) (0.043) (0.043) (0.046) Medium Potential Zone -0.849** -0.398 -0.484* Dummy variable (0.177) (0.208) (0.224) Low Potential Zone -0.802** -0.396 -0.602 Dummy variable (0.250) (0.276) (0.314) Hilly area 0.054 0.211 0.234 Dummy variable (0.140) (0.140) (0.147) Undulating area 0.165 0.191 0.211 Dummy variable (0.097) (0.117) (0.138) AEZ (range) 0.62 to 1.77 0.216 to 1.572 0.263 to 1.617 Jt. Signif. (F value) (3.319)** (3.533)** (3.263)** N 455 455 455 Adj. R 2 0.705 0.738 0.744 Test for Joint significance of District effects 9.695** 5.023** (F value) Test for the sum of Sfratio x district effects 1.737 (F value) Test for the weighted sum of sfratio x district effects 1.150 (F value) Notes: Dependent variable is log of the reconstructed 1990 value of farm production. The extension variable is the revised variable used by B&E, 1997. Standard errors reported in parentheses. * and ** indicate significance at 5 and 1% levels, respectively. District effects, with Machakos as the base case, range from -0.07 to 1.36 for model II and from -1.09 to 0.15 for model III. AW roads refer to all-weather roads. 16 References Bindlish, Vishva, and Robert E. Evenson, 1993. Evaluation of the Performance of T& V Extension in Kenya. World Bank Technical Paper No. 208, Africa Technical Department Series, World Bank, Washington, D.C. Bindlish, Vishva, and Robert E. Evenson, 1997. "The Impact of T&V Extension in Africa: The Experience of Kenya and Burkina Faso", World Bank Research Observer, 12(2): 183-201. Bindlish, Vishva, Robert E. Evenson and Mathurin Gbetibouo, 1997. Evaluation of T& V Based Extension in Burkina Faso. World Bank Technical Paper No. 226, Africa Technical Department Series, World Bank, Washington, D.C. Birkhaeuser, Dean, Robert E. Evenson, and Gershon Feder, 1991. "The Economic Impact of Agricultural Extension: A Review", Economic Development and Cultural Change, 39(3):507-21. Cleaver, Kevin M., 1993. A Strategy to Develop Agriculture in Sub-Saharan Africa and a Focus for the World Bank. World Bank Technical Paper No. 203, Africa Technical Department Series, World Bank, Washington, D.C. Greene, William H., 1990. Econometric Analysis. Macmillan, New York. Purcell, Dennis L. and Anderson, Jock R. (1997), Agricultural Extension and Research: Achievements and Problems in National Systems, A World Bank Operations Evaluation Study, World Bank, Washington, D.C. World Bank, 1994. Agricultural Extension: Lessons from Completed Projects. Report No. 13000, Operations Evaluations Department, World Bank, Washington, D.C., processed. 17 P0oDcy ReaerokD Wor0f ls li NOe AuRGmog IthorT WPS2085 Monitoring Banking Sector Fragility: As$i Demirgog-Kunt March 1999 K. Labrie A Muiv rivts Logit Approach wth Ennca Detragieche 31001 on Applicotion to the 199-97 Banking Crises WPS2088 Deregulating Technology Transfer David Gisseiquist March 1999 L. Tobada in Agricuturs: Reform's Impact Carl Pray 36893 on Turkey in the 1980s WPS2087 Male-Female Differences in Labor Peter F. Orazem March 1999 S. Fellon Market Outcomes during the Early Milan vodopivec 38009 Tranzition to Market: The Case of Estonia and Slovenia WPS2088 Expropriation of Minority Stijn Cleessens March 1999 R. Vo Shareholders: Evidence from Simeon Djankov 33722 East Asia Joseph P. H. Fan Larry H. P. Lang WPS2089 Corporate Diversification in East Asia: Stijn Claessens March 1999 R. Vo The Role of Ultimate Ownership and Simeon Djankov 33722 Group Aflilication Joseph P. H. Fan Larry H. P. Lang WPS209S Risks, Lessons Learned, and Donald F. Larson March 1999 P. Kokila Secondary Markets for Greenhouse Paul Parks 33718 Gas Reductions WPS2091 Beyond Unequal Development: Andres Solimano March 1999 A. Solimano An Overview 87671 WPS2092 What Does Aid to Africa Finance? Shantayanan Devarajan March 1999 C. Bernardo Andrew Sunil Rajkumar 3114a Vinays Swaroop WPS2093 Free Entry in Infrastructure David Ehrhardt March 1999 R. Bartolome Rebecca Burdon 35703 WPS2094 What Triggers Market Jitters? Graciela L. Kaminsky April 1999 S. Holt A Chronicle of the Asian Crisis Sergio L. Schmukler 37859 WPS2095 Ten Years of Post-Socialist Lessons Grzegorz W. Kolodko April 1999 J. Prochnow-Walker for Policy Reform 37466 WPS2096 Reforming the Urban Transport Jorge M. Rebelo April 1999 A. Turner Sedor in the Rio de Janeiro 30933 Meropolitan Region: A Case Study In Concessions Policy Research Working Paper Series Conecd Title Author Date bFo 3X WPS2097 The Theory of Access Pricing: Tommaso M. Valletti April 1999 G. Chanet-Smith An Overview for Infrastructure Antonio Estache 38370 Regulators I I