Exercise 1: True or false impacts? In the situations described below, think about the following questions: (i) Who is compared to whom? Is there a valid counterfactual? (ii) If not, what factors other than the intervention may have affected the outcome of interest? 1. Suppose you come across a study implemented in your country that evaluated the impact of program designed to increase preschool enrollment rates of children from low-income families. The program offered families monthly cash payments for every child aged between 3 - 5 that attended preschool at least 80 percent of the time. Prior to the program, researchers conducted a baseline survey among a random sample of households and measured baseline rates of preschool enrollment and preschool attendance of children. One year after the program started, the study team revisited the same exact households and measured outcomes again. The study estimates that the program increased school attendance by 30 percent on average. The researchers were careful to look for other factors that might have changed during the first year of program implementation and report that in real terms, average household income in the study area showed no change. (i) This is really a simple before-after comparison. There are other things that can vary over time besides income that could also affect preschool attendance – political and security situation, food prices, employment activities for adults, women’s wages, perceived returns to early childhood education, etc. Moreover, while average income might not have changed, income for the groups most at risk for preschool dropout might have changed. (ii) We might inquire whether the 30 percent increase was also exhibited among low-income households, as the blurb above mentions that households were randomly sampled. 2. The next study that you come across deals with a critical topic in early childhood development, stunting, a condition in which a child is too short for his/her age, which is usually associated with a host of cognitive and other developmental delays as well. A group of researchers targeted pregnant mothers coming into health clinics in a set of remote villages and counselled them on best practices during pregnancy and in the first two years of a child’s life. Once the children were born, a counsellor visited the home every two weeks and demonstrated how to best stimulate the child to improve cognitive development and school readiness. To estimate the impact of the program, they identified other mothers in the same neighborhoods who had children of the same age and who were similar on a set of other characteristics like income, educational attainment, average household income, and the baby’s gestational age at the time of delivery. They surveyed all mothers in the program and all mothers with similar characteristics in the program villages and estimated that the counselling program decreased stunting rates by 20 percent and improved cognitive scores by 0.5 of a standard deviation. Their study concludes with a recommendation that the counselling intervention be scaled up nationally. (i) This is a comparison of participants and non-participants. While the two groups of mothers may be similar with respect to these observed characteristics, they could differ on unobserved characteristics like motivation and interest in child development and nutrition, which could directly influence stunting and child development. After all, the mothers in the program are the ones who showed up to the clinics or agreed to participate in the counselling program. (ii) We should always question recommendations for a national scale-up based on evidence from a particular geographic area, even if the results had been valid. For example, stunting rates in other regions of the country could be much lower or could be driven by other factors, such as water quality or sanitation. 3. Researchers in another setting wanted to test what interventions might increase the chances that parents enroll their children in preschool. They are particularly interested in enrollment rates for low-income children. It is unknown whether parents of these children are just unaware of the long-term benefits of preschool or whether there are other obstacles for enrollment. The researchers decide to target the agency where poor families come to apply for subsidized fuel. They decide to offer an informational leaflet outlining the benefits of preschool to every third person who comes to apply for the fuel subsidy. For the next applicant, they provide a list of preschools in town that includes their locations, and they also offer to all of the enrollment paper work for a preschool of the applicant’s choice immediately after he or she finishes the paper work for the fuel subsidy. For the next applicant, they offer nothing beyond what the applicant experiences when normally applying for a fuel subsidy. Later they collect enrollment data from all local preschools and compile a list of names and residences of all enrollees. Among those offered information about preschool at the subsidy agency, 23 percent had at least one child enrolled in preschool. Among those offered immediate enrollment assistance at the subsidy agency, 43 percent had at least one child in preschool. Among the rest of the applicants, 21 percent had at least one child in preschool. The research team estimates that offering assistance with enrolment paperwork increases preschool enrolment by 20 percentage points and that informational barriers about the benefits of preschool are unlikely to be the root cause of low enrolment among poor families. (i) This is a randomized experiment. The main causal interpretation is valid. (ii) We should worry about the content of the informational leaflet. This experiment has demonstrated that that leaflet was not effective in increasing enrollment, but it is possible that the leaflet just did not do a good job in conveying the information. A better designed leaflet or information conveyed through some other way (e.g. a text message or a billboard) might have been effective, so we do not want to conclude from this experiment alone that information is not a barrier. (iii) We should also question external validity. For example, would the same impact hold for families who are not applying for a fuel subsidy? 4. Suppose your unit is interested in knowing whether a new, challenging mathematics and language curriculum for preschools improves children’s early literacy and numeracy skills. The researcher runs an experiment. Some preschools across the country are randomly selected and divided into two groups. Schools in the first group pilot the new curriculum, while schools in the second group stick to their usual teaching program. After one academic year, children’s early literacy and numeracy abilities in both groups are measured. The researcher observes that children in the treatment and control groups have had comparable cognitive outcomes. He concludes that the new curriculum does not improve children’s early literacy and numeracy skills. (i) Even if you run an experiment, your conclusions may not be valid if the intervention was not implemented well! Changes in curricula require changes in teaching practices, which in turn may impact on children’s learning outcomes. It could be that there was not enough concrete guidance for teachers on how to implement the new curriculum. It could also be that teachers would feel more comfortable implementing the new curriculum in the second year, as adapting their teaching program takes time. (ii) It could also be that some children benefitted from the challenging curriculum, while other children grew frustrated and performed worse under the new system, such that on average no positive impact could be detected when compared to the previous system.