Experimental Methods
Outline

1.What exactly do we mean
  by impact?

2.What can be learned
  from randomized
  experiments?
What do we mean by impact?

“An impact evaluation assesses
changes in the well-being of
individuals, households,
communities or firms that can be
attributed to a particular project,
program or policy.” Impact Evaluation in Practice, second edition
How do we know that the program caused
the observed effects?

What would have
happened without the
program?
                      The counterfactual
Program impact                              Is this
                                          observed?

                                                      I
                                                      M
                            What would                P
 What happens                  have
   with the                 happened                  A
   program                  without the               C
                             program                  T


                 Must be
                estimated
How can we estimate the
counterfactual?
Some options to consider


    Before-after comparisons

      Comparing participants and non-participants

    Randomized experiment
After-school
reading program
                 Time

Program starts
Average score                    Average score




                                                 Time

                Program starts



    Before-after comparisons
                                                             I
         What
                                  What would                 M
                                     have
        happens                                              P
                                  happened
        with the                                             A
                                  without the
        program
                                   program                   C
                                                                 ???
                                                             T
Average score after program   Average score before program




         Before-after comparisons
Many other things might have happened
over time




 Learning during the school day   Other child-targeted activities   Income fluctuations
                                     What                    I
        What                       would have                M
       happens                     happened
       with the
                                                             P
                                    without
       program                        the                    A
                                    program                  C
                                                             T
Average score after program   Average score before program




         Before-after comparisons
Before-after comparisons give us a biased
estimate of impact




                   BIAS
                                  Average score




            Program starts

                                  Average score




Comparing participants and non-participants
                                                                     I
         What
                                     What would                      M
                                        have
        happens                                                      P
                                     happened
        with the                                                     A
                                     without the
        program
                                      program                        C   ???

                                                                     T
 Average score of participants   Average score of non-participants




Comparing participants and non-participants
Participants and non-participants can
differ along many dimensions
      Observable            Unobservable
                                                                    I
        What
                                       What                         M
                                    would have
       happens                                                      P
                                    happened
       with the                                                     A
                                    without the
       program
                                     program                        C
                                                                    T
Average score of participants   Average score of non-participants




Comparing participants and non-participants
Comparisons of participants and non-participants
give us a biased estimate of impact




                          BIAS
                                 Selection bias
Can experiments help?




                    Cannot read   Can read
There are many different types of children
There’s even variation within type
For each child, let’s flip a coin.
Heads à Treatment
Tails à Control
For each child, let’s flip a coin.

Every child has an equal chance
of being assigned to treatment
Randomized assignment ensures balance




  TREATMENT: Gets program   CONTROL: Does not get program
Time varying elements also balanced




                 Program starts
Randomized assignment helps isolate
impact

                                                              I
        What
                                 What would                   M
                                    have
       happens                                                P
                                 happened
       with the                                               A
                                 without the
       program
                                  program                     C
                                                              T
Average score of TREATMENT   Average score of CONTROL after
after program                program

                                      Average treatment effect
Why should we do experiments?

          To know if a program is really working


            To find out how to improve a
            program

          To learn more about beneficiaries and
          service providers
Testing whether programs work
Teacher performance pay in India

                         Individual
                          bonuses       + 0.28 SD
        Government
          primary
         schools in    Group bonuses    + 0.15 SD
          Andhra
          Pradesh       Unconditional
                         block grants   + 0.08 SD

                           Control
Improving pre-primary school quality in
Ghana

                          Teacher training      Better
                                              emotional
                                             support and
      Preschools          Teacher training +   behavior
                         parental awareness management


                              Control
Improving pre-primary school quality in
Ghana
                                                 Higher
                          Teacher training       school
                                               readiness
      Preschools          Teacher training +
                         parental awareness


                              Control
Improving pre-primary school quality in
Ghana
                                               Turnover:
                          Teacher training      - 44%

      Preschools          Teacher training +
                         parental awareness


                              Control
Improving program design
Increasing college attendance in the US

                             Personalized
                             information
   Families visiting tax
   assistance offices in      Personalized      25%
       low-income          information + help   increase in
     neighborhoods            with aid form     enrollment

                               Control:
                              Brochure
Increasing secondary school enrolment in
Colombia
                           Standard
                        conditional cash
                            transfer
   Households with                          Increased
     school age         Conditional cash    attendance
      children         transfer – savings
                            variant


                           Control
Increasing secondary school enrolment in
Colombia
                           Standard
                        conditional cash
                            transfer
   Households with
     school age         Conditional cash    Increased
      children         transfer – savings
                            variant         enrollment

                           Control
Learning about beneficiaries or
service providers
Improving interest in anti-poverty programs in
the US
                                             300% increase
                          Self-affirmation   in take-up of
                                             brochure

            Guests in a
                           Funny movie
           soup kitchen


                              Control
• http://www.worldbank.org/en/programs/sief-trust-fund




                http://www.worldbank.org/sief



       THANK YOU