Estimating Causal Effects with Experimental Data

Estimating Causal Effects with

Experimental Data

Some Basic Terminology

• Start with example where X is binary (though simple to generalize):

– X=0 is control group

– X=1 is treatment group

• Causal effect sometimes called treatment effect

• Randomization implies everyone has same probability of treatment

Why is Randomization Good?

• If X allocated at random then know that X is independent of all pre-treatment

variables in whole wide world

• an amazing claim but true.

• Implies there cannot be a problem of omitted variables, reverse causality etc

• On average, only reason for difference between treatment and control group is different receipt of treatment

Why is this useful?

An Example: Racial Discrimination

• Black men earn less than white men in US

LOGWAGE | Coef. Std. Err. t ---+--- BLACK | -.1673813 .0066708 -25.09 NO_HS | -.2138331 .0077192 -27.70 SOMECOLL | .1104148 .0049139 22.47 COLLEGE | .4660205 .0048839 95.42 AGE | .0704488 .0008552 82.38 AGESQUARED | -.0007227 .0000101 -71.41 _cons | 1.088116 .0172715 63.00

• Could be discrimination or other factors unobserved by the researcher but observed by the employer?

• hard to fully resolve with non-experimental data

An Experimental Design

• Bertrand/Mullainathan “Are Emily and

Greg More Employable Than Lakisha and Jamal”, American Economic Review, 2004

• Create fake CVs and send replies to job adverts

• Allocate names at random to CVs – some given ‘black-sounding’ names, others

‘white-sounding’

• Outcome variable is call-back rates

• Interpretation – not direct measure of

racial discrimination, just effect of having a

‘black-sounding’ name – may have other connotations.

• But name uncorrelated by construction with other material on CV

The Treatment Effect

• Want estimate of:



¹  

⁰ 

E y X   E y X 

Estimating Treatment Effects: the Statistics Course Approach

• Take mean of outcome variable in treatment group

• Take mean of outcome variable in control group

• Take difference between the two

• No problems but:

– Does not generalize to where X is not binary – Does not directly compute standard errors

Estimating Treatment Effects: A Regression Approach

• Run regression:

y_i=β₀+β₁X_i+ε_i

• Proposition 2.2 The OLS estimator of β₁ is an unbiased estimator of the causal effect of X on y:

• Proof: Many ways to prove this but simplest way is perhaps:

– Proposition 1.1 says OLS estimates E(y|X)

– E(y|X=0)= β₀ so OLS estimate of intercept is consistent estimate of E(y│X=0)

– E(y|X=1)= β₀+β₁ so β₁ is consistent estimate of E(y│X=1) -E(y│X=0)

• Hence can read off estimate of treatment effect from coefficient on X

• Approach easily generalizes to where X is not binary

• Also gives estimate of standard error

Computing Standard Errors

• Unless told otherwise regression package will compute standard errors assuming errors are homoskedastic i.e.

• Even if only interested in effect of treatment on mean X may affect other aspects of distribution e.g. variance

• This will cause heteroskedasticity

• Heteroskedasticity does not make OLS

regression coefficients inconsistent but does make OLS standard errors inconsistent

‘Robust’ Standard Errors

• Also called:

– Huber standard errors – White standard errors

– Heteroskedastic-consistent standard errors

• Simple to use in practice e.g. in STATA:

. reg y x, robust

• Statistics course approach

– Get variance of estimate of mean of treatment and control group

– Sum to give estimate of variance of difference in means

Bertrand/Mullainathan:

Basic Results

Summary So Far

• Econometrics very easy if all data comes from randomized controlled experiment

• Just need to collect data on

treatment/control and outcome variables

• Just need to compare means of outcomes of treatment and control groups

• Is data on other variables of any use at all?

– Not necessary but useful

Including Other Regressors

• Can get consistent estimate of treatment effect without worrying about other variables

• Reason is that randomization ensures no problem of omitted variables bias

• But there are reasons to include other regressors:

– Improved efficiency

– Check for randomization – Improve randomization

– Control for conditional randomization – Heterogeneity in treatment effects

The Uses of Other Regressors I:

Improved Efficiency

• Don’t just want consistent estimate of causal effect – also want low standard error (or high precision or efficiency).

• Standard formula for standard error of OLS estimate of β is σ²/Var(X)

• σ² comes from variance of residual in regression – (1-R²)* Var(y)

• Include more variables and R² rises – formal proof (Proposition 2.4) a bit more complicated but this is basic idea.

The Uses of Other Regressors II:

Check for Randomization

• Randomization can go wrong

– Poor implementation of research design – Bad luck

• If randomization done well then W should be independent of X – this is testable:

– Test for differences in W in treatment/control groups

– Probit model for X on W

The Uses of Other Regressors III:

Improve Randomization

• Can also use W at stage of assigning treatment

• Can guarantee that in your sample X and W are independent instead of it being just probabiliistic

• This is what Bertrand/Mullainathan do when assigning names to CVs

The Uses of Other Regressors IV:

Adjust for Conditional Randomization

• This is case where must include W to get consistent estimates of treatment effects

• Conditional randomization is where probability of treatment is different for people with different values of W, but random conditional on W

• Why have conditional randomization?

– May have no choice

– May want to do it (c.f. stratification)

An Example: Project STAR

• Allocation of students to classes is random within schools

• But small number of classes per school

• This leads to following relationship between probability of treatment and number of kids in school:

.1.2.3.4.5Fraction in Treatment Group

40 60 80 100 120

Number of Kids in School

Controlling for Conditional Randomization

• X can know be correlated with W

• But, conditional on W, X independent of other factors

• But must get functional form of relationship between y and W correct – matching

procedures

• This is not the case with (unconditional) randomization – see class exercize

Heterogeneity in Treatment Effects

• So far have assumed causal (treatment) effect the same for everyone

• No good reason to believe this

• Start with case of no other regressors:

y_i=β₀+β_1iX_i+ε_i

• Random assignment implies X independent of β_1i

• Sometimes called random coefficients model

What treatment effect to estimate?

• Would like to estimate causal effect for everyone – this is not possible – Holland’s fundamental

problem of statistical inference

• Can only hope to estimate some average

• Average treatment effect:

• Proposition 2.5: OLS estimates ATE

 

ATE E    

Observable Heterogeneity

• Full outcomes notation:

– Outcome if in control group:

y_0i=γ₀’W_i+u_0i – Outcome if in treatment group:

y1i=γ1’Wi+u1i

• Treatment effect is (y1i-y_0i) and can be written as:

(y_1i-y_0i )=(γ₁- γ₀ )’W_i+u_1i-u_0i

• Note treatment effect has observable and unobservable component

• Can estimate as:

– Two separate equations – One single equation

Combining treatment and control groups into single regression

• We can write:

• Combining outcomes equations leads to:

• Regression includes W and interactions of W with X – these are observable part of treatment effect

• Note: error likely to be heteroskedastic

 

1 1 0

i i i i i

y  X y   X y

     

   

1 1 0 0

0 1 0 0 1 0

' 1 '

' '

i i i i i i i

i i i i i i i

y X W u X W u

W X W u X u u

 

  

    

     

Bertrand/Mullainathan

• Different treatment effect for high and low quality CVs:

Units of Measurement

• Causal effect measured in units of

‘experiment’ – not very helpful

• Often want to convert causal effects to more meaningful units e.g. in Project

STAR what is effect of reducing class size by one child

Simple estimator of this would be:

   



 



E y X E y X E S X E S X

  ^{  }

  

• where S is class size

• Takes the treatment effect on outcome variable and divides by treatment effect on class size

• Not hard to compute but how to get standard error?

IV Can Do the Job

• Can’t run regression of y on S – S influenced by factors other than treatment status

• But X is:

– Correlated with S

– Uncorrelated with unobserved stuff (because of randomization)

• Hence X can be used as an instrument for S

• IV estimator has form (just-identified case):

 

ˆ_IV X S' X y'

  ^

The Wald Estimator

• This will give estimate of standard error of treatment effect

• Where instrument is binary and no other

regressors included the IV estimate of slope coefficient can be shown to be:

   



 



E y X E y X E S X E S X

  ^{  }

  

Partial Compliance

• So far:

– in control group implies no treatment

– In treatment group implies get treatment

• Often things are not as clean as this

– Treatment is an opportunity

– Close substitutes available to those in control group

– Implementation not perfect e.g. pushy parents

An Example: Moving to Opportunity

• Designed to investigate the impact of living in bad neighbourhoods on outcomes

• Gave some residents of public housing projects chance to move out

• Two treatments:

– Voucher for private rental housing

– Voucher for private rental housing restricted for use in

‘good’ neighbourhoods

• No-one forced to move so imperfect compliance – 60% and 40% did use it

Some Terminology

• Z denotes whether in control or treatment group – ‘intention-to-treat’

• X denotes whether actually get treatment

• With perfect compliance:

– Pr(X=1│Z=1)=1 – Pr(X=1│Z=0)=0

• With imperfect compliance:

1>Pr(X=1│Z=1)>Pr(X=1│Z=0)>0

What Do We Want to Estimate?

• ‘Intention-to-Treat’:

ITT=E(y|Z=1)-E(y|Z=0)

• This can be estimated in usual way

• Treatment Effect on Treated

   



 



E y Z E y Z TOT E X Z E X Z

  

   

Estimating TOT

• Can’t use simple regression of y on Z

• But should recognize TOT as Wald estimator

• Can estimated by regressing y on X using Z as instrument

• Relationship between TOT and ITT:

   

. Pr 1 1 Pr 1 0

ITT TOT  X  Z   X  Z  

Most Important Results from MTO

• No effects on adult economic outcomes

• Improvements in adult mental health

• Beneficial outcomes for teenage girls

• Adverse outcomes for teenage boys

Sample results from MTO

• TOT approximately twice the size of ITT

• Consistent with 50% use of vouchers

IV with Heterogeneous Treatment Effects (More Difficult)

• If treatment effect same for everyone then TOT recovers this (obvious)

• But what if treatment effect heterogeneous?

• No simple answer to this question

• Suppose model for treatment effect is:

0 1

i i i i

y     X  

Proposition 2.6

The IV estimate for the heterogeneous

treatment case is a consistent estimate of:

where:

the difference in the probability of treatment for individual i when in treatment and control group

 

 ¹

ˆ1,

lim _IV ^{i i}

i

p E

E

  

 



 



i Pr Xi Zi Pr Xi Zi

      

Interpretation

• This is weighted average of treatment effects

• ‘weights’ will vary with instrument –

contrast with heterogeneous treatment case

• Some cases in which can interpret IV estimate as ATE

How will IV estimate differ from ATE

• IV is ATE if no correlation between β_1i and π_i

• Previous formula says depends on covariance of β_1i and π_i

• In some situations can sign – but not always

• Example 1: no-one gets treatment in the absence of the programme so

• If those who get treatment when in the treatment group are those with the highest returns then:

• IV>ATE

1

i

p

 

 _i, 1_i   1_i, 1_i  0

Cov    Cov p  

• Example 2: treatment is voluntary for those in the control group but compulsory for

those in the treatment group

• This implies

• If those who get treatment in control are those with highest returns then:

• IV<ATE

 _i, 1_i   0_i, 1_i  0

Cov    Cov p  

1 0

i p i

  

Angrist/Imbens Monotonicity Assumption

• Case where IV estimate is not ATE

• Assume that everyone moved in same direction by treatment – monotonicity assumption

• Then can show that IV is average of

treatment effect for those whose behaviour changed by being in treatment group

• They call this the Local Average Treatment Effect (LATE)

Problems with Experiments

• Expense

• Ethical Issues

• Threats to Internal Validity

– Failure to follow experiment

– Experimental effects (Hawthorne effects)

• Threats to External Validity

– Non-representative programme – Non-representative sample

– Scale effects

Conclusions on Experiments

• Are ‘gold standard’ of empirical research

• Are becoming more common

• Not enough of them to keep us busy

• Study of non-experimental data can deliver useful knowledge

• Some issues similar, others different