Analysis Program Evaluation and Cost-Benefit 5IE475

(1)

5IE475

Program Evaluation and Cost-Benefit Analysis

LECTURE 12

Instrumental Variable Approach (contd) Qualitative program evaluation

Klára Kalíšková

(2)

EXAMPLES OF INSTRUMENTAL VARIABLES STUDIES (CONTD)

2

(3)

Example 2: Family welfare cultures

Motivation

• Does the receipt of welfare program by parents

increases the likelihood of welfare receipt for children?

– Is there a “welfare culture”? Do children learn from their parents that living on welfare is viable strategy and thus do it themselves?

– Or is it only that welfare eligibility is correlated across generations? Children of poor parents or parents with poor health are more likely to be poor or have health problems?

• Can we simply compare children of parents who received welfare and those who did not?

– NO, poverty/health are correlated across generations.

3

(4)

Example 2: Family welfare cultures

Instrumental variable approach

• Paper: Dahl, Kostol and Mogstad (2014)

• Focus on disability pensions

– Does the receipt of disability pension by parents increases the receipt by children?

• Finding an instrument:

– In Norway, disability pension applicants who are initially denied can appeal their case to a judge – The assignment of cases to judges is random and

some judges are systematically more lenient

– Identity of the judge predicts if the parent receives disability pension or not

4

(5)

Example 2: Family welfare cultures

Is judge’s leniency a good IV?

– Is it relevant? = Correlated with X (parent’s

eligibility for disability pension)

• Yes, judge’s identity (his leniency) is correlated with parent’s disability status, because judge decides if the parent wins the appeal or not – Is it exogenous? = Not correlated with Y but

through its correlation with X

• Yes, it affects the child’s disability status only throught its effect on parent’s disability

• No direct effect of the leniency of parent’s appeal judge on child’s disability status

5

(6)

Example 2: Family welfare cultures

Estimation

• First stage:

𝑝𝑎𝑟𝑒𝑛𝑡_𝑑𝑖𝑠𝑎𝑏_𝑖 = 𝛼₀ + 𝛼₁𝑗𝑢𝑑𝑔𝑒_𝑖 + 𝛼₂𝑋_𝑖 + 𝑢_𝑖

• parent_disab is an indicator for whether parent is allowed disability pension in the appeal process

• judge captures the judge’s leniency (the average allowance rate in all the other cases a judge has handled)

• Second stage:

𝑐ℎ𝑖𝑙𝑑_𝑑𝑖𝑠𝑎𝑏_𝑖 = 𝛽₀ + 𝛽₁𝑝𝑎𝑟𝑒𝑛𝑡_ 𝑑𝑖𝑠𝑎𝑏_𝑖 + 𝛽₂𝑋_𝑖 + 𝜀_𝑖

• child_disab is an indicator for whether the child subsequently participates in disability pension program

6

(7)

Example 2: Family welfare cultures

Results

When a parent is allowed disability because of a

lenient judge, their adult child’s participation rate increases by 6 percentage points over the next five years.

7

(8)

Example 3: Public health insurance

Motivation

• Currie, Gruber (1996) Health Insurance Eligibility, Utilization of Medical Care, and Child Health. The Quarterly Journal of Economics, 111(2), 431-466.

• Effect of public health insurance eligibility on:

– Utilization of health care and health outcomes

• Taking advantage of policy reforms

– Fraction of children eligible for public insurance (Medicaid) doubled 1984-1992

– Different extent of reforms in different states – Natural experiment approach with IV

8

(9)

Example 3: Public health insurance

Health system in the U.S.

• Eligibility for public health insurance (Medicaid)

– Testing income, family situation,…

• Coverage of health insurance

– Some people are covered by public insurance (Medicaid, Medicare), some by private, some by none

– Increase in eligibility for Medicaid might not increase coverage for everyone (people have private insurance)

• Utilization of health care

– How often people visit doctors, hospitals,…

– If they are covered by health insurance, visits are mostly for free

• Health outcomes

– No info on health outcomes in the individual data – They look at child mortality at state-year-age level

9

(10)

Example 3: Public health insurance

Do we need an IV?

• Can we estimate the impact by simple comparison of Medicaid eligible/ineligible?

• Is the eligibility for Medicaid unrelated to outcome (health status)?

– Medicaid eligibility is based on income!

• Health status and income are both correlated with

education – more educated people take better care of their health and earn more

• People with worse health will go more often to doctors and this might cause lower income and thus Medicaid eligibility

People eligible for Medicaid are more likely to have worse health

10

(11)

Example 3: Public health insurance

Instrumental variable approach

– They use IV that focuses on exogenous variation in Medicaid eligibility laws

– They create a simulated instrument

• Select a national random sample of 300 children of a given age and calculate the fraction of children in this sample who would be eligible for Medicaid in a given state and year

• This simulated IV captures the generosity of Medicaid policy in a given state and year

11

(12)

Example 3: Public health insurance

Is this a good IV?

– Is it relevant? = Correlated with X (eligibility)

• Yes, it should explain well individual eligibility, as individual eligibility is given by state laws, and the IV capture differences in state laws

– Is it exogenous? = Not correlated with Y but

through its correlation with X

• Yes, it affects usage of health care only throught its effect on individual eligibility

• No direct effect of the share of eligible children on individual health outcomes

12

(13)

Example 3: Public health insurance

Estimation: impact on utilization of care

13

(14)

Example 3: Public health insurance

Results: impact on utilization of care

14

(15)

Example 3: Public health insurance

Estimation: impact on health outcomes

15

(16)

Example 3: Public health insurance

Results: impact on health outcomes

16

(17)

Examples of other IV studies

Y (outcome) X (determinant) Z (instrument) paper Earnings,

happiness

College education Proximity to college Card (1993) Earnings Military service Vietnam era draft

lottery

Angrist (1990) Crime rate Police force Timing of elections Levitt (1997) Economic

performance (GDP)

Institutions Type of colonization policy

Acemoglu, Johnson &

Robinson (2001)

17

(18)

QUALITATIVE PROGRAM METHODS

18

(19)

Quantitative vs qualitative methods

•

Quantitative:

– Numbers – Breadth

– Generalizations – Controlled

•

Qualitative:

– Words – Depth – Specific – Flexible

It would be nice if all of the data which sociologists require could be enumerated because then we could run them through IBM machines and draw charts as the economists do. However, not everything that can be counted counts, and not everything that counts can be counted.

William Bruce Cameron (1963)

19

(20)

When should we employ

qualitative/quantitative methods?

• Quantitative methods:

– When we are interested in the impact of the policy.

– When we want to know what the outcomes are.

– When we are interested in generalizability of the results.

– “How did the policy affected fertility rate?”

• Qualitative methods:

– When we want to know why the program is/is not working.

– When we want to know how it is successful – what the most important components are.

– “Why did people choose to have more children?”

– “How did the program influence family planning?”

20

(21)

Data collection

• Quantitative methods

–

Surveys, questionnaires

–

Existing databases

• Qualitative methods

–

Interviews

–

Focus groups

–

Direct observation

–

Document review

–

Participatory assessments

21

(22)

Interviews

• Asking program participants about the

program and recording their point of view on its benefits, shortcomings, …

• One-to-one interviews are suitable when:

–

The subject is sensitive.

–

People are not willing to discuss the topic in front of others.

–

Illiteracy of subjects

22

(23)

Interviews: types (1)

• Structured interview

– Interview schedule, set of written questions

– The same exact questions in the same exact order – No flexibility in wording

• Semi-structured interview

– Using interview guide – series of questions or topics

– Not necessary all questions asked to all participants and not necessarily in the same order

• Unstructured interview

– the interview proceeds at the respondent’s pace

– the subject can vary and can be chosen by the interviewee

23

(24)

Interviews: types (2)

• Face-to-face interview:

– Researcher can elicit more in‐depth responses, fill in information if a participant does not understand a question, …

– Can use visual aids, photographs, …

• Telephone interview:

– Impersonal

– Needs short introduction, well-prepared questions – Response rate may be lower

– Advantage: lower costs

24

(25)

Interviews: implementation

• Avoid long questions

• Avoid leading questions

– Try not to influence the response in a certain way – Do not use strongly positive or negative language – “Do you think this policy has increased your chances

to find a job?”

– “How do you think this policy has influences your chances of finding a job?”

• Establish time frame for the interview

• Sequence topics

• Keep neutral language

25

(26)

Focus groups

• Structured small group interviews

• Participants are usually similar in some way (certain characteristic, experience)

• Participants are required to discuss particular topic, discussion is quided by a moderator

• Ideally: 6-12 people and a moderator (note taker)

• Purpose:

– To solicit perceptions, views and opinions – In-depth analysis of certain topic

26

(27)

Focus groups: considerations

• One focus group is one observation (even if it includes 12 people)

– Need a series of focus groups

• Size of the group:

– Smaller groups if the topic is difficult or sensitive

• Homogeneity and anonymity of the group

– People are more likely to talk openly in front of people who are similar (like-minded), but who they do not know

• Leading the discussion too much vs too little

– Stay on topic, but provide enough room for participants to share their views

27

(28)

Observations

• Watching people, programs, events, …

• Involves all 5 senses (not only sight)

• Providing information about real-life situations

• Helps to understand behavior, process or situations

• Disadvantages:

– Observer’s presence can influence situation

– Potential for bias, misinterpretations, and overlook of important aspects

– Difficult to analyze

28

(29)

Observations

• Types of observations:

–

Participant

–

Unobtrusive

–

Hidden

–

Disguised

29

(30)

Qualitative data analysis

• Search for emerging themes

–

How many times has a certain issue/topic/word been mentioned?

–

How many people mentioned it?

–

Emotional response to certain questions, topics.

–

What details has been most commonly mentioned?

30

(31)

Summary of qualitative methods

• They can help us answer the “why” and “how”

questions

• They provide details, big picture, and context.

• Can help us replicate successful programs.

• Illustrates which aspects most likely

contributed to the success of the program.

31