Evaluation and the
causal revolution
Session 1
PMAP 8521: Program evaluation
Andrew Young School of Policy Studies
Plan for today
Plan for today
Data science and public service
Plan for today
Data science and public service
Evidence, evaluation, and causation
Plan for today
Data science and public service
Evidence, evaluation, and causation
Class details
Data science and
public service
Data and government
Data and government
Data and government
"To responsibly unleash the power of data to benefit all Americans"
11.4 million people + 23 days
Data shared between agencies - arresting officer checks database, sees if person has been cycling, sees if they need mental health services
Two jails in Florida closed after this
How do you use all this data
to make the world better?
Analyze it and
uncover insights!
(and take this class!)
What is "statistics"?
Collecting and analyzing data from a representative sample in order to make inferences about a whole population
What is "data science"?
Turning raw data into
understanding, insight,
and knowledge
What is "data science"?
Turning raw data into
understanding, insight,
and knowledge
Collect
Analyze
Communicate
What's the difference?
Collect
Analyze
Communicate
What is "program evaluation"?
Measuring the effect of
social programs on society
Data science
(data + statistics +
communication)
Causal inference
(DAGs + econometrics)
Evidence, evaluation,
and causation
Evidence-based medicine
Polio shots administered blindly with placebo - kids didn't know, doctor didn't know (double blind) - effect found
Why not just use those who didn't get a shot as the control group? Why placebo? Kids who opted out were poor (which actually had a lower incidence rate), so you can't compare them with the rich (which had higher incidence rate)
All drugs get the safety and efficacy requirement and double blind RCTs after that
Modern evidence-based medicine
Apply evidence to clinical
treatment decisions
Modern evidence-based medicine
Apply evidence to clinical
treatment decisions
Move away from clinical judgment
and "craft knowledge"
Modern evidence-based medicine
Apply evidence to clinical
treatment decisions
Move away from clinical judgment
and "craft knowledge"
Is this good?
No, this isn’t the greatest. Push for evidence-based practice can crowd out clinical judgment, which is valuable – we can’t ignore it or dismiss it
Doctors have seen things in the past. Lady Sybil in Downton Abbey and preeclampsia - family doctor knew more about the condition and Sybil—he had intuition and experience—but he was ignored
Evidence-based policy
Evidence-based policy
RAND health insurance study
Evidence-based policy
RAND health insurance study
Oregon Medicaid expansion
Evidence-based policy
RAND health insurance study
Oregon Medicaid expansion
HUD's Moving to Opportunity
Evidence-based policy
RAND health insurance study
Oregon Medicaid expansion
HUD's Moving to Opportunity
Tennessee STAR
- RAND: In a large-scale, multiyear experiment, participants who paid for a share of their health care used fewer health services than a comparison group given free care. It concluded that cost sharing reduced "inappropriate or unnecessary" medical care (overutilization), but also reduced "appropriate or needed" medical care.
- Oregon: ongoing
- MTO: Researchers found that voucher recipients lived in lower-crime neighborhoods and generally had better units than the control group families, but the experiment had no impact on earnings or educational attainment. It did, however, have unexpected results in health and happiness. Parents in families who moved to low-poverty areas had lower rates of obesity and depression, and positive impacts on behavior and outlook among young women (but not young men)
- STAR: smaller class sizes lead to better outcomes
Policy evidence industry
Policy evidence industry
Jameel Poverty Action Lab (J-PAL)
Policy evidence industry
Jameel Poverty Action Lab (J-PAL)
Campbell Collaboration
Should we have evidence for
every policy or program?
Should we have evidence for
every policy or program?
No!
Should we have evidence for
every policy or program?
No!
Science vs. art/craft/intuition
https://twitter.com/epiellie/status/1017622949799571456?s=12
In the 1970s, experts were saying that reducing the drinking age would be fine - 26 states did - created a natural quasi-experiment for researchers - Cook then found that highway fatalities increase by 10% for 18–20-year-olds. That wouldn't have been possible without legislators inadvertently performing the experiment, paving the way for better science
So knowledge flows between the two sides - use innovation from administrators as the basis for better science
Where does program evaluation
fit with all this?
Where does program evaluation
fit with all this?
It's a method for collecting evidence
for policies and programs
Types of evaluation
Needs assessment
Design and
theory assessment
Process evaluation
and monitoring
Impact evaluation
Efficiency evaluation (CBA)
Theories of change
Impact evaluation!
Theory → impact
Godwin's Law for statistics
Godwin's Law for statistics
Correlation does not
imply causation
Godwin's Law for statistics
Correlation does not
imply causation
Except when it does
Godwin's Law for statistics
Correlation does not
imply causation
Except when it does
Even if it doesn't,
this phrase is useless
and kills discussion
Correlation vs. causation
How do we figure out correlation?
How do we figure out causation?
Correlation vs. causation
How do we figure out correlation?
Math and statistics
How do we figure out causation?
Correlation vs. causation
How do we figure out correlation?
Math and statistics
How do we figure out causation?
Philosophy. No math.
How does we know if X causes Y?
How does we know if X causes Y?
X causes Y if…
How does we know if X causes Y?
X causes Y if…
…we intervene and change X
without changing anything else…
How does we know if X causes Y?
X causes Y if…
…we intervene and change X
without changing anything else…
…and Y changes
Y "listens to" X
Y "listens to" X
"A variable X is a cause of a variable Y if Y in any way relies on X for its value.… X is a cause of Y if Y listens to X and decides its value in response to what it hears"
(Pearl, Glymour, and Jewell 2016, 5–6)
Y "listens to" X
"A variable X is a cause of a variable Y if Y in any way relies on X for its value.… X is a cause of Y if Y listens to X and decides its value in response to what it hears"
(Pearl, Glymour, and Jewell 2016, 5–6)
Y doesn't necessarily listen only to X
Y "listens to" X
"A variable X is a cause of a variable Y if Y in any way relies on X for its value.… X is a cause of Y if Y listens to X and decides its value in response to what it hears"
(Pearl, Glymour, and Jewell 2016, 5–6)
Y doesn't necessarily listen only to X
A light switch causes a light to go on, but not
if the bulb is burned out (no Y despite X), or if
the light was already on (Y without X)
Causal relationships?
Causal relationships?
Lighting fireworks causes noise
Causal relationships?
Lighting fireworks causes noise
Rooster crows cause the sunrise
Causal relationships?
Lighting fireworks causes noise
Rooster crows cause the sunrise
Getting an MPA/MPP increases your earnings
Causal relationships?
Lighting fireworks causes noise
Rooster crows cause the sunrise
Getting an MPA/MPP increases your earnings
Colds go away a few days
after you take vitamin C
Causation
Causation =
Correlation + time order +
nonspuriousness
Causation
Causation =
Correlation + time order +
nonspuriousness
How do you know if you have it right?
Causation
Causation =
Correlation + time order +
nonspuriousness
How do you know if you have it right?
You need a philosophical model
Causation
Causation =
Correlation + time order +
nonspuriousness
How do you know if you have it right?
You need a philosophical model
That's what this class is for!
The causal revolution
Causal diagrams
Directed acyclic graphs (DAGs)
Graphical model of the process that generates the data
Maps your philosophical model
Fancy math ("do-calculus") tells you what to control for to isolate and identify causation
Class details
Plan for the class
Textbooks
Class technology
The tidyverse
The tidyverse
From "Master the Tidyverse" by RStudio
R code, but reads like English!
strike_damages_month <- bird_strikes %>% group_by(Month) %>% summarize(total_damages = sum(Cost, na.rm = TRUE), average_damages = mean(Cost, na.rm = TRUE))ggplot(data = strike_damages_month, mapping = aes(x = Month, y = total_damages)) + geom_col() + scale_y_continuous(labels = dollar) + labs(x = "Month", y = "Total damages", title = "Really expensive collisions happen in the fall?", subtitle = "Don't fly in August or October?", caption = "Source: FAA Wildlife Strike Database")Sucking
“There is no way of knowing nothing about a subject to knowing something about a subject without going through a period of much frustration and suckiness.”
“Push through. You’ll suck less.”
Hadley Wickham, author of ggplot2
Sucking
Am I making you computer scientists?
Am I making you computer scientists?
No!
Am I making you computer scientists?
No!
You don't need to be a mechanic to drive a car safely
Am I making you computer scientists?
No!
You don't need to be a mechanic to drive a car safely
You don't need to be a computer scientist
or developer to use R safely
Words of wisdom from @allison_horst to her data analysis class this quarter: You don't need to be a mechanic to drive a car safely, and you don't need to be a computer scientist or developer to use R safely. #rstats #tidytuesday
https://twitter.com/ameliaritger/status/1214682596182904832?s=12
I always teach my students: there’s three levels of skill: driver, mechanic, engineer. You’re here to get your drivers license. But being able to change a tire can be helpful. Don’t worry not every one has to be an engineer. (I talk about R in social sciences)
Learning R
You can do this.
Goals for the class
Goals for the class
Become an expert with R
Goals for the class
Become an expert with R
Speak and do causation
Goals for the class
Become an expert with R
Speak and do causation
Design rigorous evaluations
Goals for the class
Become an expert with R
Speak and do causation
Design rigorous evaluations
Change the world with data
Prerequisites
Prerequisites
Math skills
Basic algebra
Prerequisites
Math skills
Basic algebra
Computer science skills
None
Prerequisites
Math skills
Basic algebra
Computer science skills
None
Statistical skills
Regression, differences in means, and statistical significance