In 2014 municipal leaders in Flint, Michigan changed the source of the city’s drinking water and accidentally poisoned almost 100,000 people. Suddenly, drinking water in thousands of Flint homes contained toxic amounts of lead. The number of Flint children with elevated lead levels doubled.
The next year student test scores in Flint fell sharply. The Washington Post knew what to blame: “the disaster’s profound impact on a generation of children.” It’s a powerful claim, and it sounds logical; children consumed a toxin that causes cognitive impairment, and that caused their scores to suffer.
But it’s not that simple. Causation rarely is. And as Kevin Drum at Jabberwocking points out, something else happened in 2015: Michigan introduced new statewide testing. So as test scores fell in Flint they also fell in Detroit and Grand Rapids and across the state, all by about the same amount. It turns out the new test was just harder than the old one. As the head of Michigan’s Department of Education said, “With this all-new and more rigorous test, we expected statewide student scores to be lower.”
Reading scores in Flint went down about the same amount in 2015 as in Detroit and across the state. Source: Clinical Psychology in Europe via Jabberwocking.
So while there’s a strong correlation between the Flint water crisis and falling test scores, it’s not at all clear the one caused the other. Just like pirates probably don’t stop global warming and margarine is unlikely to cause divorce.
If you want to show causation, you can’t just present two facts that sound logical together — you have to do the homework and the math. The authors of this study definitely did their math. They constructed sophisticated models to control for gender, grade, socioeconomic variables, district size, and more. But it appears they didn’t do their homework. They fail to even mention the new statewide test’s impact on scores. Perhaps more importantly, they brush off a crucial problem in their data: Although children living in Flint homes with lead pipes consumed 4.5 times more lead per day than children in homes with copper pipes, the type of pipe in a student’s home (and therefore the amount of lead in their water) made no significant difference in their test scores.
Here’s a more rigorous example of what it takes to show causation from a paper by award-winning economics professors from MIT and the University of Chicago:
Nearly-indecipherable causation calculations from an academic study. Source: University of Chicago.
Yeah, I can’t do that kind of math either. And this is just a small excerpt from a 136-page paper.
Amazingly, even with that level of analysis the authors aren’t certain they’ve proved causation. They note one quirk in the data and say “we interpret [this] as reassuring that our results are picking up the causal impact of the Great Recession, rather than spurious factors correlated with the size of the Great Recession shock.”
So unless you’ve got a PhD or two (and you’ve scoured the literature for other variables), don’t assume or proclaim causality. Correlation is easy to demonstrate; causation is hard.
Thanks for reading. What’s the most dubious causation claim you’ve seen? Post it in the comments below or on LinkedIn. And if you want good data delivered to your inbox, subscribe here.