"Using weekly data from 2004 to 2013, we found a direct correlation between anti-Muslim searches and anti-Muslim hate crimes." That's what Evan Soltas and Seth Stephens-Davidowitz wrote in a powerful op-ed piece for the New York Times.
It makes sense on an intuitive level - that if more people are typing "I hate Muslims" into their search engine, you're likely to see more hate crimes.
But as a smart consumer of data, you need to look beyond intuition, and consider where the data is coming from - and what it means. For example:
- The authors appear to focus on data from Google. While Google holds 68% of the search engine market, there are certainly other search engines. Would they show the same type of data?
- What about Islamaphobes who don't type their sentiments into a search engine? How many people express their anti-Muslim thoughts through search, vs. postings in social media or other, non-digital mediums? (The authors address this selection bias in an interesting way, claiming that it may actually bolster their point of view, as people who search online are more passionate about an issue, according to a political scientist they interviewed. We're not convinced.)
- Correlation does not equal causation, yet the authors claim "we can explain some of the rise and fall of anti-Muslim hate crimes just based on what people are Googling about Muslims." This is a bold claim, and (from what we've seen so far) it's not entirely backed up by the data. To the authors' credit, they only say they can explain "some" of the rise - but even that could imply a causal relationship (rather than simply a correlation).
- Look at the sample size. In 2014, according to the authors' data (sourced from the FBI), there were fewer than 180 hate crimes against Muslims. That's still 180 too many - but, from a statistical standpoint, is it truly a large enough sample size to draw a meaningful statistical relationship?
- All hate crimes are not equal. A murder can be a hate crime. So can arson. So can vandalism. We're not condoning any of these, of course - simply pointing out how data labels can obscure deeper, relevant data.
Why does all of this matter? Consider one of the authors' conclusions: "police departments would be wise to use search data to allocate resources through predictive policing." If you accept the authors' data, this may be a perfectly logical conclusion (although we'd still have some concerns about the ability to predict the future.) But the authors' conclusion is not nearly as solid when you consider just some of the points we've addressed here.