Across the U.S., demonstrators have spent the past few weeks protesting against racial disparities in the country’s criminal justice system. There’s plenty of data to back them up: Black and Hispanic people are stopped more frequently, including traffic stops, and are more likely to be arrested. Once stopped, police are more likely to use force against, shoot and kill Black citizens. And then once in jail, Black defendants are more likely to be denied bail, which in turn makes conviction more likely. And when convicted, sentencing is also biased against Black defendants, with Black defendants more likely to be incarcerated.
The data seems to overwhelmingly point to a criminal justice system riven by racial bias. But, remarkably, it could be even more overwhelming than some studies make it seem. That’s because of a statistical quirk called “collider bias,” a kind of selection bias that means that the crime data that shows racial bias is, itself, biased by racist practices. If you thought crime data showed clear evidence of racism before, understanding how collider bias affects these analyses might make it even clearer.
To understand how all this works, we’re going to get mathy. In particular, we’re going to be talking a lot about denominators, which, if you’ll remember your fourth grade math lessons, are the numbers on the bottom of a fraction. For our purposes, though, you can think of them as the universe of people who are being studied.
[Related: Is Police Reform A Fundamentally Flawed Idea?]
Police engage with only a small subset of the population they see, so if we look at statistics about those interactions, the stats we get are informed by that smaller sample. And if there’s bias in who the police choose to interact with — if it’s not a random sample — that can change the relationships you see in the data.
There are two main ways that researchers approach this problem — by using a “population denominator” and an “encounter denominator.”
The former, for example, compares the fraction of Black people in the general population who are arrested or harmed to that same statistic for white people. That’s how you get studies that show 96 out of 100,000 Black men and boys will be killed by police over the course of their lifetimes, compared to 39 out of 100,000 white men and boys — a risk that is 2.5 times higher. Seems straightforward enough. But because Black and white people encounter the police at different rates to begin with, using population denominators might not lend itself to an apples-to-apples comparison; being more likely to encounter police means that you’re more likely to encounter police force, too.1
For this reason, many researchers choose to look at the set of people who have encountered the police. This is the “encounter denominator.” The setup is simple: You look at all the people who had recorded encounters with police — data which is not always easy to obtain — and calculate the proportion that involved the use of force. But this approach has a different issue, as a recent paper pointed out — if there’s bias in who gets stopped in the first place, then looking at discrepancies in the resulting interactions won’t give you the full picture. This is because of something called “collider bias.”2
“The vast majority — 99.9 percent of the data — we never get to see,” said Dean Knox, a professor of politics at Princeton and one of the authors of the study. “We just don’t see all the times when police officers are encountering civilians on the street. And that’s a huge problem, because among the data that you do get to see — the stops, and the arrests, and the use of force that officers record — those are already contaminated, because officers have discretion in who they choose to engage.”
The fact that researchers’ data comes from a biased sample — who the police choose to stop — rather than the full sample of possible stops might skew the conclusions we draw from it. “If we’re using administrative data, we always need to be aware of what world those data capture, and what they don’t capture,” said Allison P. Harris, a political science professor at Yale who studies racial disparities in the criminal justice system. “With policing, we just can never know what there’s no record of.”
[Related: Why It’s So Rare For Police Officers To Face Legal Consequences]
One example of an encounter denominator approach is a 2019 study by Roland Fryer, an economist at Harvard. He found that police shoot white, Black and Hispanic Americans whom they’ve stopped at equal rates.3 At first blush, that would seem like evidence that the police are not racially biased — every demographic is being treated equally, after all.
But we know that police officers are more likely to stop Black and Hispanic people than white ones — and that more of those stops are unfounded. Researchers measure this with something called the “hit rate,” or the rate at which contraband is actually found on the people who were stopped. A lower hit rate implies bias because it means that the decision to search someone was made with less evidence. White people stopped in New York City, for example, were more likely to be carrying a weapon than Black and Hispanic people who were stopped. White drivers stopped by the police were more likely to have contraband than Black and Hispanic drivers nationally.
As political scientists Knox, Will Lowe and Jonathan Mummolo, among others, have pointed out, that complicates Fryer’s findings. All of a sudden, what at first appeared to be equal treatment actually suggests unequal treatment. Because of the initial discrimination in who gets stopped, the sample of stopped people isn’t the same across races. The different hit rate indicates that stopped white people are actually more likely to have contraband, on average, than stopped Black people. In other words, in a world without discrimination in who was stopped — if the Black and white people who were stopped were equally likely to be engaged in criminal activity — you’d see an even bigger disparity in outcomes.
It’s unintuitive, I know, but let’s break it down visually.
“The data we have pick up halfway through the encounter, after officer bias very likely has already exerted an effect,” said Mummolo, a professor at Princeton. And without knowing the racial composition of the people who are sighted by police but aren’t stopped, it’s impossible to fully correct for the bias. The paper provides a suggestion for how to estimate bias in one part of the chain — use of force in encounters — but it doesn’t capture any of the potential disparities that occur beforehand that make the encounter likelier.
So if racial bias in police stops makes it hard to estimate racial disparities in how people are treated while stopped, you might be wondering: well, doesn’t this affect all of the other components of the chain, too?
“We need to think about this as this complex system,” said Knox. There are biases layered on biases — social decisions like how society allocates resources in education versus policing, and individual decisions like who police choose to stop or arrest and who is ultimately charged or convicted. “It just compounds all the way down the line,” Knox said.
[Related: The States Taking On Police Reform After The Death Of George Floyd]
The thing about systemic racism is it’s just that: systemic.
Ultimately, the fact that these biased events build on each other should be a reminder that we’re not capturing the entire process surrounding the stats we’re looking at. Essentially, all the discrimination we can actually measure at each specific stage is an underestimate of how much is actually happening.