Topic #7: Research Illiteracy

Grasping at Headlines: Research Illiteracy Fuels the Fire of Racism

Yesterday I browsed upon the following headline you may have heard about:

"No racial bias in police shootings, study by Harvard professor shows."

As a person trained in empirical research myself I decided this was worth a closer look, and the closer I looked the uglier it became. Like most things shared on social media, the claim made by this bold headline reveals more about the people who share it than the content of the actual news piece itself. The research methodology used in the study and the media interpretations of the findings are so problematic that its best purpose would be to teach graduate student researchers what not to do. So what, exactly, makes this study so full of holes?

Problem 1: The study in question is not, in fact, published research.

The study, posted on the author's homepage, shows that it is not actually a published article but a "working paper." The piece has not been accepted in a peer reviewed journal, which is the basic requirement for research to be considered valid and useful. Why? Because otherwise, this blog would be considered published by virtue of me posting it online, and the verbal survey I administered to my family about what to have for dinner would be considered research. If we put stock in every working paper, most of which never make it to peer-reviewed publication because of methodological or other flaws, then we would be consumers of ideas that are not grounded in fact and real findings. We would be following the messy trial and error learning curve associated with poorly conducted studies by graduate students and other research amateurs trying to make a name for themselves. 

I'm not suggesting that the head researcher of this project is an amateur, only that his association with Harvard and the fact that he's written a paper mean nothing until a panel of experts in his area of research have scrutinized the study with a fine tooth comb to make sure it meets the standards of a peer reviewed journal. Having submitted studies of my own, some of which have been accepted for publication and others that have been rejected, I can attest that the review process is rigorous, critical and highly necessary so that the research on which we rely for our statistics and consequential policies is accurate.

Problem 2: The researcher uses an insufficient and biased sample.

The police departments represented in this study make up a sample size called "exceedingly small" by Snopes.com (and by any researcher or first-semester graduate student). An insufficient sample size is enough to get this working paper rejected from publication outright. 

Furthermore, to understand lethal use of force the author looked at only one of the cities he studied. This is called data manipulation, and it's pretty easy to do when a researcher is seeking to support his hypothesis (hence, the requirement of peer review). According to The Guardian, "Looking at just one other city would suggest very different conclusions. In Chicago, a review of the reports of each police-involved shooting looked at fatal and non-fatal shootings. Despite the city being one third black, a disproportionate 118 black males (44 of them fatal) were involved in the 150 shootings recorded since 2010." (https://goo.gl/8D0McX). The fact that the study's findings don't apply beyond the specific places examined breaks the basic research tenet of generalizability. To be useful it must be generalizable beyond the small sample studied. 

Perhaps worst of all, this study relies on information that was volunteered by police departments. We are not getting any data from police departments that were unwilling to cooperate (read: ones that know their numbers show biases). Last but not least, the data were collected by interviewing police officers and coding their narratives rather than considering the testimonials of witnesses or suspects. This is called "self-report" and is considered by most empirical scientists the most unreliable type of data a researcher can collect.

Problem 3: The researcher's project to disprove racial bias by police is anonymously funded.

Huh? I can't even waste time explaining the problem with this.

Problem 4: The study has not demonstrated replicability.

As explained in one article, "Fryer’s conclusions about police use of lethal force will have to be reconciled relative to work by Cody Ross [and others] which shows that America’s cops are at least 3 times more likely to shoot unarmed black people than they are unarmed whites. Likewise, social psychologists and others have demonstrated that subconscious racism/implicit bias influences police to shoot black people faster than whites." (http://goo.gl/3Z5fHt). One of the most basic and important rules of research is that a study must be able to be replicated with the same results in order to be considered valid. On these grounds, multitudes of suspicious research findings have been rejected and even revoked from the literature. Fryer's single study contradicting an entire literature base (not to mention the collective testimony of the Black lived experience) will have to be tested across better samples, with the exact same or similar results for it to be considered useful in any way. 

It's generally the fault of main stream media when a research study is misinterpreted, which is why my instinctive reaction is to kindly ask folks who quote it whether or not they read the entire study for themselves. All told, I'm less concerned with the quality and applicability of one study's findings, and more concerned with the excitement expressed by the those seeking evidence to derail the movement for racial justice in our country. Unquestioningly sharing one unfounded claim by someone representing a prestigious institution reveals more about you than it reveals about the reality of racial disparity. In short, it shows a lack of critical thinking skills, inability to read and evaluate academic research, and over reliance on dangerous and erroneous media soundbites.

The mountain of evidence in support of our country's need to address and remedy racial disparity speaks for itself, and those still refusing to accept it are desperately grasping at their dying white supremacy and privilege. To their great disappointment, the paper written by Roland Fryer won't be enough, or even close to enough, to keep it alive.