ILA | New York Times Resurrects Flawed Study

Bad anti-gun research, and the sensationalist journalism that often accompanies it, provide unending opportunities for the most used (overused?) of all firearm themes: Zombies. Just when you think the thing is gone for good, it appears again.

Such was the case with a study we deconstructed last summer, “The Association of Firearm Caliber With Likelihood of Death From Gunshot Injury in Criminal Assaults.” Coverage of the research was revived by the New York Times in an article published on March 27, 2019. In the Times piece, writers Margot Sanger-Katz and Quoctrung Bui recounted the inquiry by Profs. Anthony Braga and Phillip Cook (Northeastern University and Duke University, respectively):

In Boston from 2010 to 2015, there were 221 gun homicides.

Research suggests that one change could have lowered that number by 40 percent: smaller bullets.

A study last year, published in JAMA Network Open, examined the type of weapon used in every fatal and nonfatal shooting in the city. It found that — regardless of the time of day, the number of wounds or the circumstances of the crime — the size of the bullet affected which gunshot victims lived and which ones died.

The research was better summarized, however, by NRA-ILA’s August 2018 headline: “Government-funded Researchers Discover Basic Ballistics.”

That’s right, larger-caliber firearms generally deliver a more powerful payload. Hard to imagine, but somehow the study’s authors – echoed by the Times – made the leap from that fact, confirmed by statistical analysis of shooting incidents in Boston (the study’s geographic focus), to the conclusion that prohibiting the technology- and market-based diffusion of larger-caliber semiautomatic handguns in the 1990s would have resulted in fewer deaths by shooting. Braga and Cook (2018, p. 7) stated, “the implication is that if the medium- and large-caliber guns had been replaced with small caliber (assuming everything else unchanged), the result would have been a 39.5% reduction in gun homicides.”

And, continuing to an ad absurdum conclusion, the authors hypothesized, “it is plausible that larger reductions would be associated with replacing all types of guns with knives or clubs” (emphasis added). So much for advocating for “reasonable restrictions” on guns.

The authors, of course, quickly dismiss that it is gun control measures on “Saturday Night Specials” (small, usually low caliber handguns) that likely led to an increase in the number larger caliber handguns used in crime. It’s inconvenient when your “study” that is supposed to advocate for gun control reveals the flaws of past gun control measures.

When studying firearm lethality, it helps to have even a passing knowledge of the things you claim to be studying. Authors who claim to study the relationship between handgun caliber and shooting fatality should have known better than to include in their analysis a case in which the victim was shot with a 7.62 x 39mm round.

And, even the most novice gun owners understand that caliber (and really cartridge) selection is a matter of trade-offs. All else being equal, larger caliber firearms usually result in greater recoil and less capacity. Braga and Cook should probably tell the most data-driven federal law enforcement agency that its decision to reduce the caliber of its standard service pistol is going to make its agents less effective gunfighters. 

Lack of basic firearms knowledge aside, there is potentially a more fundamental weakness with the analysis. First, a summary:

Braga and Cook (2018) assessed the relationship between characteristics of the victim (e.g., demographics, prior arraignments), circumstances of the crime (e.g., indoor vs. outdoor), and caliber used. The intent of this portion of the research was to determine whether caliber was a random variable – i.e., independent of other characteristics of the shooting – or whether it was related systematically to those characteristics. (For example, one hypothesized relationship might be that gang-related shootings consistently involved the use of large-caliber firearms.) Findings indicated no statistically significant relationships between those measured situational variables and firearm caliber, lending support to the idea that shooters’ choice of round size was independent of any characteristics of the crime or the criminal.

The key analysis was a regression to test the effect of gun caliber – not cartridge, bullet weight, projectile design, or other theoretically relevant elements – on whether or not a victim lived or died. Results showed that, “relative to shootings involving small-caliber firearms … the odds of death if the gun was a large caliber were 4.5 times higher and, if medium caliber, 2.3 times higher” (Braga & Cook, 2018, p. 6). Covariates (i.e., control variables) including victim gender, race, age, and prior arraignments all lacked statistical significance. (As mentioned earlier, indoor location did significantly boost the odds of a fatality.)

That all makes sense. As we observed in our prior critique of the study, those are “findings obvious to any middle school physics student.” But the potential weakness teased above involves control variables that the researchers did not include in the test between caliber and likelihood of fatality: the number of wounds and their location. Intuitively, one would suppose that more bullet wounds, and in more serious locations, would generally increase the potential for a fatality. And, as discussed, the researchers noted significant relationships between those variables and fatalities.

So, why didn’t the researchers attempt to control for the number of wounds and their location while measuring the association between caliber and fatality? Why did the Times writers misstate the study conclusions in this regard?

We don’t know, and we also don’t know for certain whether those factors would have mitigated the significance of the caliber-fatality connection. What we do know, however, is that you can’t answer questions that aren’t asked. And, if someone aimed to demonstrate that replacing large-caliber guns with smaller calibers would result in fewer fatalities – which the Braga and Cook (2018) model did do – including covariates that might mitigate the association probably would not be the best thing.

As even casual consumers of undead cinema know, shot placement, not caliber, is king.