I’ll be presenting a pre-circulated paper in the Johns Hopkins History of Science, Technology and Medicine colloquium series on October 31st, 2019.
Humans as a model for animals: Patterning preclinical reproducibility reform after clinical research
In 2011 and 2012, two groups of pharmaceutical industry scientists published reports on their experiences attempting to reproduce results from the biomedical literature in their in-house laboratories. The first paper, from scientists at Bayer, reported that they could not produce data that was in line with results in the published literature in 65% of cases. The second, from scientists at Amgen, reported the even more dismal finding that their attempts to reproduce published results failed in 89% of the time.
This paper asks why these two industry publications formed the discursive core of what today has become known as the “reproducibility crisis” in biomedicine. The shock value of these figures alone and the subsequent attention they received in the popular press makes their importance seem self-explanatory. And yet these were not the first publications to report dire rates of irreproducibility, or to make headlines: John Ioannidis’s now famous paper, “Why most published research findings are false,” was published in 2005, and the New Yorker published an article on reproducibility in biomedical research with the attention grabbing headline “The truth wears off” in 2010.
I argue that what made the Bayer and Amgen papers effective in garnering the attention of leaders at the National Institutes of Health was the connections they drew between clinical and preclinical research. These reports took recognized problems in the world of translational research—such as declining productivity in the pharmaceutical industry, or selective publication of clinical trial results—and reframed these as problems ones belonging to academia rather than industry. These connections, I argue, were not simply rhetorical. Clinical research reform also contributed directly to present-day preclinical reform by providing the tools reformers needed to make irreproducibility in the laboratory visible. I illustrate these connections through a case study of animal models of stroke, where researchers used meta-analysis techniques and reporting standards designed for evaluating the clinical literature to draw attention to problems in preclinical research methodology.