Review of “Statistics Done Wrong” by Alex Reinhart
This is a book review for the applied math journal SIAM Review. Comments are welcome. An short version of the book can be found here.
Most of us accept that statistics is not applied mathematics. The goal of statistics is to obtain answers from data, and mathematics just happens to be an exceptionally useful tool for doing so. However, for many applied mathematicians
statistical analysis is an integral part of daily work. Experimental studies often motivate our research, and we use data to develop and validate our models. To understand how experimental outcomes are interpreted, and to communicate with scientists in other fields, a knowledge of statistics is indispensable.
One issue that we need to take seriously is that misapplications of statistics have lead to false conclusions in much, maybe even most published studies . Although the “soft sciences” have received most scrutiny in this regard , the “hard sciences” suffer from closely related problems . Anybody who uses the results of statistical data analysis – and this includes most applied mathematicians – needs to be aware of these issues.
As the title suggests, Alex Reinhart’s “Statistics Done Wrong”  is not a textbook. Rather, its aim is to explain some of
the ways in which data analysis can, and often does go wrong. The book is related to Darrel Huff’s classic “How to Lie with Statistics”  which covers many topics that are now part of freshman courses. Huff, a journalist (and later consultant to the tobacco industry) provided a lively discussion that alerted general readers to the misuses of statistics by the media and politicians. Since the first edition of Huff’s book in 1954 computational power has increased immensely. But our increased ability to collect and analyze data has also made it easier to misapply statistics. Reinhart’s book aims to introduce the present consumers of statistics to the resulting problems, and suggests ways to avoid them.
The book starts with a review of hypothesis testing, p-values and statistical power. Here Reinhart introduces a recurrent topic of the book: the errors and “truth inflation” due to the preference of most journals and scientists to publish positive results. The ease with which we can analyze data makes the problems of multiple comparisons and double dipping particularly important. The book provides a number of thoughtful examples to illustrate these issues. The last few chapters provide good guides to data sharing, publication, and reproducibility in research. Each chapter ends with a list of useful tips.
Most of these issues are more subtle than those discussed by Huff . While not heavy on math, the book presents arguments that require reflection. The ideas are frequently illustrated using well chosen examples, making for an entertaining read. The book is thus informative, yet easy to read.
Reinhart predominantly discusses issues resulting from the misuse of frequentist statistics. This is understandable, as the frequentist approach is currently dominant in most sciences. However, it is worth noting that Bayesian approaches make it easier to deal with some of the main problems discussed in the book. Bayesian statistics makes it easier to deal with multiple comparisons, and replaces p-values with measures that are easier to interpret. However, it is not a magic bullet – as Bayesian approaches become more common over the next decades, we may need another volume describing their misuses.
What is the audience for this book? Many of the topics need to be familiar to anybody doing science today. The book could also provide good supplementary material for a second course in statistics.
Doing statistics can be tricky. Finding the right experimental design requires a careful consideration of the question to be answered. The interpretation of the results requires a good understanding of the methods that are used. All statistical models are by necessity approximate. Knowing how to verify that the underlying assumptions are reasonable, and choosing an appropriate way to analyze data is essential. A central point here is that the statistical analysis deserves as much
attention as the conclusions we draw from it. And perhaps the most important lessons of this book is that questions of statistical analysis should be addressed when the research is planned.
Reinhart’s book is not a comprehensive list of the different ways in which misuses of statistics can lead us astray. It provides no foolproof answers on how to detect problems in statistical analysis. However, it does an excellent job of introducing a range of common pitfalls, and provides sensible tips on hows to avoid them. Doing statistics means accepting that we will be wrong some of the time. The best we can do is to maximize our chances of being right, and understand how likely it is that we are not.
- Ioannidis, J. P. A. Why most published research findings are false. PLoS Medicine, 2:8, (2005) e124. http://doi.org/10.1371/journal.pmed.0020124
- Button, K. S., et al. Power failure: why small sample size undermines the reliability of neuro- science, Nat Neuroscience, 14, p. 365-376 112. (2013) http://doi.org/10.1038/nrn3475
- Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349:6251,p. 943. (2015) http://doi.org/10.1126/science.aac4716
- Reinhart, A. Statistics done wrong. No Starch Press (2014).
- Huff, D. How to Lie with Statistics. Norton, W. W. & Company, Inc. (1954).