Here is the text of another episode for Engines. For a great illustration of this idea, see this excellent web page created by undergrads at UC, Berkeley. I will try to write a second on compressive sensing, and record them together. Comments are welcome, of course:
In 1973 the University of California at Berkeley was sued for sex discrimination in graduate student admissions. The case seemed clear cut: only 35% of female compared to 44% of male applicants were admitted to graduate programs across the university. However, when statisticians looked at the data in more detail they found a surprise. When looking at the admission rates of individual departments, the apparent bias disappeared. Across departments, women were either more likely to be admitted or about equally likely to be admitted as men. Indeed, at the level of individual departments, women seemed to fare slightly better.
This is an example of Simpon’s paradox – a paradox that can affect averages whenever we combine, or pool, data. Here is another example involving two New York Yankees players, Derek Jeter and David Justice. In both 1995 and 1996 David Justice had a higher batting average than Derek Jeter. However, when we compute the batting average over both seasons, then Derek Jeter is ahead of David Justice. Again, pooling the data gives a different picture than when looking at smaller chunks.
How is this possible? Let’s look at the case of graduate school applicants to the University of California at Berkley. It turns out that more women applied to departments in the humanities, while men tended to apply in higher numbers to engineering and science departments. Humanities departments had fewer available slots, and rejected more applicants. Thus female applicants applied mostly to departments which admitted fewer students, whether male or female. As a result, the overall fraction of women admitted was lower than that of men. A bias may have existed, but it was not a bias in the rate of admissions. Rather, it was a bias in the number of women who chose to apply graduate studies in technical fields.
Simpson’s paradox can have important consequences. For example medical researchers compared a less invasive treatment for kidney stones to the established surgical methods, and found the new treatment to be better overall. However, the less invasive treatment was more frequently applied to small kidney stones. Since smaller kidney stones are easier to treat, this gave an advantage to the new, less invasive method. When the treatments were compared separately on small kidney stones and large kidney stones, the traditional treatment proved to be more successful. Taking into account kidney stone size completely changed the conclusion about which treatment is better.
The outcomes of lawsuits, promotions, and our choice of medical treatments are frequently based on numerical evidence. Yet our intuition can easily mislead us when we think about numbers. Mathematics and statistics can help – they can give us answers to the question we are asking. But it is up to us to make sure that we are asking the right questions.
The Wikipedia article on Simpson’s Paradox has a number of other good examples.
The mathematician John Tukey is credited with saying that “Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.”