Skip to content

Simpson’s paradox

Here is the text of another episode for Engines. For a great illustration of this idea, see this excellent web page created by undergrads at UC, Berkeley.  I will try to write a second  on compressive sensing, and record them together. Comments are welcome, of course:

In 1973 the University of California at Berkeley was sued for sex discrimination in graduate student admissions. The case seemed clear cut: only 35% of female compared to 44% of male applicants were admitted to graduate programs across the university. However, when statisticians looked at the data in more detail they found a surprise. When looking at the admission rates of individual departments, the apparent bias disappeared. Across departments, women were either more likely to be admitted or about equally likely to be admitted as men. Indeed, at the level of individual departments, women seemed to fare slightly better.

This is an example of Simpon’s paradox – a paradox that can affect averages whenever we combine, or pool, data. Here is another example involving two New York Yankees players, Derek Jeter and David Justice. In both 1995 and 1996 David Justice had a higher batting average than Derek Jeter. However, when we compute the batting average over both seasons, then Derek Jeter is ahead of David Justice. Again, pooling the data gives a different picture than when looking at smaller chunks.

How is this possible? Let’s look at the case of graduate school applicants to the University of California at Berkley. It turns out that more women applied to departments in the humanities, while men tended to apply in higher numbers to engineering and science departments. Humanities departments had fewer available slots, and rejected more applicants. Thus female applicants applied mostly to departments which admitted fewer students, whether male or female. As a result, the overall fraction of women admitted was lower than that of men. A bias may have existed, but it was not a bias in the rate of admissions. Rather, it was a bias in the number of women who chose to apply graduate studies in technical fields.

Simpson’s paradox can have important consequences. For example medical researchers compared a less invasive treatment for kidney stones to the established surgical methods, and found the new treatment to be better overall. However, the less invasive treatment was more frequently applied to small kidney stones. Since smaller kidney stones are easier to treat, this gave an advantage to the new, less invasive method. When the treatments were compared separately on small kidney stones and large kidney stones, the traditional treatment proved to be more successful. Taking into account kidney stone size completely changed the conclusion about which treatment is better.

The outcomes of lawsuits, promotions, and our choice of medical treatments are frequently based on numerical evidence. Yet our intuition can easily mislead us when we think about numbers. Mathematics and statistics can help – they can give us answers to the question we are asking. But it is up to us to make sure that we are asking the right questions.


The Wikipedia article on Simpson’s Paradox has a number of other good examples.

The mathematician John Tukey is credited with saying that “Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.”

How to make decisions when the environment is changing

I have recently revisited the following old problem in a new paper with Alan Veliz-Cuba and Zack Kilpatrick: Suppose that you are trying to decide between different alternatives. Maybe you are trying to find the best place to fish, behind which bush to look for your dinner, or between different products in a super market. Humans, animals, and even animal collectives decide between alternatives all the time. Such decisions are frequently not based on a single observation. Indeed, animals accumulate evidence, and combine bits of information to arrive at a better choice.

There has been a lot theoretical work to determine the best way to integrate information that arrives over time. There are also fascinating experiments that suggest that certain animals make decisions in a way that is very similar to the theoretical models. Indeed, neuroscientists have even capture the neural signature of such processes.


This figure shows the activity of neurons in area VIP of monkeys that are deciding whether a cloud of dots moves predominantly to the right or to the left.  As the animal accumulates information for one or alternative, the activity of a cell increases. Figure from Gold and Newsome, Annual Review of Neuroscience, vol. 30, p. 535 (2007).

However, in most classical studies of such decision making the correct (or better) choice is fixed during a trial (For a recent study where this is not the case, see here). Unless there is a some pressure to make a decision quickly, in such cases it is best to accumulate as much evidence as possible, that is, wait indefinitely to make sure that you are making correct choice. However, the natural world constantly changes, and what is a correct choice or better option at one instant, may no longer be so in the next. Our goal was to extend classical models to the case where the truth is not constant.

The model that we derived has some interesting features: An ideal evidence accumulator will discount prior evidence in a way that is determined by the volatility of the environment. In other words, to perform well in a changing environment it is best to forget evidence that is too old, as it is no longer pertinent. Indeed, our model shows that you want to keep evidence that has arrived over about one average environmental epoch. As a consequence, even if for some reason the best option does not change for an extended time, there is a limit to the certainty you can attain about your choice. Your certainty is limited by the amount of information you keep in memory.

The nice thing about differential equation models is that they suggest plausible neural implementations. We proposed such a model with the activity in different neural populations representing the evidence for different choices. Interestingly, unlike in classical models, the different populations here are coupled through excitation.

You can find the paper on BioRxiv.

Evolution in a test tube

The amazing variety of life on this planet is a product of evolution. However, it took billions of years for sharks, chimps and magnolia to evolve from their common ancestor. Given that evolution operates on such an enormous timescale, how could we possibly study it in a laboratory? Human life just seems too short.

But not all evolution is slow. Within our lifetime bacteria have evolved defenses against the most powerful antibiotics. Indeed, many antibiotics are themselves the result of evolution fueled warfare between different bacteria and between bacteria and the fungi they attack. We simply borrow some of their weapons. But, bacteria that are not killed by antibiotics can prosper. They give rise to new resistant generations, rendering our weapons useless. This type of evolution can occur within days, and if we don’t discover new drugs, the resulting antibiotic resistance may end up costing millions of lives by the middle of this century.

Can we control how organisms evolve? Our ancestors have done this to suit their own ends: dogs and wheat are in their current form a result of evolution that humans have been steering.

Scientists have tried to do this more deliberately. Perhaps the first was the Reverend William Dallinger. Just over 20 years after Darwin published his theory of evolution, Reverend Dallinger examined whether single celled organisms could adapt to slow changes in their environment. He started with an incubator filled with microbes that could initially only survive at room temperature. Over six years, the Reverend slowly increased the temperature inside the incubator to 158 degrees F to see whether the microbes would adapt.

More recent versions of this experiment are being carried out by many scientists, including Tim Cooper at the University of Houston, and Richard Lenski at Michigan State University. In 1988, Lenski started growing bacteria giving them just enough food to survive from day to day. He has been observing 12 different lines of bacteria ever since. This amounts to over 60,000 bacterial generations, equivalent to about 1,500,000 human years – longer than our species has been around.

The results of this experiment are giving extraordinary insights into how life changes and adapts. The 12 different lines of bacteria have all evolved to thrive on their meager diets. Looking at their genes reveals that they have often used the same tricks – the same mutations – to achieve this. But in one line something unexpected happened. The bacteria started feeding in a completely new way, a change similar to us evolving the ability to eat wood.

So the churn of mutations, and transfer of genes, keeps creating variants of organisms that have never before existed.  Most quickly disappear.  A few succeed and create offspring that inherit their parents’ characteristics.  And so – as Darwin wrote at the end of his Origin of Species – “from so simple a beginning endless forms most beautiful and most wonderful have been, and are being, evolved.”

 Additional Notes:

We have become so used to antibiotics that it is hard to imagine what medicine would look like without them. Antibiotics are not only used to treat infections, but also make many other medical treatments possible by preventing infections in the first place. Here evolution works against us – medicine without antibiotics will look very different indeed.

Here are more details about how Reverend Dallinger, a devout Methodist, came to accept evolution 

Some of Richard Lenski’s reflections on his long running experiment.




Coverage of our work in SIAM News

The last two issues of SIAM News both have an article that describes some collaborative efforts with different colleagues. Eric Shea-Brown writes about our joint effort to relate the patterns of connections between the neurons in the brain to their joint dynamics, and ultimately function. Dana Mackenzie gives a nice overview of interactions between mathematicians and synthetic biologists. This includes a description of some joint work with Will Ott (UH) and Matt Bennett (Rice). Unfortunately, the second article does not mention the two postdocs who did most of the work: Faiza Hussain (Rice) who did the majority of the experimental work, and Chinmaya Gupta (UH) who did most of the modeling. The press release from UH is here.

The El Farol Problem

When asked why he no longer frequented Ruggeri’s restaurant in St. Louis, Yogi Bera famously replied: “Nobody goes there any more. It’s too crowded.” As other Yogi Bera quotes, this one is silly, yet insightful. And it is related to an interesting problem at the interface of mathematics and economics.

The problem was inspired by the El Farol bar in Santa Fe, near the famous Santa Fe Institute for the study of complex systems. Scientists at the institute like to frequent El Farol on Thursdays to listen to live Irish music. However, the bar is too small for all of them. Let’s assume that if more than half of them decide to go, it is so packed that nobody has a good time. However, when the bar is not overcrowded, an evening of music is more fun than staying at home. Importantly, in our example everybody needs to make up their mind at the same time about whether to go or not. Nobody can call ahead to see how many people are at the bar, or coordinate with others.

How can a patron decide whether to go or not? If everybody acts the same, then everybody either stays at home, or everybody goes to the bar. Neither of these outcomes is optimal, since the bar is either completely empty or completely packed.

But these are scientists. They observe the bar and track how crowded it is from one week to the next. They look for patterns in attendance to help them decide whether to go or not. Maybe the bar was nearly empty the last couple of weeks, indicating that it might be empty again this week. Or maybe there is a cycle with the bar overcrowded one week, and nearly empty the next. Thus each scientist develops a strategy to translate these observations into a decision.

But here is the catch – there is no one strategy that guaran-tees success. If there was, everybody would be using it. But with the same strategy everybody would again be making the same decision each week, and the bar would be either overcrowded or empty. We would be where we started. The best hope is that the scientists choose different strategies. Some stay home, and each Thursday the bar is filled exactly to capacity.

This may seem like a frivolous problem. However, it has been studied extensively by mathematicians and economists as a simple model of a market. Indeed, the scientists compete for a resource – music at a bar. They are rational, as they monitor attendance and use this information strategically. But, as in a real market, they have limited information – they do not know the strategies of others, only how many show up each week. To do well, everybody needs to keep learning and adapting.

But the El Farol problem applies more widely. Suppose that instead of a bar and patrons we are think of the ocean and fisher-man. If all go out to fish, the stock will collapse. But if none do, many will go hungry. Or think of how we make use of our environment. And consider, that unlike the patrons of the El Farol bar as residents of planet Earth, we will not get a chance to try again, if we overcrowd it and overtax its resources.

Some Notes:

The original El Farol problem was proposed in a very readable paper by Brian Arthur. In the original paper, the assumption is that if more than 60% of potential patrons visit the bar, then it will be overcrowded. I have changed this to make the exposition a bit simpler, and kind of combined the El Farol problem and the Minority Game. These are similar, but not the same. You can find the origi-nal paper here.

As noted, the El Farol problem has an even simpler version – the Minority Game. The idea is that each agent plays in successive rounds of a game where there are only two choices A and B. Any agent in the minority wins a round. Everybody in the majority loses. For instance if during one round most agents chose A, then all those who chose B win a set amount. Each agent learns a strategy based on their previous choices. Here is an introduction, and an accessible overview which discusses implications for economics can be found here.

The ugly secrets behind magic tricks

A recent Radiolab episode described the magic act of the Australian couple Syd and Lesley Piddington. The two claimed to be mentalists and had a radio show on which they would communicate information “telepathically” even when separated by hundreds of miles.

The entire episode is great (as usual), and I recommend it. One thing that I found interesting is Penn Jillette’s comment about secrets behind great magic tricks: They are invariably ugly and boring. Jilette says that when you learn how the trick is done, there is no “A-ha!” moment – compared to the experiencing the trick, the revelations are mundane, uninteresting, and disappointing.

To our ancestors most of what was happening around them was magical. Imagine not knowing why it rains, what clouds are made of, or why you get sick. Science has often taken away the mystery. Sometimes what it reveled is awe inspiring – the fact that our Sun is one of countless others in an unimaginably large universe is more mind blowing than if we lived on a disk riding on the back of a turtle.

However, sometimes scientific revelations can turn the magical into the mundane. For instance, the moving rocks of Death Valley were somehow more interesting before we new they just skated around on ice. An explanation could have made us feel different if it involved something unexpected, or outside our daily experience, perhaps strong magnetic fields, or aliens with hockey sticks. However, partly it was the mystery itself, not knowing the secret, that gave the moving rocks their special aura. Once we know how the magic is done, something is lost.

The main goal of science is to understand how the world works. Some of the time what we find will be awe inspiring. At other times, the explanations will be mundane, unreducibly complicated and even ugly. We definitely crave the first kind. But if it is important to find out how the world works, should we put such a high value on esthetics.

Indeed, I fear that some of the great unanswered questions of science will have answers that we will find unsatisfying. I am reading the book Consciousness and the Social Brian by Michael Graziano (here is a shorter post about it) – roughly the idea is that awareness is the result of brain’s model of what it is paying attention to. We need to have a model of what we ourselves, as well as others, attend to. Awareness is just an abstract, communicable representation of the act of paying attention. I am not sure that this is right, but Graziano offers pretty good arguments that it is plausible.

Even if this theory is not right, it is quite possible that we will ultimately find the answer to the question of consciousness disappointing. It is arguably the greatest magic trick of all, the one that lets us experience all the rest of the magic around us. And like the magic tricks that Jillette describes, the revelation may ultimately be ugly and unsatisfying.

The Friendship Paradox

It seems to be in our nature to compare our accomplishments to those of others. Teenagers worry about popularity. Later in life we compare our success to that of friends. But, there is a mathematical reason why we usually come up short!

You may have noticed that your friends seem to have more friends than you do.  And you are right – on average, your friends are more popular than you are. This is true on Facebook and in real life, and is a consequence of what statisticians call biased sampling – you are more likely to befriend an outgoing, easy-to-get-along person than a recluse who hardly talks to anyone. Your typical friends make friends easily. Therefore you do not form friendships at random – statisticians would say that you are taking a biased sample of society.

Biased sampling is why playing a game of poker against strangers in a casino is usually a bad idea – you are more likely to meet an opponent who spends a lot of time playing the game, rather than a beginner or a complete amateur.

More surprisingly, not only are your friends more popular, but on average they are also more successful. The reason for this is that people with more friends seem to be on average more successful. We already established that because of biased sampling your friends have more friends than you do. If more friends means more success, it follows that your friends are on average more successful than you are. Hence comparing yourself to your friends, is not a good idea – you are using a biased sample of society that is likely doing better than you.

OK, you may say, but is it useful?  Indeed it can be – select a group of students at a university, and ask them to give you the name of a few of their friends.  On average they will name people who are more popular, and therefore have more social contacts than the average student. If you are interested in hearing the latest rumor, you will be well advised to go to this new group. But, since these friends are more popular, they also have more interactions with others, and may be among the first to get sick in an epidemic.

Researchers have confirmed that this is the case: they asked random students to name friends, and found that in a flu outbreak this named group got sick about two weeks earlier than the average student. To get an early warning of an epidemic, just pick people at random and ask them if their friends are sick. You can do the same if you are trying to spot a new trend.

So mathematics tells us something valuable about our friends:  Comparing our accomplishments to theirs is likely going to leave us depressed. Instead listen to your friends if you want to hear about a good place to eat, what concert to go to, or interesting new technology.  Your friends will be able to tell you about it better than the average person.

Some references: 

Popular articles about the friendship paradox can be found here.  Here is a more detailed, yet very easy to follow discussion by the inimitable Steve Strogatz.

The paradox was originally described by Scott L. Feld in  “Why your friends have more friends than you do,” American Journal of Sociology, Vol. 96:6, pp. 1464–1477 (1991).

The generalized friendship paradox – the observations that your friends are more successful than you are (on average), is described here. I have taken some liberties with the term success – the fact that friends are more successful has been shown, for example, for the number of co-authors and citations for scientific papers and and followers on Twitter. If this counts as “success” – and in scientific circles, then what I have said is strictly true.  However, it is also likely that the observation extends.

The article that describes how friends of friends can be used to track outbreaks of diseases is Christakis NA, Fowler JH (2010) Social Network Sensors for Early Detection of Contagious Outbreaks. PLoS ONE 5(9): e12948. doi:10.1371/ journal.pone.0012948, and can be found at    and a followup Garcia-Herranz M, Moro E, Cebrian M, Christakis NA, Fowler JH (2014) Using Friends as Sensors to Detect Global-Scale Contagious Outbreaks. PLoS ONE 9(4): e92413.

There is a difference between the mean and the median of the number of friends that I did not get into.  This is described in more detail here. This also provides a more detailed description of the mechanism behind the generalized friendship paradox.


Get every new post delivered to your Inbox.