How to make decisions when the environment is changing
I have recently revisited the following old problem in a new paper with Alan Veliz-Cuba and Zack Kilpatrick: Suppose that you are trying to decide between different alternatives. Maybe you are trying to find the best place to fish, behind which bush to look for your dinner, or between different products in a super market. Humans, animals, and even animal collectives decide between alternatives all the time. Such decisions are frequently not based on a single observation. Indeed, animals accumulate evidence, and combine bits of information to arrive at a better choice.
There has been a lot theoretical work to determine the best way to integrate information that arrives over time. There are also fascinating experiments that suggest that certain animals make decisions in a way that is very similar to the theoretical models. Indeed, neuroscientists have even capture the neural signature of such processes.
This figure shows the activity of neurons in area VIP of monkeys that are deciding whether a cloud of dots moves predominantly to the right or to the left. As the animal accumulates information for one or alternative, the activity of a cell increases. Figure from Gold and Newsome, Annual Review of Neuroscience, vol. 30, p. 535 (2007).
However, in most classical studies of such decision making the correct (or better) choice is fixed during a trial (For a recent study where this is not the case, see here). Unless there is a some pressure to make a decision quickly, in such cases it is best to accumulate as much evidence as possible, that is, wait indefinitely to make sure that you are making correct choice. However, the natural world constantly changes, and what is a correct choice or better option at one instant, may no longer be so in the next. Our goal was to extend classical models to the case where the truth is not constant.
The model that we derived has some interesting features: An ideal evidence accumulator will discount prior evidence in a way that is determined by the volatility of the environment. In other words, to perform well in a changing environment it is best to forget evidence that is too old, as it is no longer pertinent. Indeed, our model shows that you want to keep evidence that has arrived over about one average environmental epoch. As a consequence, even if for some reason the best option does not change for an extended time, there is a limit to the certainty you can attain about your choice. Your certainty is limited by the amount of information you keep in memory.
The nice thing about differential equation models is that they suggest plausible neural implementations. We proposed such a model with the activity in different neural populations representing the evidence for different choices. Interestingly, unlike in classical models, the different populations here are coupled through excitation.