There has been quite a bit of work on this question. For instance, when choosing a new place for a hive, honeybees will communicate to each other the location of a promising spot using a waggle dance (Seeley, et al. Science, vol. 335 (6064) pp. 108-11). Scouts will head-butt a dancing bee who is signaling a competing location. Competing factions thus mutually inhibit each other. The winning faction is the one that out-butts the others. Presumably, this faction was also larger, or more vigorous in promoting a new spot because the spot was actually better.

In a related study, a number of models have been used to examine the impact of a vocal, opinionated minority in making collective decisions (Couzin, et al. Science, vol. 334 (6062) pp. 1578-80). In a head-to-head competition with a less opinionated majority, the smaller group can carry the day. This advantage disappears in the presence of a population of individuals that do not belong to either group. This uninformed group seems to temper the influence of a strident minority. Experiments with schooling fish provided nice support for the theory.

However, there is something quite puzzling about the flow of information in groups. Consider the case of a jury in a trial. Each juror forms an independent opinion about the guilt or innocence of a defendant during the trial. During the deliberation information is exchanged, and opinions become more correlated. In fact, in the absence of complete agreement a mistrial is declared (think of the Twelve Angry Men as an example).

Would we be more likely to reach the right decision if jury members were barred from discussion between themselves, and we simply followed the decision of the majority? This may sound counterintuitive, but it is not at all clear that deliberation increases the collective wisdom of the group. A shrill minority may effectively promulgate its views in such small groups. Indeed, Daniel Kahneman suggests that the increased correlation that results from discussion decreases the quality of decisions. He urges that all participants in a meeting write down their opinions before any discussion takes place, and that these initial opinions carry significant weight in the eventual decision.

I’m somewhat new to this, but I am interested in understanding what allows for good information transfer in groups of individuals. The information available to an individual may increase, and individual decisions may become better. However, the data processing inequality suggests that however the exchange occurs, the information of the collective as a whole will only decrease. Again, this assumes that some external intelligence has access to all the information, and can make decisions based on this complete information. The problem is that groups are constrained in how they make decisions – for instance, going with the majority of a group may be suboptimal for somebody who can gather and assess information from all individuals in a group. However, it may be close to optimal in other circumstances.

Surprisingly, the process of sharing information is not always detrimental. In a recent paper, Mossel and Tamuz show that individuals in a network can reach an optimal decision (or estimate) even though they are restricted to exchanging information only with their neighbors. Surprisingly, it seems essential that the network is recurrent, i.e. that there is a path of communication between any two individuals in the network. The model is somewhat idealized, as all individuals know the exact structure of the network, as well as the belief or quality of the initial estimate of everyone else in the network. Still I find it amazing that as the estimate becomes more correlated among individuals in the group, it also becomes better. Each individual exchanges information locally. Yet, in finite time the estimate of each individual equals that of an intelligence that is able to gather complete information from all individual in the network.

This is an example of an essentially perfect exchange of information in a network. Are there other mechanisms to achieve this? Maybe more importantly, can we test whether groups of animals or social groups can integrate information locally so as to make near optimal collective decisions? I think these are very important questions, that will be examined extensively in the near future.

So let me then add the following to my previous post: There is nothing inherently wrong with journals like Nature and Science soliciting the “coolest” new research and publishing it in an easily digestible format. It is up to us to follow Sanjay Srivastava’s advice that “Our standard response to a paper in Science, Nature, or Psychological Science should be “wow, that’ll be really interesting if it replicates.”

A good example is the article Coping with Chaos: How Disordered Contexts Promote Stereotyping and Discrimination. The title pretty much tells the story: People are more prone to stereotype others if they are on a street with cracked sidewalks or litter, for instance. This paper has exactly what high-impact journals are looking for. It provides a counterintuitive, easy to understand result, and offers new, solidly documented insights about a fundamental aspect of human nature. Stapel published a number of similar studies, that told interesting, yet simple stories, and were supported by irrefutable data. The only problem was that most of the data was fabricated by Dr. Stapel.

Most of us reacted with shock when we first heard about this case. We can try to examine where the peer review process went wrong, and why Stapel’s collaborators didn’t notice that something was amiss. However, I think that perhaps we should not be surprised. Journals with a high impact factor look exactly for stories that can be told in a few pages, are simple to digest, and will be interesting and easily told in a 1 minute news story.  Is it surprising that eventually somebody decided to manufacture such stories?

Unfortunately (or fortunately), the world around us is complex. As John Muir said “When we try to pick out anything by itself, we find that it is bound fast by a thousand invisible cords that cannot be broken, to everything in the universe.” And yet as scientists we frequently try to “pick out things by themselves”: Isolate the one thing that we study from the rest, and try to pull at most one or two of those many cords at any time to see how the thing will be affected. Sometimes, we can capture the effect of the rest by “noise,” something that has a precise mathematical meaning, but frequently only a vague analog in nature. Or we try to show that pulling other strings may not have a large effect. But almost always the story that we tell is complex and full of caveats (controlled clinical studies are somewhat different, and I will address them at some later point).

The format and demands of a high-impact journal article make it difficult to communicate these complexities and caveats. I think this has changed the way we do science in certain fields. A friend who is a theorists and collaborates with high profile experimental scientists told me that often when he is trying to give a complex explanation, his experimental colleagues loose interest – not because they do not think it is right or interesting, but rather because it is unlikely to make it in a high impact paper.

Data suggests that results published in higher-impact journals are less reliable, as measured by the number retractions and sample sizes used (My caveat here: Studies in higher-impact journals are also more closely scrutinized, which probably also leads to a higher retraction rate.  Therefore, the true story is probably much more complex). I do not wish to argue against Occam’s Razor - there is good reason to go with the simpler explanation that fits the data. However, we ignore the complex web of interrelations around us at our own peril.

However, there is one point that is made in the article that I wholeheartedly agree with: Most of us at universities are paid for our teaching (typically 40% or more of our effort should be devoted to teaching, even at research universities).  However, scientists receive virtually no training in education either before or after our PhD.  It is still fairly typical for faculty members to just be given a book and thrown into the classroom when they are given a job.  After that promotion is determined largely based on our scientific output and grant dollars generated.  I find it surprising that university professors overall do a good job in the classroom given this environment.

The Turk was touted as an early robot that could play chess at the highest level. Built in Vienna in 1770 by the inventor Wolfgang von Kempelen, the machine consisted of a large pedestal, housing intricate machinery on top of which stood a chessboard. To this box was attached the upper half of a men dressed in oriental robes and a turban. After a theatrical introduction, the automaton would face a challenger. The Turk would move its pieces by itself, and would instantly recognize illegal moves.

The Turk first dazzled the court of the empress Maria Theresa in Vienna. It offered a surprisingly good game, and soon became a sensation, touring Europe and later North America.  The Turk was matched against some of the best chess players of the time, loosing some games, but winning surprisingly many. It remained popular after its inventor’s death, and it even played against Napoleon Bonaparte and Benjamin Franklin.

The secret of the Mechanical Turk was kept for over 50 years – the machine was an elaborate illusion.  It contained an ingeniously hidden compartment that housed a human operator. This hidden chess master could observe the position on the chessboard above, and manipulate the Turk. The identity of the operator that made the Turk famous is still unknown.

The original Turk was destroyed in a fire, but some of the original parts survived.  It was reconstructed in 1984 – however, at this time a hidden operator was no longer necessary (a nice video of the reconstructed machine is here).  The present incarnation of the Turk is truly autonomous, its moves guided by a chess-playing computer.

Today machines can play chess better than any human. However, there are plenty of things that humans can still do better: accurately transcribing dictations, or predicting which products other people will like.

Interestingly, Amazon has created an online service to easily harness a large human workforce for such tasks.  And they have named this service The Mechanical Turk, after the 18^th century automaton.  Businesses can use this slick computer interface behind which are hundreds of humans that actually perform the requested tasks.

The modern chess-playing Turk does not need a human operator.  And this brings us to the interesting question: How long before we can replace the human operators behind Amazon’s Mechanical Turk with machines?  I would like to believe that this will take a very long time. But given the acceleration in innovation that we are experiencing, it may take far less than 200 years.

Our current peer review system has been in place for centuries. In the past, producing an online journal article was quite expensive. To limit cost and maximize readership it was imperative that the publisher only print a small number of the very best articles. Having candidate articles reviewed by a panel of experts assured both quality and selectivity.

However, while print journals are still deposited in libraries, I would estimate that more than 95% of the time scientists access articles online. In the present environment the traditional peer review process has become very inefficient. In most disciplines, authors will send an article first to a top journal in the field. If it is rejected there, the article will be sent to a journal further down the list, and so on, until the article is eventually accepted and published. Based on polling a number of colleagues, Kravitz and Baker estimate that it takes on average 2.6 revisions and 6.3 reviews to get an manuscript published. Since the article will eventually appear anyway, more than half of these reviews are not necessary. Moreover, a lot of effort in the reviewing process goes into determining whether the manuscript contains results that are sufficiently novel, and whether it is appropriate for a particular journal. This has little bearing on the scientific content of the article – it should be the scientific community that determines how interesting the results are, and not the two or three umotivated reviewers chosen somewhat arbitrarily by the editor.

In short, nearly every manuscript eventually gets published. The review process does provide valuable feedback to the authors. But much of the process is repetitive and does not improve the quality of the eventual publication.

So what to do? Kravitz and Baker provide an interesting alternative: Let’s keep the useful part of the reviewing process. After an article is submitted, a panel of reviewers should still be asked to offer suggestions for improvement. However, there should be a single round of review. The authors then revise the article, after which the article is published. The Editorial Board provides expert comments on the article, and perhaps rates its importance. Most importantly, the scientific community is then asked to provide comments on the article, and direct questions to the authors. While the article would not undergo changes after publication (except to correct errors, I suppose), the ongoing discussion could be valuable in clarifying, and maybe even extending the published results.

I see several problems with these suggestions. First, the current system remains in place because there are strong interests keeping it in place. The publishing industry is run largely like a monopoly, and makes billions of dollars. This industry will opposed changes that could impact their profits, or potentially even eliminate the need for traditional journals. We scientists are somewhat complicit here, firstly because change requires effort. Secondly, how will we be evaluated for tenure and promotion when traditional journals are gone?

Perhaps, most importantly, who will bear the cost of these changes, and who will bear the publication costs in the future? Having authors pay the cost of publishing biases what can be published. Well funded labs could have a large impact in the field simply because they could publish more. Many mathematicians receive no external funding – should they be required to pay thousands of dollars out of pocket every time they wish to publish a paper? One solution would be that home institutions pick up the costs, instead of paying for journal subscriptions. However, some good scientists do not have university affiliations.

This summary doesn’t do justice to the ideas of Kravitz and Baker. If you are in academia, I urge you to read their article. Even if you disagree with their suggestions, I think it is important to start this conversation.

Dr. Koller makes good points: Online technology would allow us to assess student performance better, and individualize instruction. Good students could move ahead quickly, while those that struggle could get help in exactly those areas they need. This is particularly important in face of the reduction of public funding for education and soaring student/teacher ratios.

However, having taught online courses and classes with large number of students, I have to say that the news of the demise of the traditional lecture have been largely exaggerated. While online classes are convenient for the student, interactions are frequently stilted. When lecturing online, I feel like I am talking to myself. Many students choose not to attend the online lectures, so online courses are frequently nothing more than self-study courses. Even in a regular classroom, my students asked that I stop writing on a tablet, and use a whiteboard instead. They preferred it, although it meant I would not be posting the notes online. I was glad they did.

New technology is changing the way we communicate in many different ways. But I fear that there is a large human component that we don’t take into account when predicting its impact. It takes time to finely tune and meld the technology to our human quirks.  For instance, skyping, even using a large screen, doesn’t feel anything like talking to somebody in person. The visual and auditory input we receive is not that dissimilar, but a number of little (but essential) details are just not the same. Similarly, there are a number of things that just don’t feel right about online learning. And it is not the technology, it is us who are difficult.

The warden considers, and tells A that the letter was delivered to B. Prisoner A now thinks: “Well, it’s either C or me who will die tomorrow – my chance of dying has just increased to 50%! But I shouldn’t have received any more information from the warden.  What did I do wrong?”

This puzzle is actually equivalent to the Monty Hall problem (Wikipedia has an excellent discussion of that problem – perhaps one of the best entries I have seen so far).  Think of the prisoners as doors, being executed as “the prize”, and the warden as Monty Hall.  The probability of being chosen at the start is exactly 1/3 for prisoners A, B, and C – and it is 2/3 for the pair B, C.  The probability of the pair B, C is not altered by revealing that B was not chosen.  Thus A’s reasoning that his chance of dying increased to 50% is wrong – his chance is still 1/3.  However, he does know that the chances of C dying are now 2/3 (since the pair B, C has probability 2/3, but we know that B will walk free, leaving only C).

But here is a subtle point:  If prisoner A had asked outright “Will prisoner B die?” the reasoning above would have been correct.  In that case prisoner A receives actual information, if the warden answers the question.  To see that the situations are different think about this: In the case of the letter the warden had the choice of delivering the letter to B or C.  Thus when asked which one of the two will be released he had the choice of answering B or C, if both were innocent.  When faced with a direct question, such as “Will B walk free?” the warden does not have this choice in answering.   Thus the context in which the answer was obtained is of essential importance here.

However, things are starting to change. Google’s attempt to digitize all books ever published has provide a treasure trove of data.  Indeed, a team of researchers from Harvard lead by Jean-Baptiste Michel and Erez Lieberman Aiden has recently analyzed about 5% of all books ever published.  In this vast amount of text they detected trends that gave a glimpse of how language and culture have changed over time.  

The richness and variety of the insights are amazing.  For instance, they tracked how frequently different verbs appear in books and newspapers, and observed how irregular verbs become regular.  Over the last 400 years irregular verbs have become more regular. In 1800 we “chid” unruly children, but in the year 2000 we “chided” them.  The more frequently a verb is used, the more resilient it is to regularization: “spoke” will not turn into “speaked” for a long time.  However, the verb “sped” is giving way to “speeded,” a process that started around 1920 and is still going on today.  Linguists have known that verbs change in this way, but the Google book data offered detailed insight into this process of transmutation.

In 1785 the French philosopher and mathematician Marquise de Condorcet published his influential Essay on the Application of Analysis to the Probability of Majority Decisions.  Perhaps the most famous result of the work is to show how voting can be similar to a rock-paper-scissors game: With three candidates for president (A, B, and C), it is possible to obtain a situation where A can be called a better choice than B, B a better choice than C, and C better than A.

In the same work Condorcet also introduced his Jury Theorem (CJT) – I’ll paraphrase from D. Austen-Smith and J. S. Banks, The American Political Science Review, Vol. 90, No. 1, p. 34: “Suppose there are two mutually exclusive alternatives, A, B.  One of these alternatives is unequivocally better for all of n individuals in a group but the identity of the better alternative (i.e., A or B) is unknown. Suppose further that for all individuals the probability that i votes for the better alternative is p > 1/2 and is independent of the other votes. The Theorem states that the probability that a majority votes for the better alternative approaches 1 as n goes to infinity.”

The CJT has been taken as evidence for decision making by majority rule.  The argument is that the larger the jury (electorate), the more likely they are to vote for the better decision according to the CJT.  One can quibble with the assumption of independence, but the theorem can be proved even when the votes are cast dependently.

There is another thing I’d like to point out (although I am certainly not the first to do so): Note that the statement of the CJT may be interpreted as assuming that there are two states of the world, and that we are trying to infer which is the true state.  For instance, if we look at an actual jury the defendant is either guilty or innocent.  The assumptions of Condorcet’s Theorem are that individuals in the Jury make independent choices and that their probability of making the correct choice is p.  Here is where we should be careful – there are actually two distinct probabilities lumped into one here.  There is the probability of correctly declaring the defendant guilty (call it p) and correctly declaring the defendant not guilty (call it q).  These two probabilities may be the same, but may be quite different.

Suppose you have a person that has such a negative view of the world that they believe everybody to be guilty.  In that case, they would always correctly identify the guilty parties (p=1), but fail to identify the not guilty parties (q=0).  I believe it is more pertinent to ask what is the best choice to make given a particular vote, and if we know p and q for this electorate (or the individual probabilities if they differ between the voters).  Under some circumstances this will the vote of the majority will give the correct decision.  However, frequently it will not.  The optimal decision can be found using the theorem named after Condorcet’s contemporary Thomas Bayes  (And a side note about using Bayes’ Theorem.  It is necessary to assign prior probabilities on the defendant being guilty or not.  Assuming literally that a “defendant is innocent until proven guilty” would actually imply that everybody is innocent – it is necessary to assume a nonzero probability of the defendant being guilty before any evidence is presented.)

There is another situation where the classical statement of the CJT makes sense – if there is one state of the world, which is fixed, and people infer it correctly with probability p.  I think this is the less interesting case: If there is only one possible state of the world, then why would there be disagreement about it, and why would people have to vote on it?  There must be at least two probable states.  Otherwise nobody would believe the alternative.

There are other problems with calling this a Jury Theorem: In a jury people communicate and exchange information.  The theorem implicitly assumes that every individual makes a choice that is dependent only on their individual information.  In any case, I am unconvinced that the theorem says anything useful about the democratic process.  However, it does bring up some thorny issues about decision making in groups.