Skip to content

What does the case of Diederik Stapel tell us about high-profile journals

January 5, 2012

The former Dr. Diederik Stapel – he has now relinquished his Ph.D. – published a series of papers in high profile journals with simple, counterintuitive results. The articles made delightful news stories: The results were simple to explain, and offered interesting insights into human behavior.

A good example is the article Coping with Chaos: How Disordered Contexts Promote Stereotyping and Discrimination. The title pretty much tells the story: People are more prone to stereotype others if they are on a street with cracked sidewalks or litter, for instance. This paper has exactly what high-impact journals are looking for. It provides a counterintuitive, easy to understand result, and offers new, solidly documented insights about a fundamental aspect of human nature. Stapel published a number of similar studies, that told interesting, yet simple stories, and were supported by irrefutable data. The only problem was that most of the data was fabricated by Dr. Stapel.

Most of us reacted with shock when we first heard about this case. We can try to examine where the peer review process went wrong, and why Stapel’s collaborators didn’t notice that something was amiss. However, I think that perhaps we should not be surprised. Journals with a high impact factor look exactly for stories that can be told in a few pages, are simple to digest, and will be interesting and easily told in a 1 minute news story.  Is it surprising that eventually somebody decided to manufacture such stories?

Unfortunately (or fortunately), the world around us is complex. As John Muir said “When we try to pick out anything by itself, we find that it is bound fast by a thousand invisible cords that cannot be broken, to everything in the universe.” And yet as scientists we frequently try to “pick out things by themselves”: Isolate the one thing that we study from the rest, and try to pull at most one or two of those many cords at any time to see how the thing will be affected. Sometimes, we can capture the effect of the rest by “noise,” something that has a precise mathematical meaning, but frequently only a vague analog in nature. Or we try to show that pulling other strings may not have a large effect. But almost always the story that we tell is complex and full of caveats (controlled clinical studies are somewhat different, and I will address them at some later point).

The format and demands of a high-impact journal article make it difficult to communicate these complexities and caveats. I think this has changed the way we do science in certain fields. A friend who is a theorists and collaborates with high profile experimental scientists told me that often when he is trying to give a complex explanation, his experimental colleagues loose interest – not because they do not think it is right or interesting, but rather because it is unlikely to make it in a high impact paper.

Data suggests that results published in higher-impact journals are less reliable, as measured by the number retractions and sample sizes used (My caveat here: Studies in higher-impact journals are also more closely scrutinized, which probably also leads to a higher retraction rate.  Therefore, the true story is probably much more complex). I do not wish to argue against Occam’s Razor – there is good reason to go with the simpler explanation that fits the data. However, we ignore the complex web of interrelations around us at our own peril.

Advertisements

From → Uncategorized

4 Comments
  1. Kevin Lin permalink

    Suppose you are a high-profile journal, and you publish a paper, whose main result was later found to be either fabricated or at leaset irreproducible. Say the paper is retracted (the former case) or at least generates some controversy (the latter). The paper in question then gets a bunch of citations. Question: Does your impact factor go up or down?

    I don’t know the answer. An economic analog is the GDP, and apparently when disaster strikes and my house burns down due to arson, the GDP can go up due to the money I now have to spend to make repairs…

  2. josic permalink

    Kevin – you make an interesting and important point. There is a flip side – there have been a number of papers that have effectively closed an avenue of research. For instance, important conjectures have been shown to be not true. What frequently happens is that in such cases people will move on. As a result, such papers may receive very few, or even no citations – although they may be viewed as perhaps the most important papers in that field!

Trackbacks & Pingbacks

  1. The Case of Diederik Stapel, part II « Mathematics in everyday life
  2. Losing traction from retractions | The Computational Neuroscientist

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: