Skip to content

How to improve the peer review process

December 30, 2011

If you have published in an academic journal, I am sure you have been amazed by the inefficiency of the process. A new article by Dwight Kravitz and Chris Baker in the online journal Frontiers in Computational Neuroscience presents a good overview of what is wrong with the current system, and offers interesting suggestion of how to improve it.  Here is a short summary:

Our current peer review system has been in place for centuries. In the past, producing an online journal article was quite expensive. To limit cost and maximize readership it was imperative that the publisher only print a small number of the very best articles. Having candidate articles reviewed by a panel of experts assured both quality and selectivity.

However, while print journals are still deposited in libraries, I would estimate that more than 95% of the time scientists access articles online. In the present environment the traditional peer review process has become very inefficient. In most disciplines, authors will send an article first to a top journal in the field. If it is rejected there, the article will be sent to a journal further down the list, and so on, until the article is eventually accepted and published. Based on polling a number of colleagues, Kravitz and Baker estimate that it takes on average 2.6 revisions and 6.3 reviews to get an manuscript published. Since the article will eventually appear anyway, more than half of these reviews are not necessary. Moreover, a lot of effort in the reviewing process goes into determining whether the manuscript contains results that are sufficiently novel, and whether it is appropriate for a particular journal. This has little bearing on the scientific content of the article – it should be the scientific community that determines how interesting the results are, and not the two or three umotivated reviewers chosen somewhat arbitrarily by the editor.

In short, nearly every manuscript eventually gets published. The review process does provide valuable feedback to the authors. But much of the process is repetitive and does not improve the quality of the eventual publication.

So what to do? Kravitz and Baker provide an interesting alternative: Let’s keep the useful part of the reviewing process. After an article is submitted, a panel of reviewers should still be asked to offer suggestions for improvement. However, there should be a single round of review. The authors then revise the article, after which the article is published. The Editorial Board provides expert comments on the article, and perhaps rates its importance. Most importantly, the scientific community is then asked to provide comments on the article, and direct questions to the authors. While the article would not undergo changes after publication (except to correct errors, I suppose), the ongoing discussion could be valuable in clarifying, and maybe even extending the published results.

I see several problems with these suggestions. First, the current system remains in place because there are strong interests keeping it in place. The publishing industry is run largely like a monopoly, and makes billions of dollars. This industry will opposed changes that could impact their profits, or potentially even eliminate the need for traditional journals. We scientists are somewhat complicit here, firstly because change requires effort. Secondly, how will we be evaluated for tenure and promotion when traditional journals are gone?

Perhaps, most importantly, who will bear the cost of these changes, and who will bear the publication costs in the future? Having authors pay the cost of publishing biases what can be published. Well funded labs could have a large impact in the field simply because they could publish more. Many mathematicians receive no external funding – should they be required to pay thousands of dollars out of pocket every time they wish to publish a paper? One solution would be that home institutions pick up the costs, instead of paying for journal subscriptions. However, some good scientists do not have university affiliations.

This summary doesn’t do justice to the ideas of Kravitz and Baker. If you are in academia, I urge you to read their article. Even if you disagree with their suggestions, I think it is important to start this conversation.

Advertisements

From → Uncategorized

8 Comments
  1. Thanks for pointing out this article. Your point about the impact on the tenure process is very interesting, and I wonder if the ever-looming tenure revolution will occur simultaneously with the longed-for publication revolution. Like the publishing industry, the fault lines of the tenure process seem to become more and more unstable.

    You asked, “who will bear the cost of these changes?” They seem to imply that the “field” will absorb a lot. I think the article has a striking inconsistency in that they provide an extremely detailed financial analysis for some components but then completely gloss over the financial impact of the 800-pound gorilla in the room: the editorial board. In the section “Reasons for Including an Editorial Board”, they state: “The inclusion of this group adds steps and time to the process of publication and also creates a new burden on the field. Nonetheless, the benefits of the Editorial Board outweigh these costs.” Seems like they’re implying that the “field” can absorb the costs involved in the person-hours required by these editorial boards.

    I’m fairly new to the system myself, but I like Carson Chow’s quote from http://sciencehouse.wordpress.com/2011/10/28/guest-editorial/ where he said, “I don’t know what it should look like, but I do know that when it comes around I’ll wonder how I ever got along without it.”

    • josic permalink

      Your point about the Editorial Board is well taken. We can probably count on the community to do some work pro bono, but that will have its limits.

  2. Dwight Kravitz permalink

    We just posted a reply to the concerns raised here, as well as several others. The reply is on the article’s Frontiers page (comment link on the bottom of the page):

    http://bit.ly/uVggz9

    Thanks.

    • josic permalink

      Thank you for following up on the comments! My main question is what concrete steps we could take to move forward? For instance, I publish in PLoS and Frontiers whenever I can – however, the cost of each publication equals about 1 month of support for a graduate student, or a 1.5 trips to a conference for one of them.

  3. Dwight Kravitz permalink

    We completely agree that the issue of cost is critical. We don’t have a perfect answer yet because the details of the expenses incurred currently by the journals remain opaque. Further, under any new system these costs will change significantly. Nonetheless, there are several ways that publication costs are likely to be defrayed.

    1. Currently many journals have waiver policies in place. There are also institution-wide funds that have been set-up to pay for publication. See for example:
    http://www.plos.org/publish/pricing-policy/publication-fees/#pubquesthttp://www.plos.org/about/faq.html

    2. Even though the cost isn’t currently directly reflected in lab budgets there is a significant publishing cost even to labs that don’t appear to pay. The institutions pay large subscription fees for print copies of journals that are never read. At a minimum these fees will be eliminated. That money can then be placed into a general fund to cover publication fees.

    3. The proposed system will also reduce the overall amount of uncompensated time spent reviewing manuscripts and the substantial costs associated with the delays and uncertainty of the peer review process. This will free labs to be more productive and hopefully produce more and better results. As overall quality of the research improves so does its broader impact and, eventually, overall funding.

    Finally, its important to keep in mind that any new system will be very flexible and that variations for different subfields are certainly possible. The majority of the publication fees will go towards the coordination and funding of the prepublication reception. Following publication, the only cost is the maintenance of the storage system and bandwidth. If a field doesn’t need/want prepublication review, as in the case of highly circumscribed or specialized fields, they could forego it to reduce costs.

    Thanks,
    Dwight + Chris

  4. sams permalink

    Hi,

    Thank you four your nice writing on How to improve the peer review process.

    Thanks.

Trackbacks & Pingbacks

  1. A new model for publishing « Scientific Clearing House
  2. The Spamlist! » A new model for publishing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: