http://www.canonicalscience.org/publications/canonicalsciencereports/20082.html

The conclusions by Tom Jefferson about peer-review were that it is “completely useless at detecting research fraud”.

The above report also includes a list of thirty four Nobel Laureates whose awarded work was rejected by peer review. Those are some excerpts:

“The 1996 Nobel Prize in Physics was awarded to DAVID MORRIS LEE, DOUGLAS DEAN OSHEROFF, and ROBERT COLEMAN RICHARDSON for the discovery of superfluid Helium. Their key paper was rejected by the reviewers of the journal Physical Review Letters. One reviewer argued that the system «cannot do what the authors are suggesting it does».”

“WILLIAM NUNN LIPSCOMB received the 1976 Nobel Prize in Chemistry for his studies on the structure of boranes. In an interview, LIPSCOMB recalled how the Journal of the American Chemical Society rejected the first manuscript in which he used the concept of pseudorotacion to explain the structure of a boron hydride. Another manuscript in which he showed that p-dithiin was V-shaped was also rejected by the Journal of Organic Chemistry.”

ROSALYN YALOW, described how her Nobel- prize-winning paper was received by the journals, in the next terms:

“In 1955 we submitted the paper to Science…. The paper was held there for eight months before it was reviewed. It was finally rejected. We submitted it to the Journal of Clinical Investigations, which also rejected it.”

Among other problems, the mechanism that prevails is that
(a) practitioners wishing to seek tenure want to have their articles published;
(b) the primary journal in the field fills up and overflows;
(c) someone starts up a new journal to catch the overflow:
(d) there are, initially, not enough good articles to fill this new journal, and not enough good reviewers and editors:
(e) articles are published that are below publishable quality;
(f) these inferior articles gain the status won for refereed articles by Nature and Science.

Actually I know another case in which it turned out reasonably well (meaning there was no animosity) but I can definitely see how this might be the exception.

The thing is that there are times when you have a pretty good guess who a referee is anyway (particularly if you sent in a list of suggestions with your paper).

Incidentally, I find that particular practice (of sending suggestions) a little dubious. It works if authors are generally honest and don’t suggest friends and collaborators, but I’m sure there must have been papers that have squeaked through because the author suggested someone he/she knew would approve it and the editor didn’t pay careful enough attention to the paper to override the referee’s report.

I know of one specific instance where an anonymous referee disclosed their identity to an author whose work they had rejected; the results were not positive for anyone involved.

I do think we can move towards a more open system of peer review, but through more indirect steps – I’ll describe this in my future post.

I have had an instinctive dislike for the anonymous peer review system for as long as I can remember in my short career as a scientist (say 10-ish years). I am really happy to have found out recently that many others share this dislike and am still catching up with what people are saying. Some really great conversation above, but I did want to just put in a couple comments:

* I am really looking forward to your upcoming post on the future of peer review.

* I personally feel (intuitively, having done no research) that a system of fully-published, non-anonymous peer review would be much more effective than what we have now. 10 years ago it was probably not feasible, but I think now we have what we need as far as the technology for enabling it. I do hope this is part of your plan for the future.

* One thing that really has bothered me and would be solved by a fully open system is that I have really been frustrated in the past that I could not give credit to very good ideas from reviewers. It is unfair and silly that I have not been able to give credit to ideas that referees have contributed to make my publications stronger.

Perhaps this is quibbling, but I’d like to see some data that backs that up (incidentally, the reason I am quibbling about this is that my PhD is in the History of Mathematics & Physics and I have studied this extensively). You are correct if your definition of peer review is the process employed by Physical Review and similar journals (I would argue that Nature and Science do not have the same process even to this day since the editors exert more control – to prove this I would need access to their records, however, which is not likely to happen).

However, since different journals have different processes, I would argue that peer review in the sense that referees are sought to review a paper before it is “communicated” via official means, has been widespread since at least the late-eighteenth century. Again, more data would be helpful to this assertion (and such an undertaking would constitute a major research project I would think), but I would go so far as to claim that, in the broader sense, peer review is part of what defines modern science.

Even with Einstein’s papers, I would be curious to know how many of his papers were “communicated” to the publishing journal by someone else. In truth, I do not know how the German journals worked at the time. I am most familiar with the British and French journals. Nonetheless, in some cases the “communication” of a paper to a journal can represent peer review (though it is difficult to tell in any individual instance).

While I agree with some of your conclusions on this point, I really find the heading misleading since it implies, intentionally or not, that there was little or no peer-oriented oversight (if that’s the right word) to the scientific process prior to the twentieth century when, in fact, it was ubiquitous for quite some time, just perhaps in a different form.

Take a similar situation as an example of what I’m trying to get at. A glance at many papers from the early twentieth century will reveal that the act of including references or footnotes was spotty at best (this gave me quite a bit of consternation when completing my PhD and a paper a couple of years ago since following the trail of an idea proved immensely difficult). But we clearly know via correspondence, conference proceedings, and other sources that many of these authors built on each others’ work. So it would be erroneous to conclude that scientists worked nearly independently back then.

In short, all I’m saying is that the title of Myth 1 is a misleadingly strong statement. The history of science has as many nuances and subtleties as science itself.