Update (Nov 6, 2004): Lance Fortnow comments on this issue. Doron Zeilberger has recently independently commented on some related issues, from a rather different point of view. Finally, Seb gives another update.
Seb Paquet posts about the public posting of referee reports on blogs.
I’ve contemplated doing this several tmes in the past, but have held off because I don’t have satisfactory answers to key questions: will this help make referees more accountable? What are the upsdes? What are the downsides?
What do readers think?
On a related note, Cosma Shalizi has posted a whole bunch of his student evaluations online. He has my sincere admiration for posting the first set.
I don’t think it’s a good idea. Even though reports are anonymous, referees may feel that they need to be less open and frank with authors if they know the report could end up on a mailing list or a weblog. Sometimes you need the jolt of being told “look, this paper isn’t what you thought it was.” Frankness is important. I say this as someone who’s received my fair share of critical, honest reviews. I didn’t like them at first, but eventually I get over myself.
Also, as Seb Paquet notes, often you can guess who wrote a review, especially in a smaller subfield. Posting reviews seems to encourage the spectator sport of guess-the-referee. That doesn’t seem productive to me at all.
That being said, it is true that sometimes referees do a poor job. There also seems to be some disagreement sometimes as to how much “help” a referee is supposed to give, as opposed to the “gatekeeper” role of the review. I haven’t been around long enough to suggest anything concrete to fix this…but I’m skeptical that public posting of reviews will help.
I think, and some journals do that, the reviewer should be public. His review of the paper should appear alongside the paper. If he let a bad paper go through, his name will be on it. So he has a strong incentive not to let bad papers pass (as his name will be tarnished), but a very strong incentive to write a strong reviews because they will be public.
If you are worried that this would lead to not enough papers getting through: don’t worry, plenty will get through.
The current setup is ridiculous. For conferences (not journals) I repeatedly get stupid reviews. Sometimes, you get the feeling someone doesn’t want your paper accepted because you are not using his favorite methods. Sometimes, very often in fact, you get a feeling the reviewer only read your abstract, maybe the conclusion… and that’s pretty much it.
In many cases, reviewers are not useful. They do a poor job and we should not, collectively, tolerate this.
(But the problem seems to be mostly with conferences.)
I have posted, sometimes, referee reports of my papers, but I have also posted my own referee reports. This second way, to spontanesusly renounce annonimity, could be an alternative.
Computational complexity theorist Lance Fortnow says: “let’s kill that idea quickly”.
http://fortnow.com/lance/complog/2004/11/public-referee-reports.html
As someone who used to run the peer review process at a mathematical physics journal, I struggle to conceive of something an author might do that would so comprehensively undermine the current peer review process as publishing the reports. That isn’t necessarily a bad thing!
However, without a complete redesign of peer review in physics, any author doing this right now is making trouble for themselves. It’s hard enough to get reviews of papers as it is, but the prospect of having reports publicly disected and being identified as a referee (this would happen a lot) would make life very difficult for authors that choose to do this. Expect substantial delays in the refeereeing of those articles, and much less useful (and probably lenient or unduly harsh) reports.
I’m very much inclined to agree that posting referee reports online is a bad idea, for all the reasons given. I also feel there’s a big difference between that and my posting my student evaluations, seems more like a service to future students that doesn’t harm or inhibit previous ones; but it’s too early in the morning for me to be clear about that.
what about a “citebase-like” site to voluntary upload reviews (from referees or from quoted authors) of the Arxiv material?
Coming in quite fresh, it seems to me that the publish-or-perish mentality and the fact that referees are not paid is naturally going to lead to the problems of results of dubious quality being published, and occaisionally good work being knocked back. I simply don’t see that there’s so much incentive to referee thoroughly or in a timely manner.
Frankly, I don’t think publishing referee reports are going to help alleviate these causes 🙂