Comments on: How Are the Mighty Fallen

By: Mailund on the Internet » Blog Archive » This week in the blogs

Mailund on the Internet » Blog Archive » This week in the blogs — Sun, 25 Jan 2009 09:59:23 +0000

[...] How are the mighty fallen (Michael Nielsen) [...]

By: Publikationsrituale mit beschränkter Wirkung « Erich sieht

Publikationsrituale mit beschränkter Wirkung « Erich sieht — Sun, 11 Jan 2009 12:58:53 +0000

[...] Ergänzung: Michael Nielsen erklärt in seinem Blog drei Mythen über die Kollegenbegutachtung. In den Kommentaren dort findet man außerdem einen Link auf den Artikel: We Are Sorry to Inform You … (PDF), der uns die Gutachterkommentare zu berühmten und bedeutenden Aufsätzen aus der Informatik zeigt. So etwas ähnliches gibt es gibt es auch für die Wirtschaftswissenschaften. [...]

By: Carlos

Carlos — Sat, 10 Jan 2009 19:40:15 +0000

One idea I thought about for is that when a paper is accepted for a journal, the reviews, and the reviewer names, should be published with the paper. Double-blind reviews are asymmetric. When the paper is accepted, the reviews and their writers’ identities are thrown away, and so it’s easy to game the system by giving the paper an insincere score – there’s no explicit penalty for doing so.

One part of the process is already symmetric: if the paper is rejected, neither party knows the other’s identity. I have never heard a good explanation about the other part. If, at reviewing time, the reviewer knows he risks having his comments made public, there’s an incentive for making an honest effort. It’s the same incentive for not making up data in experiments – it will be there for other people to look at.

Obviously this only works for malicious reviews. Ramsey would probably be fine having “this is trivial” as his published review.

By: David Crotty

David Crotty — Sat, 10 Jan 2009 19:24:17 +0000

Sorry, that should read “final accepted forms”, not “final rejected forms”.

By: David Crotty

David Crotty — Sat, 10 Jan 2009 19:23:42 +0000

As the other commenter has noted, it’s impossible to tell whether this is a rejection of the peer review system or a great validation of the system, without a careful review of the original submissions versus their final rejected forms. Winning a Nobel Prize does not make one a good writer (a case in point being Ed Lewis who always owned up to his shortcomings as an author and an instructor). Perhaps the reviewer comments helped vastly improve the papers in question. I can’t say without seeing them and the sour grapes of the authors are not a fair way to judge.

Also, one wonders how well one can compare a field like economics to a field that is less theoretical, say, chemistry, where there is replicable data and results are perhaps less open to debate.

By: Michael Nielsen

Michael Nielsen — Sat, 10 Jan 2009 16:37:49 +0000

Carlos – Sure, that’s a very interesting question, and there’s some interesting stuff in Gans and Shepherds’ paper about improvements. On the other hand, it’s hard to believe that many papers go from unpublishable to Nobel-worthy because of referee reports.

By: Carlos

Carlos — Sat, 10 Jan 2009 00:01:42 +0000

Thanks for the good read.

I think we’re missing a control here: is there any way to measure whether a rejected paper improved prior to its eventual publication? The “buried the paper for 18 months and then resent” anecdotes suggest the cooldown period might have actually been good for the paper! Egos are fragile things, in academia especially. Maybe a way to measure the impact of rejections is by comparing the manuscript versions. But it would be subjective, personal, and bound to hurt more egos than to do good.