Does peer review work well?
Since Toni and I were getting into a discussion about the merits and flaws of peer review, I did a quick search to get a sense of the research that’s been done on the subject. Robergs (2003) attempts to summarise previous studies. He points out that peer review is a recent phenomenon in science:
Although evidence exists for some journals to have adopted a peer review system prior to the 20th century, other journals such as The Journal of the American Medical Association (JAMA) sought external opinion on manuscripts ‘only rarely’ through to the 1950s.
One reason for this was ‘the shortage of manuscripts for publication’. This situation changed drastically after World War II: ‘Journal editors experienced a transition of too few manuscripts to too many’:
the peer review system was not adopted for its ability to improve manuscript content and validity. Rather, the system was adopted, at least equally, as an answer to the realities of scientific publication where the volume of submissions out-stripped the resources of journals and professional organizations.
The implication seems to be that if scientific journals previously managed to select good articles without peer review, peer review is unnecessary. But it seems to me that, assuming that journal editors were reading the articles themselves in those days and were competent to do so, they were in effect using peer review.
Robergs notes that reviewers rarely agree on the quality of an article: ‘the available data indicate that there is minimal consensus in peer review between multiple reviewers’. Moreover, ‘blind’ review processes (which are supposed to maintain the anonymity of both authors and reviewers) are ineffective when the authors are well-known: ‘existing data indicate that most reviewers (75%) can detect the identity of a recognized researcher of a given topic’.
Armstrong (1996) provides some evidence indicating that peer review discourages the publication of ideas that challenge the conventional wisdom of the field.
One of the comments on Armstrong’s article mentions a hilarious study by Peters and Ceci (1982) in which the authors selected twelve articles that had been written by researchers at prestigious institutions and recently published in prestigious peer-reviewed psychology journals with non-blind refereeing practices, and simply re-submitted these articles to the same journals that had published them. The only change they made to the articles was to substitute fictitious authors’ names and institutions for the real ones. Only three of the resubmissions were detected; the remaining nine were reviewed and rejected, usually for ‘serious methodological flaws’. This seems to suggest that there’s some value in blind refereeing after all.
Responding to Armstrong, Miser (1998) points out a crucial problem with all attempts to critique peer review: there is no explicit model of the editorial process.
sometimes it appears that the editor of a journal is in charge, sometimes as though he were taking orders from his referees and associate editors and just publishing what they tell him to.
Since nobody has produced a scientific account of how editorial processes really work, all discussions of peer review are based on assumptions about those processes rather than real knowledge.
The proceedings of the recent International Symposium on Peer Reviewing are now available, but I haven’t looked at them yet.