The Talk.Origins Archive Post of the Month: October 2010

Subject:    | The quality (and quantity) of modern (medical) science.
Date:       | 18 Oct 2010
Message-ID: | i9i7oa$7dm$1@speranza.aioe.org

> For clarity's sake: I don't intend to feed any creationists or
> something like that. But if Ioannidis is right, it is about time to
> seriously rethink peer-reviewing and/or grant systems, methinks.
> Opinions?

First, peer review standards vary from field to field. In some branches of mathematics, reviewers are expected to check every step of a proof, and peer review can come fairly close to a confirmation of a claim. At another extreme, in some branches of experimental physics there's no way for a reviewer to check many things, short of spending a few billion dollars to recheck an experiment. A detector at the Large Hadron Collider, for instance, is complex enough that it's extremely unlikely that anyone who isn't actually on the experiment can judge some claims (e.g., how much statistical weight to ascribe to various observations). In situations like that, it's often the experimental collaboration itself that does the most rigorous review. They have a strong incentive – there is more than one detector, and it would be very embarrassing to make a strong claim only to have it disproved by your competition.

My field of theoretical physics is somewhere in between. Reviewers are not expected to strongly confirm that a paper is correct. They are basically supposed to look for:

– obvious errors ("The author claims that special relativity is disproved by observation X, but in fact the theory predicts exactly the observed outcome," or "The author claims that special relativity is disproved by observation X, but is apparently unaware that this effect has been tested in papers A, B, and C to a thousand times the author's accuracy; if he wants to claim these other observations are wrong, he ought to at least acknowledge their existence," or "The model presented here is inconsistent – it's easy to see that the only solution of equation (11) is x=0, which contradicts equation (14)," or "In section 4, the authors show that the effect they're looking for is too small to measure; why, then, do they say in the conclusion that they've found an important new test of their model?");

– conspicuous gaps ("The author provides strong evidence for hypothesis X, but nothing in the paper seems to support her much stronger claim Y," or "Equation (7) is said to follow from equation (6), and it might, but I, at least, don't see how, and since I'm nearly as bright as most readers of this journal, I expect they won't, either; a much more careful explanation is needed");

– overlaps with existing results ("If the author bothered to read the literature, he would see his claim is just a special case of the general results of [my] paper A, discussed in detail in section 2 of that paper," or "Section 4 is an interesting new result, but section 3 reproduces the material I'm currently teaching from textbook X");

– missing references ("Section 3 of this paper is based on the results of experiment A, but these were made obsolete by the much more accurate experiment B last year; the author should check that her model is consistent with the new data," or "This is a new result, but much of it is an extension of paper C, which ought to be cited," or – this one I once got in a referee report on one of my own papers – "A general discussion of this issue appeared in an obscure paper by Poincare in 1905; see if you can get someone to translate it, and cite it where it's appropriate");

– incoherent writing ("Paragraph 2 seems to only make sense if the authors are using the word "energy" to mean "entropy" and the word "mass" to mean "momentum," or "I've worked on a very closely related topic, but I find this paper incomprehensible; the authors never define their symbols, and they seem to assume that any reader will have already memorized the details of their earlier paper A");

– level of interest ("This result has already been shown for the ten most common types of black holes; it's true that no one seems to have checked this rather obscure eleventh type, but is this really important enough to publish?" or "I'm sure there's some journal out there – maybe the Journal of Mediocre Results – that would want to publish this, but it doesn't seem to meet the standards of importance required by Journal of High Prestige Physics");

Clearly, even if referees are careful – and sometimes they're not – errors will get through, and the system is certainly not designed to catch deliberate fraud. Moreover, there is such a proliferation of journals these days that a dedicated author can usually find *somewhere* to publish almost anything. But for the decent journals, at least, peer review does screen out most of the really bad papers. Typical acceptance rates in my field range from around 30% to around 60%, and from the papers I've reviewed, I'd say with some confidence that most of the rejected papers really deserved to be rejected.

(As one calibration point, I've served on a journal editorial board for which I handled appeals from authors whose papers were rejected. Of the fairly large number of appeals I received, I decided that the referees were just wrong about 5% of the time – for these cases I recommended publication, sometimes after revisions – and that about 10% of the cases were ambiguous enough to be sent out for further review. That's certainly not perfect, but it's a pretty good record for a highly selective journal. Of course, you can believe or not believe my judgment...)

At least as important, though, peer review leads to improved papers. Many submissions are initially sent back to authors for revision, and in my experience with my own papers, this has generally been a good thing. It's led me, at least, to clearer writing, to fewer gaps and fewer assumptions about what readers know or don't know, fewer missed references, and in a few cases to major improvements in the content. I've had a couple of bad experiences with referees who just missed the point, but those have been fairly rare exceptions.

Does Peer Review prevent publication of bad science?

Post of the Month: October 2010