cognitive science
and more
Publication bias in studies on publication bias?

Last week I wrote a glowing review of Ben Goldacre's "Bad Science". The book is full of amusing anecdotes and trivia, such as the section on the well known publication bias: Studies finding positive results are more likely to be published than studies finding no results. For example, a study showing that a particular treatment for (say) depression is effective may be published, whereas ten studies that fail to show an effect are not published (or published in low-impact journals), because they are considered uninteresting. This would falsely give the impression that the treatment is effective, even to those who go through the trouble of doing a literature search, whereas the evidence clearly suggests otherwise.

The amusing part is that, according to Goldacre, "a paper even found evidence of publication bias in studies of publication bias"! In other words, review papers that fail to show a publication bias are not published; The publication bias bites itself in the tail. You gotta love it, right? But when I looked up the study to which Goldacre refers, the authors actually conclude that they "found no evidence of publication bias in reports on publication bias"!

So the first time I look up a reference from Bad Science, I find a blatant misinterpretation. In all fairness, the authors did find a hint of a publication bias in studies of publication bias (and the statistical power was low, etc.), but it does seem that Goldacre was a little sloppy here... Normally I wouldn't pick on somebody for a minor glitch like this, but since Goldacre is all about accurate science reporting I thought this was kind of funny!

References

Dubben, H. H., & Beck-Bornholdt, H. P. (2005). Systematic review of publication bias in studies on publication bias. British Medical Journal, 331(7514), 433-434. [Link]