Publication bias

Publication bias arises from the tendency for researchers, editors, and companies to handle the reporting of experimental results that are positive (i.e., they show a significant finding) differently from results that are negative (i.e. supported the null hypothesis) or inconclusive.

This page provides an undergraduate-level introduction to the problem of publication bias.

Two counter-acting biases

 * Low Power: under-estimation of real effects
 * Publication Bias or File-drawer effect: over-estimation of real effects

What is publication bias?

 * Publication of results depends on their nature and direction.
 * Studies that show significant effects are more likely to be published.
 * Type I publication errors are underestimated to the extent that they are: “frightening, even calling into question the scientific basis for much published literature.” (Greenwald, 1975, p. 15)

Funnel plots

 * A scatterplot of treatment effect against study size.
 * Precision in estimating the true treatment effect increases as N increases.
 * Small studies scatter more widely at the bottom of the graph.
 * As studies become less precise, results should be more variable, scattered to both sides of the more precise larger studies … unless there is publication bias.
 * In the absence of bias the plot should resemble a symmetrical inverted funnel.
 * If there is publication bias this will cause meta-analysis to overestimate effects.
 * The more pronounced the funnel plot asymmetry, the more likely it is that the amount of bias will be substantial.

File drawer effects

 * 1) Tendency for non-sig. results to be ‘filed away’ (hidden) and not published.
 * 2) of null studies which would have to ‘filed away’ in order for a body of significant published effects to be considered doubtful.

Countering the bias

 * 1) Journal Articles in Support of the Null Hypothesis