By Ethan Sim
Empiricism and objectivity undergird the reliability of scientific research (Nature Medicine, 2001) by inspiring investigative and publicatory rigour (Prager et al., 2018). However, the nexus between significant results and career progression (Joober et al., 2012) creates publication bias: an incentive to prioritise such results for publication, which ultimately undermines scientific progress (Nature Human Behaviour, 2019). Although opinions differ about both the causes and the extent of the problem, safeguards against publication bias must become ubiquitous among both investigators and journals. Such measures will ensure perspectives and consensus are founded upon rigour, rather than results.
The notion of scientific objectivity demands that researchers remain faithful to facts (Reiss & Sprenger, 2020). Implicit in this is the expectation that study publication is independent of experimental outcome; a well-designed study finding no significant deviation from the null hypothesis should be of equivalent value to an equally well-designed study finding support for the alternative hypothesis (Button et al., 2016). Assuming all researchers publish their findings in this manner, a preponderance of significant results across study sizes and scientific fields should not be present (Sterne & Harbord, 2004). Yet, this phenomenon recurs across a plethora of disciplines, from psychology (Francis, 2012) to ecology (Jennions & Møller, 2002), and remains as pressing a problem today (Ayorinde et al., 2020) as when it was first described by Sterling (1959), who evaluated 294 published articles across four psychology journals, and found that more than 97% supported their alternative hypotheses. Such persistent, directional selection for significant results across a diverse array of fields indicates a bias intrinsic to the publication process, which all studies must undergo.
The publication process involves multiple stages and stakeholders, each of which may contribute to publication bias. A logical starting point is the researcher; given that the effort involved in writing up and publishing results is approximately equal for every experiment, and that significant results receive greater attention than non-significant results, it is unsurprising that researchers’ unwillingness to write up non-significant results is thought to be the greatest contributor to publication bias (Song et al., 2000). Rosenthal (1979) famously termed this the “file drawer problem”: non-significant results often languish in researchers’ file drawers. While researchers are certainly responsible, arguing that the problem of publication bias is essentially one of researcher discipline is myopic, as such behaviour is merely a product of a research culture biased towards significance. Mahoney (1977) found confirmation bias in the peer-review system – when randomly assigned research manuscripts differing only in the results and discussion sections, reviewers tended to rate significant results almost twice as highly as non-significant results. This seemingly irrational behaviour could arise because non-significant results also stem from methodological limitations which decrease study sensitivity (Button et al., 2016). Thus primed, reviewers subject non-significant results to more scrutiny, which increases the time between submission and publication (Ioannidis, 1998). In addition to reviewer bias, journal editors also have an incentive to prioritise significant results for publication, owing to their need to satisfy reader curiosity and expand their circulation (Song et al., 2000). Reviewers and editors thus contribute to an environment which disproportionately rewards, and therefore incentivises researchers to find, significant results. Such an incentive is further strengthened by the increasing usage of bibliographic parameters, such as the number of publications, to gauge researcher value (Fanelli, 2010), and the increasing competition for limited faculty positions (van Djik, Manor & Carey, 2014). Publication bias should therefore be understood as the inevitable outcome of the nexus between significant results and career advancement (Higginson & Munafò, 2016) amidst a “publish or perish” culture (Grimes, Bauch & Ioannidis, 2018).
Although the causes of publication bias can now be clearly identified, its effects on scientific progress are the subject of much debate. Most researchers agree that, a priori, publication bias is detrimental to scientific advancement: the deliberate omission of non-significant results hinders reproducibility and skews scientific knowledge (Begley & Ellis, 2012). Any significant result arrived at by chance is inherently difficult to replicate, and its canonisation as fact following publication (Nissen et al., 2016) may waste subsequent researchers’ funding and effort in pursuit of an ultimately futile line of research (Baker, 2016). Unfortunately, such incidents appear to be the rule, rather than the exception; most published research cannot be replicated across numerous fields, including cancer research (Begley & Ellis, 2012) and psychology (Open Science Collaboration, 2015), lending support to previously controversial assertions that most published research may be false (Ioannidis, 2005), undermining public trust in science (Anvari & Lakens, 2018), and misinforming public policy and clinical practice (Crawford, Briggs & Engeland, 2010). It is in this last domain that the effects of publication bias are especially detrimental; basing their decisions on available literature, clinicians may recommend a seemingly efficacious treatment regime, but such treatment may turn out to be futile, or worse, lethal (Joober et al., 2012). A study conducted by Driessen et al. (2015), which found that publication bias had inflated the efficacy of psychological therapy for major depressive disorder, exemplifies these concerns. It is thus incontrovertible that publication bias can exert detrimental effects on science and humanity by misinforming perspectives and consensus.
However, despite a seeming portrait of science in crisis, a growing number of researchers have refuted this perception, arguing that the effects of publication bias have been exaggerated (Fanelli, 2018). This has been supported by recent reassessments of studies which initially indicated that most published research cannot be replicated: in the case of psychology, for instance, reproducibility may be higher than previously thought, as the criteria for determining reproducibility in previous studies may have been inappropriate (Goodman, Fanelli & Ioannidis, 2016) – natural variation in both the original study and the replication was not considered (Patil, Peng & Leek, 2016). Furthermore, the simplistic classification of published studies as “significant” and “non-significant”, often based on abstract text-mining, ignores their increasing complexity (Vale, 2015); despite their disappearance from abstracts, non-significant results may remain within longer publications, inflating estimates of publication bias (Fanelli, 2018). Beyond weaknesses inherent in inferences of publication bias, various factors, such as the greater ease of disproving (rather than proving) a false claim, and the appeal of going against established fact – which may have been canonised on spurious grounds – may also function to mitigate the effects of publication bias in practice (Nissen et al., 2016). This is exemplified by the “publication loop” occurring to research linking the use of beta-blockers with depression – the canonisation of significant results as fact creates an incentive to publish nonsignificant results which disprove them, and canonisation of the latter in turn incentivises the publication of the former (Luijendijk & Koolman, 2012), preventing publication bias and its inverse from misinforming clinical practice. Although publication bias has great capacity to do harm, its extent and impact has been overstated due to weaknesses in related analyses, and the inherent robustness of science to false claims over time.
It is perhaps a reflection of the controversy surrounding the causes and extents of publication bias that a diversity of solutions has emerged, with disagreement emerging over the best method to reduce publication bias (Nature Human Behaviour, 2019). The broad perception of a replication crisis gripping science (Baker, 2016) has led to widespread awareness of publication bias, and the further development of statistical techniques to detect it (Zhu & Carriere, 2016), but this alone is merely a post hoc solution which fails to limit resource wastage on futile research. Journals such as The American Journal of Gastroenterology have become more explicit in soliciting replication attempts and non-significant results (Spiegel & Lacy, 2016), and some journals which specialise entirely in non-significant results, such as Elsevier’s New Negatives in Plant Science, have recently been launched (New Negatives in Plant Science, 2015), with the aim of reversing the preponderance of significant results. However, the extent to which journals in the latter category incentivise researchers by improving their career prospects is doubtful, as they tend to have lower impact scores (Nissen et al., 2016). Funding bodies have also established prizes which explicitly reward replications and non-significant studies to counter the strong incentive to find significant results, but this has yet to become the norm (Nature, 2017). Perhaps the most promising solution to date is the mandatory publication of results following pre-registration (Nature Human Behaviour, 2019), which forces reviewers to assess a study on its importance and analytical rigour, rather than its results; this has shown potential in the field of comparative politics (Findley et al., 2016). In concert with improved detection techniques, which reduce the impact of existing publication bias, mandatory publication following pre-registration may prevent publication bias from emerging in nascent fields by directly addressing the cause of the problem.
The notion that publication bias is detrimental to scientific progress is uncontroversial, but its causes, extent, and solutions are the subject of intense debate. Ultimately, a permanent solution to the prominence of publication bias demands a profession-wide emphasis on scientific rigour over result significance, and a deliberate effort to tie researcher career advancement to the former. Only when such measures are ubiquitous will science always inform – and never impoverish – knowledge, perspectives, and policy.
Anvari, F. & Lakens, D. (2018) The replicability crisis and public trust in psychological science. Comprehensive Results in Social Psychology. 3 (3), 266–286. Available from: https://doi.org/10.1080/23743603.2019.1684822
Ayorinde, A.A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R.J. & Chen, Y.-F. (2020) Assessment of publication bias and outcome reporting bias in systematic reviews of health services and delivery research: A meta-epidemiological study. PLOS ONE. 15, e0227580. Available from: https://doi.org/10.1371/journal.pone.0227580
Baker, M. (2016) 1,500 scientists lift the lid on reproducibility. Nature News. 533, 452. Available from: https://doi.org/10.1038/533452a
Begley, C.G. & Ellis, L.M. (2012) Raise standards for preclinical cancer research. Nature. 483, 531–533. Available from: https://doi.org/10.1038/483531a
Button, K.S., Bal, L., Clark, A. & Shipley, T. (2016) Preventing the ends from justifying the means: withholding results to address publication bias in peer-review. BMC Psychology. 4, 59. Available from: https://doi.org/10.1186/s40359-016-0167-7
Crawford, J.M., Briggs, C.L. & Engeland, C.G. (2010) Publication Bias and Its Implications for Evidence-Based Clinical Decision Making. Journal of Dental Education. 74 (6), 593–600. Available from: https://doi.org/10.1002/j.0022-0337.2010.74.6.tb04904.x
Driessen, E., Hollon, S.D., Bockting, C.L.H., Cuijpers, P. & Turner, E.H. (2015) Does Publication Bias Inflate the Apparent Efficacy of Psychological Treatment for Major Depressive Disorder? A Systematic Review and Meta-Analysis of US National Institutes of Health-Funded Trials. PLOS ONE. 10, e0137864. Available from: https://doi.org/10.1371/journal.pone.0137864
Fanelli, D. (2010) Do Pressures to Publish Increase Scientists’ Bias? An Empirical Support from US States Data. PLOS ONE. 5, e10271. Available from: https://doi.org/10.1371/journal.pone.0010271
Fanelli, D. (2018) Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences of the United States of America. 115 (11), 2628–2631. Available from: https://doi.org/10.1073/pnas.1708272114
Findley, M.G., Jensen, N.M., Malesky, E.J. & Pepinsky, T.B. (2016) Can Results-Free Review Reduce Publication Bias? The Results and Implications of a Pilot Study. Comparative Political Studies. 49 (13), 1667–1703. Available from: https://doi.org/10.1177/0010414016655539
Francis, G. (2012) Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review. 19, 975–991. Available from: https://doi.org/10.3758/s13423-012-0322-y
Goodman, S.N., Fanelli, D. & Ioannidis, J.P.A. (2016) What does research reproducibility mean? Science Translational Medicine. 8 (341), 341ps12. Available from: https://doi.org/10.1126/scitranslmed.aaf5027
Grimes, D.R., Bauch, C.T. & Ioannidis, J.P.A. (2018) Modelling science trustworthiness under publish or perish pressure. Royal Society Open Science. 5 (1), 171511. Available from: https://doi.org/10.1098/rsos.171511
Higginson, A.D. & Munafò, M.R. (2016) Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions. PLOS Biology. 14, e2000995. Available from: https://doi.org/10.1371/journal.pbio.2000995
Ioannidis, J.P.A. (1998) Effect of the Statistical Significance of Results on the Time to Completion and Publication of Randomized Efficacy Trials. JAMA. 279, 281. Available from: https://doi.org/10.1001/jama.279.4.281
Ioannidis, J.P.A. (2005) Why Most Published Research Findings Are False. PLOS Medicine. 2, e124. Available from: https://doi.org/10.1371/journal.pmed.0020124
Jennions, M.D. & Møller, A.P. (2002) Publication bias in ecology and evolution: an empirical assessment using the ‘trim and fill’ method. Biological Reviews. 77 (2), 211–222. Available from: https://doi.org/10.1017/S1464793101005875
Joober, R., Schmitz, N., Annable, L. & Boksa, P. (2012) Publication bias: What are the challenges and can they be overcome? Journal of Psychiatry & Neuroscience. 37 (3), 149–152. Available from: https://doi.org/10.1503/jpn.120065
Luijendijk, H.J. & Koolman, X. (2012) The incentive to publish negative studies: how beta-blockers and depression got stuck in the publication cycle. Journal of Clinical Epidemiology. 65 (5), 488–492. Available from: https://doi.org/10.1016/j.jclinepi.2011.06.022
Mahoney, M.J. (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research. 1, 161–175. Available from: https://doi.org/10.1007/BF01173636
Nature. (2017) Rewarding negative results keeps science on track. Nature. 551, 414. Available from: https://doi.org/10.1038/d41586-017-07325-2
Nature Human Behaviour. (2019) The importance of no evidence. Nature Human Behaviour. 3, 197. Available from: https://doi.org/10.1038/s41562-019-0569-7
Nature Medicine. (2001) In science we trust. Nature Medicine. 7, 871. Available from: https://doi.org/10.1038/90866
New Negatives in Plant Science. (2015) New Negatives in Plant Science. 1–2, A1. Available from: https://doi.org/10.1016/j.neps.2016.02.001
Nissen, S.B., Magidson, T., Gross, K. & Bergstrom, C.T. (2016) Publication bias and the canonization of false facts. eLife. 5, e21451. Available from: https://doi.org/10.7554/eLife.21451
Open Science Collaboration. (2015) Estimating the reproducibility of psychological science. Science. 349 (6251), aac4716. Available from: https://doi.org/10.1126/science.aac4716
Patil, P., Peng, R.D. & Leek, J.T. (2016) What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science. Perspectives on Psychological Science. 11, 539–544. Available from: https://doi.org/10.1177/1745691616646366
Prager, E.M., Chambers, K.E., Plotkin, J.L., McArthur, D.L., Bandrowski, A.E., Bansal, N., Martone, M.E., Bergstrom, H.C., Bespalov, A. & Graf, C. (2018) Improving transparency and scientific rigor in academic publishing. Brain and Behavior. 9 (1), e01141. Available from: https://doi.org/10.1002/brb3.1141
Reiss, J. & Sprenger, J. (2020) Scientific Objectivity. In: Zalta, E. (ed.) The Stanford Encyclopedia of Philosophy (Winter 2020 Edition). Available from: https://plato.stanford.edu/archives/win2020/entries/scientific-objectivity
Rosenthal, R. (1979) The file drawer problem and tolerance for null results. Psychological Bulletin, 86 (3), 638–641. Available from: https://doi.org/10.1037/0033-2909.86.3.638
Song, F., Eastwood, A., Gilbody, S., Duley, L. & Sutton, A. (2000) Publication and related biases: a review. Health Technology Assessment. 4 (10). Available from: https://doi.org/10.3310/hta4100
Spiegel, B. & Lacy, B.E. (2016) Negative Is Positive. The American Journal of Gastroenterology. 111 (11), 1505. Available from: https://doi.org/10.1038/ajg.2016.476
Sterling, T.D. (1959) Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa. Journal of the American Statistical Association. 54 (285), 30–34. Available from: https://doi.org/10.1080/01621459.1959.10501497
Sterne, J.A.C. & Harbord, R.M. (2004) Funnel Plots in Meta-analysis. The Stata Journal. 4, 127–141. Available from: https://doi.org/10.1177/1536867X0400400204
Vale, R.D. (2015) Accelerating scientific publication in biology. Proceedings of the National Academy of Sciences of the United States of America. 112 (44), 13439–13446. Available from: https://doi.org/10.1073/pnas.1511912112
Van Dijk, D., Manor, O. & Carey, L.B. (2014) Publication metrics and success on the academic job market. Current Biology. 24 (11), R516–R517. Available from: https://doi.org/10.1016/j.cub.2014.04.039
Zhu, Q. & Carriere, K. (2016) Detecting and correcting for publication bias in meta-analysis – A truncated normal distribution approach. Statistical Methods in Medical Research. 27 (9), 2722–2741. Available from: https://doi.org/10.1177/0962280216684671