Reason 6: The entire dataset was not considered
The scientific method is a good way to get more objective answers to questions. However, the way that we publish scientific findings is rather less objective. Scientific journals are not all equal. Some are seen as more ‘sexy’ than others. The measure of a journal’s ‘sexiness’ is its impact factor; so journals with high impact factors are seen as the best to publish in. Scientists are judged (by promotion committees, funding agencies etc.) according to the impact factors of the journals they have published in. Unfortunately, this sets up a bias against publishing negative results. Positive results (where you prove that something works) are intrinsically more ‘sexy’ than negative results. Therefore scientists are much more likely to get results published (particularly in high impact journals) if they have found a positive result. There is a much lower incentive to publish your negative or inconclusive results since these will only be accepted by low impact journals.
This phenomenon is called publication bias. For example, imagine that ten studies were carried out to test the impact of a new educational reform on children’s learning and five demonstrate a positive impact and five demonstrate a negative impact. It is conceivable that some of the researchers who found negative impacts feel that the incentive to write up their results and get them published is not high enough so they never get the negative results published (for example perhaps they have been involved in some other research which has given more ‘interesting’ results and they prioritise that instead). For somebody who examines the research literature, it may seem as if the balance of evidence shows that the intervention works when in fact the true picture is that the results are balanced for and against.
Again I have experienced this phenomenon in my own work. Early in my career, I carried out some research looking at HIV infection in a certain type of brain cell known as astrocytes. Despite my best efforts I was never able to find HIV in the astrocytes obtained from the post-mortem brains of HIV-positive individuals. I never got round to publishing this finding, mainly because it didn’t seem very interesting, but many others did publish when they found HIV in astrocytes. In this small way I contributed to the publication bias in this field.
Next — Reason 7: Part of the dataset was suppressed