The correlation bias of psychology researchers

doi:10.15406/ahoaj.2018.02.00052

eISSN: 2577-8250

Arts & Humanities Open Access Journal

Short Communication Volume 2 Issue 3

The correlation bias of psychology researchers

David Trafimow,

Verify Captcha

Regret for the inconvenience: we are taking measures to prevent fraudulent form submissions by extractors and page crawlers. Please type the correct Captcha word to see email ID.

Jenna Doddridge, Amy White

New Mexico State University, USA

Correspondence: David Trafimow, Department of Psychology, New Mexico State University, USA

Received: May 29, 2018 | Published: June 14, 2018

Citation: Trafimow D, Doddridge J, White A. The correlation bias of psychology researchers. Art Human Open Acc J. 2018;2(3):167. DOI: 10.15406/ahoaj.2018.02.00052

Download PDF

Abstract

The present manuscript investigated whether psychology researchers are biased in favor of positive correlations as opposed to negative ones. Articles in Psychological Science were searched. Consistent with the hypothesis, positive correlations were much more common than negative ones in the published literature.

Keywords: correlation bias, correlation, bias, correlation

The correlation bias of psychology researchers

There exists a voluminous literature where psychology researchers have documented the biases that people have (see Kahneman, Slovic, & Tversky, 1982; Kahneman & Tversky, 2000 for famous reviews).^1,2 However, psychology researchers also are subject to biases, though this is an area that has received much less attention.³ Our goal is to make the case that there is a bias that psychology researchers have, that we term the correlation bias. Specifically, psychology researchers are biased towards looking for positive correlations and are relatively likely to ignore the possibility of negative correlations. Put another way, the correlation bias is a tendency for psychology researchers to be more likely to attempt to obtain positive than negative correlations.

It is possible to conduct a simple test of the notion of correlation bias. It is merely necessary to search the empirical literature and record the reported correlations. Under the assumption that there is no reason to expect more positive than negative correlations in the psychological universe, it follows that if there were no bias, we should find an approximately equal number of positive and negative correlations reported in the literature. But if many more positive than negative correlations are reported, this would constitute evidence of our hypothesized correlation bias.

To test the bias, we searched the journal, Psychological Science, because it is the flagship empirical journal of the Association for Psychological Science. We gained access to all articles in 2013 and kept track of all correlations reported in all articles during that year. An exception is that some correlations were for reliability analyses, where positive correlations ought to be expected, and so we did not count these. We only counted “substantive” correlations that could have been positive or negative to ensure a fair test of our hypothesized correlation bias.

We performed three kinds of analyses. First, we counted each correlation as a separate “case.” There were 691 total correlations reported and 511 (74.0%) were positive whereas 180 (26.0%) were negative. A binomial test indicated that this difference was highly significant, p < .001, g = 0.24.

The second and third analyses treated the articles as cases rather than the correlations as cases. There were 133 articles that contained correlations. Many of these articles reported more than one correlation. Thus, in the second analysis, we coded the article as being one where there were more positive than negative correlations reported, as being one where there were more negative than positive correlations reported or as one where there was a tie between the number of positive and negative correlations reported. We found that there were 92 articles in the positive group, 30 articles in the negative group, and 11 ties. A binomial analysis on the positive versus negative group, excluding ties, was highly significant, p < .001, g = 0.25. This result held up even when we included the ties with the negative group, p < .001, g = 0.19.

The third analysis involved counting each article as a separate case, but instead of binary coding (positive or negative), we computed the percentage of positive correlations. In the case of no bias, we would expect the percentage to be approximately 50%. In fact, there was substantial bias, where the mean percentage was 74.8% (SD = 3.43%), which was significantly greater than the 50% that would be expected due to chance, t(132) = 8.34, p < .001, d = .72.

In summary, no matter how we analyzed the data, there was strong evidence that psychology researchers are subject to the correlation bias. A limitation of the present study is that we investigated a single general journal of psychology and there is the possibility that psychology researchers in different areas have different levels of bias.