Limits...
Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science.

Veldkamp CL, Nuijten MB, Dominguez-Alvarez L, van Assen MA, Wicherts JM - PLoS ONE (2014)

Bottom Line: To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors.We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard.Co-piloting was not found to be associated with reporting errors.

View Article: PubMed Central - PubMed

Affiliation: Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, The Netherlands.

ABSTRACT
Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

No MeSH data available.


The probability per journal that an article contains at least one p-value comprising an error or gross error (with 95% confidence interval).Note. JAP = Journal of Applied Psychology (n = 42); JCCP = Journal of Consulting and Clinical Psychology (n = 67); JCN = Journal of Cognitive Neuroscience (n = 107); JCPP = Journal of Child Psychology and Psychiatry (n = 39); JPSP = Journal of Personality and Social Psychology (n = 133); PP = Psychophysiology (n = 42); TOTAL = all articles together (N = 430).
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4262438&req=5

pone-0114876-g002: The probability per journal that an article contains at least one p-value comprising an error or gross error (with 95% confidence interval).Note. JAP = Journal of Applied Psychology (n = 42); JCCP = Journal of Consulting and Clinical Psychology (n = 67); JCN = Journal of Cognitive Neuroscience (n = 107); JCPP = Journal of Child Psychology and Psychiatry (n = 39); JPSP = Journal of Personality and Social Psychology (n = 133); PP = Psychophysiology (n = 42); TOTAL = all articles together (N = 430).

Mentions: We also compared the error prevalence across different journals/fields. Logistic regression analyses with journal as predictor revealed that there were differences in error rates between the journals: χ2 (5, N = 430) = 49.46, p<.001. The probability that an article contained at least one p-value that comprised an error was lower in the Journal of Applied Psychology than in all other journals (23.8%, CI [13.3−38.9], all ps ≤.002<0.05/6) except the Journal of Child Psychology and Psychiatry (51.3%, CI [36.0–66.4], p = .012>0.05/6). At the same time, this probability was higher in the Journal of Personality and Social Psychology (79.7%, CI [72.0−85.7], all ps ≤.006<0.05/6) than in all other journals except Psychophysiology (71.4%, CI [56.1–83.0], p = .264>0.05/6). These differences may be attributable to differences between journals in the mean number of reported p-values per article, as a higher number of reported p-values entails a higher probability that an article contains an error. For example, an article in the Journal of Personality and Social Psychology contains more than 30 p-values on average, whereas the average article in the Journal of Applied Psychology contains only slightly more than eight p-values. The probability that an article contained at least one p-value that comprised a gross error differed also by journal: χ2 (5, N = 430) = 15.46, p = .009, but no journal differed significantly from any other journal (all ps ≥0.012>0.05/6). The error probabilities for the sample as a whole and for each field separately are presented in Fig. 2, together with their 95% confidence intervals.


Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science.

Veldkamp CL, Nuijten MB, Dominguez-Alvarez L, van Assen MA, Wicherts JM - PLoS ONE (2014)

The probability per journal that an article contains at least one p-value comprising an error or gross error (with 95% confidence interval).Note. JAP = Journal of Applied Psychology (n = 42); JCCP = Journal of Consulting and Clinical Psychology (n = 67); JCN = Journal of Cognitive Neuroscience (n = 107); JCPP = Journal of Child Psychology and Psychiatry (n = 39); JPSP = Journal of Personality and Social Psychology (n = 133); PP = Psychophysiology (n = 42); TOTAL = all articles together (N = 430).
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4262438&req=5

pone-0114876-g002: The probability per journal that an article contains at least one p-value comprising an error or gross error (with 95% confidence interval).Note. JAP = Journal of Applied Psychology (n = 42); JCCP = Journal of Consulting and Clinical Psychology (n = 67); JCN = Journal of Cognitive Neuroscience (n = 107); JCPP = Journal of Child Psychology and Psychiatry (n = 39); JPSP = Journal of Personality and Social Psychology (n = 133); PP = Psychophysiology (n = 42); TOTAL = all articles together (N = 430).
Mentions: We also compared the error prevalence across different journals/fields. Logistic regression analyses with journal as predictor revealed that there were differences in error rates between the journals: χ2 (5, N = 430) = 49.46, p<.001. The probability that an article contained at least one p-value that comprised an error was lower in the Journal of Applied Psychology than in all other journals (23.8%, CI [13.3−38.9], all ps ≤.002<0.05/6) except the Journal of Child Psychology and Psychiatry (51.3%, CI [36.0–66.4], p = .012>0.05/6). At the same time, this probability was higher in the Journal of Personality and Social Psychology (79.7%, CI [72.0−85.7], all ps ≤.006<0.05/6) than in all other journals except Psychophysiology (71.4%, CI [56.1–83.0], p = .264>0.05/6). These differences may be attributable to differences between journals in the mean number of reported p-values per article, as a higher number of reported p-values entails a higher probability that an article contains an error. For example, an article in the Journal of Personality and Social Psychology contains more than 30 p-values on average, whereas the average article in the Journal of Applied Psychology contains only slightly more than eight p-values. The probability that an article contained at least one p-value that comprised a gross error differed also by journal: χ2 (5, N = 430) = 15.46, p = .009, but no journal differed significantly from any other journal (all ps ≥0.012>0.05/6). The error probabilities for the sample as a whole and for each field separately are presented in Fig. 2, together with their 95% confidence intervals.

Bottom Line: To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors.We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard.Co-piloting was not found to be associated with reporting errors.

View Article: PubMed Central - PubMed

Affiliation: Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, The Netherlands.

ABSTRACT
Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

No MeSH data available.