Limits...
Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science.

Veldkamp CL, Nuijten MB, Dominguez-Alvarez L, van Assen MA, Wicherts JM - PLoS ONE (2014)

Bottom Line: To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors.We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard.Co-piloting was not found to be associated with reporting errors.

View Article: PubMed Central - PubMed

Affiliation: Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, The Netherlands.

ABSTRACT
Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

No MeSH data available.


Flow chart for composition of sample.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4262438&req=5

pone-0114876-g001: Flow chart for composition of sample.

Mentions: To analyze the relationship between co-piloting and the accuracy of p-values reported in the first or only study in the corresponding articles, we merged the data file containing the retrieved p-values from each article and the data file containing the survey responses. While p-values had been retrieved from 430 out of 697 articles, and survey responses were obtained for 346 articles, these sets of articles did not completely overlap (i.e., for some articles statistical results were retrieved but no survey response was obtained, and vice versa). In total, the data of 210 articles (48.8% of the 430 articles from which statistical results had been retrieved) could be matched. Thus, we matched each statistical result retrieved from the first (or only) study reported in these articles to the survey responses given by the respondent with the lowest author number. The statistical results of the remaining 220 articles were retained in the file to analyze the effect of non-response. A schematic overview of our sample is presented in Fig. 1. Based on the study of Wicherts, Bakker, and Molenaar [14] who found a relationship between willingness to share research data and the prevalence of reporting errors in a sample of 48 articles, we expected to have enough power to detect a relationship between co-piloting and statistical reporting errors in our sample of 430 papers from which p-values were retrieved. With the 210 articles for which we obtained survey responses and had retrieved p-values, our sample was still more than four times as large as in Wicherts et al.’s study [14].


Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science.

Veldkamp CL, Nuijten MB, Dominguez-Alvarez L, van Assen MA, Wicherts JM - PLoS ONE (2014)

Flow chart for composition of sample.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4262438&req=5

pone-0114876-g001: Flow chart for composition of sample.
Mentions: To analyze the relationship between co-piloting and the accuracy of p-values reported in the first or only study in the corresponding articles, we merged the data file containing the retrieved p-values from each article and the data file containing the survey responses. While p-values had been retrieved from 430 out of 697 articles, and survey responses were obtained for 346 articles, these sets of articles did not completely overlap (i.e., for some articles statistical results were retrieved but no survey response was obtained, and vice versa). In total, the data of 210 articles (48.8% of the 430 articles from which statistical results had been retrieved) could be matched. Thus, we matched each statistical result retrieved from the first (or only) study reported in these articles to the survey responses given by the respondent with the lowest author number. The statistical results of the remaining 220 articles were retained in the file to analyze the effect of non-response. A schematic overview of our sample is presented in Fig. 1. Based on the study of Wicherts, Bakker, and Molenaar [14] who found a relationship between willingness to share research data and the prevalence of reporting errors in a sample of 48 articles, we expected to have enough power to detect a relationship between co-piloting and statistical reporting errors in our sample of 430 papers from which p-values were retrieved. With the 210 articles for which we obtained survey responses and had retrieved p-values, our sample was still more than four times as large as in Wicherts et al.’s study [14].

Bottom Line: To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors.We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard.Co-piloting was not found to be associated with reporting errors.

View Article: PubMed Central - PubMed

Affiliation: Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, The Netherlands.

ABSTRACT
Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

No MeSH data available.