Limits...
Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science.

Veldkamp CL, Nuijten MB, Dominguez-Alvarez L, van Assen MA, Wicherts JM - PLoS ONE (2014)

Bottom Line: To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors.We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard.Co-piloting was not found to be associated with reporting errors.

View Article: PubMed Central - PubMed

Affiliation: Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, The Netherlands.

ABSTRACT
Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

No MeSH data available.


The probability that a p-value in the first (or only) study reported comprises a gross error: co-piloted studies versus non-co-piloted studies.Note. statistical analyses = conducting the statistical analyses (N = 2,247); write up sample details = writing the sample details in the manuscript (N = 2,215); write up results = writing up the results in the manuscript (N = 2,231); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 2,185); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 2,228); data sharing now = having access to the data at the moment the survey was being filled in (N = 2,226).
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4262438&req=5

pone-0114876-g006: The probability that a p-value in the first (or only) study reported comprises a gross error: co-piloted studies versus non-co-piloted studies.Note. statistical analyses = conducting the statistical analyses (N = 2,247); write up sample details = writing the sample details in the manuscript (N = 2,215); write up results = writing up the results in the manuscript (N = 2,231); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 2,185); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 2,228); data sharing now = having access to the data at the moment the survey was being filled in (N = 2,226).

Mentions: Our third aim was to establish whether a relationship exists between co-piloting and the probability that a p-value comprised an error. To answer this question, we only took into account those p-values of articles of which at least one author responded to our survey. By means of a multilevel logistic regression analysis with article as random factor and journal as fixed factor we first established that in this subsample there were no differences between journals in the probability that a p-value comprised an error (χ2 (5, N = 2299) = 6.35, p = .274), nor in the probability that a p-value comprised a gross error (χ2 (5, N = 2299) = 4.15, p = .528). We then ran six different multilevel logistic regression analyses, each with article as random factor, and one of the six co-piloting variables as fixed factor. Our hypothesis that co-piloting is related to the probability that a given p-value was associated with a reduced error risk lacked support: we found no differences for any of the six processes between articles in which co-piloting had occurred and articles in which co-piloting had not occurred in the probability that a given p-value was inconsistent (all ps ≥.283>0.05/6, see Fig. 5), nor in the probability that a p-value was inconsistent to the extent that it may have affected a decision about statistical significance (all ps ≥.323>0.05/6, see Fig. 6).


Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science.

Veldkamp CL, Nuijten MB, Dominguez-Alvarez L, van Assen MA, Wicherts JM - PLoS ONE (2014)

The probability that a p-value in the first (or only) study reported comprises a gross error: co-piloted studies versus non-co-piloted studies.Note. statistical analyses = conducting the statistical analyses (N = 2,247); write up sample details = writing the sample details in the manuscript (N = 2,215); write up results = writing up the results in the manuscript (N = 2,231); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 2,185); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 2,228); data sharing now = having access to the data at the moment the survey was being filled in (N = 2,226).
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4262438&req=5

pone-0114876-g006: The probability that a p-value in the first (or only) study reported comprises a gross error: co-piloted studies versus non-co-piloted studies.Note. statistical analyses = conducting the statistical analyses (N = 2,247); write up sample details = writing the sample details in the manuscript (N = 2,215); write up results = writing up the results in the manuscript (N = 2,231); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 2,185); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 2,228); data sharing now = having access to the data at the moment the survey was being filled in (N = 2,226).
Mentions: Our third aim was to establish whether a relationship exists between co-piloting and the probability that a p-value comprised an error. To answer this question, we only took into account those p-values of articles of which at least one author responded to our survey. By means of a multilevel logistic regression analysis with article as random factor and journal as fixed factor we first established that in this subsample there were no differences between journals in the probability that a p-value comprised an error (χ2 (5, N = 2299) = 6.35, p = .274), nor in the probability that a p-value comprised a gross error (χ2 (5, N = 2299) = 4.15, p = .528). We then ran six different multilevel logistic regression analyses, each with article as random factor, and one of the six co-piloting variables as fixed factor. Our hypothesis that co-piloting is related to the probability that a given p-value was associated with a reduced error risk lacked support: we found no differences for any of the six processes between articles in which co-piloting had occurred and articles in which co-piloting had not occurred in the probability that a given p-value was inconsistent (all ps ≥.283>0.05/6, see Fig. 5), nor in the probability that a p-value was inconsistent to the extent that it may have affected a decision about statistical significance (all ps ≥.323>0.05/6, see Fig. 6).

Bottom Line: To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors.We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard.Co-piloting was not found to be associated with reporting errors.

View Article: PubMed Central - PubMed

Affiliation: Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, The Netherlands.

ABSTRACT
Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

No MeSH data available.