A counterview of 'An investigation of the false discovery rate and the misinterpretation of p-values' by Colquhoun (2014).
View Article:
PubMed Central - PubMed
Affiliation: Department of Physiology , The University of Auckland , Auckland , New Zealand ; Auckland Bioengineering Institute, The University of Auckland , Auckland , New Zealand.
We readily admit that we have no more idea than does the author regarding the true value of the ‘prevalence’ parameter for experimental science, so we have adopted three distinct approaches to estimate it. (i) First, and with reference to the author's charmingly apposite introductory quote from George Elliot's Middlemarch, we state our ‘gut instinct’ estimate to be ‘greater than 50%’. (ii) Second, and widening the scope, we have canvassed the senior investigators in our Department of Physiology for their personal estimates of the fraction of times that their explicit, experimentally testable hypotheses have proven to be supported by experimental results... The ‘guesstimates’ of predictive success by our senior investigator co-workers (N=11) ranged from 50% to 90% with the mean ± standard deviation of 69.6% ± 13.1%... With respect to the ‘literature survey’, in only three cases were the authors obliged to state that their results did not support their explicitly stated hypothesis—i.e. the hypothesis could not be rejected or, in plain English, the authors' scientific hypothesis was declared to be wrong... The complement (22 manuscripts in each of which the hypothesis was rejected) represents a ‘prevalence’ of 0.88 (a value that exceeds even our (probably inflated) ‘gut feelings’)... How are these apparently convincing results to be explained vis-à-vis Prof... Colquhoun's counter-conclusion? It seems unlikely that ‘circular reasoning’ (reflecting the unavoidable fact that, in every case, classical hypothesis testing provided the decision-basis) could have played a large role, especially given that 14 of the results would have satisfied Berger's maximum-likelihood criterion (see appendix A5 of Colquhoun )... Our investigation of this issue (performing simulations using Prof... Colquhoun's R-based software program) leads us to conclude that the resulting false discovery rate is likewise dependent on input parameters (especially the critical effect size)... Because we wish to maintain focus strictly on the input parameter: ‘prevalence’, we present the results of that investigation in appendix 2 in the electronic supplementary material... In conclusion, we find it difficult to imagine how science could have achieved its manifold successes if scientists have been wrong 90% of the time... Hence, we suspect that a number of behaviours facilitate a high probability of a real effect, thereby rendering scientific hypotheses robust against extreme probabilities of failure. No MeSH data available. |
Related In:
Results -
Collection
License getmorefigures.php?uid=PMC4555854&req=5
Mentions: Instead of such speculation, we find it instructive to present an analysis (appendix 1) and graph (figure 1), based on Prof. Colquhoun's ‘tree diagrams’ (figures 2 and 3 in the original).Figure 1. |
View Article: PubMed Central - PubMed
Affiliation: Department of Physiology , The University of Auckland , Auckland , New Zealand ; Auckland Bioengineering Institute, The University of Auckland , Auckland , New Zealand.
We readily admit that we have no more idea than does the author regarding the true value of the ‘prevalence’ parameter for experimental science, so we have adopted three distinct approaches to estimate it. (i) First, and with reference to the author's charmingly apposite introductory quote from George Elliot's Middlemarch, we state our ‘gut instinct’ estimate to be ‘greater than 50%’. (ii) Second, and widening the scope, we have canvassed the senior investigators in our Department of Physiology for their personal estimates of the fraction of times that their explicit, experimentally testable hypotheses have proven to be supported by experimental results... The ‘guesstimates’ of predictive success by our senior investigator co-workers (N=11) ranged from 50% to 90% with the mean ± standard deviation of 69.6% ± 13.1%... With respect to the ‘literature survey’, in only three cases were the authors obliged to state that their results did not support their explicitly stated hypothesis—i.e. the hypothesis could not be rejected or, in plain English, the authors' scientific hypothesis was declared to be wrong... The complement (22 manuscripts in each of which the hypothesis was rejected) represents a ‘prevalence’ of 0.88 (a value that exceeds even our (probably inflated) ‘gut feelings’)... How are these apparently convincing results to be explained vis-à-vis Prof... Colquhoun's counter-conclusion? It seems unlikely that ‘circular reasoning’ (reflecting the unavoidable fact that, in every case, classical hypothesis testing provided the decision-basis) could have played a large role, especially given that 14 of the results would have satisfied Berger's maximum-likelihood criterion (see appendix A5 of Colquhoun )... Our investigation of this issue (performing simulations using Prof... Colquhoun's R-based software program) leads us to conclude that the resulting false discovery rate is likewise dependent on input parameters (especially the critical effect size)... Because we wish to maintain focus strictly on the input parameter: ‘prevalence’, we present the results of that investigation in appendix 2 in the electronic supplementary material... In conclusion, we find it difficult to imagine how science could have achieved its manifold successes if scientists have been wrong 90% of the time... Hence, we suspect that a number of behaviours facilitate a high probability of a real effect, thereby rendering scientific hypotheses robust against extreme probabilities of failure.
No MeSH data available.