Limits...
Bias analysis applied to Agricultural Health Study publications to estimate non-random sources of uncertainty.

Lash TL - J Occup Med Toxicol (2007)

Bottom Line: For each study, I identified the prominent result and important sources of systematic error that might affect it.By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval.The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations.

View Article: PubMed Central - HTML - PubMed

Affiliation: Department of Epidemiology, Boston University School of Public Health, 715 Albany St,, TE3, Boston, MA, USA. tlash@bu.edu.

ABSTRACT

Background: The associations of pesticide exposure with disease outcomes are estimated without the benefit of a randomized design. For this reason and others, these studies are susceptible to systematic errors. I analyzed studies of the associations between alachlor and glyphosate exposure and cancer incidence, both derived from the Agricultural Health Study cohort, to quantify the bias and uncertainty potentially attributable to systematic error.

Methods: For each study, I identified the prominent result and important sources of systematic error that might affect it. I assigned probability distributions to the bias parameters that allow quantification of the bias, drew a value at random from each assigned distribution, and calculated the estimate of effect adjusted for the biases. By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval. These methods were applied without access to the primary record-level dataset.

Results: The conventional estimates of effect associating alachlor and glyphosate exposure with cancer incidence were likely biased away from the and understated the uncertainty by quantifying only random error. For example, the conventional p-value for a test of trend in the alachlor study equaled 0.02, whereas fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Similarly, the conventional fully-adjusted result associating glyphosate exposure with multiple myleoma equaled 2.6 with 95% confidence interval of 0.7 to 9.4. The frequency distribution generated by the bias analysis yielded a median hazard ratio equal to 1.5 with 95% simulation interval of 0.4 to 8.9, which was 66% wider than the conventional interval.

Conclusion: Bias analysis provides a more complete picture of true uncertainty than conventional frequentist statistical analysis accompanied by a qualitative description of study limitations. The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations. Furthermore, such analyses, once programmed, allow rapid implementation of alternative assignments of probability distributions to the bias parameters, so elevate the plane of discussion regarding study bias from characterizing studies as "valid" or "invalid" to a critical and quantitative discussion of sources of uncertainty.

No MeSH data available.


Related in: MedlinePlus

Distribution of p-values yielded by 10,000 iterations of the bias analysis for alachlor. The cumulative distribution of p-values from the 10,000 tests of trend generated by the bias analysis. This result pertains to the trend associating lifetime alachlor exposure-days with the rate of all lymphohematopoietic cancers. The p-value for this trend equaled 0.02 in the original paper, but fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Approximately 50% of the iterations yielded a p-value below 0.05 and the remainder yielded p-values greater than 0.05.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC2211481&req=5

Figure 1: Distribution of p-values yielded by 10,000 iterations of the bias analysis for alachlor. The cumulative distribution of p-values from the 10,000 tests of trend generated by the bias analysis. This result pertains to the trend associating lifetime alachlor exposure-days with the rate of all lymphohematopoietic cancers. The p-value for this trend equaled 0.02 in the original paper, but fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Approximately 50% of the iterations yielded a p-value below 0.05 and the remainder yielded p-values greater than 0.05.

Mentions: Figure 1 shows the cumulative distribution of p-values from the 10,000 tests of trend generated by the bias analysis. This result pertains to the trend associating lifetime alachlor exposure-days with the rate of all lymphohematopoietic cancers. The p-value for this trend equaled 0.02 in the original paper [2], but fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Approximately 50% of the iterations yielded a p-value below 0.05 and the remainder yielded p-values greater than 0.05.


Bias analysis applied to Agricultural Health Study publications to estimate non-random sources of uncertainty.

Lash TL - J Occup Med Toxicol (2007)

Distribution of p-values yielded by 10,000 iterations of the bias analysis for alachlor. The cumulative distribution of p-values from the 10,000 tests of trend generated by the bias analysis. This result pertains to the trend associating lifetime alachlor exposure-days with the rate of all lymphohematopoietic cancers. The p-value for this trend equaled 0.02 in the original paper, but fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Approximately 50% of the iterations yielded a p-value below 0.05 and the remainder yielded p-values greater than 0.05.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC2211481&req=5

Figure 1: Distribution of p-values yielded by 10,000 iterations of the bias analysis for alachlor. The cumulative distribution of p-values from the 10,000 tests of trend generated by the bias analysis. This result pertains to the trend associating lifetime alachlor exposure-days with the rate of all lymphohematopoietic cancers. The p-value for this trend equaled 0.02 in the original paper, but fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Approximately 50% of the iterations yielded a p-value below 0.05 and the remainder yielded p-values greater than 0.05.
Mentions: Figure 1 shows the cumulative distribution of p-values from the 10,000 tests of trend generated by the bias analysis. This result pertains to the trend associating lifetime alachlor exposure-days with the rate of all lymphohematopoietic cancers. The p-value for this trend equaled 0.02 in the original paper [2], but fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Approximately 50% of the iterations yielded a p-value below 0.05 and the remainder yielded p-values greater than 0.05.

Bottom Line: For each study, I identified the prominent result and important sources of systematic error that might affect it.By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval.The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations.

View Article: PubMed Central - HTML - PubMed

Affiliation: Department of Epidemiology, Boston University School of Public Health, 715 Albany St,, TE3, Boston, MA, USA. tlash@bu.edu.

ABSTRACT

Background: The associations of pesticide exposure with disease outcomes are estimated without the benefit of a randomized design. For this reason and others, these studies are susceptible to systematic errors. I analyzed studies of the associations between alachlor and glyphosate exposure and cancer incidence, both derived from the Agricultural Health Study cohort, to quantify the bias and uncertainty potentially attributable to systematic error.

Methods: For each study, I identified the prominent result and important sources of systematic error that might affect it. I assigned probability distributions to the bias parameters that allow quantification of the bias, drew a value at random from each assigned distribution, and calculated the estimate of effect adjusted for the biases. By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval. These methods were applied without access to the primary record-level dataset.

Results: The conventional estimates of effect associating alachlor and glyphosate exposure with cancer incidence were likely biased away from the and understated the uncertainty by quantifying only random error. For example, the conventional p-value for a test of trend in the alachlor study equaled 0.02, whereas fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Similarly, the conventional fully-adjusted result associating glyphosate exposure with multiple myleoma equaled 2.6 with 95% confidence interval of 0.7 to 9.4. The frequency distribution generated by the bias analysis yielded a median hazard ratio equal to 1.5 with 95% simulation interval of 0.4 to 8.9, which was 66% wider than the conventional interval.

Conclusion: Bias analysis provides a more complete picture of true uncertainty than conventional frequentist statistical analysis accompanied by a qualitative description of study limitations. The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations. Furthermore, such analyses, once programmed, allow rapid implementation of alternative assignments of probability distributions to the bias parameters, so elevate the plane of discussion regarding study bias from characterizing studies as "valid" or "invalid" to a critical and quantitative discussion of sources of uncertainty.

No MeSH data available.


Related in: MedlinePlus