Limits...
Bias analysis applied to Agricultural Health Study publications to estimate non-random sources of uncertainty.

Lash TL - J Occup Med Toxicol (2007)

Bottom Line: For each study, I identified the prominent result and important sources of systematic error that might affect it.By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval.The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations.

View Article: PubMed Central - HTML - PubMed

Affiliation: Department of Epidemiology, Boston University School of Public Health, 715 Albany St,, TE3, Boston, MA, USA. tlash@bu.edu.

ABSTRACT

Background: The associations of pesticide exposure with disease outcomes are estimated without the benefit of a randomized design. For this reason and others, these studies are susceptible to systematic errors. I analyzed studies of the associations between alachlor and glyphosate exposure and cancer incidence, both derived from the Agricultural Health Study cohort, to quantify the bias and uncertainty potentially attributable to systematic error.

Methods: For each study, I identified the prominent result and important sources of systematic error that might affect it. I assigned probability distributions to the bias parameters that allow quantification of the bias, drew a value at random from each assigned distribution, and calculated the estimate of effect adjusted for the biases. By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval. These methods were applied without access to the primary record-level dataset.

Results: The conventional estimates of effect associating alachlor and glyphosate exposure with cancer incidence were likely biased away from the and understated the uncertainty by quantifying only random error. For example, the conventional p-value for a test of trend in the alachlor study equaled 0.02, whereas fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Similarly, the conventional fully-adjusted result associating glyphosate exposure with multiple myleoma equaled 2.6 with 95% confidence interval of 0.7 to 9.4. The frequency distribution generated by the bias analysis yielded a median hazard ratio equal to 1.5 with 95% simulation interval of 0.4 to 8.9, which was 66% wider than the conventional interval.

Conclusion: Bias analysis provides a more complete picture of true uncertainty than conventional frequentist statistical analysis accompanied by a qualitative description of study limitations. The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations. Furthermore, such analyses, once programmed, allow rapid implementation of alternative assignments of probability distributions to the bias parameters, so elevate the plane of discussion regarding study bias from characterizing studies as "valid" or "invalid" to a critical and quantitative discussion of sources of uncertainty.

No MeSH data available.


Related in: MedlinePlus

Distribution of bias factors yielded by 10,000 iterations of the bias analysis for alachlor. The cumulative distribution of the bias factor for the dose-response slope depicts the strength of association, separate from the precision of the slope. All results from the bias analysis yielded dose-response trends less strong than the original result. Approximately 2.5% of the dose-response trends yielded by the bias analysis equaled 0 (no trend) or below (reversed trend). Approximately 70% of the dose-response trends were less than 10% of the size of the original trend.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC2211481&req=5

Figure 2: Distribution of bias factors yielded by 10,000 iterations of the bias analysis for alachlor. The cumulative distribution of the bias factor for the dose-response slope depicts the strength of association, separate from the precision of the slope. All results from the bias analysis yielded dose-response trends less strong than the original result. Approximately 2.5% of the dose-response trends yielded by the bias analysis equaled 0 (no trend) or below (reversed trend). Approximately 70% of the dose-response trends were less than 10% of the size of the original trend.

Mentions: Figure 2 shows the cumulative distribution of the bias factor for the dose-response slope. The slope depicts the strength of association, separate from the precision of the slope. These two concepts are combined in the p-value. The cumulative distribution of bias factors shows that all results from the bias analysis yielded dose-response trends less strong than the original result. Approximately 2.5% of the dose-response trends yielded by the bias analysis equaled 0 (no trend) or below (reversed trend). Approximately 70% of the dose-response trends were less than 10% of the size of the original trend.


Bias analysis applied to Agricultural Health Study publications to estimate non-random sources of uncertainty.

Lash TL - J Occup Med Toxicol (2007)

Distribution of bias factors yielded by 10,000 iterations of the bias analysis for alachlor. The cumulative distribution of the bias factor for the dose-response slope depicts the strength of association, separate from the precision of the slope. All results from the bias analysis yielded dose-response trends less strong than the original result. Approximately 2.5% of the dose-response trends yielded by the bias analysis equaled 0 (no trend) or below (reversed trend). Approximately 70% of the dose-response trends were less than 10% of the size of the original trend.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC2211481&req=5

Figure 2: Distribution of bias factors yielded by 10,000 iterations of the bias analysis for alachlor. The cumulative distribution of the bias factor for the dose-response slope depicts the strength of association, separate from the precision of the slope. All results from the bias analysis yielded dose-response trends less strong than the original result. Approximately 2.5% of the dose-response trends yielded by the bias analysis equaled 0 (no trend) or below (reversed trend). Approximately 70% of the dose-response trends were less than 10% of the size of the original trend.
Mentions: Figure 2 shows the cumulative distribution of the bias factor for the dose-response slope. The slope depicts the strength of association, separate from the precision of the slope. These two concepts are combined in the p-value. The cumulative distribution of bias factors shows that all results from the bias analysis yielded dose-response trends less strong than the original result. Approximately 2.5% of the dose-response trends yielded by the bias analysis equaled 0 (no trend) or below (reversed trend). Approximately 70% of the dose-response trends were less than 10% of the size of the original trend.

Bottom Line: For each study, I identified the prominent result and important sources of systematic error that might affect it.By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval.The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations.

View Article: PubMed Central - HTML - PubMed

Affiliation: Department of Epidemiology, Boston University School of Public Health, 715 Albany St,, TE3, Boston, MA, USA. tlash@bu.edu.

ABSTRACT

Background: The associations of pesticide exposure with disease outcomes are estimated without the benefit of a randomized design. For this reason and others, these studies are susceptible to systematic errors. I analyzed studies of the associations between alachlor and glyphosate exposure and cancer incidence, both derived from the Agricultural Health Study cohort, to quantify the bias and uncertainty potentially attributable to systematic error.

Methods: For each study, I identified the prominent result and important sources of systematic error that might affect it. I assigned probability distributions to the bias parameters that allow quantification of the bias, drew a value at random from each assigned distribution, and calculated the estimate of effect adjusted for the biases. By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval. These methods were applied without access to the primary record-level dataset.

Results: The conventional estimates of effect associating alachlor and glyphosate exposure with cancer incidence were likely biased away from the and understated the uncertainty by quantifying only random error. For example, the conventional p-value for a test of trend in the alachlor study equaled 0.02, whereas fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Similarly, the conventional fully-adjusted result associating glyphosate exposure with multiple myleoma equaled 2.6 with 95% confidence interval of 0.7 to 9.4. The frequency distribution generated by the bias analysis yielded a median hazard ratio equal to 1.5 with 95% simulation interval of 0.4 to 8.9, which was 66% wider than the conventional interval.

Conclusion: Bias analysis provides a more complete picture of true uncertainty than conventional frequentist statistical analysis accompanied by a qualitative description of study limitations. The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations. Furthermore, such analyses, once programmed, allow rapid implementation of alternative assignments of probability distributions to the bias parameters, so elevate the plane of discussion regarding study bias from characterizing studies as "valid" or "invalid" to a critical and quantitative discussion of sources of uncertainty.

No MeSH data available.


Related in: MedlinePlus