Limits...
Bias analysis applied to Agricultural Health Study publications to estimate non-random sources of uncertainty.

Lash TL - J Occup Med Toxicol (2007)

Bottom Line: For each study, I identified the prominent result and important sources of systematic error that might affect it.By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval.The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations.

View Article: PubMed Central - HTML - PubMed

Affiliation: Department of Epidemiology, Boston University School of Public Health, 715 Albany St,, TE3, Boston, MA, USA. tlash@bu.edu.

ABSTRACT

Background: The associations of pesticide exposure with disease outcomes are estimated without the benefit of a randomized design. For this reason and others, these studies are susceptible to systematic errors. I analyzed studies of the associations between alachlor and glyphosate exposure and cancer incidence, both derived from the Agricultural Health Study cohort, to quantify the bias and uncertainty potentially attributable to systematic error.

Methods: For each study, I identified the prominent result and important sources of systematic error that might affect it. I assigned probability distributions to the bias parameters that allow quantification of the bias, drew a value at random from each assigned distribution, and calculated the estimate of effect adjusted for the biases. By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval. These methods were applied without access to the primary record-level dataset.

Results: The conventional estimates of effect associating alachlor and glyphosate exposure with cancer incidence were likely biased away from the and understated the uncertainty by quantifying only random error. For example, the conventional p-value for a test of trend in the alachlor study equaled 0.02, whereas fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Similarly, the conventional fully-adjusted result associating glyphosate exposure with multiple myleoma equaled 2.6 with 95% confidence interval of 0.7 to 9.4. The frequency distribution generated by the bias analysis yielded a median hazard ratio equal to 1.5 with 95% simulation interval of 0.4 to 8.9, which was 66% wider than the conventional interval.

Conclusion: Bias analysis provides a more complete picture of true uncertainty than conventional frequentist statistical analysis accompanied by a qualitative description of study limitations. The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations. Furthermore, such analyses, once programmed, allow rapid implementation of alternative assignments of probability distributions to the bias parameters, so elevate the plane of discussion regarding study bias from characterizing studies as "valid" or "invalid" to a critical and quantitative discussion of sources of uncertainty.

No MeSH data available.


Related in: MedlinePlus

Results of glyphosate bias analysis. The stippled line shows the conventional fully-adjusted result, centered on a hazard ratio of 2.6 with 95% confidence interval of 0.7 to 9.4 associating ever-exposure to glyphosate with multiple myeloma. The solid line (random error bias analysis) shows the result of the bias analysis accounting for confounding and exposure misclassification, and incorporating random error. Its median equals 1.5 and its simulation interval was 0.4 to 8.9. The dashed line shows the bias analysis result without simultaneously incorporating random error. Its median equals 1.5, with 95% simulation interval from 0.65 to 6.3.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC2211481&req=5

Figure 3: Results of glyphosate bias analysis. The stippled line shows the conventional fully-adjusted result, centered on a hazard ratio of 2.6 with 95% confidence interval of 0.7 to 9.4 associating ever-exposure to glyphosate with multiple myeloma. The solid line (random error bias analysis) shows the result of the bias analysis accounting for confounding and exposure misclassification, and incorporating random error. Its median equals 1.5 and its simulation interval was 0.4 to 8.9. The dashed line shows the bias analysis result without simultaneously incorporating random error. Its median equals 1.5, with 95% simulation interval from 0.65 to 6.3.

Mentions: Figure 3 shows the overall results of the glyphosate bias analysis. The stippled line shows the conventional fully-adjusted result, centered on a hazard ratio of 2.6 with 95% confidence interval of 0.7 to 9.4. The solid line (random error bias analysis) shows the result of the bias analysis accounting for confounding and exposure misclassification, and incorporating random error. Its median equals 1.5, which suggests that the conventional result (2.6) was substantially biased, primarily by the adjustment for confounders. The adjustment for confounders influenced the result well beyond the confounding impact, probably by limiting the data set to those with complete data. The simulation interval about this complete bias analysis (0.4 to 8.9) was 66% wider than the conventional interval, showing that the conventional interval substantially understated the true uncertainty. The dashed line shows the bias analysis result without simultaneously incorporating random error, so depicts the additional uncertainty arising from systematic error. Its median equals 1.5, with 95% simulation interval from 0.65 to 6.3. This interval is 70% the width of the random error only conventional interval, further emphasizing that the conventional frequentist interval often substantially understates the true uncertainty of a result.


Bias analysis applied to Agricultural Health Study publications to estimate non-random sources of uncertainty.

Lash TL - J Occup Med Toxicol (2007)

Results of glyphosate bias analysis. The stippled line shows the conventional fully-adjusted result, centered on a hazard ratio of 2.6 with 95% confidence interval of 0.7 to 9.4 associating ever-exposure to glyphosate with multiple myeloma. The solid line (random error bias analysis) shows the result of the bias analysis accounting for confounding and exposure misclassification, and incorporating random error. Its median equals 1.5 and its simulation interval was 0.4 to 8.9. The dashed line shows the bias analysis result without simultaneously incorporating random error. Its median equals 1.5, with 95% simulation interval from 0.65 to 6.3.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC2211481&req=5

Figure 3: Results of glyphosate bias analysis. The stippled line shows the conventional fully-adjusted result, centered on a hazard ratio of 2.6 with 95% confidence interval of 0.7 to 9.4 associating ever-exposure to glyphosate with multiple myeloma. The solid line (random error bias analysis) shows the result of the bias analysis accounting for confounding and exposure misclassification, and incorporating random error. Its median equals 1.5 and its simulation interval was 0.4 to 8.9. The dashed line shows the bias analysis result without simultaneously incorporating random error. Its median equals 1.5, with 95% simulation interval from 0.65 to 6.3.
Mentions: Figure 3 shows the overall results of the glyphosate bias analysis. The stippled line shows the conventional fully-adjusted result, centered on a hazard ratio of 2.6 with 95% confidence interval of 0.7 to 9.4. The solid line (random error bias analysis) shows the result of the bias analysis accounting for confounding and exposure misclassification, and incorporating random error. Its median equals 1.5, which suggests that the conventional result (2.6) was substantially biased, primarily by the adjustment for confounders. The adjustment for confounders influenced the result well beyond the confounding impact, probably by limiting the data set to those with complete data. The simulation interval about this complete bias analysis (0.4 to 8.9) was 66% wider than the conventional interval, showing that the conventional interval substantially understated the true uncertainty. The dashed line shows the bias analysis result without simultaneously incorporating random error, so depicts the additional uncertainty arising from systematic error. Its median equals 1.5, with 95% simulation interval from 0.65 to 6.3. This interval is 70% the width of the random error only conventional interval, further emphasizing that the conventional frequentist interval often substantially understates the true uncertainty of a result.

Bottom Line: For each study, I identified the prominent result and important sources of systematic error that might affect it.By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval.The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations.

View Article: PubMed Central - HTML - PubMed

Affiliation: Department of Epidemiology, Boston University School of Public Health, 715 Albany St,, TE3, Boston, MA, USA. tlash@bu.edu.

ABSTRACT

Background: The associations of pesticide exposure with disease outcomes are estimated without the benefit of a randomized design. For this reason and others, these studies are susceptible to systematic errors. I analyzed studies of the associations between alachlor and glyphosate exposure and cancer incidence, both derived from the Agricultural Health Study cohort, to quantify the bias and uncertainty potentially attributable to systematic error.

Methods: For each study, I identified the prominent result and important sources of systematic error that might affect it. I assigned probability distributions to the bias parameters that allow quantification of the bias, drew a value at random from each assigned distribution, and calculated the estimate of effect adjusted for the biases. By repeating the draw and adjustment process over multiple iterations, I generated a frequency distribution of adjusted results, from which I obtained a point estimate and simulation interval. These methods were applied without access to the primary record-level dataset.

Results: The conventional estimates of effect associating alachlor and glyphosate exposure with cancer incidence were likely biased away from the and understated the uncertainty by quantifying only random error. For example, the conventional p-value for a test of trend in the alachlor study equaled 0.02, whereas fewer than 20% of the bias analysis iterations yielded a p-value of 0.02 or lower. Similarly, the conventional fully-adjusted result associating glyphosate exposure with multiple myleoma equaled 2.6 with 95% confidence interval of 0.7 to 9.4. The frequency distribution generated by the bias analysis yielded a median hazard ratio equal to 1.5 with 95% simulation interval of 0.4 to 8.9, which was 66% wider than the conventional interval.

Conclusion: Bias analysis provides a more complete picture of true uncertainty than conventional frequentist statistical analysis accompanied by a qualitative description of study limitations. The latter approach is likely to lead to overconfidence regarding the potential for causal associations, whereas the former safeguards against such overinterpretations. Furthermore, such analyses, once programmed, allow rapid implementation of alternative assignments of probability distributions to the bias parameters, so elevate the plane of discussion regarding study bias from characterizing studies as "valid" or "invalid" to a critical and quantitative discussion of sources of uncertainty.

No MeSH data available.


Related in: MedlinePlus