Limits...
Development and application of a novel metric to assess effectiveness of biomedical data.

Bloom GC, Eschrich S, Han G, Hang G, Schabath MB, Bhansali N, Hoerter AM, Morgan S, Fenstermacher DA - BMJ Open (2013)

Bottom Line: An effectiveness or E-score was obtained by calculating the conditional probabilities of the p-value and A-score within the given data set with their product equaling the effectiveness score (E-score).Conversely, elements surgery-site, histologic-type and pathological-TNM stage were down-ranked in comparison to their p values based on lower A-scores caused by significantly higher acquisition costs.Results show that an element's underlying data quality is an important consideration in addition to p value correlation to outcome when determining the element's clinical or research utility in a study.

View Article: PubMed Central - PubMed

Affiliation: Department of Biomedical Informatics, H Lee Moffitt Cancer and Research Institute, Tampa, Florida, USA.

ABSTRACT

Objective: Design a metric to assess the comparative effectiveness of biomedical data elements within a study that incorporates their statistical relatedness to a given outcome variable as well as a measurement of the quality of their underlying data.

Materials and methods: The cohort consisted of 874 patients with adenocarcinoma of the lung, each with 47 clinical data elements. The p value for each element was calculated using the Cox proportional hazard univariable regression model with overall survival as the endpoint. An attribute or A-score was calculated by quantification of an element's four quality attributes; Completeness, Comprehensiveness, Consistency and Overall-cost. An effectiveness or E-score was obtained by calculating the conditional probabilities of the p-value and A-score within the given data set with their product equaling the effectiveness score (E-score).

Results: The E-score metric provided information about the utility of an element beyond an outcome-related p value ranking. E-scores for elements age-at-diagnosis, gender and tobacco-use showed utility above what their respective p values alone would indicate due to their relative ease of acquisition, that is, higher A-scores. Conversely, elements surgery-site, histologic-type and pathological-TNM stage were down-ranked in comparison to their p values based on lower A-scores caused by significantly higher acquisition costs.

Conclusions: A novel metric termed E-score was developed which incorporates standard statistics with data quality metrics and was tested on elements from a large lung cohort. Results show that an element's underlying data quality is an important consideration in addition to p value correlation to outcome when determining the element's clinical or research utility in a study.

No MeSH data available.


Related in: MedlinePlus

Graph of the two procedure-based elements showing change in effectiveness-score with varying attribute-score. Note the Pathological_TNM_Stage and Histologic_Type have nearly overlapping lines due to similar p value ranking for these elements.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC3753524&req=5

BMJOPEN2013003220F4: Graph of the two procedure-based elements showing change in effectiveness-score with varying attribute-score. Note the Pathological_TNM_Stage and Histologic_Type have nearly overlapping lines due to similar p value ranking for these elements.

Mentions: Next, we wanted to determine whether our E-score would be able to reflect changes across a range of A-scores for a relatively stable p value as could happen when using a different data set? To address this important question we first selected five survey-type elements, and four procedure-based elements. We used their actual p values calculated within our data set but varied the A-scores to determine its effect on E-score. FigureĀ 3 shows a large range of variation between elements with changing A-scores and demonstrates the ability of the E-score to scale or reflect changes in the underlying quality of the data, that is, A-score changes. Additionally, and more importantly, these large variations in E-score are only observed for those elements for which the p value with respect to survival was significant as was the case for Age and Diagnosis, Alcohol Use and Tobacco Use. Variation of the A-score had little to no effect on the two elements that had p values that were not statistically significant, that is, Gender and Spanish Hispanic Origin. Similarly, figure 4 shows a large range of variation in E-scores between two of the elements, Pathological-TNM Stage and Histologic Type, with changing A-scores. Both of these elements had p values that were highly significant with respect to overall survival. Note that although the p values of the two elements differ by a factor of two relative p value ranking within the data set is close, thus accounting for the overlapping E-scores seen here. As can be observed in figure 3, for the survey elements the E-score is able to reflect changes in the underlying quality of the data, that is, E-score changes in a consistent and interpretable manner. Again, variation of the A-score of those elements having non-significant p values, that is, Tumor Site and Clinical Tumor Size had little to no effect on the E-score.


Development and application of a novel metric to assess effectiveness of biomedical data.

Bloom GC, Eschrich S, Han G, Hang G, Schabath MB, Bhansali N, Hoerter AM, Morgan S, Fenstermacher DA - BMJ Open (2013)

Graph of the two procedure-based elements showing change in effectiveness-score with varying attribute-score. Note the Pathological_TNM_Stage and Histologic_Type have nearly overlapping lines due to similar p value ranking for these elements.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC3753524&req=5

BMJOPEN2013003220F4: Graph of the two procedure-based elements showing change in effectiveness-score with varying attribute-score. Note the Pathological_TNM_Stage and Histologic_Type have nearly overlapping lines due to similar p value ranking for these elements.
Mentions: Next, we wanted to determine whether our E-score would be able to reflect changes across a range of A-scores for a relatively stable p value as could happen when using a different data set? To address this important question we first selected five survey-type elements, and four procedure-based elements. We used their actual p values calculated within our data set but varied the A-scores to determine its effect on E-score. FigureĀ 3 shows a large range of variation between elements with changing A-scores and demonstrates the ability of the E-score to scale or reflect changes in the underlying quality of the data, that is, A-score changes. Additionally, and more importantly, these large variations in E-score are only observed for those elements for which the p value with respect to survival was significant as was the case for Age and Diagnosis, Alcohol Use and Tobacco Use. Variation of the A-score had little to no effect on the two elements that had p values that were not statistically significant, that is, Gender and Spanish Hispanic Origin. Similarly, figure 4 shows a large range of variation in E-scores between two of the elements, Pathological-TNM Stage and Histologic Type, with changing A-scores. Both of these elements had p values that were highly significant with respect to overall survival. Note that although the p values of the two elements differ by a factor of two relative p value ranking within the data set is close, thus accounting for the overlapping E-scores seen here. As can be observed in figure 3, for the survey elements the E-score is able to reflect changes in the underlying quality of the data, that is, E-score changes in a consistent and interpretable manner. Again, variation of the A-score of those elements having non-significant p values, that is, Tumor Site and Clinical Tumor Size had little to no effect on the E-score.

Bottom Line: An effectiveness or E-score was obtained by calculating the conditional probabilities of the p-value and A-score within the given data set with their product equaling the effectiveness score (E-score).Conversely, elements surgery-site, histologic-type and pathological-TNM stage were down-ranked in comparison to their p values based on lower A-scores caused by significantly higher acquisition costs.Results show that an element's underlying data quality is an important consideration in addition to p value correlation to outcome when determining the element's clinical or research utility in a study.

View Article: PubMed Central - PubMed

Affiliation: Department of Biomedical Informatics, H Lee Moffitt Cancer and Research Institute, Tampa, Florida, USA.

ABSTRACT

Objective: Design a metric to assess the comparative effectiveness of biomedical data elements within a study that incorporates their statistical relatedness to a given outcome variable as well as a measurement of the quality of their underlying data.

Materials and methods: The cohort consisted of 874 patients with adenocarcinoma of the lung, each with 47 clinical data elements. The p value for each element was calculated using the Cox proportional hazard univariable regression model with overall survival as the endpoint. An attribute or A-score was calculated by quantification of an element's four quality attributes; Completeness, Comprehensiveness, Consistency and Overall-cost. An effectiveness or E-score was obtained by calculating the conditional probabilities of the p-value and A-score within the given data set with their product equaling the effectiveness score (E-score).

Results: The E-score metric provided information about the utility of an element beyond an outcome-related p value ranking. E-scores for elements age-at-diagnosis, gender and tobacco-use showed utility above what their respective p values alone would indicate due to their relative ease of acquisition, that is, higher A-scores. Conversely, elements surgery-site, histologic-type and pathological-TNM stage were down-ranked in comparison to their p values based on lower A-scores caused by significantly higher acquisition costs.

Conclusions: A novel metric termed E-score was developed which incorporates standard statistics with data quality metrics and was tested on elements from a large lung cohort. Results show that an element's underlying data quality is an important consideration in addition to p value correlation to outcome when determining the element's clinical or research utility in a study.

No MeSH data available.


Related in: MedlinePlus