Limits...
Tailoring the implementation of new biomarkers based on their added predictive value in subgroups of individuals.

van Giessen A, Moons KG, de Wit GA, Verschuren WM, Boer JM, Koffijberg H - PLoS ONE (2015)

Bottom Line: Using the characterizations of the typically correctly reclassified individuals, implementing SCORE only in individuals expected to benefit (n = 2,707,12.3%) improved the NRI to 5.32% (95% CI [-0.13%; 12.06%]) within the events, 0.24% (95% CI [0.10%; 0.36%]) within the nonevents, and a total NRI of 0.055 (95% CI [0.001; 0.123]).In our empirical example the presented approach successfully characterized subgroups of reclassified individuals that could be used to improve reclassification and reduce implementation burden.In particular when newly added biomarkers or imaging tests are costly or burdensome such a tailored implementation strategy may save resources and improve (cost-)effectiveness.

View Article: PubMed Central - PubMed

Affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, The Netherlands.

ABSTRACT

Background: The value of new biomarkers or imaging tests, when added to a prediction model, is currently evaluated using reclassification measures, such as the net reclassification improvement (NRI). However, these measures only provide an estimate of improved reclassification at population level. We present a straightforward approach to characterize subgroups of reclassified individuals in order to tailor implementation of a new prediction model to individuals expected to benefit from it.

Methods: In a large Dutch population cohort (n = 21,992) we classified individuals to low (< 5%) and high (≥ 5%) fatal cardiovascular disease risk by the Framingham risk score (FRS) and reclassified them based on the systematic coronary risk evaluation (SCORE). Subsequently, we characterized the reclassified individuals and, in case of heterogeneity, applied cluster analysis to identify and characterize subgroups. These characterizations were used to select individuals expected to benefit from implementation of SCORE.

Results: Reclassification after applying SCORE in all individuals resulted in an NRI of 5.00% (95% CI [-0.53%; 11.50%]) within the events, 0.06% (95% CI [-0.08%; 0.22%]) within the nonevents, and a total NRI of 0.051 (95% CI [-0.004; 0.116]). Among the correctly downward reclassified individuals cluster analysis identified three subgroups. Using the characterizations of the typically correctly reclassified individuals, implementing SCORE only in individuals expected to benefit (n = 2,707,12.3%) improved the NRI to 5.32% (95% CI [-0.13%; 12.06%]) within the events, 0.24% (95% CI [0.10%; 0.36%]) within the nonevents, and a total NRI of 0.055 (95% CI [0.001; 0.123]). Overall, the risk levels for individuals reclassified by tailored implementation of SCORE were more accurate.

Discussion: In our empirical example the presented approach successfully characterized subgroups of reclassified individuals that could be used to improve reclassification and reduce implementation burden. In particular when newly added biomarkers or imaging tests are costly or burdensome such a tailored implementation strategy may save resources and improve (cost-)effectiveness.

Show MeSH

Related in: MedlinePlus

Evaluation process of a new prediction model.Abbreviations: AUC = Area Under the (ROC-) Curve, NRI = Net Reclassification Improvement, IDI = Integrative Discrimination Improvement.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4306488&req=5

pone.0114020.g001: Evaluation process of a new prediction model.Abbreviations: AUC = Area Under the (ROC-) Curve, NRI = Net Reclassification Improvement, IDI = Integrative Discrimination Improvement.

Mentions: Prior to potential implementation, a new or extended prediction model ought to be evaluated in several stages (Fig. 1) [4–7]. First, its performance is commonly assessed by measures of discrimination and calibration [8]. Subsequently, it is essential to evaluate the incremental value of the new model, as compared to the existing model [9]. Several incremental performance measures are available, such as the difference in the area under the receiver operating characteristic curve, net reclassification improvement (NRI) and integrated discrimination improvement [10]. All these measures give indication of the average improved performance of a new or extended prediction model. However, favourable performance of one prediction model over the other may be the result of improved predictions in one (larger) group of individuals and similar or worse predictions in another group. On top of some individuals receiving worse predictions, performing additional tests in every individual may be undesirable, because of costs and invasiveness of such tests. Hence, there is a clear need to select individuals who actually benefit from a new prediction model, possibly including additional biomarkers or tests.


Tailoring the implementation of new biomarkers based on their added predictive value in subgroups of individuals.

van Giessen A, Moons KG, de Wit GA, Verschuren WM, Boer JM, Koffijberg H - PLoS ONE (2015)

Evaluation process of a new prediction model.Abbreviations: AUC = Area Under the (ROC-) Curve, NRI = Net Reclassification Improvement, IDI = Integrative Discrimination Improvement.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4306488&req=5

pone.0114020.g001: Evaluation process of a new prediction model.Abbreviations: AUC = Area Under the (ROC-) Curve, NRI = Net Reclassification Improvement, IDI = Integrative Discrimination Improvement.
Mentions: Prior to potential implementation, a new or extended prediction model ought to be evaluated in several stages (Fig. 1) [4–7]. First, its performance is commonly assessed by measures of discrimination and calibration [8]. Subsequently, it is essential to evaluate the incremental value of the new model, as compared to the existing model [9]. Several incremental performance measures are available, such as the difference in the area under the receiver operating characteristic curve, net reclassification improvement (NRI) and integrated discrimination improvement [10]. All these measures give indication of the average improved performance of a new or extended prediction model. However, favourable performance of one prediction model over the other may be the result of improved predictions in one (larger) group of individuals and similar or worse predictions in another group. On top of some individuals receiving worse predictions, performing additional tests in every individual may be undesirable, because of costs and invasiveness of such tests. Hence, there is a clear need to select individuals who actually benefit from a new prediction model, possibly including additional biomarkers or tests.

Bottom Line: Using the characterizations of the typically correctly reclassified individuals, implementing SCORE only in individuals expected to benefit (n = 2,707,12.3%) improved the NRI to 5.32% (95% CI [-0.13%; 12.06%]) within the events, 0.24% (95% CI [0.10%; 0.36%]) within the nonevents, and a total NRI of 0.055 (95% CI [0.001; 0.123]).In our empirical example the presented approach successfully characterized subgroups of reclassified individuals that could be used to improve reclassification and reduce implementation burden.In particular when newly added biomarkers or imaging tests are costly or burdensome such a tailored implementation strategy may save resources and improve (cost-)effectiveness.

View Article: PubMed Central - PubMed

Affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, The Netherlands.

ABSTRACT

Background: The value of new biomarkers or imaging tests, when added to a prediction model, is currently evaluated using reclassification measures, such as the net reclassification improvement (NRI). However, these measures only provide an estimate of improved reclassification at population level. We present a straightforward approach to characterize subgroups of reclassified individuals in order to tailor implementation of a new prediction model to individuals expected to benefit from it.

Methods: In a large Dutch population cohort (n = 21,992) we classified individuals to low (< 5%) and high (≥ 5%) fatal cardiovascular disease risk by the Framingham risk score (FRS) and reclassified them based on the systematic coronary risk evaluation (SCORE). Subsequently, we characterized the reclassified individuals and, in case of heterogeneity, applied cluster analysis to identify and characterize subgroups. These characterizations were used to select individuals expected to benefit from implementation of SCORE.

Results: Reclassification after applying SCORE in all individuals resulted in an NRI of 5.00% (95% CI [-0.53%; 11.50%]) within the events, 0.06% (95% CI [-0.08%; 0.22%]) within the nonevents, and a total NRI of 0.051 (95% CI [-0.004; 0.116]). Among the correctly downward reclassified individuals cluster analysis identified three subgroups. Using the characterizations of the typically correctly reclassified individuals, implementing SCORE only in individuals expected to benefit (n = 2,707,12.3%) improved the NRI to 5.32% (95% CI [-0.13%; 12.06%]) within the events, 0.24% (95% CI [0.10%; 0.36%]) within the nonevents, and a total NRI of 0.055 (95% CI [0.001; 0.123]). Overall, the risk levels for individuals reclassified by tailored implementation of SCORE were more accurate.

Discussion: In our empirical example the presented approach successfully characterized subgroups of reclassified individuals that could be used to improve reclassification and reduce implementation burden. In particular when newly added biomarkers or imaging tests are costly or burdensome such a tailored implementation strategy may save resources and improve (cost-)effectiveness.

Show MeSH
Related in: MedlinePlus