Limits...
Assessing risk prediction models using individual participant data from multiple studies.

Pennells L, Kaptoge S, White IR, Thompson SG, Wood AM, Emerging Risk Factors Collaborati - Am. J. Epidemiol. (2013)

Bottom Line: Various weighting approaches are compared and lead us to recommend using the number of events in each study.The concordance index and discrimination measure gave qualitatively similar results throughout.While the concordance index was very heterogeneous between studies, principally because of differing age ranges, the increments in the concordance index from adding log C-reactive protein to conventional risk factors were more homogeneous.

View Article: PubMed Central - PubMed

ABSTRACT
Individual participant time-to-event data from multiple prospective epidemiologic studies enable detailed investigation into the predictive ability of risk models. Here we address the challenges in appropriately combining such information across studies. Methods are exemplified by analyses of log C-reactive protein and conventional risk factors for coronary heart disease in the Emerging Risk Factors Collaboration, a collation of individual data from multiple prospective studies with an average follow-up duration of 9.8 years (dates varied). We derive risk prediction models using Cox proportional hazards regression analysis stratified by study and obtain estimates of risk discrimination, Harrell's concordance index, and Royston's discrimination measure within each study; we then combine the estimates across studies using a weighted meta-analysis. Various weighting approaches are compared and lead us to recommend using the number of events in each study. We also discuss the calculation of measures of reclassification for multiple studies. We further show that comparison of differences in predictive ability across subgroups should be based only on within-study information and that combining measures of risk discrimination from case-control studies and prospective studies is problematic. The concordance index and discrimination measure gave qualitatively similar results throughout. While the concordance index was very heterogeneous between studies, principally because of differing age ranges, the increments in the concordance index from adding log C-reactive protein to conventional risk factors were more homogeneous.

Show MeSH

Related in: MedlinePlus

Overall schemes for model derivation and testing of predictive ability over multiple studies. In the model derivation process, study-specific data sets are used to estimate the pooled vector of coefficients  for the included risk predictors, either by means of a 1-stage stratified model or by a 2-stage approach applying meta-analysis of study-specific estimates. In assessment of predictive ability, the pooled  is used to calculate the pooled discrimination statistic, either using a 1-stage stratified approach or by meta-analyzing study-specific estimates in a 2-stage approach. wS represents study-specific weights applied in meta-analysis approaches; possible choices are described in the text.
© Copyright Policy - creative-commons
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC3927974&req=5

KWT298F1: Overall schemes for model derivation and testing of predictive ability over multiple studies. In the model derivation process, study-specific data sets are used to estimate the pooled vector of coefficients for the included risk predictors, either by means of a 1-stage stratified model or by a 2-stage approach applying meta-analysis of study-specific estimates. In assessment of predictive ability, the pooled is used to calculate the pooled discrimination statistic, either using a 1-stage stratified approach or by meta-analyzing study-specific estimates in a 2-stage approach. wS represents study-specific weights applied in meta-analysis approaches; possible choices are described in the text.

Mentions: Fitting the stratified model (equation 1) is a 1-stage approach to model derivation across studies (Figure 1). Alternatively, a 2-stage approach could be undertaken: First, a separate Cox proportional hazards model is fitted in each study, and then its coefficients are combined over studies to obtain using either fixed- or random-effects (multivariate) meta-analysis (3, 19, 20). A 2-stage random-effects meta-analysis has the advantage of allowing for heterogeneity in the true coefficients between studies, giving a larger variance for . A multivariate meta-analysis combines estimates for the vector of correlated coefficients, taking account of its covariance matrix, over the multiple studies; separate univariate meta-analyses ignore the correlations between the coefficients. With the 2-stage approach, additional estimation is required to obtain study-specific baseline survivor functions necessary for making absolute risk predictions. The 1- and 2-stage approaches often give similar estimates (21), and hence risk scores, and the 1-stage model (equation 1) has the advantage of simplicity.Figure 1.


Assessing risk prediction models using individual participant data from multiple studies.

Pennells L, Kaptoge S, White IR, Thompson SG, Wood AM, Emerging Risk Factors Collaborati - Am. J. Epidemiol. (2013)

Overall schemes for model derivation and testing of predictive ability over multiple studies. In the model derivation process, study-specific data sets are used to estimate the pooled vector of coefficients  for the included risk predictors, either by means of a 1-stage stratified model or by a 2-stage approach applying meta-analysis of study-specific estimates. In assessment of predictive ability, the pooled  is used to calculate the pooled discrimination statistic, either using a 1-stage stratified approach or by meta-analyzing study-specific estimates in a 2-stage approach. wS represents study-specific weights applied in meta-analysis approaches; possible choices are described in the text.
© Copyright Policy - creative-commons
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC3927974&req=5

KWT298F1: Overall schemes for model derivation and testing of predictive ability over multiple studies. In the model derivation process, study-specific data sets are used to estimate the pooled vector of coefficients for the included risk predictors, either by means of a 1-stage stratified model or by a 2-stage approach applying meta-analysis of study-specific estimates. In assessment of predictive ability, the pooled is used to calculate the pooled discrimination statistic, either using a 1-stage stratified approach or by meta-analyzing study-specific estimates in a 2-stage approach. wS represents study-specific weights applied in meta-analysis approaches; possible choices are described in the text.
Mentions: Fitting the stratified model (equation 1) is a 1-stage approach to model derivation across studies (Figure 1). Alternatively, a 2-stage approach could be undertaken: First, a separate Cox proportional hazards model is fitted in each study, and then its coefficients are combined over studies to obtain using either fixed- or random-effects (multivariate) meta-analysis (3, 19, 20). A 2-stage random-effects meta-analysis has the advantage of allowing for heterogeneity in the true coefficients between studies, giving a larger variance for . A multivariate meta-analysis combines estimates for the vector of correlated coefficients, taking account of its covariance matrix, over the multiple studies; separate univariate meta-analyses ignore the correlations between the coefficients. With the 2-stage approach, additional estimation is required to obtain study-specific baseline survivor functions necessary for making absolute risk predictions. The 1- and 2-stage approaches often give similar estimates (21), and hence risk scores, and the 1-stage model (equation 1) has the advantage of simplicity.Figure 1.

Bottom Line: Various weighting approaches are compared and lead us to recommend using the number of events in each study.The concordance index and discrimination measure gave qualitatively similar results throughout.While the concordance index was very heterogeneous between studies, principally because of differing age ranges, the increments in the concordance index from adding log C-reactive protein to conventional risk factors were more homogeneous.

View Article: PubMed Central - PubMed

ABSTRACT
Individual participant time-to-event data from multiple prospective epidemiologic studies enable detailed investigation into the predictive ability of risk models. Here we address the challenges in appropriately combining such information across studies. Methods are exemplified by analyses of log C-reactive protein and conventional risk factors for coronary heart disease in the Emerging Risk Factors Collaboration, a collation of individual data from multiple prospective studies with an average follow-up duration of 9.8 years (dates varied). We derive risk prediction models using Cox proportional hazards regression analysis stratified by study and obtain estimates of risk discrimination, Harrell's concordance index, and Royston's discrimination measure within each study; we then combine the estimates across studies using a weighted meta-analysis. Various weighting approaches are compared and lead us to recommend using the number of events in each study. We also discuss the calculation of measures of reclassification for multiple studies. We further show that comparison of differences in predictive ability across subgroups should be based only on within-study information and that combining measures of risk discrimination from case-control studies and prospective studies is problematic. The concordance index and discrimination measure gave qualitatively similar results throughout. While the concordance index was very heterogeneous between studies, principally because of differing age ranges, the increments in the concordance index from adding log C-reactive protein to conventional risk factors were more homogeneous.

Show MeSH
Related in: MedlinePlus