Limits...
A strategy to build and validate a prognostic biomarker model based on RT-qPCR gene expression and clinical covariates.

Tournoud M, Larue A, Cazalis MA, Venet F, Pachot A, Monneret G, Lepape A, Veyrieras JB - BMC Bioinformatics (2015)

Bottom Line: Second, strategies for model building (covariate selection, functional relationship modeling, and statistical model) as well as performance indicators estimation are presented.On the training dataset, appropriate resampling methods are expected to prevent from any upward biases due to unaccounted technical and biological variability that may arise from the experimental and intrinsic design of the RT-qPCR assay.Moreover, the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performances are pivotal indicators to select the optimal model to be validated on the test dataset.

View Article: PubMed Central - PubMed

Affiliation: Bioinformatics Research Department, bioMérieux, Marcy L'Etoile, France. maud.tournoud@biomerieux.com.

ABSTRACT

Background: Construction and validation of a prognostic model for survival data in the clinical domain is still an active field of research. Nevertheless there is no consensus on how to develop routine prognostic tests based on a combination of RT-qPCR biomarkers and clinical or demographic variables. In particular, the estimation of the model performance requires to properly account for the RT-qPCR experimental design.

Results: We present a strategy to build, select, and validate a prognostic model for survival data based on a combination of RT-qPCR biomarkers and clinical or demographic data and we provide an illustration on a real clinical dataset. First, we compare two cross-validation schemes: a classical outcome-stratified cross-validation scheme and an alternative one that accounts for the RT-qPCR plate design, especially when samples are processed by batches. The latter is intended to limit the performance discrepancies, also called the validation surprise, between the training and the test sets. Second, strategies for model building (covariate selection, functional relationship modeling, and statistical model) as well as performance indicators estimation are presented. Since in practice several prognostic models can exhibit similar performances, complementary criteria for model selection are discussed: the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performance.

Conclusion: On the training dataset, appropriate resampling methods are expected to prevent from any upward biases due to unaccounted technical and biological variability that may arise from the experimental and intrinsic design of the RT-qPCR assay. Moreover, the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performances are pivotal indicators to select the optimal model to be validated on the test dataset.

Show MeSH
Patient level (A) vs PCR bacth level (B) resampling strategies. The training dataset includes 5 batches (on the left of the figure). The figure presents an example of patients resampling in a given fold, and a given iteration. In each batch, gene expression of survivor (open circles) and non-survivor (plain circles) patients are measured. In strategy A, samples are randomly drawn within batches to be included in the training fold-data. In strategy B, entire batches are selected and included in the training-fold data. The model building step is performed on the training-fold data and model performance are estimated on the test-fold data.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC4384357&req=5

Fig1: Patient level (A) vs PCR bacth level (B) resampling strategies. The training dataset includes 5 batches (on the left of the figure). The figure presents an example of patients resampling in a given fold, and a given iteration. In each batch, gene expression of survivor (open circles) and non-survivor (plain circles) patients are measured. In strategy A, samples are randomly drawn within batches to be included in the training fold-data. In strategy B, entire batches are selected and included in the training-fold data. The model building step is performed on the training-fold data and model performance are estimated on the test-fold data.

Mentions: As illustrated in Figure 1, two resampling strategies were evaluated: A/20 −times 5 −fold stratified cross-validation with test-fold samples drawn randomly in the training dataset, but ensuring an equal number of events in each test-fold dataset; B/20 −times 5 −fold cross-validation with samples from entire PCR batches drawn randomly to be included in the test fold data. Contrary to strategy B, strategy A does not take into account the within training sample technical variability due to the batch processing. However, with strategy B, given the unbalanced distribution of death events across batches, the outcome distributions are different between and across corresponding training and test −folds.Figure 1


A strategy to build and validate a prognostic biomarker model based on RT-qPCR gene expression and clinical covariates.

Tournoud M, Larue A, Cazalis MA, Venet F, Pachot A, Monneret G, Lepape A, Veyrieras JB - BMC Bioinformatics (2015)

Patient level (A) vs PCR bacth level (B) resampling strategies. The training dataset includes 5 batches (on the left of the figure). The figure presents an example of patients resampling in a given fold, and a given iteration. In each batch, gene expression of survivor (open circles) and non-survivor (plain circles) patients are measured. In strategy A, samples are randomly drawn within batches to be included in the training fold-data. In strategy B, entire batches are selected and included in the training-fold data. The model building step is performed on the training-fold data and model performance are estimated on the test-fold data.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC4384357&req=5

Fig1: Patient level (A) vs PCR bacth level (B) resampling strategies. The training dataset includes 5 batches (on the left of the figure). The figure presents an example of patients resampling in a given fold, and a given iteration. In each batch, gene expression of survivor (open circles) and non-survivor (plain circles) patients are measured. In strategy A, samples are randomly drawn within batches to be included in the training fold-data. In strategy B, entire batches are selected and included in the training-fold data. The model building step is performed on the training-fold data and model performance are estimated on the test-fold data.
Mentions: As illustrated in Figure 1, two resampling strategies were evaluated: A/20 −times 5 −fold stratified cross-validation with test-fold samples drawn randomly in the training dataset, but ensuring an equal number of events in each test-fold dataset; B/20 −times 5 −fold cross-validation with samples from entire PCR batches drawn randomly to be included in the test fold data. Contrary to strategy B, strategy A does not take into account the within training sample technical variability due to the batch processing. However, with strategy B, given the unbalanced distribution of death events across batches, the outcome distributions are different between and across corresponding training and test −folds.Figure 1

Bottom Line: Second, strategies for model building (covariate selection, functional relationship modeling, and statistical model) as well as performance indicators estimation are presented.On the training dataset, appropriate resampling methods are expected to prevent from any upward biases due to unaccounted technical and biological variability that may arise from the experimental and intrinsic design of the RT-qPCR assay.Moreover, the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performances are pivotal indicators to select the optimal model to be validated on the test dataset.

View Article: PubMed Central - PubMed

Affiliation: Bioinformatics Research Department, bioMérieux, Marcy L'Etoile, France. maud.tournoud@biomerieux.com.

ABSTRACT

Background: Construction and validation of a prognostic model for survival data in the clinical domain is still an active field of research. Nevertheless there is no consensus on how to develop routine prognostic tests based on a combination of RT-qPCR biomarkers and clinical or demographic variables. In particular, the estimation of the model performance requires to properly account for the RT-qPCR experimental design.

Results: We present a strategy to build, select, and validate a prognostic model for survival data based on a combination of RT-qPCR biomarkers and clinical or demographic data and we provide an illustration on a real clinical dataset. First, we compare two cross-validation schemes: a classical outcome-stratified cross-validation scheme and an alternative one that accounts for the RT-qPCR plate design, especially when samples are processed by batches. The latter is intended to limit the performance discrepancies, also called the validation surprise, between the training and the test sets. Second, strategies for model building (covariate selection, functional relationship modeling, and statistical model) as well as performance indicators estimation are presented. Since in practice several prognostic models can exhibit similar performances, complementary criteria for model selection are discussed: the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performance.

Conclusion: On the training dataset, appropriate resampling methods are expected to prevent from any upward biases due to unaccounted technical and biological variability that may arise from the experimental and intrinsic design of the RT-qPCR assay. Moreover, the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performances are pivotal indicators to select the optimal model to be validated on the test dataset.

Show MeSH