Limits...
Predicting outcome on admission and post-admission for acetaminophen-induced acute liver failure using classification and regression tree models.

Speiser JL, Lee WM, Karvellas CJ, US Acute Liver Failure Study Gro - PLoS ONE (2015)

Bottom Line: Accuracy in prediction of outcome (AC), sensitivity (SN), specificity (SP), and area under receiver-operating curve (AUROC) were compared between 3 models: KCC (INR, creatinine, coma grade, pH), CART analysis using only KCC variables (KCC-CART) and a CART model using new variables (NEW-CART).CARTs offer simple prognostic models for APAP-ALF patients, which have higher AUROC and SN than KCC, with similar AC and negligibly worse SP.Admission and post-admission predictions were developed. • Prognostication in acetaminophen-induced acute liver failure (APAP-ALF) is challenging beyond admission • Little has been published regarding the use of King's College Criteria (KCC) beyond admission and KCC has shown limited sensitivity in subsequent studies • Classification and Regression Tree (CART) methodology allows the development of predictive models using binary splits and offers an intuitive method for predicting outcome, using processes familiar to clinicians • Data from the ALFSG registry suggested that CART prognosis models for the APAP population offer improved sensitivity and model performance over traditional regression-based KCC, while maintaining similar accuracy and negligibly worse specificity • KCC-CART models offered modest improvement over traditional KCC, with NEW-CART models performing better than KCC-CART particularly at late time points.

View Article: PubMed Central - PubMed

Affiliation: Department of Public Health Sciences, Medical University of South Carolina, Charleston, South Carolina, United States of America.

ABSTRACT

Background/aim: Assessing prognosis for acetaminophen-induced acute liver failure (APAP-ALF) patients often presents significant challenges. King's College (KCC) has been validated on hospital admission, but little has been published on later phases of illness. We aimed to improve determinations of prognosis both at the time of and following admission for APAP-ALF using Classification and Regression Tree (CART) models.

Methods: CART models were applied to US ALFSG registry data to predict 21-day death or liver transplant early (on admission) and post-admission (days 3-7) for 803 APAP-ALF patients enrolled 01/1998-09/2013. Accuracy in prediction of outcome (AC), sensitivity (SN), specificity (SP), and area under receiver-operating curve (AUROC) were compared between 3 models: KCC (INR, creatinine, coma grade, pH), CART analysis using only KCC variables (KCC-CART) and a CART model using new variables (NEW-CART).

Results: Traditional KCC yielded 69% AC, 90% SP, 27% SN, and 0.58 AUROC on admission, with similar performance post-admission. KCC-CART at admission offered predictive 66% AC, 65% SP, 67% SN, and 0.74 AUROC. Post-admission, KCC-CART had predictive 82% AC, 86% SP, 46% SN and 0.81 AUROC. NEW-CART models using MELD (Model for end stage liver disease), lactate and mechanical ventilation on admission yielded predictive 72% AC, 71% SP, 77% SN and AUROC 0.79. For later stages, NEW-CART (MELD, lactate, coma grade) offered predictive AC 86%, SP 91%, SN 46%, AUROC 0.73.

Conclusion: CARTs offer simple prognostic models for APAP-ALF patients, which have higher AUROC and SN than KCC, with similar AC and negligibly worse SP. Admission and post-admission predictions were developed.

Key points: • Prognostication in acetaminophen-induced acute liver failure (APAP-ALF) is challenging beyond admission • Little has been published regarding the use of King's College Criteria (KCC) beyond admission and KCC has shown limited sensitivity in subsequent studies • Classification and Regression Tree (CART) methodology allows the development of predictive models using binary splits and offers an intuitive method for predicting outcome, using processes familiar to clinicians • Data from the ALFSG registry suggested that CART prognosis models for the APAP population offer improved sensitivity and model performance over traditional regression-based KCC, while maintaining similar accuracy and negligibly worse specificity • KCC-CART models offered modest improvement over traditional KCC, with NEW-CART models performing better than KCC-CART particularly at late time points.

Show MeSH

Related in: MedlinePlus

Plots of Confidence Intervals of Accuracy, AUROC, Sensitivity and Specificity for Admission and Post-Admission Models.Plots display confidence intervals for accuracy (AC), area under the receiver operating curve (AUROC), sensitivity (SN) and specificity (SP) from Table 2. The admission plot (top panel) illustrates non-overlapping, higher confidence intervals for both KCC-CART and NEW-CART compared to KCC for AUROC and sensitivity, and lower confidence intervals for specificity. This indicates that the CART models had significantly better AUROC and sensitivity than KCC, but had significantly worse specificity compared to KCC. Confidence intervals for accuracy for KCC, KCC-CART and NEW-CART all overlap, indicating no significant differences between the models. The post-admission plot (bottom panel) again indicated no significant differences between the models in terms of accuracy; however, the AUROC and sensitivity of KCC-CART was significantly higher than that of NEW-CART and KCC. The specificity of KCC was highest, but did no differ significantly from that of the NEW-CART. KCC demonstrated significantly higher specificity than the KCC-CART, but the difference was not significant compared to the NEW-CART.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4401567&req=5

pone.0122929.g004: Plots of Confidence Intervals of Accuracy, AUROC, Sensitivity and Specificity for Admission and Post-Admission Models.Plots display confidence intervals for accuracy (AC), area under the receiver operating curve (AUROC), sensitivity (SN) and specificity (SP) from Table 2. The admission plot (top panel) illustrates non-overlapping, higher confidence intervals for both KCC-CART and NEW-CART compared to KCC for AUROC and sensitivity, and lower confidence intervals for specificity. This indicates that the CART models had significantly better AUROC and sensitivity than KCC, but had significantly worse specificity compared to KCC. Confidence intervals for accuracy for KCC, KCC-CART and NEW-CART all overlap, indicating no significant differences between the models. The post-admission plot (bottom panel) again indicated no significant differences between the models in terms of accuracy; however, the AUROC and sensitivity of KCC-CART was significantly higher than that of NEW-CART and KCC. The specificity of KCC was highest, but did no differ significantly from that of the NEW-CART. KCC demonstrated significantly higher specificity than the KCC-CART, but the difference was not significant compared to the NEW-CART.

Mentions: Table 3 displays performance measures and their associated confidence intervals for the three post-admission prognosis models (see also Fig 4). The traditional KCC had low sensitivity (15%) and AUROC (0.56), but maintained an overall prediction accuracy of 70% and specificity of 97% using all post-admission data. Accuracy of KCC for the post-admission testing dataset was significantly higher (86%), with similar values for specificity, sensitivity and AUROC. The KCC-CART developed for post-admission had similar prediction accuracy (82%), slightly lower specificity (86%) and higher sensitivity (46%), with AUROC of 0.72 indicating improvement in prediction compared to traditional KCC for the test dataset. Accuracy for KCC and KCC-CART were comparable (confidence intervals overlap), but KCC-CART had significantly higher AUROC and sensitivity than KCC, and KCC had significantly higher specificity than KCC-CART (non-overlapping confidence intervals). For post-admission prediction, the NEW-CART model offered marginal improvement over traditional KCC, with prediction accuracy of 86%, specificity of 91% and sensitivity of 46%. AUROC for NEW-CART was 0.68, which was lower than that of the KCC-CART for post-admission prediction. Overlapping confidence intervals for accuracy and specificity indicate no significant differences between KCC-CART and NEW-CART, but non-overlapping confidence intervals for KCC-CART demonstrated significantly higher AUROC and sensitivity compared to NEW-CART.


Predicting outcome on admission and post-admission for acetaminophen-induced acute liver failure using classification and regression tree models.

Speiser JL, Lee WM, Karvellas CJ, US Acute Liver Failure Study Gro - PLoS ONE (2015)

Plots of Confidence Intervals of Accuracy, AUROC, Sensitivity and Specificity for Admission and Post-Admission Models.Plots display confidence intervals for accuracy (AC), area under the receiver operating curve (AUROC), sensitivity (SN) and specificity (SP) from Table 2. The admission plot (top panel) illustrates non-overlapping, higher confidence intervals for both KCC-CART and NEW-CART compared to KCC for AUROC and sensitivity, and lower confidence intervals for specificity. This indicates that the CART models had significantly better AUROC and sensitivity than KCC, but had significantly worse specificity compared to KCC. Confidence intervals for accuracy for KCC, KCC-CART and NEW-CART all overlap, indicating no significant differences between the models. The post-admission plot (bottom panel) again indicated no significant differences between the models in terms of accuracy; however, the AUROC and sensitivity of KCC-CART was significantly higher than that of NEW-CART and KCC. The specificity of KCC was highest, but did no differ significantly from that of the NEW-CART. KCC demonstrated significantly higher specificity than the KCC-CART, but the difference was not significant compared to the NEW-CART.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4401567&req=5

pone.0122929.g004: Plots of Confidence Intervals of Accuracy, AUROC, Sensitivity and Specificity for Admission and Post-Admission Models.Plots display confidence intervals for accuracy (AC), area under the receiver operating curve (AUROC), sensitivity (SN) and specificity (SP) from Table 2. The admission plot (top panel) illustrates non-overlapping, higher confidence intervals for both KCC-CART and NEW-CART compared to KCC for AUROC and sensitivity, and lower confidence intervals for specificity. This indicates that the CART models had significantly better AUROC and sensitivity than KCC, but had significantly worse specificity compared to KCC. Confidence intervals for accuracy for KCC, KCC-CART and NEW-CART all overlap, indicating no significant differences between the models. The post-admission plot (bottom panel) again indicated no significant differences between the models in terms of accuracy; however, the AUROC and sensitivity of KCC-CART was significantly higher than that of NEW-CART and KCC. The specificity of KCC was highest, but did no differ significantly from that of the NEW-CART. KCC demonstrated significantly higher specificity than the KCC-CART, but the difference was not significant compared to the NEW-CART.
Mentions: Table 3 displays performance measures and their associated confidence intervals for the three post-admission prognosis models (see also Fig 4). The traditional KCC had low sensitivity (15%) and AUROC (0.56), but maintained an overall prediction accuracy of 70% and specificity of 97% using all post-admission data. Accuracy of KCC for the post-admission testing dataset was significantly higher (86%), with similar values for specificity, sensitivity and AUROC. The KCC-CART developed for post-admission had similar prediction accuracy (82%), slightly lower specificity (86%) and higher sensitivity (46%), with AUROC of 0.72 indicating improvement in prediction compared to traditional KCC for the test dataset. Accuracy for KCC and KCC-CART were comparable (confidence intervals overlap), but KCC-CART had significantly higher AUROC and sensitivity than KCC, and KCC had significantly higher specificity than KCC-CART (non-overlapping confidence intervals). For post-admission prediction, the NEW-CART model offered marginal improvement over traditional KCC, with prediction accuracy of 86%, specificity of 91% and sensitivity of 46%. AUROC for NEW-CART was 0.68, which was lower than that of the KCC-CART for post-admission prediction. Overlapping confidence intervals for accuracy and specificity indicate no significant differences between KCC-CART and NEW-CART, but non-overlapping confidence intervals for KCC-CART demonstrated significantly higher AUROC and sensitivity compared to NEW-CART.

Bottom Line: Accuracy in prediction of outcome (AC), sensitivity (SN), specificity (SP), and area under receiver-operating curve (AUROC) were compared between 3 models: KCC (INR, creatinine, coma grade, pH), CART analysis using only KCC variables (KCC-CART) and a CART model using new variables (NEW-CART).CARTs offer simple prognostic models for APAP-ALF patients, which have higher AUROC and SN than KCC, with similar AC and negligibly worse SP.Admission and post-admission predictions were developed. • Prognostication in acetaminophen-induced acute liver failure (APAP-ALF) is challenging beyond admission • Little has been published regarding the use of King's College Criteria (KCC) beyond admission and KCC has shown limited sensitivity in subsequent studies • Classification and Regression Tree (CART) methodology allows the development of predictive models using binary splits and offers an intuitive method for predicting outcome, using processes familiar to clinicians • Data from the ALFSG registry suggested that CART prognosis models for the APAP population offer improved sensitivity and model performance over traditional regression-based KCC, while maintaining similar accuracy and negligibly worse specificity • KCC-CART models offered modest improvement over traditional KCC, with NEW-CART models performing better than KCC-CART particularly at late time points.

View Article: PubMed Central - PubMed

Affiliation: Department of Public Health Sciences, Medical University of South Carolina, Charleston, South Carolina, United States of America.

ABSTRACT

Background/aim: Assessing prognosis for acetaminophen-induced acute liver failure (APAP-ALF) patients often presents significant challenges. King's College (KCC) has been validated on hospital admission, but little has been published on later phases of illness. We aimed to improve determinations of prognosis both at the time of and following admission for APAP-ALF using Classification and Regression Tree (CART) models.

Methods: CART models were applied to US ALFSG registry data to predict 21-day death or liver transplant early (on admission) and post-admission (days 3-7) for 803 APAP-ALF patients enrolled 01/1998-09/2013. Accuracy in prediction of outcome (AC), sensitivity (SN), specificity (SP), and area under receiver-operating curve (AUROC) were compared between 3 models: KCC (INR, creatinine, coma grade, pH), CART analysis using only KCC variables (KCC-CART) and a CART model using new variables (NEW-CART).

Results: Traditional KCC yielded 69% AC, 90% SP, 27% SN, and 0.58 AUROC on admission, with similar performance post-admission. KCC-CART at admission offered predictive 66% AC, 65% SP, 67% SN, and 0.74 AUROC. Post-admission, KCC-CART had predictive 82% AC, 86% SP, 46% SN and 0.81 AUROC. NEW-CART models using MELD (Model for end stage liver disease), lactate and mechanical ventilation on admission yielded predictive 72% AC, 71% SP, 77% SN and AUROC 0.79. For later stages, NEW-CART (MELD, lactate, coma grade) offered predictive AC 86%, SP 91%, SN 46%, AUROC 0.73.

Conclusion: CARTs offer simple prognostic models for APAP-ALF patients, which have higher AUROC and SN than KCC, with similar AC and negligibly worse SP. Admission and post-admission predictions were developed.

Key points: • Prognostication in acetaminophen-induced acute liver failure (APAP-ALF) is challenging beyond admission • Little has been published regarding the use of King's College Criteria (KCC) beyond admission and KCC has shown limited sensitivity in subsequent studies • Classification and Regression Tree (CART) methodology allows the development of predictive models using binary splits and offers an intuitive method for predicting outcome, using processes familiar to clinicians • Data from the ALFSG registry suggested that CART prognosis models for the APAP population offer improved sensitivity and model performance over traditional regression-based KCC, while maintaining similar accuracy and negligibly worse specificity • KCC-CART models offered modest improvement over traditional KCC, with NEW-CART models performing better than KCC-CART particularly at late time points.

Show MeSH
Related in: MedlinePlus