Limits...
Discerning tumor status from unstructured MRI reports--completeness of information in existing reports and utility of automated natural language processing.

Cheng LT, Zheng J, Savova GK, Erickson BJ - J Digit Imaging (2009)

Bottom Line: Natural language processing (NLP) is an emerging technique for rapid and efficient clinical data retrieval.The NLP tool utilized a support vector machines model with statistical and rule-based outcomes.In conclusion, most reports contained sufficient information for tumor status determination, though variable features were used to describe status.

View Article: PubMed Central - PubMed

Affiliation: Department of Radiology, Mayo Clinic, Rochester, MN 55905, USA.

ABSTRACT
Information in electronic medical records is often in an unstructured free-text format. This format presents challenges for expedient data retrieval and may fail to convey important findings. Natural language processing (NLP) is an emerging technique for rapid and efficient clinical data retrieval. While proven in disease detection, the utility of NLP in discerning disease progression from free-text reports is untested. We aimed to (1) assess whether unstructured radiology reports contained sufficient information for tumor status classification; (2) develop an NLP-based data extraction tool to determine tumor status from unstructured reports; and (3) compare NLP and human tumor status classification outcomes. Consecutive follow-up brain tumor magnetic resonance imaging reports (2000--2007) from a tertiary center were manually annotated using consensus guidelines on tumor status. Reports were randomized to NLP training (70%) or testing (30%) groups. The NLP tool utilized a support vector machines model with statistical and rule-based outcomes. Most reports had sufficient information for tumor status classification, although 0.8% did not describe status despite reference to prior examinations. Tumor size was unreported in 68.7% of documents, while 50.3% lacked data on change magnitude when there was detectable progression or regression. Using retrospective human classification as the gold standard, NLP achieved 80.6% sensitivity and 91.6% specificity for tumor status determination (mean positive predictive value, 82.4%; negative predictive value, 92.0%). In conclusion, most reports contained sufficient information for tumor status determination, though variable features were used to describe status. NLP demonstrated good accuracy for tumor status classification and may have novel application for automated disease status classification from electronic databases.

Show MeSH

Related in: MedlinePlus

Receiver operating characteristic curves for tumor status determination by NLP.
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC2837158&req=5

Fig6: Receiver operating characteristic curves for tumor status determination by NLP.

Mentions: Compared to human classification for the test group (231 reports), NLP performed best for classification of tumor status, with an overall mean sensitivity and specificity of 80.6% and 91.6%, respectively (Fig. 5 and Table 5). Within the status subcategories, the highest NLP sensitivity was seen for classification of stability, while the highest specificity was obtained for classification of regression. The receiver operating characteristic (ROC) curves for NLP tumor status determination gave area under curve (AUC) values of at least 0.94 (Fig. 6). NLP performance metrics were lower for determination of magnitude and lowest for classification of significance. This trend was mirrored in the kappa values for agreement between NLP and human classification (Table 4). A similar pattern was observed for F-measures20 of NLP compared to human classification. Macro F-measure scores of 0.81, 0.77, and 0.69 were obtained for status, magnitude, and significance respectively, while micro F-measure scores were 0.86, 0.82, and 0.72, respectively.Fig 5


Discerning tumor status from unstructured MRI reports--completeness of information in existing reports and utility of automated natural language processing.

Cheng LT, Zheng J, Savova GK, Erickson BJ - J Digit Imaging (2009)

Receiver operating characteristic curves for tumor status determination by NLP.
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC2837158&req=5

Fig6: Receiver operating characteristic curves for tumor status determination by NLP.
Mentions: Compared to human classification for the test group (231 reports), NLP performed best for classification of tumor status, with an overall mean sensitivity and specificity of 80.6% and 91.6%, respectively (Fig. 5 and Table 5). Within the status subcategories, the highest NLP sensitivity was seen for classification of stability, while the highest specificity was obtained for classification of regression. The receiver operating characteristic (ROC) curves for NLP tumor status determination gave area under curve (AUC) values of at least 0.94 (Fig. 6). NLP performance metrics were lower for determination of magnitude and lowest for classification of significance. This trend was mirrored in the kappa values for agreement between NLP and human classification (Table 4). A similar pattern was observed for F-measures20 of NLP compared to human classification. Macro F-measure scores of 0.81, 0.77, and 0.69 were obtained for status, magnitude, and significance respectively, while micro F-measure scores were 0.86, 0.82, and 0.72, respectively.Fig 5

Bottom Line: Natural language processing (NLP) is an emerging technique for rapid and efficient clinical data retrieval.The NLP tool utilized a support vector machines model with statistical and rule-based outcomes.In conclusion, most reports contained sufficient information for tumor status determination, though variable features were used to describe status.

View Article: PubMed Central - PubMed

Affiliation: Department of Radiology, Mayo Clinic, Rochester, MN 55905, USA.

ABSTRACT
Information in electronic medical records is often in an unstructured free-text format. This format presents challenges for expedient data retrieval and may fail to convey important findings. Natural language processing (NLP) is an emerging technique for rapid and efficient clinical data retrieval. While proven in disease detection, the utility of NLP in discerning disease progression from free-text reports is untested. We aimed to (1) assess whether unstructured radiology reports contained sufficient information for tumor status classification; (2) develop an NLP-based data extraction tool to determine tumor status from unstructured reports; and (3) compare NLP and human tumor status classification outcomes. Consecutive follow-up brain tumor magnetic resonance imaging reports (2000--2007) from a tertiary center were manually annotated using consensus guidelines on tumor status. Reports were randomized to NLP training (70%) or testing (30%) groups. The NLP tool utilized a support vector machines model with statistical and rule-based outcomes. Most reports had sufficient information for tumor status classification, although 0.8% did not describe status despite reference to prior examinations. Tumor size was unreported in 68.7% of documents, while 50.3% lacked data on change magnitude when there was detectable progression or regression. Using retrospective human classification as the gold standard, NLP achieved 80.6% sensitivity and 91.6% specificity for tumor status determination (mean positive predictive value, 82.4%; negative predictive value, 92.0%). In conclusion, most reports contained sufficient information for tumor status determination, though variable features were used to describe status. NLP demonstrated good accuracy for tumor status classification and may have novel application for automated disease status classification from electronic databases.

Show MeSH
Related in: MedlinePlus