Limits...
Assessing the statistical significance of the achieved classification error of classifiers constructed using serum peptide profiles, and a prescription for random sampling repeated studies for massive high-throughput genomic and proteomic studies.

Lyons-Weiler J, Pelikan R, Zeh HJ, Whitcomb DC, Malehorn DE, Bigbee WL, Hauskrecht M - Cancer Inform (2005)

Bottom Line: A statistically significant ACE increases our belief that a discriminative signal was found in the data.PACE analysis does not protect researchers against confounding in the experimental design, or other sources of systematic or random error.The results show that many of these datasets indeed possess a signal that leads to a statistically significant ACE.

View Article: PubMed Central - PubMed

Affiliation: Department of Pathology, Cancer Biomarkers Laboratory, Center for Pathology Informatics, Benedum Oncology Informatics Center, University of Pittsburgh Cancer Institute University of Pittsburgh, USA. lyonsweilerj@upmc.edu

ABSTRACT
Peptide profiles generated using SELDI/MALDI time of flight mass spectrometry provide a promising source of patient-specific information with high potential impact on the early detection and classification of cancer and other diseases. The new profiling technology comes, however, with numerous challenges and concerns. Particularly important are concerns of reproducibility of classification results and their significance. In this work we describe a computational validation framework, called PACE (Permutation-Achieved Classification Error), that lets us assess, for a given classification model, the significance of the Achieved Classification Error (ACE) on the profile data. The framework compares the performance statistic of the classifier on true data samples and checks if these are consistent with the behavior of the classifier on the same data with randomly reassigned class labels. A statistically significant ACE increases our belief that a discriminative signal was found in the data. The advantage of PACE analysis is that it can be easily combined with any classification model and is relatively easy to interpret. PACE analysis does not protect researchers against confounding in the experimental design, or other sources of systematic or random error. We use PACE analysis to assess significance of classification results we have achieved on a number of published data sets. The results show that many of these datasets indeed possess a signal that leads to a statistically significant ACE.

No MeSH data available.


Related in: MedlinePlus

PACE analysis using a SVM-based classifier using the J5 scoring criterion, with a 0.6 MAC threshold. Left Panel: performance without peak selection. Right panel: performance with peak selection.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC2657655&req=5

fD3-cin-01-53: PACE analysis using a SVM-based classifier using the J5 scoring criterion, with a 0.6 MAC threshold. Left Panel: performance without peak selection. Right panel: performance with peak selection.


Assessing the statistical significance of the achieved classification error of classifiers constructed using serum peptide profiles, and a prescription for random sampling repeated studies for massive high-throughput genomic and proteomic studies.

Lyons-Weiler J, Pelikan R, Zeh HJ, Whitcomb DC, Malehorn DE, Bigbee WL, Hauskrecht M - Cancer Inform (2005)

PACE analysis using a SVM-based classifier using the J5 scoring criterion, with a 0.6 MAC threshold. Left Panel: performance without peak selection. Right panel: performance with peak selection.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC2657655&req=5

fD3-cin-01-53: PACE analysis using a SVM-based classifier using the J5 scoring criterion, with a 0.6 MAC threshold. Left Panel: performance without peak selection. Right panel: performance with peak selection.
Bottom Line: A statistically significant ACE increases our belief that a discriminative signal was found in the data.PACE analysis does not protect researchers against confounding in the experimental design, or other sources of systematic or random error.The results show that many of these datasets indeed possess a signal that leads to a statistically significant ACE.

View Article: PubMed Central - PubMed

Affiliation: Department of Pathology, Cancer Biomarkers Laboratory, Center for Pathology Informatics, Benedum Oncology Informatics Center, University of Pittsburgh Cancer Institute University of Pittsburgh, USA. lyonsweilerj@upmc.edu

ABSTRACT
Peptide profiles generated using SELDI/MALDI time of flight mass spectrometry provide a promising source of patient-specific information with high potential impact on the early detection and classification of cancer and other diseases. The new profiling technology comes, however, with numerous challenges and concerns. Particularly important are concerns of reproducibility of classification results and their significance. In this work we describe a computational validation framework, called PACE (Permutation-Achieved Classification Error), that lets us assess, for a given classification model, the significance of the Achieved Classification Error (ACE) on the profile data. The framework compares the performance statistic of the classifier on true data samples and checks if these are consistent with the behavior of the classifier on the same data with randomly reassigned class labels. A statistically significant ACE increases our belief that a discriminative signal was found in the data. The advantage of PACE analysis is that it can be easily combined with any classification model and is relatively easy to interpret. PACE analysis does not protect researchers against confounding in the experimental design, or other sources of systematic or random error. We use PACE analysis to assess significance of classification results we have achieved on a number of published data sets. The results show that many of these datasets indeed possess a signal that leads to a statistically significant ACE.

No MeSH data available.


Related in: MedlinePlus