Limits...
A Bayesian method for comparing and combining binary classifiers in the absence of a gold standard.

Keith JM, Davey CM, Boyd SE - BMC Bioinformatics (2012)

Bottom Line: In all cases, run times were feasible, and results precise.In all three test cases, the globally optimal logical combination of the classifiers was found to be their union, according to three out of four ranking criteria.We propose as a general rule of thumb that the union of classifiers will be close to optimal.

View Article: PubMed Central - HTML - PubMed

Affiliation: School of Mathematical Sciences, Monash University, Victoria 3800, Australia. jonathan.keith@monash.edu

ABSTRACT

Background: Many problems in bioinformatics involve classification based on features such as sequence, structure or morphology. Given multiple classifiers, two crucial questions arise: how does their performance compare, and how can they best be combined to produce a better classifier? A classifier can be evaluated in terms of sensitivity and specificity using benchmark, or gold standard, data, that is, data for which the true classification is known. However, a gold standard is not always available. Here we demonstrate that a Bayesian model for comparing medical diagnostics without a gold standard can be successfully applied in the bioinformatics domain, to genomic scale data sets. We present a new implementation, which unlike previous implementations is applicable to any number of classifiers. We apply this model, for the first time, to the problem of finding the globally optimal logical combination of classifiers.

Results: We compared three classifiers of protein subcellular localisation, and evaluated our estimates of sensitivity and specificity against estimates obtained using a gold standard. The method overestimated sensitivity and specificity with only a small discrepancy, and correctly ranked the classifiers. Diagnostic tests for swine flu were then compared on a small data set. Lastly, classifiers for a genome-wide association study of macular degeneration with 541094 SNPs were analysed. In all cases, run times were feasible, and results precise. The optimal logical combination of classifiers was also determined for all three data sets. Code and data are available from http://bioinformatics.monash.edu.au/downloads/.

Conclusions: The examples demonstrate the methods are suitable for both small and large data sets, applicable to the wide range of bioinformatics classification problems, and robust to dependence between classifiers. In all three test cases, the globally optimal logical combination of the classifiers was found to be their union, according to three out of four ranking criteria. We propose as a general rule of thumb that the union of classifiers will be close to optimal.

Show MeSH

Related in: MedlinePlus

Protein sub-cellular localisation results. Density plots of model variables for the chloroplast localisation data. Vertical lines show gold standard sensitivity, specificity or proportion. A, B, C: the sensitivity of the AA , DP and NCC classifiers, respectively. D, E, F: the specificity of the AA , DP and NCC classifiers, respectively. G: estimated proportion of proteins localised to chloroplasts.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC3473310&req=5

Figure 1: Protein sub-cellular localisation results. Density plots of model variables for the chloroplast localisation data. Vertical lines show gold standard sensitivity, specificity or proportion. A, B, C: the sensitivity of the AA , DP and NCC classifiers, respectively. D, E, F: the specificity of the AA , DP and NCC classifiers, respectively. G: estimated proportion of proteins localised to chloroplasts.

Mentions: The output of each classifier was converted to a sequence of 0s and 1s, indicating which proteins were localised to the chloroplast region (1) and which were not (0). For each parameter of the model, summary statistics for its marginal posterior distribution were obtained (Additional file 1: Section S3.2). Density plots (Figure 1) and time series plots (Additional file 1: Section S3.3) were also produced. Sensitivities and specificities for the classifiers were evaluated using the gold standard classification, and these are shown as vertical lines in Figure 1. It should be noted that these gold standard estimates also have standard errors (not shown) because they are based on finite sample sizes.


A Bayesian method for comparing and combining binary classifiers in the absence of a gold standard.

Keith JM, Davey CM, Boyd SE - BMC Bioinformatics (2012)

Protein sub-cellular localisation results. Density plots of model variables for the chloroplast localisation data. Vertical lines show gold standard sensitivity, specificity or proportion. A, B, C: the sensitivity of the AA , DP and NCC classifiers, respectively. D, E, F: the specificity of the AA , DP and NCC classifiers, respectively. G: estimated proportion of proteins localised to chloroplasts.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC3473310&req=5

Figure 1: Protein sub-cellular localisation results. Density plots of model variables for the chloroplast localisation data. Vertical lines show gold standard sensitivity, specificity or proportion. A, B, C: the sensitivity of the AA , DP and NCC classifiers, respectively. D, E, F: the specificity of the AA , DP and NCC classifiers, respectively. G: estimated proportion of proteins localised to chloroplasts.
Mentions: The output of each classifier was converted to a sequence of 0s and 1s, indicating which proteins were localised to the chloroplast region (1) and which were not (0). For each parameter of the model, summary statistics for its marginal posterior distribution were obtained (Additional file 1: Section S3.2). Density plots (Figure 1) and time series plots (Additional file 1: Section S3.3) were also produced. Sensitivities and specificities for the classifiers were evaluated using the gold standard classification, and these are shown as vertical lines in Figure 1. It should be noted that these gold standard estimates also have standard errors (not shown) because they are based on finite sample sizes.

Bottom Line: In all cases, run times were feasible, and results precise.In all three test cases, the globally optimal logical combination of the classifiers was found to be their union, according to three out of four ranking criteria.We propose as a general rule of thumb that the union of classifiers will be close to optimal.

View Article: PubMed Central - HTML - PubMed

Affiliation: School of Mathematical Sciences, Monash University, Victoria 3800, Australia. jonathan.keith@monash.edu

ABSTRACT

Background: Many problems in bioinformatics involve classification based on features such as sequence, structure or morphology. Given multiple classifiers, two crucial questions arise: how does their performance compare, and how can they best be combined to produce a better classifier? A classifier can be evaluated in terms of sensitivity and specificity using benchmark, or gold standard, data, that is, data for which the true classification is known. However, a gold standard is not always available. Here we demonstrate that a Bayesian model for comparing medical diagnostics without a gold standard can be successfully applied in the bioinformatics domain, to genomic scale data sets. We present a new implementation, which unlike previous implementations is applicable to any number of classifiers. We apply this model, for the first time, to the problem of finding the globally optimal logical combination of classifiers.

Results: We compared three classifiers of protein subcellular localisation, and evaluated our estimates of sensitivity and specificity against estimates obtained using a gold standard. The method overestimated sensitivity and specificity with only a small discrepancy, and correctly ranked the classifiers. Diagnostic tests for swine flu were then compared on a small data set. Lastly, classifiers for a genome-wide association study of macular degeneration with 541094 SNPs were analysed. In all cases, run times were feasible, and results precise. The optimal logical combination of classifiers was also determined for all three data sets. Code and data are available from http://bioinformatics.monash.edu.au/downloads/.

Conclusions: The examples demonstrate the methods are suitable for both small and large data sets, applicable to the wide range of bioinformatics classification problems, and robust to dependence between classifiers. In all three test cases, the globally optimal logical combination of the classifiers was found to be their union, according to three out of four ranking criteria. We propose as a general rule of thumb that the union of classifiers will be close to optimal.

Show MeSH
Related in: MedlinePlus