Limits...
A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data.

Menze BH, Kelm BM, Masuch R, Himmelreich U, Bachert P, Petrich W, Hamprecht FA - BMC Bioinformatics (2009)

Bottom Line: The random forest classifier with its associated Gini feature importance, on the other hand, allows for an explicit feature elimination, but may not be optimally adapted to spectral data due to the topology of its constituent classification trees which are based on orthogonal splits in feature space.The Gini importance of the random forest provided superior means for measuring feature relevance on spectral data, but - on an optimal subset of features - the regularized classifiers might be preferable over the random forest classifier, in spite of their limitation to model linear dependencies only.A feature selection based on Gini importance, however, may precede a regularized linear classification to identify this optimal subset of features, and to earn a double benefit of both dimensionality reduction and the elimination of noise from the classification task.

View Article: PubMed Central - HTML - PubMed

Affiliation: Interdisciplinary Center for Scientific Computing (IWR), University of Heidelberg, Heidelberg, Germany. bjoern.menze@iwr.uni-heidelberg.de

ABSTRACT

Background: Regularized regression methods such as principal component or partial least squares regression perform well in learning tasks on high dimensional spectral data, but cannot explicitly eliminate irrelevant features. The random forest classifier with its associated Gini feature importance, on the other hand, allows for an explicit feature elimination, but may not be optimally adapted to spectral data due to the topology of its constituent classification trees which are based on orthogonal splits in feature space.

Results: We propose to combine the best of both approaches, and evaluated the joint use of a feature selection based on a recursive feature elimination using the Gini importance of random forests' together with regularized classification methods on spectral data sets from medical diagnostics, chemotaxonomy, biomedical analytics, food science, and synthetically modified spectral data. Here, a feature selection using the Gini feature importance with a regularized classification by discriminant partial least squares regression performed as well as or better than a filtering according to different univariate statistical tests, or using regression coefficients in a backward feature elimination. It outperformed the direct application of the random forest classifier, or the direct application of the regularized classifiers on the full set of features.

Conclusion: The Gini importance of the random forest provided superior means for measuring feature relevance on spectral data, but - on an optimal subset of features - the regularized classifiers might be preferable over the random forest classifier, in spite of their limitation to model linear dependencies only. A feature selection based on Gini importance, however, may precede a regularized linear classification to identify this optimal subset of features, and to earn a double benefit of both dimensionality reduction and the elimination of noise from the classification task.

Show MeSH

Related in: MedlinePlus

Importance measures on NMR candida data. in the range from 0.35 to 4 ppm (indicated in the upper figure) for all 1500 spectral channels (indicated in the lower figure). Top: p-values of a t-test (black) and Wilcoxon test (gray). Below: Gini importance of a random forest with 3000 trees (gray) and 6000 trees (black). Compare t ranked measures in Fig. 3.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC2724423&req=5

Figure 2: Importance measures on NMR candida data. in the range from 0.35 to 4 ppm (indicated in the upper figure) for all 1500 spectral channels (indicated in the lower figure). Top: p-values of a t-test (black) and Wilcoxon test (gray). Below: Gini importance of a random forest with 3000 trees (gray) and 6000 trees (black). Compare t ranked measures in Fig. 3.

Mentions: Both univariate tests for significant class differences returned smooth importance vectors when employed on the spectral data (Fig. 2, top). The smoothness of the Gini importance was dependent on the size of the random forest (Fig. 2, bottom) – small forests resulted in "noisy" importance vectors, only converging towards smooth vectors when increasing the overall number of trees in the forest or the overall number of splits. As such changes influence the absolute value of this measure, the Gini importance could not be interpreted in absolute terms – like the p-values of the univariate tests – but only allowed for a relative comparison. For such a comparison between different variables and between different measures, the features were ranked according to their importance score (Fig. 3A). Here, univariate importance measures and Gini importance agreed well in many, although not all, spectral regions (Fig. 3A, rows 2 and 3). An example of the most prominent differences between univariate feature importance and multivariate Gini importance are highlighted in Fig. 3B. Spectral regions deemed unimportant by the univariate measures – with complete overlap of the marginal distributions as shown in Fig. 3B – may be attributed high importance by the multivariate importance measure (Fig. 4), indicating spectral regions with features of higher order interaction.


A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data.

Menze BH, Kelm BM, Masuch R, Himmelreich U, Bachert P, Petrich W, Hamprecht FA - BMC Bioinformatics (2009)

Importance measures on NMR candida data. in the range from 0.35 to 4 ppm (indicated in the upper figure) for all 1500 spectral channels (indicated in the lower figure). Top: p-values of a t-test (black) and Wilcoxon test (gray). Below: Gini importance of a random forest with 3000 trees (gray) and 6000 trees (black). Compare t ranked measures in Fig. 3.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC2724423&req=5

Figure 2: Importance measures on NMR candida data. in the range from 0.35 to 4 ppm (indicated in the upper figure) for all 1500 spectral channels (indicated in the lower figure). Top: p-values of a t-test (black) and Wilcoxon test (gray). Below: Gini importance of a random forest with 3000 trees (gray) and 6000 trees (black). Compare t ranked measures in Fig. 3.
Mentions: Both univariate tests for significant class differences returned smooth importance vectors when employed on the spectral data (Fig. 2, top). The smoothness of the Gini importance was dependent on the size of the random forest (Fig. 2, bottom) – small forests resulted in "noisy" importance vectors, only converging towards smooth vectors when increasing the overall number of trees in the forest or the overall number of splits. As such changes influence the absolute value of this measure, the Gini importance could not be interpreted in absolute terms – like the p-values of the univariate tests – but only allowed for a relative comparison. For such a comparison between different variables and between different measures, the features were ranked according to their importance score (Fig. 3A). Here, univariate importance measures and Gini importance agreed well in many, although not all, spectral regions (Fig. 3A, rows 2 and 3). An example of the most prominent differences between univariate feature importance and multivariate Gini importance are highlighted in Fig. 3B. Spectral regions deemed unimportant by the univariate measures – with complete overlap of the marginal distributions as shown in Fig. 3B – may be attributed high importance by the multivariate importance measure (Fig. 4), indicating spectral regions with features of higher order interaction.

Bottom Line: The random forest classifier with its associated Gini feature importance, on the other hand, allows for an explicit feature elimination, but may not be optimally adapted to spectral data due to the topology of its constituent classification trees which are based on orthogonal splits in feature space.The Gini importance of the random forest provided superior means for measuring feature relevance on spectral data, but - on an optimal subset of features - the regularized classifiers might be preferable over the random forest classifier, in spite of their limitation to model linear dependencies only.A feature selection based on Gini importance, however, may precede a regularized linear classification to identify this optimal subset of features, and to earn a double benefit of both dimensionality reduction and the elimination of noise from the classification task.

View Article: PubMed Central - HTML - PubMed

Affiliation: Interdisciplinary Center for Scientific Computing (IWR), University of Heidelberg, Heidelberg, Germany. bjoern.menze@iwr.uni-heidelberg.de

ABSTRACT

Background: Regularized regression methods such as principal component or partial least squares regression perform well in learning tasks on high dimensional spectral data, but cannot explicitly eliminate irrelevant features. The random forest classifier with its associated Gini feature importance, on the other hand, allows for an explicit feature elimination, but may not be optimally adapted to spectral data due to the topology of its constituent classification trees which are based on orthogonal splits in feature space.

Results: We propose to combine the best of both approaches, and evaluated the joint use of a feature selection based on a recursive feature elimination using the Gini importance of random forests' together with regularized classification methods on spectral data sets from medical diagnostics, chemotaxonomy, biomedical analytics, food science, and synthetically modified spectral data. Here, a feature selection using the Gini feature importance with a regularized classification by discriminant partial least squares regression performed as well as or better than a filtering according to different univariate statistical tests, or using regression coefficients in a backward feature elimination. It outperformed the direct application of the random forest classifier, or the direct application of the regularized classifiers on the full set of features.

Conclusion: The Gini importance of the random forest provided superior means for measuring feature relevance on spectral data, but - on an optimal subset of features - the regularized classifiers might be preferable over the random forest classifier, in spite of their limitation to model linear dependencies only. A feature selection based on Gini importance, however, may precede a regularized linear classification to identify this optimal subset of features, and to earn a double benefit of both dimensionality reduction and the elimination of noise from the classification task.

Show MeSH
Related in: MedlinePlus