Limits...
Comparative evaluation of gene set analysis approaches for RNA-Seq data.

Rahmatallah Y, Emmert-Streib F, Glazko G - BMC Bioinformatics (2014)

Bottom Line: Our results demonstrate that the Type I error rate and the power of multivariate tests depend only on the test statistics and are insensitive to the different normalizations.In general standard multivariate GSA tests detect pathways that do not have any bias in terms of pathways size, percentage of differentially expressed genes, or average gene length in a pathway.Our result emphasizes the importance of using self-contained non-parametric multivariate tests for detecting differentially expressed pathways for RNA-Seq data and warns against applying gene-level GSA tests, especially because of their high level of Type I error rates for both, simulated and real data.

View Article: PubMed Central - PubMed

Affiliation: Division of Biomedical Informatics, University of Arkansas for Medical Sciences, Little Rock, AR, 72205, USA. yrahmatallah@uams.edu.

ABSTRACT

Background: Over the last few years transcriptome sequencing (RNA-Seq) has almost completely taken over microarrays for high-throughput studies of gene expression. Currently, the most popular use of RNA-Seq is to identify genes which are differentially expressed between two or more conditions. Despite the importance of Gene Set Analysis (GSA) in the interpretation of the results from RNA-Seq experiments, the limitations of GSA methods developed for microarrays in the context of RNA-Seq data are not well understood.

Results: We provide a thorough evaluation of popular multivariate and gene-level self-contained GSA approaches on simulated and real RNA-Seq data. The multivariate approach employs multivariate non-parametric tests combined with popular normalizations for RNA-Seq data. The gene-level approach utilizes univariate tests designed for the analysis of RNA-Seq data to find gene-specific P-values and combines them into a pathway P-value using classical statistical techniques. Our results demonstrate that the Type I error rate and the power of multivariate tests depend only on the test statistics and are insensitive to the different normalizations. In general standard multivariate GSA tests detect pathways that do not have any bias in terms of pathways size, percentage of differentially expressed genes, or average gene length in a pathway. In contrast the Type I error rate and the power of gene-level GSA tests are heavily affected by the methods for combining P-values, and all aforementioned biases are present in detected pathways.

Conclusions: Our result emphasizes the importance of using self-contained non-parametric multivariate tests for detecting differentially expressed pathways for RNA-Seq data and warns against applying gene-level GSA tests, especially because of their high level of Type I error rates for both, simulated and real data.

Show MeSH
The power curves of gene-level GSA methods when shift alternative hypothesis(H1)holds true and the number of genes in pathwaysp = 16(N = 20).
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC4265362&req=5

Fig3: The power curves of gene-level GSA methods when shift alternative hypothesis(H1)holds true and the number of genes in pathwaysp = 16(N = 20).

Mentions: Figure 3 presents the power estimates for gene-level GSA approaches that use univariate tests (edgeR, DESeq, and eBayes) and employ different methods for combining P-values (FM, SM, and GM with STT = 0.05) when H1 is true (N = 20, p = 16). When the percentage of truly differentially expressed genes is small (γ = 1/8), all three tests that apply GM have slightly higher power than those tests with FM, while the power of tests with SM is much smaller. When γ increases (from the top to the bottom on each panel of Figure 3) the difference between tests with GM and tests with FM diminishes, and the power of tests with SM becomes very close to the power of tests with FM and GM. The results when N = 20 and p = 100 (Additional file 3: Figure S6), N = 40 and p = 16 (Additional file 3: Figure S7) and N = 40 and p = 100 (Additional file 3: Figure S8) are similar, but the power to detect even small fold changes is higher for all tests. Comparing the performance of the three univariate tests under each P-value combining method shows that edgeR has slightly higher power than DESeq and eBayes, with both FM and GM, while eBayes has slightly higher power than edgeR and DESeq with SM (Additional file 3: Figure S9). Additional file 3: Figure S10 (N = 20 and p = 100), Additional file 3: Figure S11 (N = 40 and p = 16), and Additional file 3: Figure S12 (N = 40 and p = 100) demonstrate a similar pattern with even more insignificant differences.Figure 3


Comparative evaluation of gene set analysis approaches for RNA-Seq data.

Rahmatallah Y, Emmert-Streib F, Glazko G - BMC Bioinformatics (2014)

The power curves of gene-level GSA methods when shift alternative hypothesis(H1)holds true and the number of genes in pathwaysp = 16(N = 20).
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC4265362&req=5

Fig3: The power curves of gene-level GSA methods when shift alternative hypothesis(H1)holds true and the number of genes in pathwaysp = 16(N = 20).
Mentions: Figure 3 presents the power estimates for gene-level GSA approaches that use univariate tests (edgeR, DESeq, and eBayes) and employ different methods for combining P-values (FM, SM, and GM with STT = 0.05) when H1 is true (N = 20, p = 16). When the percentage of truly differentially expressed genes is small (γ = 1/8), all three tests that apply GM have slightly higher power than those tests with FM, while the power of tests with SM is much smaller. When γ increases (from the top to the bottom on each panel of Figure 3) the difference between tests with GM and tests with FM diminishes, and the power of tests with SM becomes very close to the power of tests with FM and GM. The results when N = 20 and p = 100 (Additional file 3: Figure S6), N = 40 and p = 16 (Additional file 3: Figure S7) and N = 40 and p = 100 (Additional file 3: Figure S8) are similar, but the power to detect even small fold changes is higher for all tests. Comparing the performance of the three univariate tests under each P-value combining method shows that edgeR has slightly higher power than DESeq and eBayes, with both FM and GM, while eBayes has slightly higher power than edgeR and DESeq with SM (Additional file 3: Figure S9). Additional file 3: Figure S10 (N = 20 and p = 100), Additional file 3: Figure S11 (N = 40 and p = 16), and Additional file 3: Figure S12 (N = 40 and p = 100) demonstrate a similar pattern with even more insignificant differences.Figure 3

Bottom Line: Our results demonstrate that the Type I error rate and the power of multivariate tests depend only on the test statistics and are insensitive to the different normalizations.In general standard multivariate GSA tests detect pathways that do not have any bias in terms of pathways size, percentage of differentially expressed genes, or average gene length in a pathway.Our result emphasizes the importance of using self-contained non-parametric multivariate tests for detecting differentially expressed pathways for RNA-Seq data and warns against applying gene-level GSA tests, especially because of their high level of Type I error rates for both, simulated and real data.

View Article: PubMed Central - PubMed

Affiliation: Division of Biomedical Informatics, University of Arkansas for Medical Sciences, Little Rock, AR, 72205, USA. yrahmatallah@uams.edu.

ABSTRACT

Background: Over the last few years transcriptome sequencing (RNA-Seq) has almost completely taken over microarrays for high-throughput studies of gene expression. Currently, the most popular use of RNA-Seq is to identify genes which are differentially expressed between two or more conditions. Despite the importance of Gene Set Analysis (GSA) in the interpretation of the results from RNA-Seq experiments, the limitations of GSA methods developed for microarrays in the context of RNA-Seq data are not well understood.

Results: We provide a thorough evaluation of popular multivariate and gene-level self-contained GSA approaches on simulated and real RNA-Seq data. The multivariate approach employs multivariate non-parametric tests combined with popular normalizations for RNA-Seq data. The gene-level approach utilizes univariate tests designed for the analysis of RNA-Seq data to find gene-specific P-values and combines them into a pathway P-value using classical statistical techniques. Our results demonstrate that the Type I error rate and the power of multivariate tests depend only on the test statistics and are insensitive to the different normalizations. In general standard multivariate GSA tests detect pathways that do not have any bias in terms of pathways size, percentage of differentially expressed genes, or average gene length in a pathway. In contrast the Type I error rate and the power of gene-level GSA tests are heavily affected by the methods for combining P-values, and all aforementioned biases are present in detected pathways.

Conclusions: Our result emphasizes the importance of using self-contained non-parametric multivariate tests for detecting differentially expressed pathways for RNA-Seq data and warns against applying gene-level GSA tests, especially because of their high level of Type I error rates for both, simulated and real data.

Show MeSH