Limits...
Parameterization of disorder predictors for large-scale applications requiring high specificity by using an extended benchmark dataset.

Sirota FL, Ooi HS, Gattermayer T, Schneider G, Eisenhaber F, Maurer-Stroh S - BMC Genomics (2010)

Bottom Line: We identify settings, in which the different predictors have the same false positive rate.This is useful in the framework of proteome-wide applications where high specificity is required such as in our in-house sequence analysis pipeline and the ANNIE webserver.This work identifies parameter settings and thresholds for a selection of disorder predictors to produce comparable results at a desired level of specificity over a newly derived benchmark dataset that accounts equally for ordered and disordered regions of different lengths.

View Article: PubMed Central - HTML - PubMed

Affiliation: Biomolecular Function Discovery Division, Bioinformatics Institute (BII), Agency for Science Technology and Research (A*STAR), Matrix, Singapore. fernanda@bii.a-star.edu.sg

ABSTRACT

Background: Algorithms designed to predict protein disorder play an important role in structural and functional genomics, as disordered regions have been reported to participate in important cellular processes. Consequently, several methods with different underlying principles for disorder prediction have been independently developed by various groups. For assessing their usability in automated workflows, we are interested in identifying parameter settings and threshold selections, under which the performance of these predictors becomes directly comparable.

Results: First, we derived a new benchmark set that accounts for different flavours of disorder complemented with a similar amount of order annotation derived for the same protein set. We show that, using the recommended default parameters, the programs tested are producing a wide range of predictions at different levels of specificity and sensitivity. We identify settings, in which the different predictors have the same false positive rate. We assess conditions when sets of predictors can be run together to derive consensus or complementary predictions. This is useful in the framework of proteome-wide applications where high specificity is required such as in our in-house sequence analysis pipeline and the ANNIE webserver.

Conclusions: This work identifies parameter settings and thresholds for a selection of disorder predictors to produce comparable results at a desired level of specificity over a newly derived benchmark dataset that accounts equally for ordered and disordered regions of different lengths.

Show MeSH
Performance of combined algorithms. Consensus and complementary predictions at highest MCC and false positive rate at ~0.05. (a) SL dataset. (b) Remark 465 dataset. ROC curves for DISOPRED2 and IUPred long and short were used as reference. Only the data points closer and above the DISOPRED2 curve are labelled. FPR and TPR are false positive rate and true positive rate, respectively.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC2822529&req=5

Figure 5: Performance of combined algorithms. Consensus and complementary predictions at highest MCC and false positive rate at ~0.05. (a) SL dataset. (b) Remark 465 dataset. ROC curves for DISOPRED2 and IUPred long and short were used as reference. Only the data points closer and above the DISOPRED2 curve are labelled. FPR and TPR are false positive rate and true positive rate, respectively.

Mentions: In this work, we explored the combined performance of any pair of disorder prediction algorithms. In contrast to previous work [33,36], we used the parameters that reproduce the same level of specificity for each method at a false positive rate of 0.05 (Tables 3 and 4). In addition, we also combined them applying the parameters where the highest MCC was obtained (see Tables 7 and 8). The results of this investigation are summarized in Figure 5. As a trend, the combination of two methods either through consensus or complementary predictions results in a slight improvement of performance compared to single methods. We find that DISOPRED2, which has ranked quite well in the individual comparison to other methods, can only be slightly improved through combination with almost any method but, if at all, the best effect is achieved with IUPred long, CAST [23] or DisEMBL Remark 465. On the other hand, only the combination of IUPred long with either CAST (for the SL dataset) or DisEMBL Remark 465 (for the Remark 465 dataset) reaches the single method performance of DISOPRED2. This is of interest due to the long computation time required for DISOPRED2 compared to other methods.


Parameterization of disorder predictors for large-scale applications requiring high specificity by using an extended benchmark dataset.

Sirota FL, Ooi HS, Gattermayer T, Schneider G, Eisenhaber F, Maurer-Stroh S - BMC Genomics (2010)

Performance of combined algorithms. Consensus and complementary predictions at highest MCC and false positive rate at ~0.05. (a) SL dataset. (b) Remark 465 dataset. ROC curves for DISOPRED2 and IUPred long and short were used as reference. Only the data points closer and above the DISOPRED2 curve are labelled. FPR and TPR are false positive rate and true positive rate, respectively.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC2822529&req=5

Figure 5: Performance of combined algorithms. Consensus and complementary predictions at highest MCC and false positive rate at ~0.05. (a) SL dataset. (b) Remark 465 dataset. ROC curves for DISOPRED2 and IUPred long and short were used as reference. Only the data points closer and above the DISOPRED2 curve are labelled. FPR and TPR are false positive rate and true positive rate, respectively.
Mentions: In this work, we explored the combined performance of any pair of disorder prediction algorithms. In contrast to previous work [33,36], we used the parameters that reproduce the same level of specificity for each method at a false positive rate of 0.05 (Tables 3 and 4). In addition, we also combined them applying the parameters where the highest MCC was obtained (see Tables 7 and 8). The results of this investigation are summarized in Figure 5. As a trend, the combination of two methods either through consensus or complementary predictions results in a slight improvement of performance compared to single methods. We find that DISOPRED2, which has ranked quite well in the individual comparison to other methods, can only be slightly improved through combination with almost any method but, if at all, the best effect is achieved with IUPred long, CAST [23] or DisEMBL Remark 465. On the other hand, only the combination of IUPred long with either CAST (for the SL dataset) or DisEMBL Remark 465 (for the Remark 465 dataset) reaches the single method performance of DISOPRED2. This is of interest due to the long computation time required for DISOPRED2 compared to other methods.

Bottom Line: We identify settings, in which the different predictors have the same false positive rate.This is useful in the framework of proteome-wide applications where high specificity is required such as in our in-house sequence analysis pipeline and the ANNIE webserver.This work identifies parameter settings and thresholds for a selection of disorder predictors to produce comparable results at a desired level of specificity over a newly derived benchmark dataset that accounts equally for ordered and disordered regions of different lengths.

View Article: PubMed Central - HTML - PubMed

Affiliation: Biomolecular Function Discovery Division, Bioinformatics Institute (BII), Agency for Science Technology and Research (A*STAR), Matrix, Singapore. fernanda@bii.a-star.edu.sg

ABSTRACT

Background: Algorithms designed to predict protein disorder play an important role in structural and functional genomics, as disordered regions have been reported to participate in important cellular processes. Consequently, several methods with different underlying principles for disorder prediction have been independently developed by various groups. For assessing their usability in automated workflows, we are interested in identifying parameter settings and threshold selections, under which the performance of these predictors becomes directly comparable.

Results: First, we derived a new benchmark set that accounts for different flavours of disorder complemented with a similar amount of order annotation derived for the same protein set. We show that, using the recommended default parameters, the programs tested are producing a wide range of predictions at different levels of specificity and sensitivity. We identify settings, in which the different predictors have the same false positive rate. We assess conditions when sets of predictors can be run together to derive consensus or complementary predictions. This is useful in the framework of proteome-wide applications where high specificity is required such as in our in-house sequence analysis pipeline and the ANNIE webserver.

Conclusions: This work identifies parameter settings and thresholds for a selection of disorder predictors to produce comparable results at a desired level of specificity over a newly derived benchmark dataset that accounts equally for ordered and disordered regions of different lengths.

Show MeSH