Limits...
Semi-automated curation of protein subcellular localization: a text mining-based approach to Gene Ontology (GO) Cellular Component curation.

Van Auken K, Jaffery J, Chan J, Müller HM, Sternberg PW - BMC Bioinformatics (2009)

Bottom Line: We compared the results of manual curation to that of Textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated C. elegans protein, and found that Textpresso searches identified curatable papers with recall and precision rates of 79.1% and 61.8%, respectively (F-score of 69.5%), when compared to manual curation.Textpresso is an effective tool for improving the efficiency of manual, experimentally based curation.Continued development of curation task-specific Textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation.

View Article: PubMed Central - HTML - PubMed

Affiliation: Division of Biology, California Institute of Technology, Pasadena, CA 91125, USA. vanauken@caltech.edu

ABSTRACT

Background: Manual curation of experimental data from the biomedical literature is an expensive and time-consuming endeavor. Nevertheless, most biological knowledge bases still rely heavily on manual curation for data extraction and entry. Text mining software that can semi- or fully automate information retrieval from the literature would thus provide a significant boost to manual curation efforts.

Results: We employ the Textpresso category-based information retrieval and extraction system (http://www.textpresso.org), developed by WormBase to explore how Textpresso might improve the efficiency with which we manually curate C. elegans proteins to the Gene Ontology's Cellular Component Ontology. Using a training set of sentences that describe results of localization experiments in the published literature, we generated three new curation task-specific categories (Cellular Components, Assay Terms, and Verbs) containing words and phrases associated with reports of experimentally determined subcellular localization. We compared the results of manual curation to that of Textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated C. elegans protein, and found that Textpresso searches identified curatable papers with recall and precision rates of 79.1% and 61.8%, respectively (F-score of 69.5%), when compared to manual curation. Within those documents, Textpresso identified relevant sentences with recall and precision rates of 30.3% and 80.1% (F-score of 44.0%). From returned sentences, curators were able to make 66.2% of all possible experimentally supported GO Cellular Component annotations with 97.3% precision (F-score of 78.8%). Measuring the relative efficiencies of Textpresso-based versus manual curation we find that Textpresso has the potential to increase curation efficiency by at least 8-fold, and perhaps as much as 15-fold, given differences in individual curatorial speed.

Conclusion: Textpresso is an effective tool for improving the efficiency of manual, experimentally based curation. Incorporating a Textpresso-based Cellular Component curation pipeline at WormBase has allowed us to transition from strictly manual curation of this data type to a more efficient pipeline of computer-assisted validation. Continued development of curation task-specific Textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation.

Show MeSH
Textpresso-based curation is more efficient than manual curation. Three different curators recorded the amount of time it took to identify cellular component information from a set of 20 papers either read manually or searched via Textpresso using the three new cellular component categories. Textpresso-based curation results in an 8–15-fold improvement in curation efficiency depending upon the individual curator and the paper set.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC2719631&req=5

Figure 3: Textpresso-based curation is more efficient than manual curation. Three different curators recorded the amount of time it took to identify cellular component information from a set of 20 papers either read manually or searched via Textpresso using the three new cellular component categories. Textpresso-based curation results in an 8–15-fold improvement in curation efficiency depending upon the individual curator and the paper set.

Mentions: The results of this curation efficiency test are presented in Figure 3. The time required for each curator to manually evaluate and record subcellular localization information from their test set of 20 papers was 102, 82, and 90 minutes each, respectively. Using Textpresso, however, the time required to make annotations was approximately 7, 10, and 6 minutes, respectively. Thus, Textpresso-based Cellular Component curation has the potential to improve curatorial efficiency by at least a factor of 8, and possibly as much as 15, given differences in individual curatorial speed. Therefore, even though our new Textpresso categories are not yet able to recover every annotation from the literature, we believe that Textpresso-based curation can still greatly improve the efficiency with which information is extracted from the literature and thus, affords a significant improvement to our GO curation pipeline.


Semi-automated curation of protein subcellular localization: a text mining-based approach to Gene Ontology (GO) Cellular Component curation.

Van Auken K, Jaffery J, Chan J, Müller HM, Sternberg PW - BMC Bioinformatics (2009)

Textpresso-based curation is more efficient than manual curation. Three different curators recorded the amount of time it took to identify cellular component information from a set of 20 papers either read manually or searched via Textpresso using the three new cellular component categories. Textpresso-based curation results in an 8–15-fold improvement in curation efficiency depending upon the individual curator and the paper set.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC2719631&req=5

Figure 3: Textpresso-based curation is more efficient than manual curation. Three different curators recorded the amount of time it took to identify cellular component information from a set of 20 papers either read manually or searched via Textpresso using the three new cellular component categories. Textpresso-based curation results in an 8–15-fold improvement in curation efficiency depending upon the individual curator and the paper set.
Mentions: The results of this curation efficiency test are presented in Figure 3. The time required for each curator to manually evaluate and record subcellular localization information from their test set of 20 papers was 102, 82, and 90 minutes each, respectively. Using Textpresso, however, the time required to make annotations was approximately 7, 10, and 6 minutes, respectively. Thus, Textpresso-based Cellular Component curation has the potential to improve curatorial efficiency by at least a factor of 8, and possibly as much as 15, given differences in individual curatorial speed. Therefore, even though our new Textpresso categories are not yet able to recover every annotation from the literature, we believe that Textpresso-based curation can still greatly improve the efficiency with which information is extracted from the literature and thus, affords a significant improvement to our GO curation pipeline.

Bottom Line: We compared the results of manual curation to that of Textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated C. elegans protein, and found that Textpresso searches identified curatable papers with recall and precision rates of 79.1% and 61.8%, respectively (F-score of 69.5%), when compared to manual curation.Textpresso is an effective tool for improving the efficiency of manual, experimentally based curation.Continued development of curation task-specific Textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation.

View Article: PubMed Central - HTML - PubMed

Affiliation: Division of Biology, California Institute of Technology, Pasadena, CA 91125, USA. vanauken@caltech.edu

ABSTRACT

Background: Manual curation of experimental data from the biomedical literature is an expensive and time-consuming endeavor. Nevertheless, most biological knowledge bases still rely heavily on manual curation for data extraction and entry. Text mining software that can semi- or fully automate information retrieval from the literature would thus provide a significant boost to manual curation efforts.

Results: We employ the Textpresso category-based information retrieval and extraction system (http://www.textpresso.org), developed by WormBase to explore how Textpresso might improve the efficiency with which we manually curate C. elegans proteins to the Gene Ontology's Cellular Component Ontology. Using a training set of sentences that describe results of localization experiments in the published literature, we generated three new curation task-specific categories (Cellular Components, Assay Terms, and Verbs) containing words and phrases associated with reports of experimentally determined subcellular localization. We compared the results of manual curation to that of Textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated C. elegans protein, and found that Textpresso searches identified curatable papers with recall and precision rates of 79.1% and 61.8%, respectively (F-score of 69.5%), when compared to manual curation. Within those documents, Textpresso identified relevant sentences with recall and precision rates of 30.3% and 80.1% (F-score of 44.0%). From returned sentences, curators were able to make 66.2% of all possible experimentally supported GO Cellular Component annotations with 97.3% precision (F-score of 78.8%). Measuring the relative efficiencies of Textpresso-based versus manual curation we find that Textpresso has the potential to increase curation efficiency by at least 8-fold, and perhaps as much as 15-fold, given differences in individual curatorial speed.

Conclusion: Textpresso is an effective tool for improving the efficiency of manual, experimentally based curation. Incorporating a Textpresso-based Cellular Component curation pipeline at WormBase has allowed us to transition from strictly manual curation of this data type to a more efficient pipeline of computer-assisted validation. Continued development of curation task-specific Textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation.

Show MeSH