Limits...
Semi-automated curation of protein subcellular localization: a text mining-based approach to Gene Ontology (GO) Cellular Component curation.

Van Auken K, Jaffery J, Chan J, Müller HM, Sternberg PW - BMC Bioinformatics (2009)

Bottom Line: We compared the results of manual curation to that of Textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated C. elegans protein, and found that Textpresso searches identified curatable papers with recall and precision rates of 79.1% and 61.8%, respectively (F-score of 69.5%), when compared to manual curation.Textpresso is an effective tool for improving the efficiency of manual, experimentally based curation.Continued development of curation task-specific Textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation.

View Article: PubMed Central - HTML - PubMed

Affiliation: Division of Biology, California Institute of Technology, Pasadena, CA 91125, USA. vanauken@caltech.edu

ABSTRACT

Background: Manual curation of experimental data from the biomedical literature is an expensive and time-consuming endeavor. Nevertheless, most biological knowledge bases still rely heavily on manual curation for data extraction and entry. Text mining software that can semi- or fully automate information retrieval from the literature would thus provide a significant boost to manual curation efforts.

Results: We employ the Textpresso category-based information retrieval and extraction system (http://www.textpresso.org), developed by WormBase to explore how Textpresso might improve the efficiency with which we manually curate C. elegans proteins to the Gene Ontology's Cellular Component Ontology. Using a training set of sentences that describe results of localization experiments in the published literature, we generated three new curation task-specific categories (Cellular Components, Assay Terms, and Verbs) containing words and phrases associated with reports of experimentally determined subcellular localization. We compared the results of manual curation to that of Textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated C. elegans protein, and found that Textpresso searches identified curatable papers with recall and precision rates of 79.1% and 61.8%, respectively (F-score of 69.5%), when compared to manual curation. Within those documents, Textpresso identified relevant sentences with recall and precision rates of 30.3% and 80.1% (F-score of 44.0%). From returned sentences, curators were able to make 66.2% of all possible experimentally supported GO Cellular Component annotations with 97.3% precision (F-score of 78.8%). Measuring the relative efficiencies of Textpresso-based versus manual curation we find that Textpresso has the potential to increase curation efficiency by at least 8-fold, and perhaps as much as 15-fold, given differences in individual curatorial speed.

Conclusion: Textpresso is an effective tool for improving the efficiency of manual, experimentally based curation. Incorporating a Textpresso-based Cellular Component curation pipeline at WormBase has allowed us to transition from strictly manual curation of this data type to a more efficient pipeline of computer-assisted validation. Continued development of curation task-specific Textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation.

Show MeSH
Sample true positive sentences from the training set. Three different sentences from the training set are shown [19-21], illustrating the types of sentences selected by curators and the individual terms selected for each of the categories. C. elegans proteins are shown in upper-case bold type, Cellular Components in blue, Assay Terms in red, and Verbs in green.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC2719631&req=5

Figure 2: Sample true positive sentences from the training set. Three different sentences from the training set are shown [19-21], illustrating the types of sentences selected by curators and the individual terms selected for each of the categories. C. elegans proteins are shown in upper-case bold type, Cellular Components in blue, Assay Terms in red, and Verbs in green.

Mentions: Our approach to developing Textpresso categories for GO Cellular Component Curation is outlined in Figure 1 and described in detail in the Methods. To identify words and phrases relevant to reports of subcellular localization experiments, we collected ~1,700 sentences from papers reporting experimentally determined subcellular localization, and then analyzed lists of words and phrases used in the sentences, as well as the frequency with which the words and phrases occur, to manually select terms that authors use to describe their experimental results (Figure 2[19-21], see Additional file 1). Words and phrases identified by our word usage and frequency analysis were then manually sorted into three categories: Cellular Components, Assay Terms, and Verbs, and included terms such as: nucleus, cell body, centrosomal; expression, antibody, throughout; and detect, exhibited, revealed, respectively (see Additional file 2).


Semi-automated curation of protein subcellular localization: a text mining-based approach to Gene Ontology (GO) Cellular Component curation.

Van Auken K, Jaffery J, Chan J, Müller HM, Sternberg PW - BMC Bioinformatics (2009)

Sample true positive sentences from the training set. Three different sentences from the training set are shown [19-21], illustrating the types of sentences selected by curators and the individual terms selected for each of the categories. C. elegans proteins are shown in upper-case bold type, Cellular Components in blue, Assay Terms in red, and Verbs in green.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC2719631&req=5

Figure 2: Sample true positive sentences from the training set. Three different sentences from the training set are shown [19-21], illustrating the types of sentences selected by curators and the individual terms selected for each of the categories. C. elegans proteins are shown in upper-case bold type, Cellular Components in blue, Assay Terms in red, and Verbs in green.
Mentions: Our approach to developing Textpresso categories for GO Cellular Component Curation is outlined in Figure 1 and described in detail in the Methods. To identify words and phrases relevant to reports of subcellular localization experiments, we collected ~1,700 sentences from papers reporting experimentally determined subcellular localization, and then analyzed lists of words and phrases used in the sentences, as well as the frequency with which the words and phrases occur, to manually select terms that authors use to describe their experimental results (Figure 2[19-21], see Additional file 1). Words and phrases identified by our word usage and frequency analysis were then manually sorted into three categories: Cellular Components, Assay Terms, and Verbs, and included terms such as: nucleus, cell body, centrosomal; expression, antibody, throughout; and detect, exhibited, revealed, respectively (see Additional file 2).

Bottom Line: We compared the results of manual curation to that of Textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated C. elegans protein, and found that Textpresso searches identified curatable papers with recall and precision rates of 79.1% and 61.8%, respectively (F-score of 69.5%), when compared to manual curation.Textpresso is an effective tool for improving the efficiency of manual, experimentally based curation.Continued development of curation task-specific Textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation.

View Article: PubMed Central - HTML - PubMed

Affiliation: Division of Biology, California Institute of Technology, Pasadena, CA 91125, USA. vanauken@caltech.edu

ABSTRACT

Background: Manual curation of experimental data from the biomedical literature is an expensive and time-consuming endeavor. Nevertheless, most biological knowledge bases still rely heavily on manual curation for data extraction and entry. Text mining software that can semi- or fully automate information retrieval from the literature would thus provide a significant boost to manual curation efforts.

Results: We employ the Textpresso category-based information retrieval and extraction system (http://www.textpresso.org), developed by WormBase to explore how Textpresso might improve the efficiency with which we manually curate C. elegans proteins to the Gene Ontology's Cellular Component Ontology. Using a training set of sentences that describe results of localization experiments in the published literature, we generated three new curation task-specific categories (Cellular Components, Assay Terms, and Verbs) containing words and phrases associated with reports of experimentally determined subcellular localization. We compared the results of manual curation to that of Textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated C. elegans protein, and found that Textpresso searches identified curatable papers with recall and precision rates of 79.1% and 61.8%, respectively (F-score of 69.5%), when compared to manual curation. Within those documents, Textpresso identified relevant sentences with recall and precision rates of 30.3% and 80.1% (F-score of 44.0%). From returned sentences, curators were able to make 66.2% of all possible experimentally supported GO Cellular Component annotations with 97.3% precision (F-score of 78.8%). Measuring the relative efficiencies of Textpresso-based versus manual curation we find that Textpresso has the potential to increase curation efficiency by at least 8-fold, and perhaps as much as 15-fold, given differences in individual curatorial speed.

Conclusion: Textpresso is an effective tool for improving the efficiency of manual, experimentally based curation. Incorporating a Textpresso-based Cellular Component curation pipeline at WormBase has allowed us to transition from strictly manual curation of this data type to a more efficient pipeline of computer-assisted validation. Continued development of curation task-specific Textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation.

Show MeSH