Limits...
Understanding the limits of animal models as predictors of human biology: lessons learned from the sbv IMPROVER Species Translation Challenge.

Rhrissorrakrai K, Belcastro V, Bilal E, Norel R, Poussin C, Mathis C, Dulize RH, Ivanov NV, Alexopoulos L, Rice JJ, Peitsch MC, Stolovitzky G, Meyer P, Hoeng J - Bioinformatics (2014)

Bottom Line: Participating teams submitted 49 different solutions across the sub-challenges, two-thirds of which were statistically significantly better than random.Additionally, similar computational methods were found to range widely in their performance within the same challenge, and no single method emerged as a clear winner across all sub-challenges.Finally, computational methods were able to effectively translate some specific stimuli and biological processes in the lung epithelial system, such as DNA synthesis, cytoskeleton and extracellular matrix, translation, immune/inflammation and growth factor/proliferation pathways, better than the expected response similarity between species. pmeyerr@us.ibm.com or Julia.Hoeng@pmi.com Supplementary data are available at Bioinformatics online.

View Article: PubMed Central - PubMed

Affiliation: IBM T.J. Watson Research Center, Computational Biology Center, Yorktown Heights, NY 10003, USA, Philip Morris International R&D, Philip Morris Products S.A., 2000 Neuchâtel, Switzerland, Telethon Institute of Genetics and Medicine, Via Pietro Castellino, 111, 80131 Naples, Italy, ProtATonce Ltd, Scientific Park Lefkippos, Patriarchou Grigoriou & Neapoleos 15343 Ag. Paraskevi, Attiki and National Technical University of Athens, Heroon Polytechniou 9, Zografou 15780, Greece.

Show MeSH

Related in: MedlinePlus

Predictability versus species similarity for gene sets and phosphoproteins. (A) The y-axis indicates for each gene set the mean Prg of all team predictions when considering response to 26 stimuli in SC3. The x-axis is Sg of gene set activation. In red are stimuli where Prg > Sg > 0. (B) The y-axis indicates for each protein the mean Prp of all team predictions when considering response to 26 stimuli in SC2. The x-axis is Sp for phosphoprotein activation. (C and D) Plots showing the percentage of teams where Prg > Sg (C) and Prp > Sp gene sets and phosphoproteins are ordered by number of active calls, indicated on top of each black dot
© Copyright Policy - creative-commons
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4325540&req=5

btu611-F4: Predictability versus species similarity for gene sets and phosphoproteins. (A) The y-axis indicates for each gene set the mean Prg of all team predictions when considering response to 26 stimuli in SC3. The x-axis is Sg of gene set activation. In red are stimuli where Prg > Sg > 0. (B) The y-axis indicates for each protein the mean Prp of all team predictions when considering response to 26 stimuli in SC2. The x-axis is Sp for phosphoprotein activation. (C and D) Plots showing the percentage of teams where Prg > Sg (C) and Prp > Sp gene sets and phosphoproteins are ordered by number of active calls, indicated on top of each black dot

Mentions: Figure 4A and B show the mean Prp and Prg for all participants plotted against Sp and Sg, respectively, based on activation in stimuli. A total of 49 of 246 gene sets were predicted better than expected by Sg (Prg > Sg > 0, Fig. 4A). Prediction performance per phosphoprotein Prp showed a ribosomal protein S6 kinase (KS6A1) and mitogen-activated protein kinases (MK09 and MP2K6) were predicted better than expected by Sp (Fig. 4B). Although aggregating all teams’ results did not yield a better overall prediction for protein phosphorylation activity, the aggregate of the five best teams performed better than individual predictions (Supplementary Fig. S5B). The high correlation between Prp and Sp (PCC = 0.71, P-value < 0.0087) reveals that most of the pathways defined by the protein phosphorylation activation were predicted with an accuracy expected by species similarity. We observed a similar situation for gene set activation prediction, with a lower but still significant correlation (PCC = 0.38, P-value < 1e-6). These results again suggested a slightly higher predictability in the protein phosphorylation data, though the prediction space was smaller. The individual team values for Prp and found that participants’ predictions were well translated for 71 of 176 active gene sets and for 8 of 16 phosphorylated proteins (Fig. 4A and B). Overall a higher percentage of teams performed better than species similarity when predicting protein phosphorylation activation (55%) versus predicting gene set activation (41%; see Fig. 4C and D). Nevertheless, when looking specifically at the set of active gene set and stimulus pairs (n = 560), 30% were correctly predicted by at least three teams (Fig. 5A), and in contrast to phosphorylation activation, six of seven teams in SC3 were better at globally translating the effects of stimuli than gene set activity (Fig. 5B).Fig. 4.


Understanding the limits of animal models as predictors of human biology: lessons learned from the sbv IMPROVER Species Translation Challenge.

Rhrissorrakrai K, Belcastro V, Bilal E, Norel R, Poussin C, Mathis C, Dulize RH, Ivanov NV, Alexopoulos L, Rice JJ, Peitsch MC, Stolovitzky G, Meyer P, Hoeng J - Bioinformatics (2014)

Predictability versus species similarity for gene sets and phosphoproteins. (A) The y-axis indicates for each gene set the mean Prg of all team predictions when considering response to 26 stimuli in SC3. The x-axis is Sg of gene set activation. In red are stimuli where Prg > Sg > 0. (B) The y-axis indicates for each protein the mean Prp of all team predictions when considering response to 26 stimuli in SC2. The x-axis is Sp for phosphoprotein activation. (C and D) Plots showing the percentage of teams where Prg > Sg (C) and Prp > Sp gene sets and phosphoproteins are ordered by number of active calls, indicated on top of each black dot
© Copyright Policy - creative-commons
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4325540&req=5

btu611-F4: Predictability versus species similarity for gene sets and phosphoproteins. (A) The y-axis indicates for each gene set the mean Prg of all team predictions when considering response to 26 stimuli in SC3. The x-axis is Sg of gene set activation. In red are stimuli where Prg > Sg > 0. (B) The y-axis indicates for each protein the mean Prp of all team predictions when considering response to 26 stimuli in SC2. The x-axis is Sp for phosphoprotein activation. (C and D) Plots showing the percentage of teams where Prg > Sg (C) and Prp > Sp gene sets and phosphoproteins are ordered by number of active calls, indicated on top of each black dot
Mentions: Figure 4A and B show the mean Prp and Prg for all participants plotted against Sp and Sg, respectively, based on activation in stimuli. A total of 49 of 246 gene sets were predicted better than expected by Sg (Prg > Sg > 0, Fig. 4A). Prediction performance per phosphoprotein Prp showed a ribosomal protein S6 kinase (KS6A1) and mitogen-activated protein kinases (MK09 and MP2K6) were predicted better than expected by Sp (Fig. 4B). Although aggregating all teams’ results did not yield a better overall prediction for protein phosphorylation activity, the aggregate of the five best teams performed better than individual predictions (Supplementary Fig. S5B). The high correlation between Prp and Sp (PCC = 0.71, P-value < 0.0087) reveals that most of the pathways defined by the protein phosphorylation activation were predicted with an accuracy expected by species similarity. We observed a similar situation for gene set activation prediction, with a lower but still significant correlation (PCC = 0.38, P-value < 1e-6). These results again suggested a slightly higher predictability in the protein phosphorylation data, though the prediction space was smaller. The individual team values for Prp and found that participants’ predictions were well translated for 71 of 176 active gene sets and for 8 of 16 phosphorylated proteins (Fig. 4A and B). Overall a higher percentage of teams performed better than species similarity when predicting protein phosphorylation activation (55%) versus predicting gene set activation (41%; see Fig. 4C and D). Nevertheless, when looking specifically at the set of active gene set and stimulus pairs (n = 560), 30% were correctly predicted by at least three teams (Fig. 5A), and in contrast to phosphorylation activation, six of seven teams in SC3 were better at globally translating the effects of stimuli than gene set activity (Fig. 5B).Fig. 4.

Bottom Line: Participating teams submitted 49 different solutions across the sub-challenges, two-thirds of which were statistically significantly better than random.Additionally, similar computational methods were found to range widely in their performance within the same challenge, and no single method emerged as a clear winner across all sub-challenges.Finally, computational methods were able to effectively translate some specific stimuli and biological processes in the lung epithelial system, such as DNA synthesis, cytoskeleton and extracellular matrix, translation, immune/inflammation and growth factor/proliferation pathways, better than the expected response similarity between species. pmeyerr@us.ibm.com or Julia.Hoeng@pmi.com Supplementary data are available at Bioinformatics online.

View Article: PubMed Central - PubMed

Affiliation: IBM T.J. Watson Research Center, Computational Biology Center, Yorktown Heights, NY 10003, USA, Philip Morris International R&D, Philip Morris Products S.A., 2000 Neuchâtel, Switzerland, Telethon Institute of Genetics and Medicine, Via Pietro Castellino, 111, 80131 Naples, Italy, ProtATonce Ltd, Scientific Park Lefkippos, Patriarchou Grigoriou & Neapoleos 15343 Ag. Paraskevi, Attiki and National Technical University of Athens, Heroon Polytechniou 9, Zografou 15780, Greece.

Show MeSH
Related in: MedlinePlus