Limits...
Network topology and parameter estimation: from experimental design methods to gene regulatory network kinetics using a community based approach.

Meyer P, Cokelaer T, Chandran D, Kim KH, Loh PR, Tucker G, Lipson M, Berger B, Kreutz C, Raue A, Steiert B, Timmer J, Bilal E - BMC Syst Biol (2014)

Bottom Line: While significant progress has been made, effective experimental strategies for parameter identification and for distinguishing among alternative network topologies remain unclear.We created an in silico test framework under which participants could probe a network with hidden parameters by requesting a range of experimental assays; results of these experiments were simulated according to a model of network dynamics only partially revealed to participants.Importantly, we found that aggregating independent parameter predictions and network topology across submissions creates a solution that can be better than the one from the best-performing submission.

View Article: PubMed Central - HTML - PubMed

ABSTRACT

Background: Accurate estimation of parameters of biochemical models is required to characterize the dynamics of molecular processes. This problem is intimately linked to identifying the most informative experiments for accomplishing such tasks. While significant progress has been made, effective experimental strategies for parameter identification and for distinguishing among alternative network topologies remain unclear. We approached these questions in an unbiased manner using a unique community-based approach in the context of the DREAM initiative (Dialogue for Reverse Engineering Assessment of Methods). We created an in silico test framework under which participants could probe a network with hidden parameters by requesting a range of experimental assays; results of these experiments were simulated according to a model of network dynamics only partially revealed to participants.

Results: We proposed two challenges; in the first, participants were given the topology and underlying biochemical structure of a 9-gene regulatory network and were asked to determine its parameter values. In the second challenge, participants were given an incomplete topology with 11 genes and asked to find three missing links in the model. In both challenges, a budget was provided to buy experimental data generated in silico with the model and mimicking the features of different common experimental techniques, such as microarrays and fluorescence microscopy. Data could be bought at any stage, allowing participants to implement an iterative loop of experiments and computation.

Conclusions: A total of 19 teams participated in this competition. The results suggest that the combination of state-of-the-art parameter estimation and a varied set of experimental methods using a few datasets, mostly fluorescence imaging data, can accurately determine parameters of biochemical models of gene regulation. However, the task is considerably more difficult if the gene network topology is not completely defined, as in challenge 2. Importantly, we found that aggregating independent parameter predictions and network topology across submissions creates a solution that can be better than the one from the best-performing submission.

Show MeSH

Related in: MedlinePlus

Scores of aggregated participant results. A. Protein concentrations of participants’ predictions (in blue) and the solution (green) are plotted against time for proteins p3, p5 and p8 under the perturbed conditions considered for scoring. B. Participant submissions are aggregated by averaging each protein concentration for individual time points, starting from the 2 best performing teams until all 12 teams are included. Each aggregated result is plotted in blue and the solution is plotted in green. C. Log scale distance to the solution of parameter predictions is plotted for participant teams ordered by rank (blue line) and geometric means of parameter predictions from teams ordered by number of aggregated teams following parameter distance rank (green line) or inverse rank order (red line). D. Log-scale distance to the solution of proteins p3, p5 and p8 under perturbed conditions is plotted for participant teams ordered by rank (blue line) and aggregated teams. Aggregations were computed for the predictions of the teams, ordered by number of aggregated teams ranging from 1 to 12, following prediction distance rank (green line) or inverse order (red line).
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC3927870&req=5

Figure 3: Scores of aggregated participant results. A. Protein concentrations of participants’ predictions (in blue) and the solution (green) are plotted against time for proteins p3, p5 and p8 under the perturbed conditions considered for scoring. B. Participant submissions are aggregated by averaging each protein concentration for individual time points, starting from the 2 best performing teams until all 12 teams are included. Each aggregated result is plotted in blue and the solution is plotted in green. C. Log scale distance to the solution of parameter predictions is plotted for participant teams ordered by rank (blue line) and geometric means of parameter predictions from teams ordered by number of aggregated teams following parameter distance rank (green line) or inverse rank order (red line). D. Log-scale distance to the solution of proteins p3, p5 and p8 under perturbed conditions is plotted for participant teams ordered by rank (blue line) and aggregated teams. Aggregations were computed for the predictions of the teams, ordered by number of aggregated teams ranging from 1 to 12, following prediction distance rank (green line) or inverse order (red line).

Mentions: For model 1, most participants’ time-course predictions of proteins p3, p5 and p8 are close to the solution (Figure 3A blue lines) but, as seen in other DREAM challenges [25,26], aggregated participant submissions are robust, as the prediction is close to the gold standard and ‘buffers’ outliers (Figure 3B blue lines). Predictions were aggregated by averaging each protein concentration for individual time-points starting from the best performing team, followed by averaging the first and second best performing teams, and so on until all 12 teams were included.


Network topology and parameter estimation: from experimental design methods to gene regulatory network kinetics using a community based approach.

Meyer P, Cokelaer T, Chandran D, Kim KH, Loh PR, Tucker G, Lipson M, Berger B, Kreutz C, Raue A, Steiert B, Timmer J, Bilal E - BMC Syst Biol (2014)

Scores of aggregated participant results. A. Protein concentrations of participants’ predictions (in blue) and the solution (green) are plotted against time for proteins p3, p5 and p8 under the perturbed conditions considered for scoring. B. Participant submissions are aggregated by averaging each protein concentration for individual time points, starting from the 2 best performing teams until all 12 teams are included. Each aggregated result is plotted in blue and the solution is plotted in green. C. Log scale distance to the solution of parameter predictions is plotted for participant teams ordered by rank (blue line) and geometric means of parameter predictions from teams ordered by number of aggregated teams following parameter distance rank (green line) or inverse rank order (red line). D. Log-scale distance to the solution of proteins p3, p5 and p8 under perturbed conditions is plotted for participant teams ordered by rank (blue line) and aggregated teams. Aggregations were computed for the predictions of the teams, ordered by number of aggregated teams ranging from 1 to 12, following prediction distance rank (green line) or inverse order (red line).
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC3927870&req=5

Figure 3: Scores of aggregated participant results. A. Protein concentrations of participants’ predictions (in blue) and the solution (green) are plotted against time for proteins p3, p5 and p8 under the perturbed conditions considered for scoring. B. Participant submissions are aggregated by averaging each protein concentration for individual time points, starting from the 2 best performing teams until all 12 teams are included. Each aggregated result is plotted in blue and the solution is plotted in green. C. Log scale distance to the solution of parameter predictions is plotted for participant teams ordered by rank (blue line) and geometric means of parameter predictions from teams ordered by number of aggregated teams following parameter distance rank (green line) or inverse rank order (red line). D. Log-scale distance to the solution of proteins p3, p5 and p8 under perturbed conditions is plotted for participant teams ordered by rank (blue line) and aggregated teams. Aggregations were computed for the predictions of the teams, ordered by number of aggregated teams ranging from 1 to 12, following prediction distance rank (green line) or inverse order (red line).
Mentions: For model 1, most participants’ time-course predictions of proteins p3, p5 and p8 are close to the solution (Figure 3A blue lines) but, as seen in other DREAM challenges [25,26], aggregated participant submissions are robust, as the prediction is close to the gold standard and ‘buffers’ outliers (Figure 3B blue lines). Predictions were aggregated by averaging each protein concentration for individual time-points starting from the best performing team, followed by averaging the first and second best performing teams, and so on until all 12 teams were included.

Bottom Line: While significant progress has been made, effective experimental strategies for parameter identification and for distinguishing among alternative network topologies remain unclear.We created an in silico test framework under which participants could probe a network with hidden parameters by requesting a range of experimental assays; results of these experiments were simulated according to a model of network dynamics only partially revealed to participants.Importantly, we found that aggregating independent parameter predictions and network topology across submissions creates a solution that can be better than the one from the best-performing submission.

View Article: PubMed Central - HTML - PubMed

ABSTRACT

Background: Accurate estimation of parameters of biochemical models is required to characterize the dynamics of molecular processes. This problem is intimately linked to identifying the most informative experiments for accomplishing such tasks. While significant progress has been made, effective experimental strategies for parameter identification and for distinguishing among alternative network topologies remain unclear. We approached these questions in an unbiased manner using a unique community-based approach in the context of the DREAM initiative (Dialogue for Reverse Engineering Assessment of Methods). We created an in silico test framework under which participants could probe a network with hidden parameters by requesting a range of experimental assays; results of these experiments were simulated according to a model of network dynamics only partially revealed to participants.

Results: We proposed two challenges; in the first, participants were given the topology and underlying biochemical structure of a 9-gene regulatory network and were asked to determine its parameter values. In the second challenge, participants were given an incomplete topology with 11 genes and asked to find three missing links in the model. In both challenges, a budget was provided to buy experimental data generated in silico with the model and mimicking the features of different common experimental techniques, such as microarrays and fluorescence microscopy. Data could be bought at any stage, allowing participants to implement an iterative loop of experiments and computation.

Conclusions: A total of 19 teams participated in this competition. The results suggest that the combination of state-of-the-art parameter estimation and a varied set of experimental methods using a few datasets, mostly fluorescence imaging data, can accurately determine parameters of biochemical models of gene regulation. However, the task is considerably more difficult if the gene network topology is not completely defined, as in challenge 2. Importantly, we found that aggregating independent parameter predictions and network topology across submissions creates a solution that can be better than the one from the best-performing submission.

Show MeSH
Related in: MedlinePlus