Limits...
Spartan: a comprehensive tool for understanding uncertainty in simulations of biological systems.

Alden K, Read M, Timmis J, Andrews PS, Veiga-Fernandes H, Coles M - PLoS Comput. Biol. (2013)

Bottom Line: Success requires the relationship between simulation and the real-world system to be established: substantial aspects of the biological system are typically unknown, and the abstract nature of simulation can complicate interpretation of in silico results in terms of the biology.The tools comprising spartan help identify which simulation results can be attributed to the dynamics of the modelled biological system, rather than artefacts of biological uncertainty or parametrisation, or simulation stochasticity.We demonstrate the power of spartan in providing critical insight into aspects of lymphoid tissue development in the small intestine through simulation.

View Article: PubMed Central - PubMed

Affiliation: Centre for Systems Biology, School of Biosciences, University of Birmingham, Birmingham, United Kingdom. k.j.alden@bham.ac.uk

ABSTRACT
Integrating computer simulation with conventional wet-lab research has proven to have much potential in furthering the understanding of biological systems. Success requires the relationship between simulation and the real-world system to be established: substantial aspects of the biological system are typically unknown, and the abstract nature of simulation can complicate interpretation of in silico results in terms of the biology. Here we present spartan (Simulation Parameter Analysis RToolkit ApplicatioN), a package of statistical techniques specifically designed to help researchers understand this relationship and provide novel biological insight. The tools comprising spartan help identify which simulation results can be attributed to the dynamics of the modelled biological system, rather than artefacts of biological uncertainty or parametrisation, or simulation stochasticity. Statistical analyses reveal the influence that pathways and components have on simulation behaviour, offering valuable biological insight into aspects of the system under study. We demonstrate the power of spartan in providing critical insight into aspects of lymphoid tissue development in the small intestine through simulation. Spartan is released under a GPLv2 license, implemented within the open source R statistical environment, and freely available from both the Comprehensive R Archive Network (CRAN) and http://www.cs.york.ac.uk/spartan. The techniques within the package can be applied to traditional ordinary or partial differential equation simulations as well as agent-based implementations. Manuals, comprehensive tutorials, and example simulation data upon which spartan can be applied are available from the website.

Show MeSH
Use of spartan to mitigate aleatory uncertainty in stochastic simulations.Spartan's consistency analysis technique that can determine the number of runs required to generate a representative result from a stochastic simulation. A, B, C: A-Test scores for sample sizes of 5, 50, and 300 runs respectively. D. Maximum A Test score for each simulation response over 20 result sets for all sample sizes analysed. Scores below 0.5 are assigned corresponding values above 0.5 as direction of effect is not important. The effect magnitude thresholds are indicated.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC3585389&req=5

pcbi-1002916-g001: Use of spartan to mitigate aleatory uncertainty in stochastic simulations.Spartan's consistency analysis technique that can determine the number of runs required to generate a representative result from a stochastic simulation. A, B, C: A-Test scores for sample sizes of 5, 50, and 300 runs respectively. D. Maximum A Test score for each simulation response over 20 result sets for all sample sizes analysed. Scores below 0.5 are assigned corresponding values above 0.5 as direction of effect is not important. The effect magnitude thresholds are indicated.

Mentions: To determine the number of simulation runs required to obtain a representative result, we analysed sample sizes of 1, 5, 50, 100, 300, and 500 runs. Parameter values were kept constant at their calibrated values. Each sample size is analysed in turn using the procedure described, with the generation of 20 subsets of each sample size. This analysis thus required 19,120 individual runs. The online tutorial examines the first five sample sizes. Spartan produces the plots in Figure 1, where Figures 1(a,b,c) show the A-Test scores for all simulation output responses in each of the 20 result sets, for 5, 50, and 100 samples respectively. Figure 1(d) shows the maximum A Test score for each simulation response over the 20 result sets, for all sample sizes analysed. The latter indicates that reducing the effect magnitude of aleatory uncertainty on simulation results to less than ‘small’ (the desired level) requires more than 300 samples when compiling results, thus a sample size of 500 was chosen. It is important to note however that this is specific to our simulation, and unlikely to apply in all cases where spartan is applied. A full analysis for this simulation is found in Patel et al [9]; the online tutorial and results in Figure 1 stop at 300 runs to ensure the tutorial data is of manageable size for download.


Spartan: a comprehensive tool for understanding uncertainty in simulations of biological systems.

Alden K, Read M, Timmis J, Andrews PS, Veiga-Fernandes H, Coles M - PLoS Comput. Biol. (2013)

Use of spartan to mitigate aleatory uncertainty in stochastic simulations.Spartan's consistency analysis technique that can determine the number of runs required to generate a representative result from a stochastic simulation. A, B, C: A-Test scores for sample sizes of 5, 50, and 300 runs respectively. D. Maximum A Test score for each simulation response over 20 result sets for all sample sizes analysed. Scores below 0.5 are assigned corresponding values above 0.5 as direction of effect is not important. The effect magnitude thresholds are indicated.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC3585389&req=5

pcbi-1002916-g001: Use of spartan to mitigate aleatory uncertainty in stochastic simulations.Spartan's consistency analysis technique that can determine the number of runs required to generate a representative result from a stochastic simulation. A, B, C: A-Test scores for sample sizes of 5, 50, and 300 runs respectively. D. Maximum A Test score for each simulation response over 20 result sets for all sample sizes analysed. Scores below 0.5 are assigned corresponding values above 0.5 as direction of effect is not important. The effect magnitude thresholds are indicated.
Mentions: To determine the number of simulation runs required to obtain a representative result, we analysed sample sizes of 1, 5, 50, 100, 300, and 500 runs. Parameter values were kept constant at their calibrated values. Each sample size is analysed in turn using the procedure described, with the generation of 20 subsets of each sample size. This analysis thus required 19,120 individual runs. The online tutorial examines the first five sample sizes. Spartan produces the plots in Figure 1, where Figures 1(a,b,c) show the A-Test scores for all simulation output responses in each of the 20 result sets, for 5, 50, and 100 samples respectively. Figure 1(d) shows the maximum A Test score for each simulation response over the 20 result sets, for all sample sizes analysed. The latter indicates that reducing the effect magnitude of aleatory uncertainty on simulation results to less than ‘small’ (the desired level) requires more than 300 samples when compiling results, thus a sample size of 500 was chosen. It is important to note however that this is specific to our simulation, and unlikely to apply in all cases where spartan is applied. A full analysis for this simulation is found in Patel et al [9]; the online tutorial and results in Figure 1 stop at 300 runs to ensure the tutorial data is of manageable size for download.

Bottom Line: Success requires the relationship between simulation and the real-world system to be established: substantial aspects of the biological system are typically unknown, and the abstract nature of simulation can complicate interpretation of in silico results in terms of the biology.The tools comprising spartan help identify which simulation results can be attributed to the dynamics of the modelled biological system, rather than artefacts of biological uncertainty or parametrisation, or simulation stochasticity.We demonstrate the power of spartan in providing critical insight into aspects of lymphoid tissue development in the small intestine through simulation.

View Article: PubMed Central - PubMed

Affiliation: Centre for Systems Biology, School of Biosciences, University of Birmingham, Birmingham, United Kingdom. k.j.alden@bham.ac.uk

ABSTRACT
Integrating computer simulation with conventional wet-lab research has proven to have much potential in furthering the understanding of biological systems. Success requires the relationship between simulation and the real-world system to be established: substantial aspects of the biological system are typically unknown, and the abstract nature of simulation can complicate interpretation of in silico results in terms of the biology. Here we present spartan (Simulation Parameter Analysis RToolkit ApplicatioN), a package of statistical techniques specifically designed to help researchers understand this relationship and provide novel biological insight. The tools comprising spartan help identify which simulation results can be attributed to the dynamics of the modelled biological system, rather than artefacts of biological uncertainty or parametrisation, or simulation stochasticity. Statistical analyses reveal the influence that pathways and components have on simulation behaviour, offering valuable biological insight into aspects of the system under study. We demonstrate the power of spartan in providing critical insight into aspects of lymphoid tissue development in the small intestine through simulation. Spartan is released under a GPLv2 license, implemented within the open source R statistical environment, and freely available from both the Comprehensive R Archive Network (CRAN) and http://www.cs.york.ac.uk/spartan. The techniques within the package can be applied to traditional ordinary or partial differential equation simulations as well as agent-based implementations. Manuals, comprehensive tutorials, and example simulation data upon which spartan can be applied are available from the website.

Show MeSH