Limits...
A unified framework for spiking and gap-junction interactions in distributed neuronal network simulations.

Hahne J, Helias M, Kunkel S, Igarashi J, Bolten M, Frommer A, Diesmann M - Front Neuroinform (2015)

Bottom Line: This approach is well-suited for simulations that employ only chemical synapses but it has so far impeded the incorporation of gap-junction models, which require instantaneous neuronal interactions.To show that the unified framework for gap-junction and spiking interactions achieves high performance and delivers high accuracy in the presence of gap junctions, we present benchmarks for workstations, clusters, and supercomputers.Finally, we discuss limitations of the novel technology.

View Article: PubMed Central - PubMed

Affiliation: Department of Mathematics and Science, Bergische Universität Wuppertal Wuppertal, Germany.

ABSTRACT
Contemporary simulators for networks of point and few-compartment model neurons come with a plethora of ready-to-use neuron and synapse models and support complex network topologies. Recent technological advancements have broadened the spectrum of application further to the efficient simulation of brain-scale networks on supercomputers. In distributed network simulations the amount of spike data that accrues per millisecond and process is typically low, such that a common optimization strategy is to communicate spikes at relatively long intervals, where the upper limit is given by the shortest synaptic transmission delay in the network. This approach is well-suited for simulations that employ only chemical synapses but it has so far impeded the incorporation of gap-junction models, which require instantaneous neuronal interactions. Here, we present a numerical algorithm based on a waveform-relaxation technique which allows for network simulations with gap junctions in a way that is compatible with the delayed communication strategy. Using a reference implementation in the NEST simulator, we demonstrate that the algorithm and the required data structures can be smoothly integrated with existing code such that they complement the infrastructure for spiking connections. To show that the unified framework for gap-junction and spiking interactions achieves high performance and delivers high accuracy in the presence of gap junctions, we present benchmarks for workstations, clusters, and supercomputers. Finally, we discuss limitations of the novel technology.

No MeSH data available.


Related in: MedlinePlus

Costs of the gap-junction dynamics. Open symbols show the results with h-step communication ( = h) while filled symbols show the results with the original NEST communication scheme ( = dmin, here dmin = 1 ms). The solid curves with triangles indicate the simulation time Tsim in the absence of gap junctions. The corresponding darker blue curves with asterisks show the ratio ρ of Tsim with and without gap junctions, while gray curves with asterisks show the difference Tgap of both simulation times. Simulations represent 50 ms of biological time for (A,B) and 100 ms for (C) at a step size of h = 0.05 ms. All simulations use only a single iteration per time interval. (A) Weak scaling of Test case 1b on JUQUEEN with N = 185·VP neurons. (B) Strong scaling of Test case 1b on JUQUEEN with N = 185·16384 = 3,031,040 neurons. (C) Strong scaling of Test case 1b run on the shared memory cluster node with N = 100,000 neurons.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4563270&req=5

Figure 13: Costs of the gap-junction dynamics. Open symbols show the results with h-step communication ( = h) while filled symbols show the results with the original NEST communication scheme ( = dmin, here dmin = 1 ms). The solid curves with triangles indicate the simulation time Tsim in the absence of gap junctions. The corresponding darker blue curves with asterisks show the ratio ρ of Tsim with and without gap junctions, while gray curves with asterisks show the difference Tgap of both simulation times. Simulations represent 50 ms of biological time for (A,B) and 100 ms for (C) at a step size of h = 0.05 ms. All simulations use only a single iteration per time interval. (A) Weak scaling of Test case 1b on JUQUEEN with N = 185·VP neurons. (B) Strong scaling of Test case 1b on JUQUEEN with N = 185·16384 = 3,031,040 neurons. (C) Strong scaling of Test case 1b run on the shared memory cluster node with N = 100,000 neurons.

Mentions: Figure 13 investigates the slowdown due to gap-junction dynamics. This is done by simulating Test case 1b with a single iteration per time interval. The obtained results are then compared to the run time of a simulation without gap junctions but otherwise identical setup. This way the difference of the two run times Tgap can be interpreted as the time required for the additional computational load and communication. Figure 13A is a weak-scaling scenario. It demonstrates that the scalability of the method is impaired by the additional communication. Despite the constant number of neurons per virtual process and constant MPI send-buffer size the run time increases substantially. This is due to the increasing total number of neurons, which has an effect on the MPI receive buffer size and thereby on the communication time. Figure 13B studies the same setup in strong scaling with N ≈ 3 · 107 neurons. In this scenario the receive buffer size is constant, while the size of the send buffer shrinks with increasing number of virtual processes. Here the additional time required for MPI communication is almost constant. Tgap decreases at first and then stagnates for more than 1024 virtual processes. The saturation is explained by the additional MPI-communication, which constitutes the major contribution to Tgap in this setup. As the simulation without gap junctions uses exactly the same pattern of MPI communication this is not an issue of latency but an issue of bandwidth. The initial decrease is due to the parallelization of the gap-junction dynamics: the computations on the single-neuron level, like the handling of incoming gap events, the calculation of the interpolation coefficients and their central storage in the scheduler is parallelized. In conclusion the additional time required by simulations with gap junctions on JUQUEEN is determined by the total number of neurons N. As the increase in run time is dominated by MPI bandwidth it cannot be eliminated by using more virtual processes VP. Therefore, it is advisable to use as few compute nodes as possible. In this setting the communication required for gap junctions increases the simulation time of one iteration for a network of N ≈ 3 · 107 neurons by a factor of ρ = 2.5. One can multiply this factor ρ from Figure 13 with the average number of iterations ıh, respectively ıdmin to receive an estimate of the overall increase in run time. Figure 13C shows a strong scaling scenario for a smaller network with N = 100,000 neurons simulated on a shared memory compute node. This setup differs from the one in panels A and B as the parallelization is implemented by OpenMP and no MPI communication is needed. Here the impact of additional virtual processes on ρ is more moderate. ρ increases from ρ = 2 for 2 threads up to ρ = 3 for 48 threads for the case where communication takes place in intervals of the minimal delay. The scalability of NEST is preserved and the time for a single iteration per time interval decreases from 1366 s with 1 thread to 56 s with 48 threads. In contrast to Figure 13B the additional time Tgap is not dominated by a constant overhead and decreases due to parallelization of the gap-junction dynamics. In the case of h-step communication, however, again a limit of scaling is observed. The limit is not dominated by the communication between threads but due to the serial component of event delivery in NEST; all threads inspect all incoming events.


A unified framework for spiking and gap-junction interactions in distributed neuronal network simulations.

Hahne J, Helias M, Kunkel S, Igarashi J, Bolten M, Frommer A, Diesmann M - Front Neuroinform (2015)

Costs of the gap-junction dynamics. Open symbols show the results with h-step communication ( = h) while filled symbols show the results with the original NEST communication scheme ( = dmin, here dmin = 1 ms). The solid curves with triangles indicate the simulation time Tsim in the absence of gap junctions. The corresponding darker blue curves with asterisks show the ratio ρ of Tsim with and without gap junctions, while gray curves with asterisks show the difference Tgap of both simulation times. Simulations represent 50 ms of biological time for (A,B) and 100 ms for (C) at a step size of h = 0.05 ms. All simulations use only a single iteration per time interval. (A) Weak scaling of Test case 1b on JUQUEEN with N = 185·VP neurons. (B) Strong scaling of Test case 1b on JUQUEEN with N = 185·16384 = 3,031,040 neurons. (C) Strong scaling of Test case 1b run on the shared memory cluster node with N = 100,000 neurons.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4563270&req=5

Figure 13: Costs of the gap-junction dynamics. Open symbols show the results with h-step communication ( = h) while filled symbols show the results with the original NEST communication scheme ( = dmin, here dmin = 1 ms). The solid curves with triangles indicate the simulation time Tsim in the absence of gap junctions. The corresponding darker blue curves with asterisks show the ratio ρ of Tsim with and without gap junctions, while gray curves with asterisks show the difference Tgap of both simulation times. Simulations represent 50 ms of biological time for (A,B) and 100 ms for (C) at a step size of h = 0.05 ms. All simulations use only a single iteration per time interval. (A) Weak scaling of Test case 1b on JUQUEEN with N = 185·VP neurons. (B) Strong scaling of Test case 1b on JUQUEEN with N = 185·16384 = 3,031,040 neurons. (C) Strong scaling of Test case 1b run on the shared memory cluster node with N = 100,000 neurons.
Mentions: Figure 13 investigates the slowdown due to gap-junction dynamics. This is done by simulating Test case 1b with a single iteration per time interval. The obtained results are then compared to the run time of a simulation without gap junctions but otherwise identical setup. This way the difference of the two run times Tgap can be interpreted as the time required for the additional computational load and communication. Figure 13A is a weak-scaling scenario. It demonstrates that the scalability of the method is impaired by the additional communication. Despite the constant number of neurons per virtual process and constant MPI send-buffer size the run time increases substantially. This is due to the increasing total number of neurons, which has an effect on the MPI receive buffer size and thereby on the communication time. Figure 13B studies the same setup in strong scaling with N ≈ 3 · 107 neurons. In this scenario the receive buffer size is constant, while the size of the send buffer shrinks with increasing number of virtual processes. Here the additional time required for MPI communication is almost constant. Tgap decreases at first and then stagnates for more than 1024 virtual processes. The saturation is explained by the additional MPI-communication, which constitutes the major contribution to Tgap in this setup. As the simulation without gap junctions uses exactly the same pattern of MPI communication this is not an issue of latency but an issue of bandwidth. The initial decrease is due to the parallelization of the gap-junction dynamics: the computations on the single-neuron level, like the handling of incoming gap events, the calculation of the interpolation coefficients and their central storage in the scheduler is parallelized. In conclusion the additional time required by simulations with gap junctions on JUQUEEN is determined by the total number of neurons N. As the increase in run time is dominated by MPI bandwidth it cannot be eliminated by using more virtual processes VP. Therefore, it is advisable to use as few compute nodes as possible. In this setting the communication required for gap junctions increases the simulation time of one iteration for a network of N ≈ 3 · 107 neurons by a factor of ρ = 2.5. One can multiply this factor ρ from Figure 13 with the average number of iterations ıh, respectively ıdmin to receive an estimate of the overall increase in run time. Figure 13C shows a strong scaling scenario for a smaller network with N = 100,000 neurons simulated on a shared memory compute node. This setup differs from the one in panels A and B as the parallelization is implemented by OpenMP and no MPI communication is needed. Here the impact of additional virtual processes on ρ is more moderate. ρ increases from ρ = 2 for 2 threads up to ρ = 3 for 48 threads for the case where communication takes place in intervals of the minimal delay. The scalability of NEST is preserved and the time for a single iteration per time interval decreases from 1366 s with 1 thread to 56 s with 48 threads. In contrast to Figure 13B the additional time Tgap is not dominated by a constant overhead and decreases due to parallelization of the gap-junction dynamics. In the case of h-step communication, however, again a limit of scaling is observed. The limit is not dominated by the communication between threads but due to the serial component of event delivery in NEST; all threads inspect all incoming events.

Bottom Line: This approach is well-suited for simulations that employ only chemical synapses but it has so far impeded the incorporation of gap-junction models, which require instantaneous neuronal interactions.To show that the unified framework for gap-junction and spiking interactions achieves high performance and delivers high accuracy in the presence of gap junctions, we present benchmarks for workstations, clusters, and supercomputers.Finally, we discuss limitations of the novel technology.

View Article: PubMed Central - PubMed

Affiliation: Department of Mathematics and Science, Bergische Universität Wuppertal Wuppertal, Germany.

ABSTRACT
Contemporary simulators for networks of point and few-compartment model neurons come with a plethora of ready-to-use neuron and synapse models and support complex network topologies. Recent technological advancements have broadened the spectrum of application further to the efficient simulation of brain-scale networks on supercomputers. In distributed network simulations the amount of spike data that accrues per millisecond and process is typically low, such that a common optimization strategy is to communicate spikes at relatively long intervals, where the upper limit is given by the shortest synaptic transmission delay in the network. This approach is well-suited for simulations that employ only chemical synapses but it has so far impeded the incorporation of gap-junction models, which require instantaneous neuronal interactions. Here, we present a numerical algorithm based on a waveform-relaxation technique which allows for network simulations with gap junctions in a way that is compatible with the delayed communication strategy. Using a reference implementation in the NEST simulator, we demonstrate that the algorithm and the required data structures can be smoothly integrated with existing code such that they complement the infrastructure for spiking connections. To show that the unified framework for gap-junction and spiking interactions achieves high performance and delivers high accuracy in the presence of gap junctions, we present benchmarks for workstations, clusters, and supercomputers. Finally, we discuss limitations of the novel technology.

No MeSH data available.


Related in: MedlinePlus