Limits...
Back-propagation operation for analog neural network hardware with synapse components having hysteresis characteristics.

Ueda M, Nishitani Y, Kaneko Y, Omote A - PLoS ONE (2014)

Bottom Line: We examined the probability that the error decreased to a designated value within a predetermined loop number.The ferroelectric has the characteristics that the magnitude of polarization does not become smaller when voltages having the same polarity are applied.These characteristics greatly improved the probability even if the learning rate was small, if the magnitude of the dispersion is adequate.

View Article: PubMed Central - PubMed

Affiliation: Advanced Research Division, Panasonic Corporation, Soraku, Kyoto, Japan.

ABSTRACT
To realize an analog artificial neural network hardware, the circuit element for synapse function is important because the number of synapse elements is much larger than that of neuron elements. One of the candidates for this synapse element is a ferroelectric memristor. This device functions as a voltage controllable variable resistor, which can be applied to a synapse weight. However, its conductance shows hysteresis characteristics and dispersion to the input voltage. Therefore, the conductance values vary according to the history of the height and the width of the applied pulse voltage. Due to the difficulty of controlling the accurate conductance, it is not easy to apply the back-propagation learning algorithm to the neural network hardware having memristor synapses. To solve this problem, we proposed and simulated a learning operation procedure as follows. Employing a weight perturbation technique, we derived the error change. When the error reduced, the next pulse voltage was updated according to the back-propagation learning algorithm. If the error increased the amplitude of the next voltage pulse was set in such way as to cause similar memristor conductance but in the opposite voltage scanning direction. By this operation, we could eliminate the hysteresis and confirmed that the simulation of the learning operation converged. We also adopted conductance dispersion numerically in the simulation. We examined the probability that the error decreased to a designated value within a predetermined loop number. The ferroelectric has the characteristics that the magnitude of polarization does not become smaller when voltages having the same polarity are applied. These characteristics greatly improved the probability even if the learning rate was small, if the magnitude of the dispersion is adequate. Because the dispersion of analog circuit elements is inevitable, this learning operation procedure is useful for analog neural network hardware.

Show MeSH
Schematic of a neuron circuit and its input-output characteristics.The neuron circuit is based on an op-amp adder circuit as shown in (a). Synapse circuits are constructed with a FeMEM. To realize positive and negative synapse weights, we adopted excitatory and inhibitory synapses. The inner potential (u) is calculated according to (3). The relation between u and output voltage (Vout) of the op-amp resembles the input-output characteristics of a sigmoidal function as shown in (b).
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4231062&req=5

pone-0112659-g002: Schematic of a neuron circuit and its input-output characteristics.The neuron circuit is based on an op-amp adder circuit as shown in (a). Synapse circuits are constructed with a FeMEM. To realize positive and negative synapse weights, we adopted excitatory and inhibitory synapses. The inner potential (u) is calculated according to (3). The relation between u and output voltage (Vout) of the op-amp resembles the input-output characteristics of a sigmoidal function as shown in (b).

Mentions: To realize an analog neuron device, we examined a circuit based on an operational amplifier (op-amp) adder circuit. Using FeMEMs and an op-amp, a neuron circuit was constructed as shown in Figure 2(a). RF is a fixed resistance, whose conductance is GR. To achieve a synapse function using an FeMEM, the synaptic circuit modules were devised that consist of inhibitory/excitatory synapse pairs. As the op-amp adder circuit is an inverting amplifier circuit, the inhibitory pairs receive raw input directly and the excitatory ones receive inverted copies of the raw input voltage via a unity gain inverting amplifier. Although this synapse circuit construction needs two FeMEMs, a highly functional neuron circuit can be realized, because the modulation of synapse weight is easier to control individually with two FeMEMs. Here, we denote the channel conductance of FeMEM as GF. Also, we denote GF for the excitatory synapse as GE(i) and GF for the inhibitory synapse as GI(i). The sum of amplified voltages, or the inner potential (u), is calculated as


Back-propagation operation for analog neural network hardware with synapse components having hysteresis characteristics.

Ueda M, Nishitani Y, Kaneko Y, Omote A - PLoS ONE (2014)

Schematic of a neuron circuit and its input-output characteristics.The neuron circuit is based on an op-amp adder circuit as shown in (a). Synapse circuits are constructed with a FeMEM. To realize positive and negative synapse weights, we adopted excitatory and inhibitory synapses. The inner potential (u) is calculated according to (3). The relation between u and output voltage (Vout) of the op-amp resembles the input-output characteristics of a sigmoidal function as shown in (b).
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4231062&req=5

pone-0112659-g002: Schematic of a neuron circuit and its input-output characteristics.The neuron circuit is based on an op-amp adder circuit as shown in (a). Synapse circuits are constructed with a FeMEM. To realize positive and negative synapse weights, we adopted excitatory and inhibitory synapses. The inner potential (u) is calculated according to (3). The relation between u and output voltage (Vout) of the op-amp resembles the input-output characteristics of a sigmoidal function as shown in (b).
Mentions: To realize an analog neuron device, we examined a circuit based on an operational amplifier (op-amp) adder circuit. Using FeMEMs and an op-amp, a neuron circuit was constructed as shown in Figure 2(a). RF is a fixed resistance, whose conductance is GR. To achieve a synapse function using an FeMEM, the synaptic circuit modules were devised that consist of inhibitory/excitatory synapse pairs. As the op-amp adder circuit is an inverting amplifier circuit, the inhibitory pairs receive raw input directly and the excitatory ones receive inverted copies of the raw input voltage via a unity gain inverting amplifier. Although this synapse circuit construction needs two FeMEMs, a highly functional neuron circuit can be realized, because the modulation of synapse weight is easier to control individually with two FeMEMs. Here, we denote the channel conductance of FeMEM as GF. Also, we denote GF for the excitatory synapse as GE(i) and GF for the inhibitory synapse as GI(i). The sum of amplified voltages, or the inner potential (u), is calculated as

Bottom Line: We examined the probability that the error decreased to a designated value within a predetermined loop number.The ferroelectric has the characteristics that the magnitude of polarization does not become smaller when voltages having the same polarity are applied.These characteristics greatly improved the probability even if the learning rate was small, if the magnitude of the dispersion is adequate.

View Article: PubMed Central - PubMed

Affiliation: Advanced Research Division, Panasonic Corporation, Soraku, Kyoto, Japan.

ABSTRACT
To realize an analog artificial neural network hardware, the circuit element for synapse function is important because the number of synapse elements is much larger than that of neuron elements. One of the candidates for this synapse element is a ferroelectric memristor. This device functions as a voltage controllable variable resistor, which can be applied to a synapse weight. However, its conductance shows hysteresis characteristics and dispersion to the input voltage. Therefore, the conductance values vary according to the history of the height and the width of the applied pulse voltage. Due to the difficulty of controlling the accurate conductance, it is not easy to apply the back-propagation learning algorithm to the neural network hardware having memristor synapses. To solve this problem, we proposed and simulated a learning operation procedure as follows. Employing a weight perturbation technique, we derived the error change. When the error reduced, the next pulse voltage was updated according to the back-propagation learning algorithm. If the error increased the amplitude of the next voltage pulse was set in such way as to cause similar memristor conductance but in the opposite voltage scanning direction. By this operation, we could eliminate the hysteresis and confirmed that the simulation of the learning operation converged. We also adopted conductance dispersion numerically in the simulation. We examined the probability that the error decreased to a designated value within a predetermined loop number. The ferroelectric has the characteristics that the magnitude of polarization does not become smaller when voltages having the same polarity are applied. These characteristics greatly improved the probability even if the learning rate was small, if the magnitude of the dispersion is adequate. Because the dispersion of analog circuit elements is inevitable, this learning operation procedure is useful for analog neural network hardware.

Show MeSH