Limits...
Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments.

Amemori K, Gibb LG, Graybiel AM - Front Hum Neurosci (2011)

Bottom Line: We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways.Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals.Our modeling functionally unites the modular compartmental organization of the striatum with the direct-indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications.

View Article: PubMed Central - PubMed

Affiliation: McGovern Institute for Brain Research, Massachusetts Institute of Technology Cambridge, MA, USA.

ABSTRACT
We propose here that the modular organization of the striatum reflects a context-sensitive modular learning architecture in which clustered striosome-matrisome domains participate in modular reinforcement learning (RL). Based on anatomical and physiological evidence, it has been suggested that the modular organization of the striatum could represent a learning architecture. There is not, however, a coherent view of how such a learning architecture could relate to the organization of striatal outputs into the direct and indirect pathways of the basal ganglia, nor a clear formulation of how such a modular architecture relates to the RL functions attributed to the striatum. Here, we hypothesize that striosome-matrisome modules not only learn to bias behavior toward specific actions, as in standard RL, but also learn to assess their own relevance to the environmental context and modulate their own learning and activity on this basis. We further hypothesize that the contextual relevance or "responsibility" of modules is determined by errors in predictions of environmental features and that such responsibility is assigned by striosomes and conveyed to matrisomes via local circuit interneurons. To examine these hypotheses and to identify the general requirements for realizing this architecture in the nervous system, we developed a simple modular RL model. We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways. Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals. Our modeling functionally unites the modular compartmental organization of the striatum with the direct-indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications.

No MeSH data available.


Influences of input cortex and responsibility signaling on striatal matrix MSN activity in the network model. We model two neurons of the input region of the cortex, each representing information related to the value of action “a” or “b.” These neurons project, respectively, to a D1 and a D2 matrix MSN in each of two modules in the striatum. (A) Cortical and striatal activity (arbitrary units) for simulation using only positive responsibility signals. The effect of positive responsibility signals is labeled “↑ DA,” based on the possibility that they may involve an increase in phasic local striatal dopamine release (triggered by a decrease in local acetylcholine release). Responsibility is assigned to module B at time 50 and module A at time 200 (yellow boxes). Such responsibility signaling transiently increases D1 MSN activity and decreases D2 MSN activity. (B) Cortical and striatal activity for simulation using both positive and negative responsibility signals. The effect of negative responsibility signals is labeled “↑ ACh,” based on the possibility that they may involve an increase in local striatal acetylcholine release. Positive responsibility is again assigned to module B at time 50 and module A at time 200 (yellow boxes). Additionally, negative responsibility is assigned to module A at time 50 and module B at time 200 (gray boxes). Negative responsibility signaling transiently increases D2 MSN activity.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC3105240&req=5

Figure 6: Influences of input cortex and responsibility signaling on striatal matrix MSN activity in the network model. We model two neurons of the input region of the cortex, each representing information related to the value of action “a” or “b.” These neurons project, respectively, to a D1 and a D2 matrix MSN in each of two modules in the striatum. (A) Cortical and striatal activity (arbitrary units) for simulation using only positive responsibility signals. The effect of positive responsibility signals is labeled “↑ DA,” based on the possibility that they may involve an increase in phasic local striatal dopamine release (triggered by a decrease in local acetylcholine release). Responsibility is assigned to module B at time 50 and module A at time 200 (yellow boxes). Such responsibility signaling transiently increases D1 MSN activity and decreases D2 MSN activity. (B) Cortical and striatal activity for simulation using both positive and negative responsibility signals. The effect of negative responsibility signals is labeled “↑ ACh,” based on the possibility that they may involve an increase in local striatal acetylcholine release. Positive responsibility is again assigned to module B at time 50 and module A at time 200 (yellow boxes). Additionally, negative responsibility is assigned to module A at time 50 and module B at time 200 (gray boxes). Negative responsibility signaling transiently increases D2 MSN activity.

Mentions: The model striatum contains two modules: module A, containing the first half of the MSNs according to their index, and module B, containing the second half. Each module contains 150 pairs of D1 and D2 MSNs. We assume that the two modules are differentially modulated by different responsibility signals even while receiving the same input from the input cortex and the same phasic dopamine signal. To illustrate module selection by the model, we generate identical cortical activity patterns at two different times (Figure 6A, left). At the first time, striatal module B is influenced by a responsibility signal (Figure 6A, middle); at the second time, striatal module A is influenced instead (Figure 6A, right). Based on the possibility that these positive responsibility signals may involve an increase in phasic local striatal dopamine release resulting from a decrease in local acetylcholine release (see “Discussion.”; Rice and Cragg, 2004; Cragg, 2006), their effect is labeled “↑ DA.”


Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments.

Amemori K, Gibb LG, Graybiel AM - Front Hum Neurosci (2011)

Influences of input cortex and responsibility signaling on striatal matrix MSN activity in the network model. We model two neurons of the input region of the cortex, each representing information related to the value of action “a” or “b.” These neurons project, respectively, to a D1 and a D2 matrix MSN in each of two modules in the striatum. (A) Cortical and striatal activity (arbitrary units) for simulation using only positive responsibility signals. The effect of positive responsibility signals is labeled “↑ DA,” based on the possibility that they may involve an increase in phasic local striatal dopamine release (triggered by a decrease in local acetylcholine release). Responsibility is assigned to module B at time 50 and module A at time 200 (yellow boxes). Such responsibility signaling transiently increases D1 MSN activity and decreases D2 MSN activity. (B) Cortical and striatal activity for simulation using both positive and negative responsibility signals. The effect of negative responsibility signals is labeled “↑ ACh,” based on the possibility that they may involve an increase in local striatal acetylcholine release. Positive responsibility is again assigned to module B at time 50 and module A at time 200 (yellow boxes). Additionally, negative responsibility is assigned to module A at time 50 and module B at time 200 (gray boxes). Negative responsibility signaling transiently increases D2 MSN activity.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC3105240&req=5

Figure 6: Influences of input cortex and responsibility signaling on striatal matrix MSN activity in the network model. We model two neurons of the input region of the cortex, each representing information related to the value of action “a” or “b.” These neurons project, respectively, to a D1 and a D2 matrix MSN in each of two modules in the striatum. (A) Cortical and striatal activity (arbitrary units) for simulation using only positive responsibility signals. The effect of positive responsibility signals is labeled “↑ DA,” based on the possibility that they may involve an increase in phasic local striatal dopamine release (triggered by a decrease in local acetylcholine release). Responsibility is assigned to module B at time 50 and module A at time 200 (yellow boxes). Such responsibility signaling transiently increases D1 MSN activity and decreases D2 MSN activity. (B) Cortical and striatal activity for simulation using both positive and negative responsibility signals. The effect of negative responsibility signals is labeled “↑ ACh,” based on the possibility that they may involve an increase in local striatal acetylcholine release. Positive responsibility is again assigned to module B at time 50 and module A at time 200 (yellow boxes). Additionally, negative responsibility is assigned to module A at time 50 and module B at time 200 (gray boxes). Negative responsibility signaling transiently increases D2 MSN activity.
Mentions: The model striatum contains two modules: module A, containing the first half of the MSNs according to their index, and module B, containing the second half. Each module contains 150 pairs of D1 and D2 MSNs. We assume that the two modules are differentially modulated by different responsibility signals even while receiving the same input from the input cortex and the same phasic dopamine signal. To illustrate module selection by the model, we generate identical cortical activity patterns at two different times (Figure 6A, left). At the first time, striatal module B is influenced by a responsibility signal (Figure 6A, middle); at the second time, striatal module A is influenced instead (Figure 6A, right). Based on the possibility that these positive responsibility signals may involve an increase in phasic local striatal dopamine release resulting from a decrease in local acetylcholine release (see “Discussion.”; Rice and Cragg, 2004; Cragg, 2006), their effect is labeled “↑ DA.”

Bottom Line: We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways.Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals.Our modeling functionally unites the modular compartmental organization of the striatum with the direct-indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications.

View Article: PubMed Central - PubMed

Affiliation: McGovern Institute for Brain Research, Massachusetts Institute of Technology Cambridge, MA, USA.

ABSTRACT
We propose here that the modular organization of the striatum reflects a context-sensitive modular learning architecture in which clustered striosome-matrisome domains participate in modular reinforcement learning (RL). Based on anatomical and physiological evidence, it has been suggested that the modular organization of the striatum could represent a learning architecture. There is not, however, a coherent view of how such a learning architecture could relate to the organization of striatal outputs into the direct and indirect pathways of the basal ganglia, nor a clear formulation of how such a modular architecture relates to the RL functions attributed to the striatum. Here, we hypothesize that striosome-matrisome modules not only learn to bias behavior toward specific actions, as in standard RL, but also learn to assess their own relevance to the environmental context and modulate their own learning and activity on this basis. We further hypothesize that the contextual relevance or "responsibility" of modules is determined by errors in predictions of environmental features and that such responsibility is assigned by striosomes and conveyed to matrisomes via local circuit interneurons. To examine these hypotheses and to identify the general requirements for realizing this architecture in the nervous system, we developed a simple modular RL model. We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways. Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals. Our modeling functionally unites the modular compartmental organization of the striatum with the direct-indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications.

No MeSH data available.