Limits...
Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments.

Amemori K, Gibb LG, Graybiel AM - Front Hum Neurosci (2011)

Bottom Line: We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways.Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals.Our modeling functionally unites the modular compartmental organization of the striatum with the direct-indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications.

View Article: PubMed Central - PubMed

Affiliation: McGovern Institute for Brain Research, Massachusetts Institute of Technology Cambridge, MA, USA.

ABSTRACT
We propose here that the modular organization of the striatum reflects a context-sensitive modular learning architecture in which clustered striosome-matrisome domains participate in modular reinforcement learning (RL). Based on anatomical and physiological evidence, it has been suggested that the modular organization of the striatum could represent a learning architecture. There is not, however, a coherent view of how such a learning architecture could relate to the organization of striatal outputs into the direct and indirect pathways of the basal ganglia, nor a clear formulation of how such a modular architecture relates to the RL functions attributed to the striatum. Here, we hypothesize that striosome-matrisome modules not only learn to bias behavior toward specific actions, as in standard RL, but also learn to assess their own relevance to the environmental context and modulate their own learning and activity on this basis. We further hypothesize that the contextual relevance or "responsibility" of modules is determined by errors in predictions of environmental features and that such responsibility is assigned by striosomes and conveyed to matrisomes via local circuit interneurons. To examine these hypotheses and to identify the general requirements for realizing this architecture in the nervous system, we developed a simple modular RL model. We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways. Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals. Our modeling functionally unites the modular compartmental organization of the striatum with the direct-indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications.

No MeSH data available.


Schematic summary of proposed effects of striatal responsibility signals on module and action selection. Top: If the sets of actions influenced by module A are appropriate to the environmental context, its striosome (S) assigns high responsibility by sending a signal to adjacent matrisomes (M) via local circuit interneurons. This results in relatively low activity in the indirect pathway and high activity in the direct pathway, which permits the direct pathway to promote selection of an action. Bottom: If the sets of actions influenced by module B are inappropriate to the environmental context, its striosome assigns low responsibility. This results in relatively high activity in the indirect pathway, which suppresses the associated set of candidate actions (behavioral module).
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC3105240&req=5

Figure 8: Schematic summary of proposed effects of striatal responsibility signals on module and action selection. Top: If the sets of actions influenced by module A are appropriate to the environmental context, its striosome (S) assigns high responsibility by sending a signal to adjacent matrisomes (M) via local circuit interneurons. This results in relatively low activity in the indirect pathway and high activity in the direct pathway, which permits the direct pathway to promote selection of an action. Bottom: If the sets of actions influenced by module B are inappropriate to the environmental context, its striosome assigns low responsibility. This results in relatively high activity in the indirect pathway, which suppresses the associated set of candidate actions (behavioral module).

Mentions: The anatomical modularity of the striatum is a plausible substrate for a functional modularity of the kind expressed in our model (Figure 8). The two basic compartments of the striatum, striosomes, and matrix, are distinguished from each other on the basis of neurochemical markers, input and output connectivity with other brain structures and local connectivity (Graybiel and Ragsdale, 1978; Herkenham and Pert, 1981; Gerfen, 1984; Gerfen et al., 1987; Bolam et al., 1988; Ragsdale and Graybiel, 1988; Gimenez-Amaya and Graybiel, 1990; Eblen and Graybiel, 1995; Graybiel, 1995; Holt et al., 1997; Joel and Weiner, 2000; Saka et al., 2002; Mikula et al., 2009). The dendritic and axonal arborizations of the majority of MSNs are mostly, but not entirely, restricted to their compartment of origin (Penny et al., 1988; Kawaguchi et al., 1989; Walker et al., 1993); and striatal interneurons also tend to follow striosome–matrix divisions, notably with the cholinergic interneurons (putative tonically active neurons, TANs) and somatostatin-containing interneurons (putatively the LTS interneurons) tending to lie near the borders of striosomes, but having much of their arborizations in the matrix compartment (Graybiel et al., 1981, 1986; Gerfen, 1984; Chesselet and Graybiel, 1986; Penny et al., 1988; Kawaguchi, 1992; Aosaki et al., 1995; Kawaguchi et al., 1995; Kreitzer, 2009). Striosomes form a labyrinthine reticulum, as if to provide functional coverage throughout the volume of the striatum (Graybiel and Ragsdale, 1978; Graybiel, 1984; Groves et al., 1988; Mikula et al., 2009). Finally, the input and output connections of the matrix itself also have a modular organization in which inputs and outputs of the large matrix compartment are divided up into clustered domains called matrisomes (Gimenez-Amaya and Graybiel, 1991; Flaherty and Graybiel, 1994; Kincaid and Wilson, 1996; Parthasarathy and Graybiel, 1997). Thus, modularity appears to be a fundamental principle of the anatomical organization of the striatum.


Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments.

Amemori K, Gibb LG, Graybiel AM - Front Hum Neurosci (2011)

Schematic summary of proposed effects of striatal responsibility signals on module and action selection. Top: If the sets of actions influenced by module A are appropriate to the environmental context, its striosome (S) assigns high responsibility by sending a signal to adjacent matrisomes (M) via local circuit interneurons. This results in relatively low activity in the indirect pathway and high activity in the direct pathway, which permits the direct pathway to promote selection of an action. Bottom: If the sets of actions influenced by module B are inappropriate to the environmental context, its striosome assigns low responsibility. This results in relatively high activity in the indirect pathway, which suppresses the associated set of candidate actions (behavioral module).
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC3105240&req=5

Figure 8: Schematic summary of proposed effects of striatal responsibility signals on module and action selection. Top: If the sets of actions influenced by module A are appropriate to the environmental context, its striosome (S) assigns high responsibility by sending a signal to adjacent matrisomes (M) via local circuit interneurons. This results in relatively low activity in the indirect pathway and high activity in the direct pathway, which permits the direct pathway to promote selection of an action. Bottom: If the sets of actions influenced by module B are inappropriate to the environmental context, its striosome assigns low responsibility. This results in relatively high activity in the indirect pathway, which suppresses the associated set of candidate actions (behavioral module).
Mentions: The anatomical modularity of the striatum is a plausible substrate for a functional modularity of the kind expressed in our model (Figure 8). The two basic compartments of the striatum, striosomes, and matrix, are distinguished from each other on the basis of neurochemical markers, input and output connectivity with other brain structures and local connectivity (Graybiel and Ragsdale, 1978; Herkenham and Pert, 1981; Gerfen, 1984; Gerfen et al., 1987; Bolam et al., 1988; Ragsdale and Graybiel, 1988; Gimenez-Amaya and Graybiel, 1990; Eblen and Graybiel, 1995; Graybiel, 1995; Holt et al., 1997; Joel and Weiner, 2000; Saka et al., 2002; Mikula et al., 2009). The dendritic and axonal arborizations of the majority of MSNs are mostly, but not entirely, restricted to their compartment of origin (Penny et al., 1988; Kawaguchi et al., 1989; Walker et al., 1993); and striatal interneurons also tend to follow striosome–matrix divisions, notably with the cholinergic interneurons (putative tonically active neurons, TANs) and somatostatin-containing interneurons (putatively the LTS interneurons) tending to lie near the borders of striosomes, but having much of their arborizations in the matrix compartment (Graybiel et al., 1981, 1986; Gerfen, 1984; Chesselet and Graybiel, 1986; Penny et al., 1988; Kawaguchi, 1992; Aosaki et al., 1995; Kawaguchi et al., 1995; Kreitzer, 2009). Striosomes form a labyrinthine reticulum, as if to provide functional coverage throughout the volume of the striatum (Graybiel and Ragsdale, 1978; Graybiel, 1984; Groves et al., 1988; Mikula et al., 2009). Finally, the input and output connections of the matrix itself also have a modular organization in which inputs and outputs of the large matrix compartment are divided up into clustered domains called matrisomes (Gimenez-Amaya and Graybiel, 1991; Flaherty and Graybiel, 1994; Kincaid and Wilson, 1996; Parthasarathy and Graybiel, 1997). Thus, modularity appears to be a fundamental principle of the anatomical organization of the striatum.

Bottom Line: We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways.Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals.Our modeling functionally unites the modular compartmental organization of the striatum with the direct-indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications.

View Article: PubMed Central - PubMed

Affiliation: McGovern Institute for Brain Research, Massachusetts Institute of Technology Cambridge, MA, USA.

ABSTRACT
We propose here that the modular organization of the striatum reflects a context-sensitive modular learning architecture in which clustered striosome-matrisome domains participate in modular reinforcement learning (RL). Based on anatomical and physiological evidence, it has been suggested that the modular organization of the striatum could represent a learning architecture. There is not, however, a coherent view of how such a learning architecture could relate to the organization of striatal outputs into the direct and indirect pathways of the basal ganglia, nor a clear formulation of how such a modular architecture relates to the RL functions attributed to the striatum. Here, we hypothesize that striosome-matrisome modules not only learn to bias behavior toward specific actions, as in standard RL, but also learn to assess their own relevance to the environmental context and modulate their own learning and activity on this basis. We further hypothesize that the contextual relevance or "responsibility" of modules is determined by errors in predictions of environmental features and that such responsibility is assigned by striosomes and conveyed to matrisomes via local circuit interneurons. To examine these hypotheses and to identify the general requirements for realizing this architecture in the nervous system, we developed a simple modular RL model. We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways. Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals. Our modeling functionally unites the modular compartmental organization of the striatum with the direct-indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications.

No MeSH data available.