Limits...
Parallel Representation of Value-Based and Finite State-Based Strategies in the Ventral and Dorsal Striatum.

Ito M, Doya K - PLoS Comput. Biol. (2015)

Bottom Line: Analyses of neural activity recorded from the dorsolateral striatum (DLS), the dorsomedial striatum (DMS), and the ventral striatum (VS) identified significant fractions of neurons in all three subareas for which activities were correlated with individual states of the finite state-based strategy.The signal of internal states at the time of choice was found in DMS, and for clusters of states was found in VS.These results suggest that both the value-based strategy and the finite state-based strategy are implemented in the striatum.

View Article: PubMed Central - PubMed

Affiliation: Okinawa Institute of Science and Technology Graduate University, Onna-son Okinawa, Japan.

ABSTRACT
Previous theoretical studies of animal and human behavioral learning have focused on the dichotomy of the value-based strategy using action value functions to predict rewards and the model-based strategy using internal models to predict environmental states. However, animals and humans often take simple procedural behaviors, such as the "win-stay, lose-switch" strategy without explicit prediction of rewards or states. Here we consider another strategy, the finite state-based strategy, in which a subject selects an action depending on its discrete internal state and updates the state depending on the action chosen and the reward outcome. By analyzing choice behavior of rats in a free-choice task, we found that the finite state-based strategy fitted their behavioral choices more accurately than value-based and model-based strategies did. When fitted models were run autonomously with the same task, only the finite state-based strategy could reproduce the key feature of choice sequences. Analyses of neural activity recorded from the dorsolateral striatum (DLS), the dorsomedial striatum (DMS), and the ventral striatum (VS) identified significant fractions of neurons in all three subareas for which activities were correlated with individual states of the finite state-based strategy. The signal of internal states at the time of choice was found in DMS, and for clusters of states was found in VS. In addition, action values and state values of the value-based strategy were encoded in DMS and VS, respectively. These results suggest that both the value-based strategy and the finite state-based strategy are implemented in the striatum.

No MeSH data available.


Related in: MedlinePlus

Breakdowns of state-coding neurons shown in Fig 7E and 7F.(A, B), The proportion of neurons coding x(t) during the 500 ms before entry into the L/R hole, and x(t+1) during 500 ms after exit from L/R hole, respectively. The color for each state showing a significant proportion (p < 0.05, binominal test) corresponds to the color in the simplified diagram of the state transition in the FSA model with 8 states shown in (C). Populations with less than chance probabilities are shown in gray.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4631489&req=5

pcbi.1004540.g008: Breakdowns of state-coding neurons shown in Fig 7E and 7F.(A, B), The proportion of neurons coding x(t) during the 500 ms before entry into the L/R hole, and x(t+1) during 500 ms after exit from L/R hole, respectively. The color for each state showing a significant proportion (p < 0.05, binominal test) corresponds to the color in the simplified diagram of the state transition in the FSA model with 8 states shown in (C). Populations with less than chance probabilities are shown in gray.

Mentions: Interestingly, not all states were equally coded in the striatum (Fig 8). During action execution (Fig 8A), only the proportion of state-4- and state-5-coding neurons in DMS and VS (also state 6 and 8 in DMS) were statistically significant, and both states preceded an exploratory action in the keep-left and keep-right clusters (Fig 4C). After execution of an action and reward feedback (Fig 8B), representations of most subsequent states appeared in DLS and DMS, while representations of the same state x5, persisted in VS. Interestingly, states 2 and 7 are major transition targets from states 4 and 5, and these signals, especially, the signal of state 7, were prominent in DLS.


Parallel Representation of Value-Based and Finite State-Based Strategies in the Ventral and Dorsal Striatum.

Ito M, Doya K - PLoS Comput. Biol. (2015)

Breakdowns of state-coding neurons shown in Fig 7E and 7F.(A, B), The proportion of neurons coding x(t) during the 500 ms before entry into the L/R hole, and x(t+1) during 500 ms after exit from L/R hole, respectively. The color for each state showing a significant proportion (p < 0.05, binominal test) corresponds to the color in the simplified diagram of the state transition in the FSA model with 8 states shown in (C). Populations with less than chance probabilities are shown in gray.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4631489&req=5

pcbi.1004540.g008: Breakdowns of state-coding neurons shown in Fig 7E and 7F.(A, B), The proportion of neurons coding x(t) during the 500 ms before entry into the L/R hole, and x(t+1) during 500 ms after exit from L/R hole, respectively. The color for each state showing a significant proportion (p < 0.05, binominal test) corresponds to the color in the simplified diagram of the state transition in the FSA model with 8 states shown in (C). Populations with less than chance probabilities are shown in gray.
Mentions: Interestingly, not all states were equally coded in the striatum (Fig 8). During action execution (Fig 8A), only the proportion of state-4- and state-5-coding neurons in DMS and VS (also state 6 and 8 in DMS) were statistically significant, and both states preceded an exploratory action in the keep-left and keep-right clusters (Fig 4C). After execution of an action and reward feedback (Fig 8B), representations of most subsequent states appeared in DLS and DMS, while representations of the same state x5, persisted in VS. Interestingly, states 2 and 7 are major transition targets from states 4 and 5, and these signals, especially, the signal of state 7, were prominent in DLS.

Bottom Line: Analyses of neural activity recorded from the dorsolateral striatum (DLS), the dorsomedial striatum (DMS), and the ventral striatum (VS) identified significant fractions of neurons in all three subareas for which activities were correlated with individual states of the finite state-based strategy.The signal of internal states at the time of choice was found in DMS, and for clusters of states was found in VS.These results suggest that both the value-based strategy and the finite state-based strategy are implemented in the striatum.

View Article: PubMed Central - PubMed

Affiliation: Okinawa Institute of Science and Technology Graduate University, Onna-son Okinawa, Japan.

ABSTRACT
Previous theoretical studies of animal and human behavioral learning have focused on the dichotomy of the value-based strategy using action value functions to predict rewards and the model-based strategy using internal models to predict environmental states. However, animals and humans often take simple procedural behaviors, such as the "win-stay, lose-switch" strategy without explicit prediction of rewards or states. Here we consider another strategy, the finite state-based strategy, in which a subject selects an action depending on its discrete internal state and updates the state depending on the action chosen and the reward outcome. By analyzing choice behavior of rats in a free-choice task, we found that the finite state-based strategy fitted their behavioral choices more accurately than value-based and model-based strategies did. When fitted models were run autonomously with the same task, only the finite state-based strategy could reproduce the key feature of choice sequences. Analyses of neural activity recorded from the dorsolateral striatum (DLS), the dorsomedial striatum (DMS), and the ventral striatum (VS) identified significant fractions of neurons in all three subareas for which activities were correlated with individual states of the finite state-based strategy. The signal of internal states at the time of choice was found in DMS, and for clusters of states was found in VS. In addition, action values and state values of the value-based strategy were encoded in DMS and VS, respectively. These results suggest that both the value-based strategy and the finite state-based strategy are implemented in the striatum.

No MeSH data available.


Related in: MedlinePlus