Limits...
Learning to control a brain-machine interface for reaching and grasping by primates.

Carmena JM, Lebedev MA, Crist RE, O'Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MA - PLoS Biol. (2003)

Bottom Line: Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance.Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move.Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.

View Article: PubMed Central - PubMed

Affiliation: Department of Neurobiology, Duke University, Durham, North Carolina, USA.

ABSTRACT
Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain-machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.

Show MeSH
Ensemble Encoding of Gripping Force, Plasticity of Directional Tuning, and Neuronal Contribution to Model Performance during Learning to Control the BMIc for Reaching and Grasping(A) Perievent time histograms (PETHs) in task 2 for the neuronal population sampled in monkey 1. The plots on top are color-coded (red shows high values; blue, low values). Each horizontal row represents a PETH for a single-neuron or multiunit activity. PETHs have been normalized by subtracting the mean and then dividing by the standard deviation. PETHs are aligned on the gripping force onset (crossing a threshold). Plots at the bottom show the corresponding average traces of gripping force. Note the general similarity of PETHs in pole (left) and brain (right) control in this relatively easy task. Cortical location of neurons is indicated by the bar on the top left. Note the distinct pattern of activation for different areas.(B) Changes in the mean contribution of neurons from different cortical areas to model predictions during training of monkey 1 in task 2.(C) Increases in directional tuning for six cortical areas during training in pole control in task 3.(D and E) Increases in neuronal contribution to linear models predicting hand position (blue), hand velocity (red), and gripping force (black) during learning task 3 in both monkeys.(F and G) Representative robot trajectories and gripping force profiles in an advanced stage of training in task 3 during both pole and brain control. The bottom graphs show trajectories and the amount of the gripping force developed during grasping each virtual object. The dotted vertical lines in the panels indicate the end of reach phase and the beginning of grasp phase. Note that during both modes of BMIc operation, the patterns of reaching and grasping movements (displacement followed by force increase) were preserved.
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC261882&req=5

pbio.0000042-g006: Ensemble Encoding of Gripping Force, Plasticity of Directional Tuning, and Neuronal Contribution to Model Performance during Learning to Control the BMIc for Reaching and Grasping(A) Perievent time histograms (PETHs) in task 2 for the neuronal population sampled in monkey 1. The plots on top are color-coded (red shows high values; blue, low values). Each horizontal row represents a PETH for a single-neuron or multiunit activity. PETHs have been normalized by subtracting the mean and then dividing by the standard deviation. PETHs are aligned on the gripping force onset (crossing a threshold). Plots at the bottom show the corresponding average traces of gripping force. Note the general similarity of PETHs in pole (left) and brain (right) control in this relatively easy task. Cortical location of neurons is indicated by the bar on the top left. Note the distinct pattern of activation for different areas.(B) Changes in the mean contribution of neurons from different cortical areas to model predictions during training of monkey 1 in task 2.(C) Increases in directional tuning for six cortical areas during training in pole control in task 3.(D and E) Increases in neuronal contribution to linear models predicting hand position (blue), hand velocity (red), and gripping force (black) during learning task 3 in both monkeys.(F and G) Representative robot trajectories and gripping force profiles in an advanced stage of training in task 3 during both pole and brain control. The bottom graphs show trajectories and the amount of the gripping force developed during grasping each virtual object. The dotted vertical lines in the panels indicate the end of reach phase and the beginning of grasp phase. Note that during both modes of BMIc operation, the patterns of reaching and grasping movements (displacement followed by force increase) were preserved.

Mentions: In addition to reproducing hand trajectories with great accuracy, linear models also allowed the reconstruction of fine variations in gripping force produced by both monkeys in tasks 2 and 3. Figure 6A shows that during execution of task 2, most of the recorded cortical neurons contained information about gripping force. In this figure, normalization was achieved by dividing the firing rate of each individual neuron by its standard deviation. In this way, force-related modulations are expressed relative to the overall variability of the neuron's firing rate. Both monkeys mastered task 2 in seven to eight sessions. Figure 6B displays the evolution of the average contribution of neurons from different areas of monkey 1 to model predictions during this period. Contribution of contralateral M1 (R = 0.77, slope = 0.02, p < 0.05) and S1 (R = 0.85, slope = 0.02, p < 0.002) increased significantly, while that of PMd (R = 0.19, slope = 2 × 10−3), SMA (R = 0.34, slope = 0.01), and ipsilateral M1 (R = 0.38, slope = −0.01) did not change substantially. For the whole ensemble combined, there was a significant increase in contribution in both monkey 1 (R = 0.95, slope = 0.02, p < 0.001) and monkey 2 (R = 0.54, slope = 0.01, p < 0.05). By comparing Figure 3B–3F and Figure 6B, we can see that while M1 and S1 neurons showed changes during both tasks 1 and 2, PMd and SMA neurons showed changes in task 1, but not in task 2. This may reflect the greater involvement of these cortical areas in learning visuomotor spatial relationships than in the production of muscle force.


Learning to control a brain-machine interface for reaching and grasping by primates.

Carmena JM, Lebedev MA, Crist RE, O'Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MA - PLoS Biol. (2003)

Ensemble Encoding of Gripping Force, Plasticity of Directional Tuning, and Neuronal Contribution to Model Performance during Learning to Control the BMIc for Reaching and Grasping(A) Perievent time histograms (PETHs) in task 2 for the neuronal population sampled in monkey 1. The plots on top are color-coded (red shows high values; blue, low values). Each horizontal row represents a PETH for a single-neuron or multiunit activity. PETHs have been normalized by subtracting the mean and then dividing by the standard deviation. PETHs are aligned on the gripping force onset (crossing a threshold). Plots at the bottom show the corresponding average traces of gripping force. Note the general similarity of PETHs in pole (left) and brain (right) control in this relatively easy task. Cortical location of neurons is indicated by the bar on the top left. Note the distinct pattern of activation for different areas.(B) Changes in the mean contribution of neurons from different cortical areas to model predictions during training of monkey 1 in task 2.(C) Increases in directional tuning for six cortical areas during training in pole control in task 3.(D and E) Increases in neuronal contribution to linear models predicting hand position (blue), hand velocity (red), and gripping force (black) during learning task 3 in both monkeys.(F and G) Representative robot trajectories and gripping force profiles in an advanced stage of training in task 3 during both pole and brain control. The bottom graphs show trajectories and the amount of the gripping force developed during grasping each virtual object. The dotted vertical lines in the panels indicate the end of reach phase and the beginning of grasp phase. Note that during both modes of BMIc operation, the patterns of reaching and grasping movements (displacement followed by force increase) were preserved.
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC261882&req=5

pbio.0000042-g006: Ensemble Encoding of Gripping Force, Plasticity of Directional Tuning, and Neuronal Contribution to Model Performance during Learning to Control the BMIc for Reaching and Grasping(A) Perievent time histograms (PETHs) in task 2 for the neuronal population sampled in monkey 1. The plots on top are color-coded (red shows high values; blue, low values). Each horizontal row represents a PETH for a single-neuron or multiunit activity. PETHs have been normalized by subtracting the mean and then dividing by the standard deviation. PETHs are aligned on the gripping force onset (crossing a threshold). Plots at the bottom show the corresponding average traces of gripping force. Note the general similarity of PETHs in pole (left) and brain (right) control in this relatively easy task. Cortical location of neurons is indicated by the bar on the top left. Note the distinct pattern of activation for different areas.(B) Changes in the mean contribution of neurons from different cortical areas to model predictions during training of monkey 1 in task 2.(C) Increases in directional tuning for six cortical areas during training in pole control in task 3.(D and E) Increases in neuronal contribution to linear models predicting hand position (blue), hand velocity (red), and gripping force (black) during learning task 3 in both monkeys.(F and G) Representative robot trajectories and gripping force profiles in an advanced stage of training in task 3 during both pole and brain control. The bottom graphs show trajectories and the amount of the gripping force developed during grasping each virtual object. The dotted vertical lines in the panels indicate the end of reach phase and the beginning of grasp phase. Note that during both modes of BMIc operation, the patterns of reaching and grasping movements (displacement followed by force increase) were preserved.
Mentions: In addition to reproducing hand trajectories with great accuracy, linear models also allowed the reconstruction of fine variations in gripping force produced by both monkeys in tasks 2 and 3. Figure 6A shows that during execution of task 2, most of the recorded cortical neurons contained information about gripping force. In this figure, normalization was achieved by dividing the firing rate of each individual neuron by its standard deviation. In this way, force-related modulations are expressed relative to the overall variability of the neuron's firing rate. Both monkeys mastered task 2 in seven to eight sessions. Figure 6B displays the evolution of the average contribution of neurons from different areas of monkey 1 to model predictions during this period. Contribution of contralateral M1 (R = 0.77, slope = 0.02, p < 0.05) and S1 (R = 0.85, slope = 0.02, p < 0.002) increased significantly, while that of PMd (R = 0.19, slope = 2 × 10−3), SMA (R = 0.34, slope = 0.01), and ipsilateral M1 (R = 0.38, slope = −0.01) did not change substantially. For the whole ensemble combined, there was a significant increase in contribution in both monkey 1 (R = 0.95, slope = 0.02, p < 0.001) and monkey 2 (R = 0.54, slope = 0.01, p < 0.05). By comparing Figure 3B–3F and Figure 6B, we can see that while M1 and S1 neurons showed changes during both tasks 1 and 2, PMd and SMA neurons showed changes in task 1, but not in task 2. This may reflect the greater involvement of these cortical areas in learning visuomotor spatial relationships than in the production of muscle force.

Bottom Line: Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance.Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move.Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.

View Article: PubMed Central - PubMed

Affiliation: Department of Neurobiology, Duke University, Durham, North Carolina, USA.

ABSTRACT
Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain-machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.

Show MeSH