Limits...
Learning to control a brain-machine interface for reaching and grasping by primates.

Carmena JM, Lebedev MA, Crist RE, O'Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MA - PLoS Biol. (2003)

Bottom Line: Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance.Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move.Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.

View Article: PubMed Central - PubMed

Affiliation: Department of Neurobiology, Duke University, Durham, North Carolina, USA.

ABSTRACT
Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain-machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.

Show MeSH
Directional Tuning in Frontoparietal Ensemble during Different Modes of Operation in Task 1(A–D) Directional tuning during pole control (A), brain control with arm movements (tuning assessed from cursor movements) (B), brain control without arm movements (tuning assessed from cursor movements) (C), and brain control with arm movements (tuning assessed from pole movements) (D). In each of the color-coded diagrams (red shows high values and blue low values; see color scale), the rows depict normalized directional tuning for individual cells. Because of the high directional tuning values of some cells (e.g., that shown in [H]), a color scale limit was set at 0.3 to allow color representation of the largest possible number of cells. Each tuning curve contains eight points that have been interpolated for visual clarity. Correspondence of tuning patterns under different conditions has been quantified using correlation coefficients (shown near lines connecting the diagrams). The highest correspondence was between tuning during pole control and brain control with arm movements. A much less similar pattern of direction tuning emerged during brain control without arm movements. Polar plots near each diagram show average directional tuning for the whole neural ensemble recorded. They indicate an average decrease in tuning strength and shifts in the preferred direction of tuning as the operation mode was switched from pole to brain control. Spread of preferred directions (90° corresponds to uniformly random distribution) is indicated near each polar plot.(E–G) Scatterplots comparing DTD (maximum minus minimum values of tuning curves) during pole control versus brain control with and without arm movements. DTD during brain control was consistently lower than during pole control. This effect was particularly evident during brain control without arm movements.(H–J) Changes in directional tuning for individual neurons under different conditions. Blue shows pole control; red, brain control with arm movements (tuning assessed from pole movements); and green, brain control without arm movements. The first illustrated cell (H) was tuned only when the monkey moved its arm, more so during pole control. The second cell (I) had similar tuning during all modes of operation, but tuning depth was the highest for pole control and the lowest for brain control without arm movements. The third cell (J) was better tuned during brain control.
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC261882&req=5

pbio.0000042-g004: Directional Tuning in Frontoparietal Ensemble during Different Modes of Operation in Task 1(A–D) Directional tuning during pole control (A), brain control with arm movements (tuning assessed from cursor movements) (B), brain control without arm movements (tuning assessed from cursor movements) (C), and brain control with arm movements (tuning assessed from pole movements) (D). In each of the color-coded diagrams (red shows high values and blue low values; see color scale), the rows depict normalized directional tuning for individual cells. Because of the high directional tuning values of some cells (e.g., that shown in [H]), a color scale limit was set at 0.3 to allow color representation of the largest possible number of cells. Each tuning curve contains eight points that have been interpolated for visual clarity. Correspondence of tuning patterns under different conditions has been quantified using correlation coefficients (shown near lines connecting the diagrams). The highest correspondence was between tuning during pole control and brain control with arm movements. A much less similar pattern of direction tuning emerged during brain control without arm movements. Polar plots near each diagram show average directional tuning for the whole neural ensemble recorded. They indicate an average decrease in tuning strength and shifts in the preferred direction of tuning as the operation mode was switched from pole to brain control. Spread of preferred directions (90° corresponds to uniformly random distribution) is indicated near each polar plot.(E–G) Scatterplots comparing DTD (maximum minus minimum values of tuning curves) during pole control versus brain control with and without arm movements. DTD during brain control was consistently lower than during pole control. This effect was particularly evident during brain control without arm movements.(H–J) Changes in directional tuning for individual neurons under different conditions. Blue shows pole control; red, brain control with arm movements (tuning assessed from pole movements); and green, brain control without arm movements. The first illustrated cell (H) was tuned only when the monkey moved its arm, more so during pole control. The second cell (I) had similar tuning during all modes of operation, but tuning depth was the highest for pole control and the lowest for brain control without arm movements. The third cell (J) was better tuned during brain control.

Mentions: As animals learned to operate the BMIc, we also observed short-term changes in neuronal directional tuning, within a single recording session, after switching the BMIc mode of operation from pole to brain control. Directional tuning curves (DTCs) reflected dependency of the neuronal firing rate on movement direction of either the pole or the cursor. DTCs were normalized by dividing average firing rates for particular movement directions by the standard deviation of the whole firing rate record and then subtracting the DTC mean. Using that normalization, changes in firing rate with movement direction were compared with the overall variation of firing rate. Average directional tuning of neural ensembles (DTE) was also assessed, and the spread of the preferred tuning directions was evaluated as the ensemble average of the angles between preferred directions in pairs of neurons. Color-coded population histograms (Figure 4A–4D) displayed the DTCs of all recorded neurons. Polar plots (magenta figures in Figure 4A–4D) showed the DTE and preferred direction spread. Figure 4A–4D shows that DTCs and DTEs for the same neural ensemble had distinct features during pole control (Figure 4A), during brain control with the presence of arm movements (Figure 4B and 4D), and during brain control without arm movements (Figure 4C). Even if the animal was still making arm movements after switching to brain control and direction tuning was calculated in relation to pole movements (compare Figure 4A with 4D), DTC and DTE changed significantly when compared to curves obtained during pole control (R = 0.57 using pole movements as a reference direction, R = 0.70 using cursor movements as a reference). The changes in DTC and DTE became greater as the animal ceased to produce arm movements in brain control (Figure 4C) (R = 0.48). Notice, however, that the pattern for brain control without arm movements (Figure 4C) was also distinct from that for brain control with arm movements (Figure 4B) (R = 0.57). These findings suggest that both the cursor and pole movements influenced the definition of directional tuning profiles in multiple cortical areas.


Learning to control a brain-machine interface for reaching and grasping by primates.

Carmena JM, Lebedev MA, Crist RE, O'Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MA - PLoS Biol. (2003)

Directional Tuning in Frontoparietal Ensemble during Different Modes of Operation in Task 1(A–D) Directional tuning during pole control (A), brain control with arm movements (tuning assessed from cursor movements) (B), brain control without arm movements (tuning assessed from cursor movements) (C), and brain control with arm movements (tuning assessed from pole movements) (D). In each of the color-coded diagrams (red shows high values and blue low values; see color scale), the rows depict normalized directional tuning for individual cells. Because of the high directional tuning values of some cells (e.g., that shown in [H]), a color scale limit was set at 0.3 to allow color representation of the largest possible number of cells. Each tuning curve contains eight points that have been interpolated for visual clarity. Correspondence of tuning patterns under different conditions has been quantified using correlation coefficients (shown near lines connecting the diagrams). The highest correspondence was between tuning during pole control and brain control with arm movements. A much less similar pattern of direction tuning emerged during brain control without arm movements. Polar plots near each diagram show average directional tuning for the whole neural ensemble recorded. They indicate an average decrease in tuning strength and shifts in the preferred direction of tuning as the operation mode was switched from pole to brain control. Spread of preferred directions (90° corresponds to uniformly random distribution) is indicated near each polar plot.(E–G) Scatterplots comparing DTD (maximum minus minimum values of tuning curves) during pole control versus brain control with and without arm movements. DTD during brain control was consistently lower than during pole control. This effect was particularly evident during brain control without arm movements.(H–J) Changes in directional tuning for individual neurons under different conditions. Blue shows pole control; red, brain control with arm movements (tuning assessed from pole movements); and green, brain control without arm movements. The first illustrated cell (H) was tuned only when the monkey moved its arm, more so during pole control. The second cell (I) had similar tuning during all modes of operation, but tuning depth was the highest for pole control and the lowest for brain control without arm movements. The third cell (J) was better tuned during brain control.
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC261882&req=5

pbio.0000042-g004: Directional Tuning in Frontoparietal Ensemble during Different Modes of Operation in Task 1(A–D) Directional tuning during pole control (A), brain control with arm movements (tuning assessed from cursor movements) (B), brain control without arm movements (tuning assessed from cursor movements) (C), and brain control with arm movements (tuning assessed from pole movements) (D). In each of the color-coded diagrams (red shows high values and blue low values; see color scale), the rows depict normalized directional tuning for individual cells. Because of the high directional tuning values of some cells (e.g., that shown in [H]), a color scale limit was set at 0.3 to allow color representation of the largest possible number of cells. Each tuning curve contains eight points that have been interpolated for visual clarity. Correspondence of tuning patterns under different conditions has been quantified using correlation coefficients (shown near lines connecting the diagrams). The highest correspondence was between tuning during pole control and brain control with arm movements. A much less similar pattern of direction tuning emerged during brain control without arm movements. Polar plots near each diagram show average directional tuning for the whole neural ensemble recorded. They indicate an average decrease in tuning strength and shifts in the preferred direction of tuning as the operation mode was switched from pole to brain control. Spread of preferred directions (90° corresponds to uniformly random distribution) is indicated near each polar plot.(E–G) Scatterplots comparing DTD (maximum minus minimum values of tuning curves) during pole control versus brain control with and without arm movements. DTD during brain control was consistently lower than during pole control. This effect was particularly evident during brain control without arm movements.(H–J) Changes in directional tuning for individual neurons under different conditions. Blue shows pole control; red, brain control with arm movements (tuning assessed from pole movements); and green, brain control without arm movements. The first illustrated cell (H) was tuned only when the monkey moved its arm, more so during pole control. The second cell (I) had similar tuning during all modes of operation, but tuning depth was the highest for pole control and the lowest for brain control without arm movements. The third cell (J) was better tuned during brain control.
Mentions: As animals learned to operate the BMIc, we also observed short-term changes in neuronal directional tuning, within a single recording session, after switching the BMIc mode of operation from pole to brain control. Directional tuning curves (DTCs) reflected dependency of the neuronal firing rate on movement direction of either the pole or the cursor. DTCs were normalized by dividing average firing rates for particular movement directions by the standard deviation of the whole firing rate record and then subtracting the DTC mean. Using that normalization, changes in firing rate with movement direction were compared with the overall variation of firing rate. Average directional tuning of neural ensembles (DTE) was also assessed, and the spread of the preferred tuning directions was evaluated as the ensemble average of the angles between preferred directions in pairs of neurons. Color-coded population histograms (Figure 4A–4D) displayed the DTCs of all recorded neurons. Polar plots (magenta figures in Figure 4A–4D) showed the DTE and preferred direction spread. Figure 4A–4D shows that DTCs and DTEs for the same neural ensemble had distinct features during pole control (Figure 4A), during brain control with the presence of arm movements (Figure 4B and 4D), and during brain control without arm movements (Figure 4C). Even if the animal was still making arm movements after switching to brain control and direction tuning was calculated in relation to pole movements (compare Figure 4A with 4D), DTC and DTE changed significantly when compared to curves obtained during pole control (R = 0.57 using pole movements as a reference direction, R = 0.70 using cursor movements as a reference). The changes in DTC and DTE became greater as the animal ceased to produce arm movements in brain control (Figure 4C) (R = 0.48). Notice, however, that the pattern for brain control without arm movements (Figure 4C) was also distinct from that for brain control with arm movements (Figure 4B) (R = 0.57). These findings suggest that both the cursor and pole movements influenced the definition of directional tuning profiles in multiple cortical areas.

Bottom Line: Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance.Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move.Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.

View Article: PubMed Central - PubMed

Affiliation: Department of Neurobiology, Duke University, Durham, North Carolina, USA.

ABSTRACT
Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain-machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.

Show MeSH