Limits...
The effect of target and non-target similarity on neural classification performance: a boost from confidence.

Marathe AR, Ries AJ, Lawhern VJ, Lance BJ, Touryan J, McDowell K, Cecotti H - Front Neurosci (2015)

Bottom Line: In most real-world environments there are likely to be many shared features between targets and non-targets resulting in similar neural activity between the two classes.It is unknown how current neural-based target classification algorithms perform when qualitatively similar target and non-target images are presented.This study address this question by comparing behavioral and neural classification performance across two conditions: first, when targets were the only infrequent stimulus presented amongst frequent background distracters; and second when targets were presented together with infrequent non-targets containing similar visual features to the targets.

View Article: PubMed Central - PubMed

Affiliation: Translational Neuroscience Branch, US Army Research Laboratory, Human Research and Engineering Directorate Aberdeen Proving Grounds, MD, USA.

ABSTRACT
Brain computer interaction (BCI) technologies have proven effective in utilizing single-trial classification algorithms to detect target images in rapid serial visualization presentation tasks. While many factors contribute to the accuracy of these algorithms, a critical aspect that is often overlooked concerns the feature similarity between target and non-target images. In most real-world environments there are likely to be many shared features between targets and non-targets resulting in similar neural activity between the two classes. It is unknown how current neural-based target classification algorithms perform when qualitatively similar target and non-target images are presented. This study address this question by comparing behavioral and neural classification performance across two conditions: first, when targets were the only infrequent stimulus presented amongst frequent background distracters; and second when targets were presented together with infrequent non-targets containing similar visual features to the targets. The resulting findings show that behavior is slower and less accurate when targets are presented together with similar non-targets; moreover, single-trial classification yielded high levels of misclassification when infrequent non-targets are included. Furthermore, we present an approach to mitigate the image misclassification. We use confidence measures to assess the quality of single-trial classification, and demonstrate that a system in which low confidence trials are reclassified through a secondary process can result in improved performance.

No MeSH data available.


Confidence. (A) Confidence levels by stimulus type. (B) Az for trials as a function of confidence threshold. Solid line shows the Az for trials exceeding the confidence threshold given. Dashed line shows Az when trials below the confidence threshold are manually labeled while trials above the threshold are labeled through the neural classification. In both cases, as confidence increases, Az increases. (C) Misclassification rates for trials that exceed a given confidence threshold. Solid lines show misclassification rates for neural classification only. As confidence increases, the misclassification rates for target and background distractor stimuli fall to nearly 0. Non-target misclassification rates remain high regardless of confidence levels. Dashed lines show misclassification rates when trials below the threshold are manually labeled, while trials above the threshold use neural classification. Misclassification rates for all three stimulus classes are reduced through the manual labeling process. The inset graph show zooms in on the lower portion of the graph to highlight the decrease in misclassification rates for target and background stimuli. (D) Percent of trials that exceed a given confidence threshold.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4544215&req=5

Figure 7: Confidence. (A) Confidence levels by stimulus type. (B) Az for trials as a function of confidence threshold. Solid line shows the Az for trials exceeding the confidence threshold given. Dashed line shows Az when trials below the confidence threshold are manually labeled while trials above the threshold are labeled through the neural classification. In both cases, as confidence increases, Az increases. (C) Misclassification rates for trials that exceed a given confidence threshold. Solid lines show misclassification rates for neural classification only. As confidence increases, the misclassification rates for target and background distractor stimuli fall to nearly 0. Non-target misclassification rates remain high regardless of confidence levels. Dashed lines show misclassification rates when trials below the threshold are manually labeled, while trials above the threshold use neural classification. Misclassification rates for all three stimulus classes are reduced through the manual labeling process. The inset graph show zooms in on the lower portion of the graph to highlight the decrease in misclassification rates for target and background stimuli. (D) Percent of trials that exceed a given confidence threshold.

Mentions: The increase in misclassification rates in the non-target condition is potentially problematic for many real-world applications of this technology where users will encounter instances of non-target stimuli that share the same physical and semantic features as target stimuli. To address this issue, we explored applying confidence measures to the classifier outputs as a means to mitigate the misclassification rate (Figures 6, 7).


The effect of target and non-target similarity on neural classification performance: a boost from confidence.

Marathe AR, Ries AJ, Lawhern VJ, Lance BJ, Touryan J, McDowell K, Cecotti H - Front Neurosci (2015)

Confidence. (A) Confidence levels by stimulus type. (B) Az for trials as a function of confidence threshold. Solid line shows the Az for trials exceeding the confidence threshold given. Dashed line shows Az when trials below the confidence threshold are manually labeled while trials above the threshold are labeled through the neural classification. In both cases, as confidence increases, Az increases. (C) Misclassification rates for trials that exceed a given confidence threshold. Solid lines show misclassification rates for neural classification only. As confidence increases, the misclassification rates for target and background distractor stimuli fall to nearly 0. Non-target misclassification rates remain high regardless of confidence levels. Dashed lines show misclassification rates when trials below the threshold are manually labeled, while trials above the threshold use neural classification. Misclassification rates for all three stimulus classes are reduced through the manual labeling process. The inset graph show zooms in on the lower portion of the graph to highlight the decrease in misclassification rates for target and background stimuli. (D) Percent of trials that exceed a given confidence threshold.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4544215&req=5

Figure 7: Confidence. (A) Confidence levels by stimulus type. (B) Az for trials as a function of confidence threshold. Solid line shows the Az for trials exceeding the confidence threshold given. Dashed line shows Az when trials below the confidence threshold are manually labeled while trials above the threshold are labeled through the neural classification. In both cases, as confidence increases, Az increases. (C) Misclassification rates for trials that exceed a given confidence threshold. Solid lines show misclassification rates for neural classification only. As confidence increases, the misclassification rates for target and background distractor stimuli fall to nearly 0. Non-target misclassification rates remain high regardless of confidence levels. Dashed lines show misclassification rates when trials below the threshold are manually labeled, while trials above the threshold use neural classification. Misclassification rates for all three stimulus classes are reduced through the manual labeling process. The inset graph show zooms in on the lower portion of the graph to highlight the decrease in misclassification rates for target and background stimuli. (D) Percent of trials that exceed a given confidence threshold.
Mentions: The increase in misclassification rates in the non-target condition is potentially problematic for many real-world applications of this technology where users will encounter instances of non-target stimuli that share the same physical and semantic features as target stimuli. To address this issue, we explored applying confidence measures to the classifier outputs as a means to mitigate the misclassification rate (Figures 6, 7).

Bottom Line: In most real-world environments there are likely to be many shared features between targets and non-targets resulting in similar neural activity between the two classes.It is unknown how current neural-based target classification algorithms perform when qualitatively similar target and non-target images are presented.This study address this question by comparing behavioral and neural classification performance across two conditions: first, when targets were the only infrequent stimulus presented amongst frequent background distracters; and second when targets were presented together with infrequent non-targets containing similar visual features to the targets.

View Article: PubMed Central - PubMed

Affiliation: Translational Neuroscience Branch, US Army Research Laboratory, Human Research and Engineering Directorate Aberdeen Proving Grounds, MD, USA.

ABSTRACT
Brain computer interaction (BCI) technologies have proven effective in utilizing single-trial classification algorithms to detect target images in rapid serial visualization presentation tasks. While many factors contribute to the accuracy of these algorithms, a critical aspect that is often overlooked concerns the feature similarity between target and non-target images. In most real-world environments there are likely to be many shared features between targets and non-targets resulting in similar neural activity between the two classes. It is unknown how current neural-based target classification algorithms perform when qualitatively similar target and non-target images are presented. This study address this question by comparing behavioral and neural classification performance across two conditions: first, when targets were the only infrequent stimulus presented amongst frequent background distracters; and second when targets were presented together with infrequent non-targets containing similar visual features to the targets. The resulting findings show that behavior is slower and less accurate when targets are presented together with similar non-targets; moreover, single-trial classification yielded high levels of misclassification when infrequent non-targets are included. Furthermore, we present an approach to mitigate the image misclassification. We use confidence measures to assess the quality of single-trial classification, and demonstrate that a system in which low confidence trials are reclassified through a secondary process can result in improved performance.

No MeSH data available.