Limits...
Relation between belief and performance in perceptual decision making.

Drugowitsch J, Moreno-Bote R, Pouget A - PLoS ONE (2014)

Bottom Line: Prediction of future outcomes and self-monitoring are only effective if belief closely matches behavioral performance.We furthermore show that belief and performance do not match when conditioned on task difficulty--as is common practice when plotting the psychometric curve--highlighting common pitfalls in previous neuroscience work.These results have important implications for experimental design and are of relevance for theories that aim to unravel the nature of meta-cognition.

View Article: PubMed Central - PubMed

Affiliation: Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York, United States of America; Institut National de la Santé et de la Recherche Médicale, École Normale Supérieure, Paris, France; Département des Neurosciences Fondamentales, Université de Genève, Geneva, Switzerland.

ABSTRACT
In an uncertain and ambiguous world, effective decision making requires that subjects form and maintain a belief about the correctness of their choices, a process called meta-cognition. Prediction of future outcomes and self-monitoring are only effective if belief closely matches behavioral performance. Equality between belief and performance is also critical for experimentalists to gain insight into the subjects' belief by simply measuring their performance. Assuming that the decision maker holds the correct model of the world, one might indeed expect that belief and performance should go hand in hand. Unfortunately, we show here that this is rarely the case when performance is defined as the percentage of correct responses for a fixed stimulus, a standard definition in psychophysics. In this case, belief equals performance only for a very narrow family of tasks, whereas in others they will only be very weakly correlated. As we will see it is possible to restore this equality in specific circumstances but this remedy is only effective for a decision-maker, not for an experimenter. We furthermore show that belief and performance do not match when conditioned on task difficulty--as is common practice when plotting the psychometric curve--highlighting common pitfalls in previous neuroscience work. Finally, we demonstrate that miscalibration and the hard-easy effect observed in humans' and other animals' certainty judgments could be explained by a mismatch between the experimenter's and decision maker's expected distribution of task difficulties. These results have important implications for experimental design and are of relevance for theories that aim to unravel the nature of meta-cognition.

Show MeSH

Related in: MedlinePlus

Illustration of framework with a three-sided coin example.(a) In each trial of a sequence, a hidden state  is picked by the experimenter, based on which the observation  is generated. The decision maker only observes  but not  and chooses option  where  is a deterministic function that maps observations into decisions. In this 2-AFC example there are two possible hidden state, causing  to be sampled either according to a biased 3-sided coin , or a fair 3-sided coin . (b) For the given decision function, which maximizes the number of correct decisions for  and , the resulting belief and performance are shown for either choice/hidden state. Belief and performance only match if , that is, when .
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4016031&req=5

pone-0096511-g001: Illustration of framework with a three-sided coin example.(a) In each trial of a sequence, a hidden state is picked by the experimenter, based on which the observation is generated. The decision maker only observes but not and chooses option where is a deterministic function that maps observations into decisions. In this 2-AFC example there are two possible hidden state, causing to be sampled either according to a biased 3-sided coin , or a fair 3-sided coin . (b) For the given decision function, which maximizes the number of correct decisions for and , the resulting belief and performance are shown for either choice/hidden state. Belief and performance only match if , that is, when .

Mentions: In general, we consider -alternative forced choice (-AFC) tasks () with a sequence of independent trials, in each of which an experimenter determines the hidden state of the world, and the aim of the decision maker is to identify this state based on limited information (Fig. 1). At the beginning of each trial, the experimenter draws the hidden state from the prior probability distribution . This state can take one of values out of the set . Consider, for example, an orientation categorization task, in which a displayed orientation is generated stochastically from one of two categories, and the decision maker's task is to identify this category upon observing the orientation. In this example, we would have , such that the generative category can take values out of the set . Furthermore, if each category is a-priory equally likely, we would have .


Relation between belief and performance in perceptual decision making.

Drugowitsch J, Moreno-Bote R, Pouget A - PLoS ONE (2014)

Illustration of framework with a three-sided coin example.(a) In each trial of a sequence, a hidden state  is picked by the experimenter, based on which the observation  is generated. The decision maker only observes  but not  and chooses option  where  is a deterministic function that maps observations into decisions. In this 2-AFC example there are two possible hidden state, causing  to be sampled either according to a biased 3-sided coin , or a fair 3-sided coin . (b) For the given decision function, which maximizes the number of correct decisions for  and , the resulting belief and performance are shown for either choice/hidden state. Belief and performance only match if , that is, when .
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4016031&req=5

pone-0096511-g001: Illustration of framework with a three-sided coin example.(a) In each trial of a sequence, a hidden state is picked by the experimenter, based on which the observation is generated. The decision maker only observes but not and chooses option where is a deterministic function that maps observations into decisions. In this 2-AFC example there are two possible hidden state, causing to be sampled either according to a biased 3-sided coin , or a fair 3-sided coin . (b) For the given decision function, which maximizes the number of correct decisions for and , the resulting belief and performance are shown for either choice/hidden state. Belief and performance only match if , that is, when .
Mentions: In general, we consider -alternative forced choice (-AFC) tasks () with a sequence of independent trials, in each of which an experimenter determines the hidden state of the world, and the aim of the decision maker is to identify this state based on limited information (Fig. 1). At the beginning of each trial, the experimenter draws the hidden state from the prior probability distribution . This state can take one of values out of the set . Consider, for example, an orientation categorization task, in which a displayed orientation is generated stochastically from one of two categories, and the decision maker's task is to identify this category upon observing the orientation. In this example, we would have , such that the generative category can take values out of the set . Furthermore, if each category is a-priory equally likely, we would have .

Bottom Line: Prediction of future outcomes and self-monitoring are only effective if belief closely matches behavioral performance.We furthermore show that belief and performance do not match when conditioned on task difficulty--as is common practice when plotting the psychometric curve--highlighting common pitfalls in previous neuroscience work.These results have important implications for experimental design and are of relevance for theories that aim to unravel the nature of meta-cognition.

View Article: PubMed Central - PubMed

Affiliation: Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York, United States of America; Institut National de la Santé et de la Recherche Médicale, École Normale Supérieure, Paris, France; Département des Neurosciences Fondamentales, Université de Genève, Geneva, Switzerland.

ABSTRACT
In an uncertain and ambiguous world, effective decision making requires that subjects form and maintain a belief about the correctness of their choices, a process called meta-cognition. Prediction of future outcomes and self-monitoring are only effective if belief closely matches behavioral performance. Equality between belief and performance is also critical for experimentalists to gain insight into the subjects' belief by simply measuring their performance. Assuming that the decision maker holds the correct model of the world, one might indeed expect that belief and performance should go hand in hand. Unfortunately, we show here that this is rarely the case when performance is defined as the percentage of correct responses for a fixed stimulus, a standard definition in psychophysics. In this case, belief equals performance only for a very narrow family of tasks, whereas in others they will only be very weakly correlated. As we will see it is possible to restore this equality in specific circumstances but this remedy is only effective for a decision-maker, not for an experimenter. We furthermore show that belief and performance do not match when conditioned on task difficulty--as is common practice when plotting the psychometric curve--highlighting common pitfalls in previous neuroscience work. Finally, we demonstrate that miscalibration and the hard-easy effect observed in humans' and other animals' certainty judgments could be explained by a mismatch between the experimenter's and decision maker's expected distribution of task difficulties. These results have important implications for experimental design and are of relevance for theories that aim to unravel the nature of meta-cognition.

Show MeSH
Related in: MedlinePlus