Limits...
Relation between belief and performance in perceptual decision making.

Drugowitsch J, Moreno-Bote R, Pouget A - PLoS ONE (2014)

Bottom Line: Prediction of future outcomes and self-monitoring are only effective if belief closely matches behavioral performance.We furthermore show that belief and performance do not match when conditioned on task difficulty--as is common practice when plotting the psychometric curve--highlighting common pitfalls in previous neuroscience work.These results have important implications for experimental design and are of relevance for theories that aim to unravel the nature of meta-cognition.

View Article: PubMed Central - PubMed

Affiliation: Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York, United States of America; Institut National de la Santé et de la Recherche Médicale, École Normale Supérieure, Paris, France; Département des Neurosciences Fondamentales, Université de Genève, Geneva, Switzerland.

ABSTRACT
In an uncertain and ambiguous world, effective decision making requires that subjects form and maintain a belief about the correctness of their choices, a process called meta-cognition. Prediction of future outcomes and self-monitoring are only effective if belief closely matches behavioral performance. Equality between belief and performance is also critical for experimentalists to gain insight into the subjects' belief by simply measuring their performance. Assuming that the decision maker holds the correct model of the world, one might indeed expect that belief and performance should go hand in hand. Unfortunately, we show here that this is rarely the case when performance is defined as the percentage of correct responses for a fixed stimulus, a standard definition in psychophysics. In this case, belief equals performance only for a very narrow family of tasks, whereas in others they will only be very weakly correlated. As we will see it is possible to restore this equality in specific circumstances but this remedy is only effective for a decision-maker, not for an experimenter. We furthermore show that belief and performance do not match when conditioned on task difficulty--as is common practice when plotting the psychometric curve--highlighting common pitfalls in previous neuroscience work. Finally, we demonstrate that miscalibration and the hard-easy effect observed in humans' and other animals' certainty judgments could be explained by a mismatch between the experimenter's and decision maker's expected distribution of task difficulties. These results have important implications for experimental design and are of relevance for theories that aim to unravel the nature of meta-cognition.

Show MeSH
Comparing estimated belief with performance and trial-by-trial belief.(a) A DM with a non-uniform prior of  as in Fig. 3b. Trial-by-trial belief differs from performance because of the asymmetric prior. By contrast, the estimated belief using Eq. (13) matches the trial-by-trial belief, because the decision maker's state is fully observable in a DM. (b) A two race model with uniform priors as in Fig 3c. This time, the decision maker's state is not fully observable because the state of the losing race is unknown to the experimenter. As a consequence, the belief estimated by Eq. (13) no longer matches the trial-by-trial belief of the observer but only the averaged belief, where the average is performed over the state of the losing race. Details of the model simulations are described in Methods: Generating Figures 3 and 4.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4016031&req=5

pone-0096511-g004: Comparing estimated belief with performance and trial-by-trial belief.(a) A DM with a non-uniform prior of as in Fig. 3b. Trial-by-trial belief differs from performance because of the asymmetric prior. By contrast, the estimated belief using Eq. (13) matches the trial-by-trial belief, because the decision maker's state is fully observable in a DM. (b) A two race model with uniform priors as in Fig 3c. This time, the decision maker's state is not fully observable because the state of the losing race is unknown to the experimenter. As a consequence, the belief estimated by Eq. (13) no longer matches the trial-by-trial belief of the observer but only the averaged belief, where the average is performed over the state of the losing race. Details of the model simulations are described in Methods: Generating Figures 3 and 4.

Mentions: (a) In a DM, a particle drifts and diffuses over time. A decision is performed as soon as this particle reaches one of the two boundaries. The mean drift rate , which is unknown to the decision maker, determines which of the two choices is correct. In this illustration, the drift is towards the upper boundary, corresponding to hidden state , such that is the correct choice. We show eight (solid) trajectories leading to the correct choice () and two (dashed) trajectories leading to the wrong choice (). Our framework allows for time-varying boundaries, as shown here and used to generate Figs. 3a/b and 4a/b. (b) A race model features races (here ) that compete against each other in a race towards a boundary of height . The race that first reaches its associated boundary determines the decision. The set of all races is described by a drifting/diffusing particle in -dimensional space. In our illustration this particle drifts towards the upper boundary (thus ) and diffuses in both dimensions. Thus, four (solid) trajectories lead to the correct choice (), and one (dashed) trajectory leads to the incorrect choice ().


Relation between belief and performance in perceptual decision making.

Drugowitsch J, Moreno-Bote R, Pouget A - PLoS ONE (2014)

Comparing estimated belief with performance and trial-by-trial belief.(a) A DM with a non-uniform prior of  as in Fig. 3b. Trial-by-trial belief differs from performance because of the asymmetric prior. By contrast, the estimated belief using Eq. (13) matches the trial-by-trial belief, because the decision maker's state is fully observable in a DM. (b) A two race model with uniform priors as in Fig 3c. This time, the decision maker's state is not fully observable because the state of the losing race is unknown to the experimenter. As a consequence, the belief estimated by Eq. (13) no longer matches the trial-by-trial belief of the observer but only the averaged belief, where the average is performed over the state of the losing race. Details of the model simulations are described in Methods: Generating Figures 3 and 4.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4016031&req=5

pone-0096511-g004: Comparing estimated belief with performance and trial-by-trial belief.(a) A DM with a non-uniform prior of as in Fig. 3b. Trial-by-trial belief differs from performance because of the asymmetric prior. By contrast, the estimated belief using Eq. (13) matches the trial-by-trial belief, because the decision maker's state is fully observable in a DM. (b) A two race model with uniform priors as in Fig 3c. This time, the decision maker's state is not fully observable because the state of the losing race is unknown to the experimenter. As a consequence, the belief estimated by Eq. (13) no longer matches the trial-by-trial belief of the observer but only the averaged belief, where the average is performed over the state of the losing race. Details of the model simulations are described in Methods: Generating Figures 3 and 4.
Mentions: (a) In a DM, a particle drifts and diffuses over time. A decision is performed as soon as this particle reaches one of the two boundaries. The mean drift rate , which is unknown to the decision maker, determines which of the two choices is correct. In this illustration, the drift is towards the upper boundary, corresponding to hidden state , such that is the correct choice. We show eight (solid) trajectories leading to the correct choice () and two (dashed) trajectories leading to the wrong choice (). Our framework allows for time-varying boundaries, as shown here and used to generate Figs. 3a/b and 4a/b. (b) A race model features races (here ) that compete against each other in a race towards a boundary of height . The race that first reaches its associated boundary determines the decision. The set of all races is described by a drifting/diffusing particle in -dimensional space. In our illustration this particle drifts towards the upper boundary (thus ) and diffuses in both dimensions. Thus, four (solid) trajectories lead to the correct choice (), and one (dashed) trajectory leads to the incorrect choice ().

Bottom Line: Prediction of future outcomes and self-monitoring are only effective if belief closely matches behavioral performance.We furthermore show that belief and performance do not match when conditioned on task difficulty--as is common practice when plotting the psychometric curve--highlighting common pitfalls in previous neuroscience work.These results have important implications for experimental design and are of relevance for theories that aim to unravel the nature of meta-cognition.

View Article: PubMed Central - PubMed

Affiliation: Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York, United States of America; Institut National de la Santé et de la Recherche Médicale, École Normale Supérieure, Paris, France; Département des Neurosciences Fondamentales, Université de Genève, Geneva, Switzerland.

ABSTRACT
In an uncertain and ambiguous world, effective decision making requires that subjects form and maintain a belief about the correctness of their choices, a process called meta-cognition. Prediction of future outcomes and self-monitoring are only effective if belief closely matches behavioral performance. Equality between belief and performance is also critical for experimentalists to gain insight into the subjects' belief by simply measuring their performance. Assuming that the decision maker holds the correct model of the world, one might indeed expect that belief and performance should go hand in hand. Unfortunately, we show here that this is rarely the case when performance is defined as the percentage of correct responses for a fixed stimulus, a standard definition in psychophysics. In this case, belief equals performance only for a very narrow family of tasks, whereas in others they will only be very weakly correlated. As we will see it is possible to restore this equality in specific circumstances but this remedy is only effective for a decision-maker, not for an experimenter. We furthermore show that belief and performance do not match when conditioned on task difficulty--as is common practice when plotting the psychometric curve--highlighting common pitfalls in previous neuroscience work. Finally, we demonstrate that miscalibration and the hard-easy effect observed in humans' and other animals' certainty judgments could be explained by a mismatch between the experimenter's and decision maker's expected distribution of task difficulties. These results have important implications for experimental design and are of relevance for theories that aim to unravel the nature of meta-cognition.

Show MeSH