Limits...
Scaling prediction errors to reward variability benefits error-driven learning in humans.

Diederen KM, Schultz W - J. Neurophysiol. (2015)

Bottom Line: In addition, participants who scaled prediction errors relative to standard deviation also presented with more similar performance for different standard deviations, indicating that increases in standard deviation did not substantially decrease "adapters'" accuracy in predicting the means of reward distributions.However, exaggerated scaling beyond the standard deviation resulted in impaired performance.Thus efficient adaptation makes learning more robust to changing variability.

View Article: PubMed Central - PubMed

Affiliation: Department of Physiology, Development, and Neuroscience, University of Cambridge, Cambridge, United Kingdom k.diederen@gmail.com.

No MeSH data available.


Related in: MedlinePlus

A, left: overall performance error (/prediction − EV/ averaged over all trials) varied significantly with the estimated degree of prediction error scaling. Whereas performance error decreased for adaptation indexes up to ν = 0.4–0.6 (i.e., approximately half the logarithm of the SD), higher adaptation indexes were associated with increases in performance error. Right: relationship between learning rate (LR) decay and performance error. Performance errors slightly decreased for small increases in learning rate decay (η) but increased substantially for larger decays (>0.6–0.8). Adaptation indexes and learning rate decays were divided into 5 bins of equal width. Subsequently, performance errors were averaged over all adaptation/learning rate decay indexes in a certain bin. B: increases in performance error in those individuals who scaled prediction error with a quantity greater than the log SD. C, left: dissimilarity in performance error across SD conditions was lower for individuals who scaled prediction errors to a value up to ν = 0.4–0.6 (i.e., approximately half the SD) but not for those who adapted with larger values. Right: relationship between initial learning rate and performance error. Performance error was more similar for initial learning rates (α1) of ∼0.2–0.4 but became more dissimilar with smaller and larger learning rates. Adaptation indexes and initial learning rates were divided into 5 bins of equal width. Subsequently, similarity in performance error was averaged over all adaptation indexes/initial learning rates in a certain bin.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4563025&req=5

Figure 4: A, left: overall performance error (/prediction − EV/ averaged over all trials) varied significantly with the estimated degree of prediction error scaling. Whereas performance error decreased for adaptation indexes up to ν = 0.4–0.6 (i.e., approximately half the logarithm of the SD), higher adaptation indexes were associated with increases in performance error. Right: relationship between learning rate (LR) decay and performance error. Performance errors slightly decreased for small increases in learning rate decay (η) but increased substantially for larger decays (>0.6–0.8). Adaptation indexes and learning rate decays were divided into 5 bins of equal width. Subsequently, performance errors were averaged over all adaptation/learning rate decay indexes in a certain bin. B: increases in performance error in those individuals who scaled prediction error with a quantity greater than the log SD. C, left: dissimilarity in performance error across SD conditions was lower for individuals who scaled prediction errors to a value up to ν = 0.4–0.6 (i.e., approximately half the SD) but not for those who adapted with larger values. Right: relationship between initial learning rate and performance error. Performance error was more similar for initial learning rates (α1) of ∼0.2–0.4 but became more dissimilar with smaller and larger learning rates. Adaptation indexes and initial learning rates were divided into 5 bins of equal width. Subsequently, similarity in performance error was averaged over all adaptation indexes/initial learning rates in a certain bin.

Mentions: We observed a significant quadratic relationship between the individual degree of prediction error scaling and overall performance error (P = 0.0067; Table 2; Fig. 4A, left). Whereas performance error decreased for adaptation indexes up to ν = ∼0.5 (i.e., half the logarithm of the SD), higher adaptation indexes were associated with increases in performance error (Fig. 4A, left). Analyses using the extent of SD-dependent changes in learning rate (Fig. 3B, right) as an alternative measure for adaptation confirmed this result [β12 = 0.1614, T(24) = 3.1066, P = 0.0048]. These results imply that efficient adaptation required scaling of prediction errors relative to, but smaller than, the (log)SD, in line with the simulated data (Fig. 2F). The tight relationship between the simulated and experimental data suggests that participants tended to scale their prediction errors in an optimal manner. This relationship furthermore implies that the estimated adaptation parameters provided a good fit of participants' behavior, i.e., unreliable fits might have resulted in erroneous adaptation parameters unlikely to correlate with (raw) performance error data. To further investigate the extent of prediction error scaling in relation to performance, we repeated model estimation for the log adaptive model without any constraints on the adaptation parameter. Seven of the 31 participants scaled prediction errors with a quantity larger than the log SD. These participants presented with significantly larger performance errors compared with individuals who scaled prediction errors with a quantity smaller than the SD [T(29) = 1.9937, P = 0.0278; Fig. 4B]. This result shows how participants can make errors and deviate from theoretical predictions.


Scaling prediction errors to reward variability benefits error-driven learning in humans.

Diederen KM, Schultz W - J. Neurophysiol. (2015)

A, left: overall performance error (/prediction − EV/ averaged over all trials) varied significantly with the estimated degree of prediction error scaling. Whereas performance error decreased for adaptation indexes up to ν = 0.4–0.6 (i.e., approximately half the logarithm of the SD), higher adaptation indexes were associated with increases in performance error. Right: relationship between learning rate (LR) decay and performance error. Performance errors slightly decreased for small increases in learning rate decay (η) but increased substantially for larger decays (>0.6–0.8). Adaptation indexes and learning rate decays were divided into 5 bins of equal width. Subsequently, performance errors were averaged over all adaptation/learning rate decay indexes in a certain bin. B: increases in performance error in those individuals who scaled prediction error with a quantity greater than the log SD. C, left: dissimilarity in performance error across SD conditions was lower for individuals who scaled prediction errors to a value up to ν = 0.4–0.6 (i.e., approximately half the SD) but not for those who adapted with larger values. Right: relationship between initial learning rate and performance error. Performance error was more similar for initial learning rates (α1) of ∼0.2–0.4 but became more dissimilar with smaller and larger learning rates. Adaptation indexes and initial learning rates were divided into 5 bins of equal width. Subsequently, similarity in performance error was averaged over all adaptation indexes/initial learning rates in a certain bin.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4563025&req=5

Figure 4: A, left: overall performance error (/prediction − EV/ averaged over all trials) varied significantly with the estimated degree of prediction error scaling. Whereas performance error decreased for adaptation indexes up to ν = 0.4–0.6 (i.e., approximately half the logarithm of the SD), higher adaptation indexes were associated with increases in performance error. Right: relationship between learning rate (LR) decay and performance error. Performance errors slightly decreased for small increases in learning rate decay (η) but increased substantially for larger decays (>0.6–0.8). Adaptation indexes and learning rate decays were divided into 5 bins of equal width. Subsequently, performance errors were averaged over all adaptation/learning rate decay indexes in a certain bin. B: increases in performance error in those individuals who scaled prediction error with a quantity greater than the log SD. C, left: dissimilarity in performance error across SD conditions was lower for individuals who scaled prediction errors to a value up to ν = 0.4–0.6 (i.e., approximately half the SD) but not for those who adapted with larger values. Right: relationship between initial learning rate and performance error. Performance error was more similar for initial learning rates (α1) of ∼0.2–0.4 but became more dissimilar with smaller and larger learning rates. Adaptation indexes and initial learning rates were divided into 5 bins of equal width. Subsequently, similarity in performance error was averaged over all adaptation indexes/initial learning rates in a certain bin.
Mentions: We observed a significant quadratic relationship between the individual degree of prediction error scaling and overall performance error (P = 0.0067; Table 2; Fig. 4A, left). Whereas performance error decreased for adaptation indexes up to ν = ∼0.5 (i.e., half the logarithm of the SD), higher adaptation indexes were associated with increases in performance error (Fig. 4A, left). Analyses using the extent of SD-dependent changes in learning rate (Fig. 3B, right) as an alternative measure for adaptation confirmed this result [β12 = 0.1614, T(24) = 3.1066, P = 0.0048]. These results imply that efficient adaptation required scaling of prediction errors relative to, but smaller than, the (log)SD, in line with the simulated data (Fig. 2F). The tight relationship between the simulated and experimental data suggests that participants tended to scale their prediction errors in an optimal manner. This relationship furthermore implies that the estimated adaptation parameters provided a good fit of participants' behavior, i.e., unreliable fits might have resulted in erroneous adaptation parameters unlikely to correlate with (raw) performance error data. To further investigate the extent of prediction error scaling in relation to performance, we repeated model estimation for the log adaptive model without any constraints on the adaptation parameter. Seven of the 31 participants scaled prediction errors with a quantity larger than the log SD. These participants presented with significantly larger performance errors compared with individuals who scaled prediction errors with a quantity smaller than the SD [T(29) = 1.9937, P = 0.0278; Fig. 4B]. This result shows how participants can make errors and deviate from theoretical predictions.

Bottom Line: In addition, participants who scaled prediction errors relative to standard deviation also presented with more similar performance for different standard deviations, indicating that increases in standard deviation did not substantially decrease "adapters'" accuracy in predicting the means of reward distributions.However, exaggerated scaling beyond the standard deviation resulted in impaired performance.Thus efficient adaptation makes learning more robust to changing variability.

View Article: PubMed Central - PubMed

Affiliation: Department of Physiology, Development, and Neuroscience, University of Cambridge, Cambridge, United Kingdom k.diederen@gmail.com.

No MeSH data available.


Related in: MedlinePlus