{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T08:37:51Z","timestamp":1778575071197,"version":"3.51.4"},"reference-count":25,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2021,8,1]],"date-time":"2021-08-01T00:00:00Z","timestamp":1627776000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,8,2]],"date-time":"2021-08-02T00:00:00Z","timestamp":1627862400000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003246","name":"Nederlandse Organisatie voor Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["15989"],"award-info":[{"award-number":["15989"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Biol Cybern"],"published-print":{"date-parts":[[2021,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>When learning a movement based on binary success information, one is more variable following failure than following success. Theoretically, the additional variability post-failure might reflect exploration of possibilities to obtain success. When average behavior is changing (as in learning), variability can be estimated from differences between subsequent movements. Can one estimate exploration reliably from such trial-to-trial changes when studying reward-based motor learning? To answer this question, we tried to reconstruct the exploration underlying learning as described by four existing reward-based motor learning models. We simulated learning for various learner and task characteristics. If we simply determined the additional change post-failure, estimates of exploration were sensitive to learner and task characteristics. We identified two pitfalls in quantifying exploration based on trial-to-trial changes. Firstly, performance-dependent feedback can cause correlated samples of motor noise and exploration on successful trials, which biases exploration estimates. Secondly, the trial relative to which trial-to-trial change is calculated may also contain exploration, which causes underestimation. As a solution, we developed the additional trial-to-trial change (ATTC) method. By moving the reference trial one trial back and subtracting trial-to-trial changes following specific sequences of trial outcomes, exploration can be estimated reliably for the three models that explore based on the outcome of only the previous trial. Since ATTC estimates are based on a selection of trial sequences, this method requires many trials. In conclusion, if exploration is a binary function of previous trial outcome, the ATTC method allows for a model-free quantification of exploration.<\/jats:p>","DOI":"10.1007\/s00422-021-00884-8","type":"journal-article","created":{"date-parts":[[2021,8,2]],"date-time":"2021-08-02T17:02:45Z","timestamp":1627923765000},"page":"365-382","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them"],"prefix":"10.1007","volume":"115","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8531-9568","authenticated-orcid":false,"given":"Nina M.","family":"van Mastrigt","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4190-7827","authenticated-orcid":false,"given":"Katinka","family":"van der Kooij","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3794-0579","authenticated-orcid":false,"given":"Jeroen B. J.","family":"Smeets","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,8,2]]},"reference":[{"key":"884_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3389\/fnhum.2013.00225","volume":"7","author":"MO Abe","year":"2013","unstructured":"Abe MO, Sternad D (2013) Directionality in distribution and temporal structure of variability in skill acquisition. Front Hum Neurosci 7:1\u201315. https:\/\/doi.org\/10.3389\/fnhum.2013.00225","journal-title":"Front Hum Neurosci"},{"key":"884_CR2","doi-asserted-by":"publisher","DOI":"10.1101\/2020.09.14.297143","author":"G Avraham","year":"2020","unstructured":"Avraham G, Taylor JA, Ivry RB, Mcdougle SD (2020) An associative learning account of sensorimotor adaptation. BioRxiv. https:\/\/doi.org\/10.1101\/2020.09.14.297143","journal-title":"BioRxiv"},{"issue":"3","key":"884_CR3","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1006839","volume":"15","author":"JGA Cashaback","year":"2019","unstructured":"Cashaback JGA, Lao CK, Palidis DJ, Coltman SK, McGregor HR, Gribble PL (2019) The gradient of the reinforcement landscape influences sensorimotor learning. PLoS Comput Biol 15(3):e1006839. https:\/\/doi.org\/10.1371\/journal.pcbi.1006839","journal-title":"PLoS Comput Biol"},{"issue":"4","key":"884_CR4","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1005503","volume":"13","author":"X Chen","year":"2017","unstructured":"Chen X, Mohr K, Galea JM (2017) Predicting explorative motor learning using decision-making and motor noise. PLoS Comput Biol 13(4):e1005503. https:\/\/doi.org\/10.1371\/journal.pcbi.1005503","journal-title":"PLoS Comput Biol"},{"issue":"4","key":"884_CR5","doi-asserted-by":"publisher","first-page":"760","DOI":"10.1162\/neco.2006.18.4.760","volume":"18","author":"S Cheng","year":"2006","unstructured":"Cheng S, Sabes PN (2006) Modeling sensorimotor learning with linear dynamical systems. Neural Comput 18(4):760\u2013793. https:\/\/doi.org\/10.1162\/neco.2006.18.4.760","journal-title":"Neural Comput"},{"key":"884_CR6","doi-asserted-by":"publisher","unstructured":"Codol O, Holland, P. J., & Galea, J. M. (2018). The relationship between reinforcement and explicit control during visuomotor adaptation. Scientific Reports, 8(9121). https:\/\/doi.org\/10.1038\/s41598-018-27378-1","DOI":"10.1038\/s41598-018-27378-1"},{"issue":"1","key":"884_CR7","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1146\/annurev-neuro-072116-031548","volume":"40","author":"AK Dhawale","year":"2017","unstructured":"Dhawale AK, Smith MA, \u00d6lveczky BP (2017) The role of variability in motor learning. Annu Rev Neurosci 40(1):479\u2013498. https:\/\/doi.org\/10.1146\/annurev-neuro-072116-031548","journal-title":"Annu Rev Neurosci"},{"issue":"21","key":"884_CR8","doi-asserted-by":"publisher","first-page":"3551","DOI":"10.1016\/j.cub.2019.08.052","volume":"29","author":"AK Dhawale","year":"2019","unstructured":"Dhawale AK, Miyamoto YR, Smith MA, \u00d6lveczky BP (2019) Adaptive regulation of motor variability. Curr Biol 29(21):3551-3562.e7. https:\/\/doi.org\/10.1016\/j.cub.2019.08.052","journal-title":"Curr Biol"},{"issue":"6","key":"884_CR9","doi-asserted-by":"publisher","first-page":"2241","DOI":"10.1152\/jn.00901.2017","volume":"119","author":"P Holland","year":"2018","unstructured":"Holland P, Codol O, Galea JM (2018) Contribution of explicit processes to reinforcement-based motor learning. J Neurophysiol 119(6):2241\u20132255. https:\/\/doi.org\/10.1152\/jn.00901.2017","journal-title":"J Neurophysiol"},{"issue":"3","key":"884_CR10","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1002012","volume":"7","author":"J Izawa","year":"2011","unstructured":"Izawa J, Shadmehr R (2011) Learning from sensory and reward prediction errors during motor adaptation. PLoS Comput Biol 7(3):e1002012. https:\/\/doi.org\/10.1371\/journal.pcbi.1002012","journal-title":"PLoS Comput Biol"},{"issue":"4","key":"884_CR11","doi-asserted-by":"publisher","first-page":"1561","DOI":"10.1152\/jn.00792.2018","volume":"121","author":"DJ Palidis","year":"2019","unstructured":"Palidis DJ, Cashaback JGA, Gribble PL (2019) Neural signatures of reward and sensory error feedback processing in motor learning. J Neurophysiol 121(4):1561\u20131574. https:\/\/doi.org\/10.1152\/jn.00792.2018","journal-title":"J Neurophysiol"},{"issue":"9","key":"884_CR12","doi-asserted-by":"publisher","first-page":"4015","DOI":"10.1523\/JNEUROSCI.3244-14.2015","volume":"35","author":"SE Pekny","year":"2015","unstructured":"Pekny SE, Izawa J, Shadmehr R (2015) Reward-dependent modulation of movement variability. J Neurosci 35(9):4015\u20134024. https:\/\/doi.org\/10.1523\/JNEUROSCI.3244-14.2015","journal-title":"J Neurosci"},{"key":"884_CR13","unstructured":"Ranjan, T., & Smith, M. (2018). Cancellation of internally-generated errors from the signal driving motor adaptation. Motor Learning and Motor Control. http:\/\/www.motor-conference.org\/abstracts\/228.pdf"},{"key":"884_CR14","doi-asserted-by":"publisher","first-page":"3275","DOI":"10.1152\/jn.00442.2018","volume":"120","author":"A Sidarta","year":"2018","unstructured":"Sidarta A, van Vugt F, Ostry DJ (2018) Somatosensory working memory in human reinforcement-based motor learning. J Neurophysiol 120:3275\u20133286. https:\/\/doi.org\/10.1152\/jn.00442.2018","journal-title":"J Neurophysiol"},{"key":"884_CR15","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1016\/j.cobeha.2018.01.004","volume":"20","author":"D Sternad","year":"2018","unstructured":"Sternad D (2018) It\u2019s not (only) the mean that matters: variability, noise and exploration in skill learning. Curr Opin Behav Sci 20:183\u2013195. https:\/\/doi.org\/10.1016\/j.cobeha.2018.01.004","journal-title":"Curr Opin Behav Sci"},{"key":"884_CR16","doi-asserted-by":"publisher","unstructured":"Sutton, R. S., & Barto, A. G. (2017). Reinforcement learning: an introduction (2nd ed.). MIT Press. https:\/\/doi.org\/10.1016\/S1364-6613(99)01331-5","DOI":"10.1016\/S1364-6613(99)01331-5"},{"issue":"1","key":"884_CR17","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1093\/brain\/awv329","volume":"139","author":"AS Therrien","year":"2016","unstructured":"Therrien AS, Wolpert DM, Bastian AJ (2016) Effective Reinforcement learning following cerebellar damage requires a balance between exploration and motor noise. Brain 139(1):101\u2013114. https:\/\/doi.org\/10.1093\/brain\/awv329","journal-title":"Brain"},{"key":"884_CR18","doi-asserted-by":"publisher","unstructured":"Therrien, A. S., Wolpert, D. M., & Bastian, A. J. (2018). Increasing Motor Noise Impairs Reinforcement Learning in Healthy Individuals. Eneuro, 5(3), e0050\u201318.2018. https:\/\/doi.org\/10.1523\/ENEURO.0050-18.2018","DOI":"10.1523\/ENEURO.0050-18.2018"},{"key":"884_CR19","unstructured":"Thirey, B., & Hickman, R. (2015). Distribution of Euclidean Distances Between Randomly Distributed Gaussian Points in n-Space. http:\/\/arxiv.org\/abs\/1508.02238"},{"key":"884_CR20","doi-asserted-by":"publisher","first-page":"797","DOI":"10.1152\/jn.00390.2018","volume":"122","author":"S Uehara","year":"2019","unstructured":"Uehara S, Mawase F, Therrien AS, Cherry-Allen KM, Celnik PA (2019) Interactions between motor exploration and reinforcement learning. J Neurophysiol 122:797\u2013808. https:\/\/doi.org\/10.1152\/jn.00390.2018","journal-title":"J Neurophysiol"},{"issue":"3","key":"884_CR21","doi-asserted-by":"publisher","first-page":"406","DOI":"10.1016\/j.neuron.2009.06.025","volume":"63","author":"RJ van Beers","year":"2009","unstructured":"van Beers RJ (2009) Motor learning is optimally tuned to the properties of motor noise. Neuron 63(3):406\u2013417. https:\/\/doi.org\/10.1016\/j.neuron.2009.06.025","journal-title":"Neuron"},{"issue":"4","key":"884_CR22","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0226789","volume":"15","author":"NM van Mastrigt","year":"2020","unstructured":"van Mastrigt NM, Smeets JBJ, van der Kooij K (2020) Quantifying exploration in reward-based motor learning. PLoS ONE 15(4):e0226789. https:\/\/doi.org\/10.1371\/journal.pone.0226789","journal-title":"PLoS ONE"},{"issue":"1","key":"884_CR23","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1037\/xlm0000573","volume":"45","author":"K van der Kooij","year":"2018","unstructured":"van der Kooij K, Smeets JBJ (2018) Reward-based motor adaptation can generalize across actions. J Exp Psychol Learn Mem Cogn 45(1):71\u201381. https:\/\/doi.org\/10.1037\/xlm0000573","journal-title":"J Exp Psychol Learn Mem Cogn"},{"issue":"3","key":"884_CR24","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0193002","volume":"13","author":"K van der Kooij","year":"2018","unstructured":"van der Kooij K, Oostwoud Wijdenes L, Rigterink T, Overvliet KE, Smeets JBJ (2018) Reward abundance interferes with error-based learning in a visuomotor adaptation task. PLoS ONE 13(3):e0193002. https:\/\/doi.org\/10.1371\/journal.pone.0193002","journal-title":"PLoS ONE"},{"key":"884_CR25","doi-asserted-by":"publisher","unstructured":"van der Vliet, R., Frens, M. A., de Vreede, L., Jonker, Z. D., Ribbers, G. M., Selles, R. W., van der Geest, J. N., & Donchin, O. (2018). Individual Differences in Motor Noise and Adaptation Rate Are Optimally Related. Eneuro, 5(4), ENEURO.0170\u201318.2018. https:\/\/doi.org\/10.1523\/ENEURO.0170-18.2018","DOI":"10.1523\/ENEURO.0170-18.2018"}],"container-title":["Biological Cybernetics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00422-021-00884-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00422-021-00884-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00422-021-00884-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,8,23]],"date-time":"2021-08-23T15:16:13Z","timestamp":1629731773000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00422-021-00884-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8]]},"references-count":25,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,8]]}},"alternative-id":["884"],"URL":"https:\/\/doi.org\/10.1007\/s00422-021-00884-8","relation":{},"ISSN":["0340-1200","1432-0770"],"issn-type":[{"value":"0340-1200","type":"print"},{"value":"1432-0770","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8]]},"assertion":[{"value":"22 March 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 June 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 August 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}