{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T06:04:54Z","timestamp":1779689094606,"version":"3.53.1"},"reference-count":69,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T00:00:00Z","timestamp":1779148800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Behavioral Sciences"],"abstract":"<jats:p>As artificial intelligence systems continue to overcome evermore challenging tasks, researchers have suggested that the time is ripe to begin evaluating these systems along more psychologically inspired lines. This study seeks to build upon these recommendations by evaluating two machine learning models, A-Learning and Proximal Policy Optimisation, for the cognitive capability known as numerosity. In our experiment, these two models were embodied in a three-dimensional virtual environment, known as Animal-AI, and tested in a psychologically inspired numerosity experiment. In contrast to previous research, A-Learning failed to reliably express numerosity capabilities, as did Proximal Policy Optimisation. Both models displayed a tendency to overfit to the first policy that provided rewarding feedback. These results suggest that predicting the cognitive capabilities of machine learning models once embodied is non-trivial, and confounding factors such as environmental properties and perceptual processes complicate the expression of numerosity capabilities. Building on these findings, it is suggested that future researchers pay greater attention to the influence of environmental factors and perceptual mechanisms on the machine learning models they are developing, especially if such models are to be embodied in a virtual- or real-world environment.<\/jats:p>","DOI":"10.3390\/bs16050813","type":"journal-article","created":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T05:33:54Z","timestamp":1779687234000},"page":"813","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Discerning Quantity: Numerosity in Two Embodied Machine Learning Agents"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0098-7701","authenticated-orcid":false,"given":"Niall","family":"Donnelly","sequence":"first","affiliation":[{"name":"Faculty of Environment, Science and Economy, University of Exeter, Exeter EX4 4RN, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3650-6487","authenticated-orcid":false,"given":"Edward","family":"Keedwell","sequence":"additional","affiliation":[{"name":"Faculty of Environment, Science and Economy, University of Exeter, Exeter EX4 4RN, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,5,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.tics.2020.10.009","article-title":"A sensorimotor numerosity system","volume":"25","author":"Anobile","year":"2021","journal-title":"Trends in Cognitive Sciences"},{"key":"ref_2","unstructured":"Asad, R., Babanezhad, R., and Vaswani, S. (2025). Revisiting actor-critic methods in discrete action off-policy reinforcement learning. arXiv."},{"key":"ref_3","unstructured":"Beyret, B., Hern\u00e1ndez-Orallo, J., Cheke, L., Halina, M., Shanahan, M., and Crosby, M. (2019). The animal-AI environment: Training and testing animal-like artificial cognition. arXiv."},{"key":"ref_4","unstructured":"Brannon, E. M. (2005). What animals know about numbers. The handbook of mathematical cognition, Psychology Press."},{"key":"ref_5","unstructured":"Burden, J., Voudouris, K., Burnell, R., Rutar, D., Cheke, L., and Hern\u00e1ndez-Orallo, J. (2023). Inferring capabilities from task performance with bayesian triangulation. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Burnell, R., Burden, J., Rutar, D., Voudouris, K., Cheke, L., and Hern\u00e1ndez-Orallo, J. (2022). Not a number: Identifying instance features for capability-oriented evaluation. Thirty-first international joint conference on artificial intelligence IJCAI-22, IJCAI.","DOI":"10.24963\/ijcai.2022\/392"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1126\/science.adf6369","article-title":"Rethink reporting of evaluation results in AI","volume":"380","author":"Burnell","year":"2023","journal-title":"Science"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Call, J. E., Burghardt, G. M., Pepperberg, I. M., Snowdon, C. T., and Zentall, T. E. (2017). Apa handbook of comparative psychology: Perception, learning, and cognition, American Psychological Association.","DOI":"10.1037\/0000012-000"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Capel, T., and Brereton, M. (2023). What is human-centered about human-centered AI? A map of the research landscape. Proceedings of the 2023 chi conference on human factors in computing systems, ACM.","DOI":"10.1145\/3544548.3580959"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Cardoso, R. P., Donnelly, N., Keedwell, E., Cheke, L., and Shanahan, M. (2023). What is a stimulus? A computational perspective on an associative learning model. Alife 2023: Ghost in the machine: Proceedings of the 2023 artificial life conference, MIT.","DOI":"10.1162\/isal_a_00600"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"12536","DOI":"10.1038\/ncomms12536","article-title":"Spontaneous perception of numerosity in humans","volume":"7","author":"Cicchini","year":"2016","journal-title":"Nature Communications"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1038\/s42256-019-0050-3","article-title":"The animal-AI olympics","volume":"1","author":"Crosby","year":"2019","journal-title":"Nature Machine Intelligence"},{"key":"ref_13","first-page":"164","article-title":"The animal-AI testbed and competition. Neurips 2019 competition and demonstration track","volume":"123","author":"Crosby","year":"2020","journal-title":"PMLR"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/j.neuron.2015.09.026","article-title":"The infancy of the human brain","volume":"88","author":"Spelke","year":"2015","journal-title":"Neuron"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Deowan, M. E., Yousha, M. S. Y., Hossain, T. M., Hassan, S., and Marxer, R. (2025). Optimizing underwater robot navigation: A study of DRL algorithms and multi-modal sensor fusion. IEEE international conference on robotics & automation (ICRA), IEEE.","DOI":"10.1109\/ICRA55743.2025.11127836"},{"key":"ref_16","unstructured":"Donnelly, N. (2025). Resisting temptation: A delayed gratification experiment of two embodied reinforcement learning agents. Superintelligence conference, University of Exeter."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/3477.484436","article-title":"Ant system: Optimization by a colony of cooperating agents","volume":"26","author":"Dorigo","year":"1996","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Elallid, B. B., Benamar, N., Bagaa, M., and Hadjadj-Aoul, Y. (2024). Enhancing autonomous driving navigation using soft actor-critic. Future Internet, 16, (7).","DOI":"10.3390\/fi16070238"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1016\/j.tics.2004.05.002","article-title":"Core systems of number","volume":"8","author":"Feigenson","year":"2004","journal-title":"Trends in Cognitive Sciences"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1166","DOI":"10.3758\/s13423-020-01749-0","article-title":"A-learning: A new formulation of associative learning theory","volume":"27","author":"Ghirlanda","year":"2020","journal-title":"Psychonomic Bulletin & Review"},{"key":"ref_21","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International conference on machine learning, PMLR."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1037\/a0040180","article-title":"Overcoming associative learning","volume":"130","author":"Haselgrove","year":"2016","journal-title":"Journal of Comparative Psychology"},{"key":"ref_23","unstructured":"Heiman, G. W. (2001). Understanding research methods and statistics: An integrated introduction for psychology, Houghton, Mifflin and Company."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1007\/s10462-016-9505-7","article-title":"Evaluation in artificial intelligence: From task-oriented to ability-oriented measurement","volume":"48","year":"2017","journal-title":"Artificial Intelligence Review"},{"key":"ref_25","unstructured":"Hern\u00e1ndez-Orallo, J. (2020). AI evaluation: On broken yardsticks and measurement scales. Workshop on evaluating evaluation of AI systems at AAAI, AAAI."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1037\/h0076219","article-title":"Second-order conditioning with food unconditioned stimulus","volume":"88","author":"Holland","year":"1975","journal-title":"Journal of Comparative and Physiological Psychology"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1338","DOI":"10.1037\/rev0000368","article-title":"Higher-order conditioning: A critical review and computational model","volume":"129","author":"Honey","year":"2022","journal-title":"Psychological Review"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"141","DOI":"10.2307\/249115","article-title":"Methodological issues in experimental IS research: Experiences and recommendations","volume":"9","author":"Jarvenpaa","year":"1985","journal-title":"MIS Quarterly"},{"key":"ref_29","unstructured":"Johnson, B., and Weitzenfeld, A. (2025). Hierarchical reinforcement learning in multi-goal spatial navigation with autonomous mobile robots. arXiv."},{"key":"ref_30","unstructured":"Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., and Lange, D. (2018). Unity: A general platform for intelligent agents. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1167\/16.3.30","article-title":"Numerosity as a topological invariant","volume":"16","author":"Kluth","year":"2016","journal-title":"Journal of Vision"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"e253","DOI":"10.1017\/S0140525X16001837","article-title":"Building machines that learn and think like people","volume":"40","author":"Lake","year":"2017","journal-title":"Behavioral and Brain Sciences"},{"key":"ref_33","unstructured":"Lehuger, A., and Crosby, M. (2021). Fixed \u03b2 -VAE Encoding for curious exploration in complex 3D environments. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"e164","DOI":"10.1017\/S0140525X16000960","article-title":"From \u201csense of number\u201d to \u201csense of magnitude\u201d: The role of continuous magnitudes in numerical cognition","volume":"40","author":"Leibovich","year":"2017","journal-title":"Behavioral and Brain Sciences"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Lind, J., and Vinken, V. (2021). Can associative learning be the general process for intelligent behavior in non-human animals?. bioRxiv.","DOI":"10.1101\/2021.12.15.472737"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1007\/s11571-022-09882-3","article-title":"Memristive neural network circuit implementation of associative learning with overshadowing and blocking","volume":"17","author":"Liu","year":"2023","journal-title":"Cognitive Neurodynamics"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Liu, Y., Wang, C., Zhao, C., Wu, H., and Wei, Y. (2024). A soft actor-critic deep reinforcement-learning-based robot navigation method using LiDAR. Remote Sensing, 16, (12).","DOI":"10.3390\/rs16122072"},{"key":"ref_38","unstructured":"Lucas, J., and Pr\u00e9mont-Schwarz, I. (2024). Articulated Animal AI: An environment for animal-like cognition in a limbed agent. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Mackworth, A. K. (1993). On seeing robots. Computer vision: Systems, theory and applications, World Scientific.","DOI":"10.1142\/9789814343312_0001"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1145\/3392663","article-title":"Insights for AI from the human mind","volume":"64","author":"Marcus","year":"2020","journal-title":"Communications of the ACM"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"407","DOI":"10.4103\/aca.ACA_94_19","article-title":"Application of student\u2019s t-test, analysis of variance, and covariance","volume":"22","author":"Mishra","year":"2019","journal-title":"Annals of Cardiac Anaesthesia"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.lmot.2015.01.003","article-title":"Analysis of the role of stimulus comparison in discrimination learning in Pigeons","volume":"49","author":"Hall","year":"2015","journal-title":"Learning and Motivation"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"jeb218289","DOI":"10.1242\/jeb.218289","article-title":"Neuroethology of number sense across the animal kingdom","volume":"224","author":"Nieder","year":"2021","journal-title":"Journal of Experimental Biology"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2495","DOI":"10.1007\/s00426-020-01428-8","article-title":"A robot that counts like a child: A developmental model of counting and pointing","volume":"86","author":"Pecyna","year":"2022","journal-title":"Psychological Research"},{"key":"ref_45","first-page":"1","article-title":"Stable-baselines3: Reliable reinforcement learning implementations","volume":"22","author":"Raffin","year":"2021","journal-title":"Journal of Machine Learning Research"},{"key":"ref_46","unstructured":"Rutar, D., Markelius, A., Voudouris, K., Hern\u00e1ndez-Orallo, J., and Cheke, L. (2025). Cognitive science-inspired evaluation of core capabilities for object understanding in AI. arXiv."},{"key":"ref_47","unstructured":"Sabathiel, S., McClelland, J. L., and Solstad, T. (1, January July). A computational model of learning to count in a multimodal, interactive environment. Annual Meeting of the Cognitive Science Society, Virtual."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Sabathiel, S., Solstad, T., Testolin, A., and Petruzzellis, F. (2022). Self-communicating deep reinforcement learning agents develop external number representations. Proceedings of the northern lights deep learning workshop, Septentrio Academic Publishing.","DOI":"10.7557\/18.6291"},{"key":"ref_49","first-page":"42","article-title":"Machine learning","volume":"62","author":"Samuel","year":"1959a","journal-title":"The Technology Review"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1147\/rd.33.0210","article-title":"Some studies in machine learning using the game of checkers","volume":"3","author":"Samuel","year":"1959b","journal-title":"IBM Journal of Research and Development"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Schmidt, M. S. (2022). Numerosity. Encyclopedia of animal cognition and behavior, Springer.","DOI":"10.1007\/978-3-319-55065-7_78"},{"key":"ref_52","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1080\/14786445008521796","article-title":"XXII. Programming a computer for playing chess","volume":"41","author":"Shannon","year":"1950","journal-title":"The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Shneiderman, B. (2022). Human-centered AI, Oxford University Press.","DOI":"10.1093\/oso\/9780192845290.001.0001"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of Go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1140","DOI":"10.1126\/science.aar6404","article-title":"A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play","volume":"362","author":"Silver","year":"2018","journal-title":"Science"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the game of go without human knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"key":"ref_58","first-page":"23","article-title":"On the problem of scales for the measurement of psychological magnitudes","volume":"22","author":"Stevens","year":"1939","journal-title":"Proceedings of Fechner Day"},{"key":"ref_59","unstructured":"Sutton, R. S., and Barto, A. G. (2018). Reinforcement learning: An introduction, MIT press."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"10045","DOI":"10.1038\/s41598-020-66838-5","article-title":"Visual sense of number vs. sense of magnitude in humans and machines","volume":"10","author":"Testolin","year":"2020","journal-title":"Scientific Reports"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"79","DOI":"10.3758\/BF03334444","article-title":"Relative numerousness judgments by squirrel monkeys","volume":"16","author":"Thomas","year":"1980","journal-title":"Bulletin of the Psychonomic Society"},{"key":"ref_62","unstructured":"Tsividis, P., Pouncy, T., Xu, J. L., Tenenbaum, J. B., and Gershman, S. J. (2017). Human learning in Atari. AAAI spring symposia, AAAI."},{"key":"ref_63","unstructured":"Upadhyay, N., and Varma, S. (, January December). CNN models\u2019 sensitivity to numerosity concepts. 3rd Workshop on Mathematical Reasoning and AI at NeurIPS, New Orleans, LA, USA."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"105","DOI":"10.3819\/CCBR.2024.190009","article-title":"The future is computational comparative cognition","volume":"19","author":"Voudouris","year":"2024","journal-title":"Comparative Cognition and Behavior Reviews"},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Voudouris, K., Crosby, M., Beyret, B., Hern\u00e1ndez-Orallo, J., Shanahan, M., Halina, M., and Cheke, L. G. (2022). Direct human-AI comparison in the animal-AI environment. Frontiers in Psychology, 13.","DOI":"10.3389\/fpsyg.2022.711821"},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"107","DOI":"10.3758\/s13428-025-02616-3","article-title":"The Animal-AI Environment: A virtual laboratory for comparative cognition and artificial intelligence research","volume":"57","author":"Voudouris","year":"2025","journal-title":"Behavior Research Methods"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1037\/h0059328","article-title":"Effect of amount of reward and consummative activity on learning in chickens","volume":"31","author":"Wolfe","year":"1941","journal-title":"Journal of Comparative Psychology"},{"key":"ref_68","first-page":"1820","article-title":"On numerosity of deep neural networks","volume":"33","author":"Zhang","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_69","doi-asserted-by":"crossref","unstructured":"Zhu, J., Wu, F., and Zhao, J. (2021). An overview of the action space for deep reinforcement learning. Proceedings of the 2021 4th international conference on algorithms, computing and artificial intelligence, ACM.","DOI":"10.1145\/3508546.3508598"}],"container-title":["Behavioral Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-328X\/16\/5\/813\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T05:56:48Z","timestamp":1779688608000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-328X\/16\/5\/813"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,19]]},"references-count":69,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,5]]}},"alternative-id":["bs16050813"],"URL":"https:\/\/doi.org\/10.3390\/bs16050813","relation":{},"ISSN":["2076-328X"],"issn-type":[{"value":"2076-328X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,19]]}}}