{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T14:30:52Z","timestamp":1776090652927,"version":"3.50.1"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2025,3,21]],"date-time":"2025-03-21T00:00:00Z","timestamp":1742515200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,3,21]],"date-time":"2025-03-21T00:00:00Z","timestamp":1742515200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004063","name":"Knut och Alice Wallenbergs Stiftelse","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004063","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J of Soc Robotics"],"published-print":{"date-parts":[[2025,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    This paper introduces UJI-Butler, an innovative multi-robot framework that blends symbolic and non-symbolic artificial intelligence methods. Unlike previous systems, UJI-Butler integrates large language models (LLMs) with a knowledge base akin to RAG-based systems, while imposing logical reasoning on LLM-generated results. It facilitates multi-modal interaction with human users through speech, sign language, and physical interaction, fostering a human-in-the-loop learning paradigm. By acquiring new knowledge through verbal communication and mastering manipulation skills via human-lead-through programming, UJI-Butler enhances transparency and trust by incorporating human feedback during operations. Experimental results demonstrate that UJI-Butler\u2019s combination of symbolic and non-symbolic AI offers intuitive interaction and accelerates the learning process with experience. It adeptly stores and utilizes knowledge gained from verbal communication, recognizing hand gestures for requests. Additionally, UJI-Butler successfully performs user-taught physical skills and generalizes them to varying object sizes and locations. The explicit nature of acquired knowledge enables seamless transferability to other platforms and modification by human users. The code of the whole project is available on\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/orgs\/UR5-Robotic-Intelligence\/repositories\" ext-link-type=\"uri\">Github<\/jats:ext-link>\n                    , in addition, video demonstrations of the UJI-Butler system are available online in a\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/youtube.com\/playlist?list=PLKYWqKMe8hVKP9UAvhe-WZa0OisrNtL-v\" ext-link-type=\"uri\">Youtube Playlist<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1007\/s12369-025-01234-5","type":"journal-article","created":{"date-parts":[[2025,3,22]],"date-time":"2025-03-22T22:04:46Z","timestamp":1742681086000},"page":"2883-2903","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["UJI-Butler: A Symbolic\/Non-symbolic Robotic System that Learns Through Multi-modal Interaction"],"prefix":"10.1007","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7206-4551","authenticated-orcid":false,"given":"Abdelrhman","family":"Bassiouny","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9094-3238","authenticated-orcid":false,"given":"Ahmed H.","family":"Elsayed","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6398-8488","authenticated-orcid":false,"given":"Zoe","family":"Falomir","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6227-3758","authenticated-orcid":false,"given":"Angel P.","family":"del Pobil","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,3,21]]},"reference":[{"issue":"1","key":"1234_CR1","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1038\/scientificamerican0107-58","volume":"296","author":"B Gates","year":"2007","unstructured":"Gates B (2007) A robot in every home. Sci Am 296(1):58\u201365","journal-title":"Sci Am"},{"key":"1234_CR2","doi-asserted-by":"crossref","unstructured":"Martinez-Martin E, Pobil AP (2018). In: Costa A, Julian V, Novais P (eds) Personal robot assistants for elderly care: an overview. Springer, Cham, pp 77\u201391","DOI":"10.1007\/978-3-319-62530-0_5"},{"issue":"3","key":"1234_CR3","doi-asserted-by":"publisher","first-page":"79","DOI":"10.3390\/robotics12030079","volume":"12","author":"C Taesi","year":"2023","unstructured":"Taesi C, Aggogeri F, Pellegrini N (2023) Cobot applications-recent advances and challenges. Robotics 12(3):79","journal-title":"Robotics"},{"issue":"1\u20132","key":"1234_CR4","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1016\/0921-8890(95)00004-Y","volume":"15","author":"S Thrun","year":"1995","unstructured":"Thrun S, Mitchell TM (1995) Lifelong robot learning. Robot Auton Syst 15(1\u20132):25\u201346","journal-title":"Robot Auton Syst"},{"key":"1234_CR5","unstructured":"Schlimmer JC, Fisher D (1986) A case study of incremental concept induction. In: Proceedings of the Fifth AAAI National Conference on Artificial Intelligence, pp 496\u2013501"},{"key":"1234_CR6","doi-asserted-by":"crossref","unstructured":"Sutton RS, Whitehead SD (1993) Online learning with random representations. In: Proceedings of the Tenth International Conference on International Conference on Machine Learning. ICML\u201993, pp 314\u2013321. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA","DOI":"10.1016\/B978-1-55860-307-3.50047-2"},{"issue":"1","key":"1234_CR7","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1023\/A:1007331723572","volume":"28","author":"MB Ring","year":"1997","unstructured":"Ring MB (1997) Child: a first step towards continual learning. Mach Learn 28(1):77\u2013104","journal-title":"Mach Learn"},{"key":"1234_CR8","doi-asserted-by":"crossref","unstructured":"McCloskey M, Cohen NJ (1989) Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation, vol 24, pp 109\u2013165. Academic Press","DOI":"10.1016\/S0079-7421(08)60536-8"},{"key":"1234_CR9","doi-asserted-by":"crossref","unstructured":"Tenorth M, Beetz M (2009) Knowrob-knowledge processing for autonomous personal robots. In: 2009 IEEE\/RSJ International Conference on Intelligent Robots and Systems, pp 4261\u20134266","DOI":"10.1109\/IROS.2009.5354602"},{"key":"1234_CR10","first-page":"1877","volume":"33","author":"T Brown","year":"2020","unstructured":"Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877\u20131901","journal-title":"Adv Neural Inf Process Syst"},{"key":"1234_CR11","unstructured":"Bechhofer S, Harmelen F, Hendler J, Horrocks I, McGuinness D, Patel-Schneijder P, Stein LA (2004) OWL web ontology language reference. Recommendation, World Wide Web Consortium (W3C)"},{"issue":"1\u20132","key":"1234_CR12","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1017\/S1471068411000494","volume":"12","author":"J Wielemaker","year":"2012","unstructured":"Wielemaker J, Schrijvers T, Triska M, Lager T (2012) Swi-prolog. Theory Pract Logic Program 12(1\u20132):67\u201396","journal-title":"Theory Pract Logic Program"},{"key":"1234_CR13","unstructured":"Thosar M, Zug S, Skaria A, Jain A (2018) A review of knowledge bases for service robots in household environments. In: 6th International Workshop on Artificial Intelligence and Cognition"},{"key":"1234_CR14","first-page":"9459","volume":"33","author":"P Lewis","year":"2020","unstructured":"Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, K\u00fcttler H, Lewis M, Yih W-T, Rockt\u00e4schel T et al (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst 33:9459\u20139474","journal-title":"Adv Neural Inf Process Syst"},{"key":"1234_CR15","first-page":"707","volume":"10","author":"VI Levenshtein","year":"1966","unstructured":"Levenshtein VI et al (1966) Binary codes capable of correcting deletions, insertions, and reversals. Soviet Phys Doklady 10:707\u2013710","journal-title":"Soviet Phys Doklady"},{"key":"1234_CR16","doi-asserted-by":"crossref","unstructured":"Wang C-Y, Bochkovskiy A, Liao H-YM (2023) Yolov7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp 7464\u20137475","DOI":"10.1109\/CVPR52729.2023.00721"},{"key":"1234_CR17","unstructured":"Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J et al (2021) Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp 8748\u20138763"},{"issue":"6","key":"1234_CR18","doi-asserted-by":"publisher","first-page":"381","DOI":"10.1145\/358669.358692","volume":"24","author":"MA Fischler","year":"1981","unstructured":"Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381\u2013395","journal-title":"Commun ACM"},{"key":"1234_CR19","unstructured":"McGlinn I (2021) Sign language alphabets from around the world - ASL - AI-media. Ai-Media creating accessibility, one word at a time"},{"key":"1234_CR20","unstructured":"Lee D (2022) American sign language letters object detection dataset. https:\/\/public.roboflow.com\/object-detection\/american-sign-language-letters"},{"key":"1234_CR21","unstructured":"SmartHandsCA (2017) Easiest way to learn your ASL ABCS | slowest alphabet lesson. YouTube"},{"issue":"1","key":"1234_CR22","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/TRO.2006.889486","volume":"23","author":"G Grisetti","year":"2007","unstructured":"Grisetti G, Stachniss C, Burgard W (2007) Improved techniques for grid mapping with Rao-Blackwellized particle filters. IEEE Trans Robot 23(1):34\u201346","journal-title":"IEEE Trans Robot"},{"key":"1234_CR23","doi-asserted-by":"crossref","unstructured":"Murphy K, Russell S (2001). In: Doucet A, Freitas N, Gordon N (eds) Rao-Blackwellised particle filtering for dynamic Bayesian networks. Springer, New York, NY, pp 499\u2013515","DOI":"10.1007\/978-1-4757-3437-9_24"},{"key":"1234_CR24","doi-asserted-by":"crossref","unstructured":"Chen C, Wu X, Han L, Ou Y (2011) Butler robot. In: 2011 IEEE International Conference on Information and Automation, pp 732\u2013737","DOI":"10.1109\/ICINFA.2011.5949090"},{"key":"1234_CR25","doi-asserted-by":"crossref","unstructured":"Moore RK (2013). In: Trappl R (ed) Spoken language processing: Where do we go from here? Springer, Berlin, Heidelberg, pp 119\u2013133","DOI":"10.1007\/978-3-642-37346-6_10"},{"key":"1234_CR26","doi-asserted-by":"crossref","unstructured":"Lee M, Heo Y, Park J, Yang H-D, Jang H-D, Benz P, Park H, Kweon IS, Oh J-H (2019) Fast perception, planning, and execution for a robotic butler: Wheeled humanoid m-hubo. In: 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 5444\u20135451","DOI":"10.1109\/IROS40897.2019.8968064"},{"key":"1234_CR27","doi-asserted-by":"crossref","unstructured":"Gunawan AAS, Clemons B, Halim IF, Anderson K, Adianti MP (2023) Development of e-butler: introduction of robot system in hospitality with mobile application. Procedia Comput Sci 216:67\u201376","DOI":"10.1016\/j.procs.2022.12.112"},{"key":"1234_CR28","unstructured":"Brohan A, Chebotar Y, Finn C, Hausman K, Herzog A, Ho D, Ibarz J, Irpan A, Jang E, Julian R et al (2023) Do as I can, not as I say: grounding language in robotic affordances. In: Conference on Robot Learning, pp 287\u2013318"},{"key":"1234_CR29","doi-asserted-by":"publisher","first-page":"118669","DOI":"10.1016\/j.eswa.2022.118669","volume":"212","author":"A Salaberria","year":"2023","unstructured":"Salaberria A, Azkune G, Lacalle OL, Soroa A, Agirre E (2023) Image captioning for effective use of language models in knowledge-based visual question answering. Expert Syst Appl 212:118669","journal-title":"Expert Syst Appl"},{"key":"1234_CR30","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1007\/s11023-020-09548-1","volume":"30","author":"L Floridi","year":"2020","unstructured":"Floridi L, Chiriatti M (2020) GPT-3: its nature, scope, limits, and consequences. Minds Mach 30:681\u2013694","journal-title":"Minds Mach"},{"key":"1234_CR31","unstructured":"Li F, UK A, Hogg DC, Cohn AG (2022) Ontology knowledge-enhanced in-context learning for action-effect prediction. In: Advances in Cognitive Systems"},{"issue":"11","key":"1234_CR32","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1109\/MC.2023.3305206","volume":"56","author":"M Jovanovi\u0107","year":"2023","unstructured":"Jovanovi\u0107 M, Campbell M (2023) Connecting AI: merging large language models and knowledge graph. Computer 56(11):103\u2013108","journal-title":"Computer"},{"key":"1234_CR33","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1016\/j.future.2022.05.014","volume":"135","author":"X Wu","year":"2022","unstructured":"Wu X, Xiao L, Sun Y, Zhang J, Ma T, He L (2022) A survey of human-in-the-loop for machine learning. Futur Gener Comput Syst 135:364\u2013381","journal-title":"Futur Gener Comput Syst"},{"key":"1234_CR34","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1016\/j.aiopen.2023.08.012","volume":"5","author":"X Liu","year":"2023","unstructured":"Liu X, Zheng Y, Du Z, Ding M, Qian Y, Yang Z, Tang J (2023) GPT understands, too. AI Open 5:208","journal-title":"AI Open"},{"key":"1234_CR35","doi-asserted-by":"publisher","first-page":"102085","DOI":"10.1016\/j.rcim.2020.102085","volume":"68","author":"C Nuzzi","year":"2021","unstructured":"Nuzzi C, Pasinetti S, Pagani R, Ghidini S, Beschi M, Coffetti G, Sansoni G (2021) Meguru: a gesture-based robot program builder for meta-collaborative workstations. Robot Comput Integr Manuf 68:102085","journal-title":"Robot Comput Integr Manuf"},{"key":"1234_CR36","doi-asserted-by":"publisher","first-page":"571","DOI":"10.1007\/s12369-015-0307-x","volume":"7","author":"P Uluer","year":"2015","unstructured":"Uluer P, Akal\u0131n N, K\u00f6se H (2015) A new robotic platform for sign language tutoring: Humanoid robots as assistive game companions for teaching sign language. Int J Soc Robot 7:571\u2013585","journal-title":"Int J Soc Robot"},{"key":"1234_CR37","doi-asserted-by":"crossref","unstructured":"Mazhar O, Ramdani S, Navarro B, Passama R, Cherubini A (2018) Towards real-time physical human-robot interaction using skeleton information and hand gestures. In: 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 1\u20136","DOI":"10.1109\/IROS.2018.8594385"},{"key":"1234_CR38","doi-asserted-by":"crossref","unstructured":"Islam MJ, Ho M, Sattar J (2018) Dynamic reconfiguration of mission parameters in underwater human-robot collaboration. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp 6212\u20136219","DOI":"10.1109\/ICRA.2018.8461197"},{"issue":"8","key":"1234_CR39","doi-asserted-by":"publisher","first-page":"1899","DOI":"10.1007\/s11760-021-01930-5","volume":"15","author":"Y Jiang","year":"2021","unstructured":"Jiang Y, Zhao M, Wang C, Wei F, Wang K, Qi H (2021) Diver\u2019s hand gesture recognition and segmentation for human-robot interaction on AUV. Signal Image Video Process 15(8):1899\u20131906","journal-title":"Signal Image Video Process"},{"issue":"3","key":"1234_CR40","doi-asserted-by":"publisher","first-page":"12197","DOI":"10.1111\/exsy.12197","volume":"34","author":"S Ameen","year":"2017","unstructured":"Ameen S, Vadera S (2017) A convolutional neural network to classify American sign language fingerspelling from depth and colour images. Expert Syst 34(3):12197","journal-title":"Expert Syst"},{"key":"1234_CR41","doi-asserted-by":"publisher","first-page":"565","DOI":"10.1016\/j.neucom.2014.06.086","volume":"151","author":"S-Z Li","year":"2015","unstructured":"Li S-Z, Yu B, Wu W, Su S-Z, Ji R-R (2015) Feature learning based on SAE-PCA network for human gesture recognition in RGBD images. Neurocomputing 151:565\u2013573","journal-title":"Neurocomputing"},{"issue":"2","key":"1234_CR42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2735952","volume":"6","author":"A Tang","year":"2015","unstructured":"Tang A, Lu K, Wang Y, Huang J, Li H (2015) A real-time hand posture recognition system using deep neural networks. ACM Trans Intel Syst Technol 6(2):1\u201323","journal-title":"ACM Trans Intel Syst Technol"},{"key":"1234_CR43","doi-asserted-by":"crossref","unstructured":"Koller O, Ney H, Bowden R (2016) Deep hand: How to train a CNN on 1 million hand images when your data is continuous and weakly labelled. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3793\u20133802","DOI":"10.1109\/CVPR.2016.412"}],"container-title":["International Journal of Social Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12369-025-01234-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12369-025-01234-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12369-025-01234-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,24]],"date-time":"2025-12-24T08:41:14Z","timestamp":1766565674000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12369-025-01234-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,21]]},"references-count":43,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2025,12]]}},"alternative-id":["1234"],"URL":"https:\/\/doi.org\/10.1007\/s12369-025-01234-5","relation":{},"ISSN":["1875-4791","1875-4805"],"issn-type":[{"value":"1875-4791","type":"print"},{"value":"1875-4805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,21]]},"assertion":[{"value":"11 February 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 March 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}