{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,25]],"date-time":"2025-03-25T18:09:15Z","timestamp":1742926155276,"version":"3.40.3"},"publisher-location":"Cham","reference-count":23,"publisher":"Springer Nature Switzerland","isbn-type":[{"type":"print","value":"9783031442063"},{"type":"electronic","value":"9783031442070"}],"license":[{"start":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T00:00:00Z","timestamp":1672531200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,22]],"date-time":"2023-09-22T00:00:00Z","timestamp":1695340800000},"content-version":"vor","delay-in-days":264,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Multimodal integration is a key component of allowing robots to perceive the world. Multimodality comes with multiple challenges that have to be considered, such as how to integrate and fuse the data. In this paper, we compare different possibilities of fusing visual, tactile and proprioceptive data. The data is directly recorded on the NICOL robot in an experimental setup in which the robot has to classify containers and their content. Due to the different nature of the containers, the use of the modalities can wildly differ between the classes. We demonstrate the superiority of multimodal solutions in this use case and evaluate three fusion strategies that integrate the data at different time steps. We find that the accuracy of the best fusion strategy is 15% higher than the best strategy using only one singular sense.<\/jats:p>","DOI":"10.1007\/978-3-031-44207-0_37","type":"book-chapter","created":{"date-parts":[[2023,9,21]],"date-time":"2023-09-21T14:03:51Z","timestamp":1695305031000},"page":"444-456","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Clarifying the\u00a0Half Full or\u00a0Half Empty Question: Multimodal Container Classification"],"prefix":"10.1007","author":[{"given":"Josua","family":"Spisak","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthias","family":"Kerzel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefan","family":"Wermter","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,9,22]]},"reference":[{"issue":"3","key":"37_CR1","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1109\/TAMD.2011.2106782","volume":"3","author":"C Castellini","year":"2011","unstructured":"Castellini, C., Tommasi, T., Noceti, N., Odone, F., Caputo, B.: Using object affordances to improve object recognition. IEEE transactions on autonomous mental development 3(3), 207\u2013215 (2011)","journal-title":"IEEE transactions on autonomous mental development"},{"key":"37_CR2","doi-asserted-by":"crossref","unstructured":"Chitta, S., Piccoli, M., Sturm, J.: Tactile object class and internal state recognition for mobile manipulation. In: 2010 IEEE International Conference on Robotics and Automation. pp. 2342\u20132348. IEEE (2010)","DOI":"10.1109\/ROBOT.2010.5509923"},{"key":"37_CR3","unstructured":"Cui, Z.J., Wang, Y., Shafiullah, N.M.M., Pinto, L.: From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10047 (2022)"},{"key":"37_CR4","doi-asserted-by":"crossref","unstructured":"Do, C., Schubert, T., Burgard, W.: A probabilistic approach to liquid level detection in cups using an rgb-d camera. In: 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 2075\u20132080. IEEE (2016)","DOI":"10.1109\/IROS.2016.7759326"},{"issue":"2","key":"37_CR5","first-page":"67","volume":"1","author":"JJ Gibson","year":"1977","unstructured":"Gibson, J.J.: The theory of affordances. Hilldale, USA 1(2), 67\u201382 (1977)","journal-title":"Hilldale, USA"},{"key":"37_CR6","doi-asserted-by":"crossref","unstructured":"G\u00fcler, P., Bekiroglu, Y., Gratal, X., Pauwels, K., Kragic, D.: What\u2019s in the container? classifying object contents from vision and touch. In: 2014 IEEE\/RSJ International Conference on Intelligent Robots and Systems. pp. 3961\u20133968. IEEE (2014)","DOI":"10.1109\/IROS.2014.6943119"},{"issue":"1","key":"37_CR7","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1109\/5.554205","volume":"85","author":"DL Hall","year":"1997","unstructured":"Hall, D.L., Llinas, J.: An introduction to multisensor data fusion. Proceedings of the IEEE 85(1), 6\u201323 (1997)","journal-title":"Proceedings of the IEEE"},{"key":"37_CR8","doi-asserted-by":"crossref","unstructured":"Jonetzko, Y., Fiedler, N., Eppe, M., Zhang, J.: Multimodal object analysis with auditory and tactile sensing using recurrent neural networks. In: International Conference on Cognitive Systems and Signal Processing. pp. 253\u2013265. Springer (2020)","DOI":"10.1007\/978-981-16-2336-3_23"},{"key":"37_CR9","doi-asserted-by":"crossref","unstructured":"Kerzel, M., Allgeuer, P., Strahl, E., Frick, N., Habekost, J.G., Eppe, M., Wermter, S.: Nicol: A neuro-inspired collaborative semi-humanoid robot that bridges social interaction and reliable manipulation. arXiv preprint arXiv:2305.08528 (2023)","DOI":"10.1109\/ACCESS.2023.3329370"},{"issue":"9","key":"37_CR10","doi-asserted-by":"publisher","first-page":"1449","DOI":"10.1109\/JPROC.2015.2460697","volume":"103","author":"D Lahat","year":"2015","unstructured":"Lahat, D., Adali, T., Jutten, C.: Multimodal data fusion: an overview of methods, challenges, and prospects. Proceedings of the IEEE 103(9), 1449\u20131477 (2015)","journal-title":"Proceedings of the IEEE"},{"key":"37_CR11","doi-asserted-by":"crossref","unstructured":"Lopes, M., Melo, F.S., Montesano, L.: Affordance-based imitation learning in robots. In: 2007 IEEE\/RSJ international conference on intelligent robots and systems. pp. 1015\u20131021. IEEE (2007)","DOI":"10.1109\/IROS.2007.4399517"},{"issue":"4","key":"37_CR12","doi-asserted-by":"publisher","first-page":"293","DOI":"10.4103\/0256-4602.64604","volume":"27","author":"UG Mangai","year":"2010","unstructured":"Mangai, U.G., Samanta, S., Das, S., Chowdhury, P.R.: A survey of decision fusion and feature fusion strategies for pattern classification. IETE Technical review 27(4), 293\u2013307 (2010)","journal-title":"IETE Technical review"},{"issue":"1","key":"37_CR13","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1109\/TRO.2007.914848","volume":"24","author":"L Montesano","year":"2008","unstructured":"Montesano, L., Lopes, M., Bernardino, A., Santos-Victor, J.: Learning object affordances: from sensory-motor coordination to imitation. IEEE Transactions on Robotics 24(1), 15\u201326 (2008)","journal-title":"IEEE Transactions on Robotics"},{"key":"37_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.dib.2020.106472","volume":"33","author":"D Pau","year":"2020","unstructured":"Pau, D., Kumar, B.P., Namekar, P., Dhande, G., Simonetta, L.: Dataset of sodium chloride sterile liquid in bottles for intravenous administration and fill level monitoring. Data in Brief 33, 106472 (2020)","journal-title":"Data in Brief"},{"key":"37_CR15","doi-asserted-by":"crossref","unstructured":"Piacenza, P., Lee, D., Isler, V.: Pouring by feel: An analysis of tactile and proprioceptive sensing for accurate pouring. In: 2022 International Conference on Robotics and Automation (ICRA). pp. 10248\u201310254. IEEE (2022)","DOI":"10.1109\/ICRA46639.2022.9811898"},{"key":"37_CR16","doi-asserted-by":"crossref","unstructured":"Pieropan, A., Salvi, G., Pauwels, K., Kjellstr\u00f6m, H.: Audio-visual classification and detection of human manipulation actions. In: 2014 IEEE\/RSJ International Conference on Intelligent Robots and Systems. pp. 3045\u20133052. IEEE (2014)","DOI":"10.1109\/IROS.2014.6942983"},{"key":"37_CR17","unstructured":"Pithadiya, K.J., Modi, C.K., Chauhan, J.D.: Selecting the most favourable edge detection technique for liquid level inspection in bottles. International Journal of Computer Information Systems and Industrial Management Applications (IJCISIM) ISSN pp. 2150\u20137988 (2011)"},{"issue":"13","key":"37_CR18","doi-asserted-by":"publisher","first-page":"2115","DOI":"10.1016\/S0167-8655(03)00079-5","volume":"24","author":"A Ross","year":"2003","unstructured":"Ross, A., Jain, A.: Information fusion in biometrics. Pattern recognition letters 24(13), 2115\u20132125 (2003)","journal-title":"Pattern recognition letters"},{"issue":"5","key":"37_CR19","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1016\/j.dsp.2004.05.001","volume":"14","author":"C Sanderson","year":"2004","unstructured":"Sanderson, C., Paliwal, K.K.: Identity verification using speech and face information. Digital Signal Processing 14(5), 449\u2013480 (2004)","journal-title":"Digital Signal Processing"},{"issue":"1","key":"37_CR20","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/MTS.2018.2795095","volume":"37","author":"A Sciutti","year":"2018","unstructured":"Sciutti, A., Mara, M., Tagliasco, V., Sandini, G.: Humanizing human-robot interaction: On the importance of mutual understanding. IEEE Technology and Society Magazine 37(1), 22\u201329 (2018)","journal-title":"IEEE Technology and Society Magazine"},{"key":"37_CR21","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1007\/s12559-017-9536-7","volume":"10","author":"S Toprak","year":"2018","unstructured":"Toprak, S., Navarro-Guerrero, N., Wermter, S.: Evaluating integration strategies for visuo-haptic object recognition. Cognitive computation 10, 408\u2013425 (2018)","journal-title":"Cognitive computation"},{"key":"37_CR22","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1016\/j.patrec.2013.07.003","volume":"36","author":"M Turk","year":"2014","unstructured":"Turk, M.: Multimodal interaction: A review. Pattern recognition letters 36, 189\u2013195 (2014)","journal-title":"Pattern recognition letters"},{"issue":"1\u20132","key":"37_CR23","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1163\/22134808-00002390","volume":"26","author":"S Zmigrod","year":"2013","unstructured":"Zmigrod, S., Hommel, B.: Feature integration across multimodal perception and action: a review. Multisensory research 26(1\u20132), 143\u2013157 (2013)","journal-title":"Multisensory research"}],"container-title":["Lecture Notes in Computer Science","Artificial Neural Networks and Machine Learning \u2013 ICANN 2023"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-44207-0_37","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,22]],"date-time":"2023-12-22T08:25:51Z","timestamp":1703233551000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-44207-0_37"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"ISBN":["9783031442063","9783031442070"],"references-count":23,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-44207-0_37","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2023]]},"assertion":[{"value":"22 September 2023","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"ICANN","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Artificial Neural Networks","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Heraklion","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Greece","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2023","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26 September 2023","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"29 September 2023","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"32","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"icann2023","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/e-nns.org\/icann2023\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Single-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"easyacademia.org","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"947","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"426","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"22","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"45% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"2.4","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"4","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"No","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"type of other papers accepted : 9 Abstract","order":10,"name":"additional_info_on_review_process","label":"Additional Info on Review Process","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}