{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,4]],"date-time":"2026-03-04T02:58:54Z","timestamp":1772593134052,"version":"3.50.1"},"reference-count":89,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T00:00:00Z","timestamp":1695772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100013348","name":"Innosuisse - Schweizerische Agentur f\u00fcr Innovationsf\u00f6rderung","doi-asserted-by":"publisher","award":["#48342.1 IP-ICT"],"award-info":[{"award-number":["#48342.1 IP-ICT"]}],"id":[{"id":"10.13039\/501100013348","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2023,9,27]]},"abstract":"<jats:p>The increasing number of objects in ubiquitous computing environments creates a need for effective object detection and identification mechanisms that permit users to intuitively initiate interactions with these objects. While multiple approaches to such object detection -- including through visual object detection, fiducial markers, relative localization, or absolute spatial referencing -- are available, each of these suffers from drawbacks that limit their applicability. In this paper, we propose ODIF, an architecture that permits the fusion of object situation information from such heterogeneous sources and that remains vertically and horizontally modular to allow extending and upgrading systems that are constructed accordingly. We furthermore present BLEARVIS, a prototype system that builds on the proposed architecture and integrates computer-vision (CV) based object detection with radio-frequency (RF) angle of arrival (AoA) estimation to identify BLE-tagged objects. In our system, the front camera of a Mixed Reality (MR) head-mounted display (HMD) provides a live image stream to a vision-based object detection module, while an antenna array that is mounted on the HMD collects AoA information from ambient devices. In this way, BLEARVIS is able to differentiate between visually identical objects in the same environment and can provide an MR overlay of information (data and controls) that relates to them. We include experimental evaluations of both, the CV-based object detection and the RF-based AoA estimation, and discuss the applicability of the combined RF and CV pipelines in different ubiquitous computing scenarios. This research can form a starting point to spawn the integration of diverse object detection, identification, and interaction approaches that function across the electromagnetic spectrum, and beyond.<\/jats:p>","DOI":"10.1145\/3610879","type":"journal-article","created":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T15:45:03Z","timestamp":1695829503000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["MR Object Identification and Interaction"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7607-8064","authenticated-orcid":false,"given":"Jannis","family":"Strecker","sequence":"first","affiliation":[{"name":"University of St. Gallen, St. Gallen, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9585-6722","authenticated-orcid":false,"given":"Khakim","family":"Akhunov","sequence":"additional","affiliation":[{"name":"University of Trento, Trento, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3559-906X","authenticated-orcid":false,"given":"Federico","family":"Carbone","sequence":"additional","affiliation":[{"name":"University of Trento, Trento, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4971-2944","authenticated-orcid":false,"given":"Kimberly","family":"Garc\u00eda","sequence":"additional","affiliation":[{"name":"University of St. Gallen, St. Gallen, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2937-0542","authenticated-orcid":false,"given":"Kenan","family":"Bekta\u015f","sequence":"additional","affiliation":[{"name":"University of St. Gallen, St. Gallen, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5825-3567","authenticated-orcid":false,"given":"Andres","family":"Gomez","sequence":"additional","affiliation":[{"name":"TU Braunschweig, Braunschweig, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6367-3454","authenticated-orcid":false,"given":"Simon","family":"Mayer","sequence":"additional","affiliation":[{"name":"University of St. Gallen, St. Gallen, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9528-6923","authenticated-orcid":false,"given":"Kasim Sinan","family":"Yildirim","sequence":"additional","affiliation":[{"name":"University of Trento, Trento, Italy"}]}],"member":"320","published-online":{"date-parts":[[2023,9,27]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Khadeeja Gamal Mohammed.","author":"Amer","year":"2020","unstructured":"Amer, Mohammed M T and Atteya, Khadeeja Gamal Mohammed. 2020. Indoor Positioning Bluetooth Angle of Arrival. http:\/\/lup.lub.lu.se\/student-papers\/record\/9005557"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/38.963459"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.array.2019.100009"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450341.3458766"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379157.3391657"},{"key":"e_1_2_2_6_1","volume-title":"CHI'22","author":"Bektas Kenan","year":"2022","unstructured":"Kenan Bektas, Jannis Rene Strecker, Simon Mayer, and Markus Stolze. 2022. EToS-1: Eye Tracking on Shopfloors for User Engagement with Automation. In AutomationXP22: Engaging with Automation, CHI'22. CEUR Workshop Proceedings. http:\/\/www.alexandria.unisg. ch\/266339\/"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3560905.3568095"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000049"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC46164.2021.9630186"},{"key":"e_1_2_2_10_1","unstructured":"Bluetooth Special Interest Group. 2019. Bluetooth Specification Version 5.1. https:\/\/www.bluetooth.com\/bluetooth-resources\/bluetooth-core-specification-v5-1-feature-overview\/."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TWC.2020.3038565"},{"key":"e_1_2_2_12_1","volume-title":"YOLOv4: Optimal Speed and Accuracy of Object Detection. CoRR abs\/2004.10934","author":"Bochkovskiy Alexey","year":"2020","unstructured":"Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. YOLOv4: Optimal Speed and Accuracy of Object Detection. CoRR abs\/2004.10934 (2020). arXiv:2004.10934 https:\/\/arxiv.org\/abs\/2004.10934"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3264904"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485730.3485944"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.1986.4767851"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3161173"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858177"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.164"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501951"},{"key":"e_1_2_2_20_1","volume-title":"International Conference on Health Informatics.","author":"Eckert Martin","unstructured":"Martin Eckert, Matthias Blex, and C. Friedrich. 2018. Object Detection Featuring 3D Audio Localization for Microsoft HoloLens - A Deep Learning based Sensor Substitution Approach for the Blind. In International Conference on Health Informatics."},{"key":"e_1_2_2_21_1","unstructured":"epishine. 2023. Indoor solar cell. https:\/\/www.epishine.com\/product."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s43154-020-00006-5"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2005.74"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR.2007.4538858"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2022.3157058"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.169"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.81"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3384419.3430440"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1080\/00207543.2020.1779370"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2889431"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/PerComWorkshops51409.2021.9431115"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453157"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544549.3583923"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/SECON55815.2022.9918581"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.3390\/s19194330"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13673-018-0162-5"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298990"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1670252.1670305"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWAR.1999.803809"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rser.2021.110755"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCE.2019.2897758"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/79.526899"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/0165-1684(95)00126-3"},{"key":"e_1_2_2_44_1","unstructured":"L\u00e5ngberg Felix and Thurborg Jonas. 2020. Design of a Size Reduced Antenna Array for Angle of Arrival (AoA) Estimation for BLE 5.1. https:\/\/lup.lub.lu.se\/student-papers\/search\/publication\/9031557"},{"key":"e_1_2_2_45_1","unstructured":"Silicon Laboratories. 2021. Custom Direction-Finding Solutions using the Silicon Labs Bluetooth Stack. Technical Report. Silicon Laboratories. https:\/\/www.silabs.com\/documents\/public\/application-notes\/an1297-custom-direction-finding-solutions-silicon-labs-bluetooth.pdf"},{"key":"e_1_2_2_46_1","unstructured":"Silicon Laboratories. 2022. iBeacon. https:\/\/docs.silabs.com\/rs9116-wiseconnect\/2.6\/wifibt-wc-snippet-examples\/ble-ble-ibeacon-readme#i-beacon"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3294109.3295659"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jksuci.2023.01.019"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2669062.2669077"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2494091.2494168"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/BSN.Workshops.2014.17"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCE.2018.2797638"},{"key":"e_1_2_2_54_1","first-page":"1321","article-title":"A taxonomy of mixed reality visual displays","volume":"77","author":"Milgram Paul","year":"1994","unstructured":"Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321--1329.","journal-title":"IEICE TRANSACTIONS on Information and Systems"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","unstructured":"Alexis Morris Jie Guan and Amna Azhar. 2021. An XRI Mixed-Reality Internet-of-Things Architectural Framework Toward Immersive and Adaptive Smart Environments. In 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (2021-10). 68--74. https:\/\/doi.org\/10.1109\/ISMAR-Adjunct54149.2021.00024","DOI":"10.1109\/ISMAR-Adjunct54149.2021.00024"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-04110-6_1"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807442.2807497"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.3389\/frvir.2021.763340"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jag.2021.102456"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1002\/adfm.202007952"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3307334.3326079"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491207"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3490149.3502263"},{"key":"e_1_2_2_64_1","unstructured":"Theodoros Prokic. 2019. Antenna Design for Angle of Arrival Measurement in Access Control Applications. Ph. D. Dissertation. http:\/\/urn.kb.se\/resolve?urn=urn:nbn:se:kth:diva-247508"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","unstructured":"E. Fantin Irudaya Raj M. Appadurai and K. Athiappan. 2021. Precision Farming in Modern Agriculture. In Smart Agriculture Automation Using Advanced Technologies: Data Analytics and Machine Learning Cloud Architecture Automation and IoT Amitava Choudhury Arindam Biswas T. P. Singh and Santanu Kumar Ghosh (Eds.). Springer Singapore 61--87. https:\/\/doi.org\/10.1007\/978-981-16-6124-2_4","DOI":"10.1007\/978-981-16-6124-2_4"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9053895"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984751.2985729"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.91"},{"key":"e_1_2_2_69_1","volume-title":"YOLOv3: An Incremental Improvement. CoRR abs\/1804.02767","author":"Redmon Joseph","year":"2018","unstructured":"Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement. CoRR abs\/1804.02767 (2018). arXiv:1804.02767 http:\/\/arxiv.org\/abs\/1804.02767"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1016\/j"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/354666.354667"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/215585.215639"},{"key":"e_1_2_2_73_1","volume-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems 28","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems 28 (2015)."},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3536221.3556591"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAP.1986.1143830"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/2994551.2996526"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494322.3494326"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300905"},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3567445.3567453"},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/1476589.1476686"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300431"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSEN.2019.2922409"},{"key":"e_1_2_2_83_1","unstructured":"Chien-Yao Wang Alexey Bochkovskiy and Hong-Yuan Mark Liao. 2022. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv:2207.02696 http:\/\/arxiv.org\/abs\/2207.02696"},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2022.3169328"},{"key":"e_1_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2018.2857812"},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.17743\/jaes.2022.0048"},{"key":"e_1_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/3569482"},{"key":"e_1_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.01021"},{"key":"e_1_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2023.3238524"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610879","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3610879","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T16:28:06Z","timestamp":1753720086000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610879"}},"subtitle":["Fusing Object Situation Information from Heterogeneous Sources"],"short-title":[],"issued":{"date-parts":[[2023,9,27]]},"references-count":89,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,9,27]]}},"alternative-id":["10.1145\/3610879"],"URL":"https:\/\/doi.org\/10.1145\/3610879","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,27]]},"assertion":[{"value":"2023-09-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}