{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T15:10:08Z","timestamp":1759158608954,"version":"3.44.0"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"5","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:p>Knowing when a user picks up an object plays a vital role in many context-aware applications. For example, tracking water consumption, counting calories consumed, or reminding you to bring your keys are all context-centered scenarios involving picking up objects. In this project, we propose Contextra, a wrist-worn system that uses sensor fusion to recognize when a user grasps objects. Sensor fusion allows all parts of the grasp to be sensed in ways single channels cannot alone. In our wristband, we fuse EMG and IMU data with video captured from three low-power IR cameras. These cameras maintain privacy by using an active-illumination technique to only capture features close to the sensors. Beyond grasps alone, we see Contextra as playing a foundational role in providing continuous awareness of context triggers to extend the functionality of existing AI devices that cannot run continuously due to power and privacy concerns.<\/jats:p>","DOI":"10.1145\/3743741","type":"journal-article","created":{"date-parts":[[2025,9,9]],"date-time":"2025-09-09T14:28:48Z","timestamp":1757428128000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Contextra: Detecting Object Grasps With Low-Power Cameras and Sensor Fusion On the Wrist MHCI006"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7498-2567","authenticated-orcid":false,"given":"Nathan","family":"DeVrio","sequence":"first","affiliation":[{"name":"Human-Computer Interaction Institute","place":["Pittsburgh, USA"]},{"name":"Carnegie Mellon University","place":["Pittsburgh, USA"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3745-3516","authenticated-orcid":false,"given":"Roger","family":"Boldu","sequence":"additional","affiliation":[{"name":"Meta Reality Labs","place":["Redmond, USA"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7715-7557","authenticated-orcid":false,"given":"Eric","family":"Whitmire","sequence":"additional","affiliation":[{"name":"Meta Reality Labs","place":["Redmond, USA"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4962-751X","authenticated-orcid":false,"given":"Wolf","family":"Kienzle","sequence":"additional","affiliation":[{"name":"Meta","place":["Seattle, USA"]},{"name":"Reality Labs","place":["Seattle, USA"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,9,9]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Farshid Amirabdollahian and Michael Walters. 2017. Application of support vector machines to detect hand and wrist gestures using a myoelectric armband. (July 2017). http:\/\/uhra.herts.ac.uk\/handle\/2299\/19095 Accepted: 2017-07-26T10:22:36Z."},{"key":"e_1_3_2_3_2","unstructured":"Apple. 2024. Apple Vision Pro. https:\/\/www.apple.com\/apple-vision-pro\/"},{"key":"e_1_3_2_4_2","unstructured":"Apple. 2024. Connect and charge Apple Vision Pro battery. https:\/\/support.apple.com\/en-us\/117740"},{"key":"e_1_3_2_5_2","unstructured":"Arducam. 2023. Arducam 12MP IMX708 Fixed Focus HDR High SNR Camera Module for Raspberry Pi. https:\/\/www.arducam.com\/product\/arducam-12mp-imx708-fixed-focus-hdr-high-snr-camera-module-for-raspberry-pi\/"},{"key":"e_1_3_2_6_2","unstructured":"Arducam. 2023. Arducam OV2640 Camera Module 2MP Mini CCM Compact Camera Modules Compatible with Arduino ESP32 ESP8266 Development Board with DVP 24 Pin Interface. https:\/\/www.arducam.com\/product\/arducam-ov2640-camera-module-2mp-mini-ccm-compact-camera-modules-compatible-with-arduino_m0031esp32-esp8266-development-board-with-dvp-24-pin-interface_\/"},{"key":"e_1_3_2_7_2","unstructured":"Arducam. 2023. Arducam OV7670 Camera Module VGA Mini CCM Compact Camera Modules Compatible with Arduino ARM FPGA with DVP 24 Pin Interface. https:\/\/www.arducam.com\/product\/arducam_ov7670_camera_module_vga_mini_ccm_compact_camera_modules_m0030\/"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/1709886.1709898"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3311823.3311841"},{"key":"e_1_3_2_10_2","unstructured":"Minjie Cai Kris\u00a0M. Kitani and Yoichi Sato. 2018. Understanding hand-object manipulation by modeling the contextual relationship between actions grasp types and object attributes. ArXiv (July 2018). https:\/\/www.semanticscholar.org\/paper\/Understanding-hand-object-manipulation-by-modeling-Cai-Kitani\/9c1b526dc9adba029809e45693f758c2af913f60"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2807442.2807450"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","unstructured":"Savannah Cofer Tyler\u00a0N. Chen Jackie\u00a0Junrui Yang and Sean Follmer. 2022. Detecting Touch and Grasp Gestures Using a Wrist-Worn Optical and Inertial Sensing Network. IEEE Robotics and Automation Letters 7 4 (Oct. 2022) 10842\u201310849. 10.1109\/LRA.2022.3191173 Number: 4 Conference Name: IEEE Robotics and Automation Letters.","DOI":"10.1109\/LRA.2022.3191173"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","unstructured":"Shrivatsa Deshmukh Vitthal Khatik and Anupam Saxena. 2023. Robust Fusion Model for Handling EMG and Computer Vision Data in Prosthetic Hand Control. IEEE Sensors Letters 7 9 (Sept. 2023) 1\u20134. 10.1109\/LSENS.2023.3301837 Conference Name: IEEE Sensors Letters.","DOI":"10.1109\/LSENS.2023.3301837"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3526113.3545634"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580962"},{"key":"e_1_3_2_16_2","unstructured":"ELP. 2020. ELP High Speed USB3.0 USB Camera 2MP USB Camera Module with IMX291 Image Sensor 1920 * 1080@50fps Webcam with 100 Degree No Distortion Lens for Android Windows Linux Plug&Play Webcam [ELP-SUSB1080P01-LC1100] - $60.90 : ELP USB Webcam. http:\/\/www.webcamerausb.com\/elp-high-speed-usb30-usb-camera-2mp-usb-camera-module-with-imx291-image-sensor-1920-108050fps-webcam-with-100-degree-no-distortion-lens-for-android-windows-linuxplugplay-webcam-p-249.html"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","unstructured":"Junjun Fan Xiangmin Fan Feng Tian Yang Li Zitao Liu Wei Sun and Hongan Wang. 2018. What is That in Your Hand? Recognizing Grasped Objects via Forearm Electromyography Sensing. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies 2 4 (Dec. 2018) 161:1\u2013161:24. 10.1145\/3287039 Number: 4.","DOI":"10.1145\/3287039"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISWC.2005.44"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISWC.2005.25"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","unstructured":"Fang Hu Peng He Songlin Xu Yin Li and Cheng Zhang. 2020. FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies 4 2 (June 2020) 71:1\u201371:24. 10.1145\/3397306","DOI":"10.1145\/3397306"},{"key":"e_1_3_2_21_2","unstructured":"Humane. 2024. Humane Ai Pin | See the World Not Your Screen. | Humane. https:\/\/humane.com\/"},{"key":"e_1_3_2_22_2","unstructured":"Apple Inc.2022. Deploying Transformers on the Apple Neural Engine. https:\/\/machinelearning.apple.com\/research\/neural-engine-transformers"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","unstructured":"Xianta Jiang Lukas-Karim Merhi and Carlo Menon. 2018. Force Exertion Affects Grasp Classification Using Force Myography. IEEE Transactions on Human-Machine Systems 48 2 (April 2018) 219\u2013226. 10.1109\/THMS.2017.2693245","DOI":"10.1109\/THMS.2017.2693245"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2019.8793751"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380139"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984582"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21726-518"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-31205-215"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/PerComW.2012.6197551"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/PerComW.2012.6197551"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-12654-315"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMC.2018.00035"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","unstructured":"J.\u00a0A. Mercer N. Bezodis D. DeLion T. Zachry and M.\u00a0D. Rubley. 2006. EMG sensor location: Does it influence the ability to detect differences in muscle contraction conditions? Journal of Electromyography and Kinesiology 16 2 (April 2006) 198\u2013204. 10.1016\/j.jelekin.2005.07.002","DOI":"10.1016\/j.jelekin.2005.07.002"},{"key":"e_1_3_2_34_2","unstructured":"Meta. 2023. Meta Quest 3: Mixed Reality VR Headset - Shop Now. https:\/\/www.meta.com\/quest\/quest-3\/"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2017.7989370"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300906"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","unstructured":"Angkoon Phinyomark and Erik Scheme. 2018. EMG Pattern Recognition in the Era of Big Data and Deep Learning. Big Data and Cognitive Computing 2 3 (Sept. 2018) 21. 10.3390\/bdcc2030021 Number: 3 Publisher: Multidisciplinary Digital Publishing Institute.","DOI":"10.3390\/bdcc2030021"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","unstructured":"Tobias Pistohl Thomas Sebastian\u00a0Benedikt Schmidt Tonio Ball Andreas Schulze-Bonhage Ad Aertsen and Carsten Mehring. 2013. Grasp Detection from Human ECoG during Natural Reach-to-Grasp Movements. PLOS ONE 8 1 (Jan. 2013) e54658. 10.1371\/journal.pone.0054658 Publisher: Public Library of Science.","DOI":"10.1371\/journal.pone.0054658"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490149.3501320"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISWC.2000.888497"},{"key":"e_1_3_2_41_2","unstructured":"Sensel. 2021. Sensel Morph. https:\/\/morph.sensel.com\/"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROBIO49542.2019.8961582"},{"key":"e_1_3_2_43_2","unstructured":"STMicroelectronics. 2022. VL53L8CX - Low-power high-performance 8x8 multizone Time-of-Flight sensor (ToF) - STMicroelectronics. https:\/\/www.st.com\/en\/imaging-and-photonics-solutions\/vl53l8cx.html"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2875194.2875207"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","unstructured":"Radu-Daniel Vatavu and Ionu\u0163\u00a0Alexandru Zai\u0163i. 2013. Automatic recognition of object size and shape via user-dependent measurements of the grasping hand. International Journal of Human-Computer Studies 71 5 (2013) 590\u2013607. 10.1016\/j.ijhcs.2013.01.002","DOI":"10.1016\/j.ijhcs.2013.01.002"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3607822.3614538"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2804271"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/1935701.1935745"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/1517664.1517736"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415897"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","unstructured":"Zhen\u00a0Gang Xiao and Carlo Menon. 2017. Counting Grasping Action Using Force Myography: An Exploratory Study With Healthy Individuals. JMIR rehabilitation and assistive technologies 4 1 (May 2017) e5. 10.2196\/rehab.6901 Number: 1.","DOI":"10.2196\/rehab.6901"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581264"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39062-3_52"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","unstructured":"Mehrshad Zandigohar Mo Han Mohammadreza Sharif Sezen\u00a0Ya\u011fmur G\u00fcnay Mariusz\u00a0P. Furmanek Mathew Yarossi Paolo Bonato Cagdas Onal Ta\u015fk\u0131n Pad\u0131r Deniz Erdo\u011fmu\u015f and Gunar Schirner. 2024. Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control. Frontiers in Robotics and AI 11 (Feb. 2024). 10.3389\/frobt.2024.1312554 Publisher: Frontiers.","DOI":"10.3389\/frobt.2024.1312554"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376313"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3743741","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T14:49:17Z","timestamp":1759157357000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3743741"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,9]]},"references-count":54,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,9,30]]}},"alternative-id":["10.1145\/3743741"],"URL":"https:\/\/doi.org\/10.1145\/3743741","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2025,9,9]]},"assertion":[{"value":"2025-09-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}