{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T00:31:52Z","timestamp":1784334712242,"version":"3.55.0"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,28]],"date-time":"2024-03-28T00:00:00Z","timestamp":1711584000000},"content-version":"vor","delay-in-days":367,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["IIS-1763827"],"award-info":[{"award-number":["IIS-1763827"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"NIH\/NLM","award":["2R01LM011834-05"],"award-info":[{"award-number":["2R01LM011834-05"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2023,3,27]]},"abstract":"<jats:p>In clinical settings, most automatic recognition systems use visual or sensory data to recognize activities. These systems cannot recognize activities that rely on verbal assessment, lack visual cues, or do not use medical devices. We examined speech-based activity and activity-stage recognition in a clinical domain, making the following contributions. (1) We collected a high-quality dataset representing common activities and activity stages during actual trauma resuscitation events-the initial evaluation and treatment of critically injured patients. (2) We introduced a novel multimodal network based on audio signal and a set of keywords that does not require a high-performing automatic speech recognition (ASR) engine. (3) We designed novel contextual modules to capture dynamic dependencies in team conversations about activities and stages during a complex workflow. (4) We introduced a data augmentation method, which simulates team communication by combining selected utterances and their audio clips, and showed that this method contributed to performance improvement in our data-limited scenario. In offline experiments, our proposed context-aware multimodal model achieved F1-scores of 73.2\u00b10.8% and 78.1\u00b11.1% for activity and activity-stage recognition, respectively. In online experiments, the performance declined about 10% for both recognition types when using utterance-level segmentation of the ASR output. The performance declined about 15% when we omitted the utterance-level segmentation. Our experiments showed the feasibility of speech-based activity and activity-stage recognition during dynamic clinical events.<\/jats:p>","DOI":"10.1145\/3580798","type":"journal-article","created":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T14:57:51Z","timestamp":1680015471000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Real-time Context-Aware Multimodal Network for Activity and Activity-Stage Recognition from Team Communication in Dynamic Clinical Settings"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0280-7882","authenticated-orcid":false,"given":"Chenyang","family":"Gao","sequence":"first","affiliation":[{"name":"Rutgers University, Piscataway, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1033-6865","authenticated-orcid":false,"given":"Ivan","family":"Marsic","sequence":"additional","affiliation":[{"name":"Rutgers University, Piscataway, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2126-8527","authenticated-orcid":false,"given":"Aleksandra","family":"Sarcevic","sequence":"additional","affiliation":[{"name":"Drexel University, Philadelphia, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9954-4523","authenticated-orcid":false,"given":"Waverly","family":"Gestrich-Thompson","sequence":"additional","affiliation":[{"name":"Children's National Medical Center, Washington, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4465-9117","authenticated-orcid":false,"given":"Randall S.","family":"Burd","sequence":"additional","affiliation":[{"name":"Children's National Medical Center, Washington, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,3,28]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICHI48887.2020.9374372"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448090"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331375"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-1909"},{"key":"e_1_2_1_5_1","first-page":"1","volume-title":"23th International conference on architecture of computing systems","author":"Avci Akin","year":"2010","unstructured":"Akin Avci, Stephan Bosch, Mihai Marin-Perianu, Raluca Marin-Perianu, and Paul Havinga. 2010. Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: A survey, in 23th International conference on architecture of computing systems 2010. VDE, p. 1--10."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1607.06450"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","unstructured":"Chandrakant Bothe Sven Magg Cornelius Weber and Stefan Wermter. 2018. Conversational analysis using utterance-level attention-based bidirectional recurrent neural networks. INTERSPEECH. https:\/\/doi.org\/10.21437\/Interspeech.2018-2527.","DOI":"10.21437\/Interspeech.2018-2527"},{"key":"e_1_2_1_8_1","volume-title":"Connectionist speech recognition: a hybrid approach","author":"Bourlard Herve A","year":"2012","unstructured":"Herve A Bourlard and Nelson Morgan, Connectionist speech recognition: a hybrid approach. Vol. 247. 2012: Springer Science & Business Media."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASRU51503.2021.9687874"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/FG.2013.6553758"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2016.7472621"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSTSP.2022.3188113"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.195"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2370216.2370288"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6259"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1612.08083"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2010.11929"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1001\/archsurg.2010.333"},{"key":"e_1_2_1_20_1","first-page":"208","volume-title":"Canadian journal of surgery. 49(3)","author":"Fitzgerald Mark","unstructured":"Mark Fitzgerald, Rob Gocentas, Linas Dziukas, Peter Cameron, Colin Mackenzie, and Nathan Farrow. 2006. Using video audit to improve trauma resuscitation---time for a new approach, in Canadian journal of surgery. 49(3), p. 208."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1097\/01.sla.0000234655.83517.56"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-57351-9_30"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1207"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICHI.2019.8904713"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2020-3015"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","unstructured":"Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus) in arXiv preprint arXiv:1606.08415. https:\/\/doi.org\/10.48550\/arXiv.1606.08415.","DOI":"10.48550\/arXiv.1606.08415"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1097\/00005373-200206000-00009"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2021.3122291"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1502.03167"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3512920"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3312983"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240925.3240941"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2019.08.100"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1412.6980"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11701"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2804262"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1062"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","unstructured":"Keyi Li Sen Yang Travis M Sullivan Randall S Burd and Ivan Marsic. 2022. Exploring Runtime Decision Support for Trauma Resuscitation in arXiv preprint arXiv:2207.02922. https:\/\/doi.org\/10.48550\/arXiv.2207.02922.","DOI":"10.48550\/arXiv.2207.02922"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K19-1036"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/RFID.2016.7488002"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/UEMCON.2016.7777912"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2994551.2994569"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3123266.3123365"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3314404"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.3115\/1118108.1118117"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","unstructured":"Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space in arXiv preprint arXiv:1301.3781. https:\/\/doi.org\/10.48550\/arXiv.1301.3781.","DOI":"10.48550\/arXiv.1301.3781"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","unstructured":"Abdelrahman Mohamed Dmytro Okhonko and Luke Zettlemoyer. 2019. Transformers with convolutional context for ASR. https:\/\/doi.org\/10.48550\/arXiv.1904.11660.","DOI":"10.48550\/arXiv.1904.11660"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1542\/peds.2004-1803"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2015.7178964"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2019-2680"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1912.01703"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1373"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","unstructured":"Prajit Ramachandran Barret Zoph and Quoc V Le. 2017. Searching for activation functions in arXiv preprint arXiv:1710.05941. https:\/\/doi.org\/10.48550\/arXiv.1710.05941.","DOI":"10.48550\/arXiv.1710.05941"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/TETCI.2017.2762739"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-59716-0_37"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1107"},{"key":"e_1_2_1_57_1","first-page":"1929","volume-title":"The journal of machine learning research. 15(1)","author":"Srivastava Nitish","unstructured":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting, in The journal of machine learning research. 15(1), p. 1929--1958."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-48989-2_14"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1097\/TA.0b013e31828b82f5"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.proeng.2014.03.129"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1656"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1542\/peds.103.1.20"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9054345"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2018-1456"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098174"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2021-1775"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02842477"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2010.08.066"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2021.102224"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.11"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3580798","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3580798","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3580798","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:45:40Z","timestamp":1752468340000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3580798"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,27]]},"references-count":70,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,3,27]]}},"alternative-id":["10.1145\/3580798"],"URL":"https:\/\/doi.org\/10.1145\/3580798","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,27]]},"assertion":[{"value":"2023-03-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}