{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T16:53:57Z","timestamp":1776099237838,"version":"3.50.1"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,1,11]],"date-time":"2023-01-11T00:00:00Z","timestamp":1673395200000},"content-version":"vor","delay-in-days":21,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP21J20353"],"award-info":[{"award-number":["JP21J20353"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency","doi-asserted-by":"publisher","award":["JPMJAX200R"],"award-info":[{"award-number":["JPMJAX200R"]}],"id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2022,12,21]]},"abstract":"<jats:p>A user often needs training and guidance while performing several daily life procedures, e.g., cooking, setting up a new appliance, or doing a COVID test. Watch-based human activity recognition (HAR) can track users' actions during these procedures. However, out of the box, state-of-the-art HAR struggles from noisy data and less-expressive actions that are often part of daily life tasks. This paper proposes PrISM-Tracker, a procedure-tracking framework that augments existing HAR models with (1) graph-based procedure representation and (2) a user-interaction module to handle model uncertainty. Specifically, PrISM-Tracker extends a Viterbi algorithm to update state probabilities based on time-series HAR outputs by leveraging the graph representation that embeds time information as prior. Moreover, the model identifies moments or classes of uncertainty and asks the user for guidance to improve tracking accuracy. We tested PrISM-Tracker in two procedures: latte-making in an engineering lab study and wound care for skin cancer patients at a clinic. The results showed the effectiveness of the proposed algorithm utilizing transition graphs in tracking steps and the efficacy of using simulated human input to enhance performance. This work is the first step toward human-in-the-loop intelligent systems for guiding users while performing new and complicated procedural tasks.<\/jats:p>","DOI":"10.1145\/3569504","type":"journal-article","created":{"date-parts":[[2023,1,11]],"date-time":"2023-01-11T15:34:01Z","timestamp":1673451241000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["PrISM-Tracker"],"prefix":"10.1145","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7868-4754","authenticated-orcid":false,"given":"Riku","family":"Arakawa","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2558-735X","authenticated-orcid":false,"given":"Hiromu","family":"Yakura","sequence":"additional","affiliation":[{"name":"University of Tsukuba, Tsukuba, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9085-8830","authenticated-orcid":false,"given":"Vimal","family":"Mollyn","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8423-7464","authenticated-orcid":false,"given":"Suzanne","family":"Nie","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0114-9902","authenticated-orcid":false,"given":"Emma","family":"Russell","sequence":"additional","affiliation":[{"name":"Case Western Reserve University, Cleveland, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6469-7255","authenticated-orcid":false,"given":"Dustin P.","family":"DeMeo","sequence":"additional","affiliation":[{"name":"Case Western Reserve University, Cleveland, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7583-6484","authenticated-orcid":false,"given":"Haarika A.","family":"Reddy","sequence":"additional","affiliation":[{"name":"Case Western Reserve University, Cleveland, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6332-9100","authenticated-orcid":false,"given":"Alexander K.","family":"Maytin","sequence":"additional","affiliation":[{"name":"Boston University, Boston, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5712-6307","authenticated-orcid":false,"given":"Bryan T.","family":"Carroll","sequence":"additional","affiliation":[{"name":"University Hospitals of Cleveland Department of Dermatology, Cleveland, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7906-4899","authenticated-orcid":false,"given":"Jill Fain","family":"Lehman","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1237-7545","authenticated-orcid":false,"given":"Mayank","family":"Goel","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,1,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300233"},{"key":"e_1_2_1_3_1","unstructured":"Apple. 2022. Handwashing on Apple Watch. https:\/\/support.apple.com\/guide\/watch\/set-up-handwashing-apdc9b9f04a8\/watchos"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of 2nd Workshop on Human-Robot Teaming Beyond Human Operational Speeds and Robot Teammates Operating in Dynamic, Unstructured Environments. 2 pages.","author":"Arakawa Riku","year":"2018","unstructured":"Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi, and Shin-ichi Maeda. 2018. DQN-TAMER: Human-in-the-loop reinforcement learning with intractable feedback. In Proceedings of 2nd Workshop on Human-Robot Teaming Beyond Human Operational Speeds and Robot Teammates Operating in Dynamic, Unstructured Environments. 2 pages."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3490099.3511164"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341163.3347735"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2017.2726187"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534582"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2021.102062"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/p14-1139"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447744"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 2017 Annual Conference on Neural Information Processing Systems. Neural Information Processing Systems Foundation","author":"Christiano Paul F.","year":"2017","unstructured":"Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the 2017 Annual Conference on Neural Information Processing Systems. Neural Information Processing Systems Foundation, La Jolla, CA, 4299--4307."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2105.12949"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/n19-1423"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3185517"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.socscimed.2020.113174"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/PROC.1973.9030"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-48746-5_45"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3463501"},{"key":"e_1_2_1_20_1","unstructured":"Google. 2022. Text-to-Speech. https:\/\/cloud.google.com\/text-to-speech"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8460699"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40708-016-0042-6"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445283"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2020.102738"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCE.2012.6227479"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 2016 Annual Conference on Neural Information Processing Systems. 2946--2954","author":"Johnson Matthew J.","unstructured":"Matthew J. Johnson, David Duvenaud, Alexander B. Wiltschko, Ryan P. Adams, and Sandeep R. Datta. 2016. Composing graphical models with neural networks for structured representations and fast inference. In Proceedings of the 2016 Annual Conference on Neural Information Processing Systems. 2946--2954."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287063"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01285"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1080\/08839514.2016.1156461"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3326458.3326932"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411841"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242609"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984582"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/SURV.2012.110112.00192"},{"key":"e_1_2_1_35_1","volume-title":"Self-care in health. Annual review of public health 4, 1","author":"Levin Lowell S","year":"1983","unstructured":"Lowell S Levin and Ellen L Idler. 1983. Self-care in health. Annual review of public health 4, 1 (1983), 181--201."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.3390\/s19143213"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the 7th International Conference on Learning Representations. OpenReview.net, 18 pages.","author":"Loshchilov Ilya","year":"2019","unstructured":"Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations. OpenReview.net, 18 pages."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01022"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5220\/0006902405040511"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550284"},{"key":"e_1_2_1_41_1","volume-title":"Human-in-the-loop machine learning: Active learning and annotation for human-centered AI","author":"Monarch Robert Munro","unstructured":"Robert Munro Monarch. 2021. Human-in-the-loop machine learning: Active learning and annotation for human-centered AI. Simon and Schuster, New York, NY."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557116"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517658"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236112.3236149"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2009.5326050"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2018.03.056"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327566ijce0403_2"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.21437\/IberSPEECH.2021-55"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445687"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370024.2020.1734931"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2540930.2540952"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.4028\/www.scientific.net\/KEM.502.25"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.puhe.2016.02.030"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP39728.2021.9414560"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2009.5204354"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2012.6343802"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3517252"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1177\/1477153517693887"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3319502.3374779"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207695"},{"key":"e_1_2_1_62_1","first-page":"43","article-title":"Video education provides effective wound care instruction pre-or post-Mohs micrographic surgery","volume":"7","author":"Van Acker Monica M","year":"2014","unstructured":"Monica M Van Acker and MArk A Kuriata. 2014. Video education provides effective wound care instruction pre-or post-Mohs micrographic surgery. The Journal of Clinical and Aesthetic Dermatology 7, 4 (2014), 43.","journal-title":"The Journal of Clinical and Aesthetic Dermatology"},{"key":"e_1_2_1_63_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. [n.d.]."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2015.00028"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/IEEECONF49454.2021.9382750"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/BHI.2016.7455925"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328931"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209889.3209897"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415819"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/SII52469.2022.9708836"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/884"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJSNET.2019.100218"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3569504","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3569504","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3569504","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T20:51:47Z","timestamp":1752612707000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3569504"}},"subtitle":["A Framework for Multimodal Procedure Tracking Using Wearable Sensors and State Transition Information with User-Driven Handling of Errors and Uncertainty"],"short-title":[],"issued":{"date-parts":[[2022,12,21]]},"references-count":72,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,12,21]]}},"alternative-id":["10.1145\/3569504"],"URL":"https:\/\/doi.org\/10.1145\/3569504","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,21]]},"assertion":[{"value":"2023-01-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}