{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T17:26:50Z","timestamp":1780421210280,"version":"3.54.1"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2026,2,9]],"date-time":"2026-02-09T00:00:00Z","timestamp":1770595200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003052","name":"MOTIE","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003052","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,2,28]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Worker safety in modern industrial environments is becoming increasingly critical owing to the complexities introduced by human\u2013machine collaborations, making real-time activity recognition essential. Conventional unimodal human activity recognition (HAR), which relies solely on inertial sensors or cameras, often exhibits degraded performance in dynamic settings owing to occlusion, sensor faults, and poor scalability in multi-worker scenarios. To address these limitations, we propose a real-time multimodal HAR system as the core component of a human digital twin (HDT) for safety management. The system integrates inertial measurement unit (IMU) data from workers\u2019 smartphones with vision data from existing closed-circuit television (CCTV) infrastructure, eliminating the need for additional wearables. Worker identity is maintained by linking smartphone identifiers with face recognition results obtained from video streams, enabling consistent tracking even when workers temporarily leave the camera\u2019s field of view. An adaptive score fusion strategy dynamically balances modalities according to confidence, which improves the robustness of the system to missing or noisy data. This approach advances HAR beyond controlled laboratory settings and provides a practical and computationally efficient foundation for HDT-based safety management in industrial environments.<\/jats:p>","DOI":"10.1093\/jcde\/qwag011","type":"journal-article","created":{"date-parts":[[2026,2,6]],"date-time":"2026-02-06T12:49:44Z","timestamp":1770382184000},"page":"111-122","source":"Crossref","is-referenced-by-count":1,"title":["Real-time multi-worker identification and action recognition system employing multimodal deep learning for human digital twin"],"prefix":"10.1093","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5767-5792","authenticated-orcid":false,"given":"Donggun","family":"Lee","sequence":"first","affiliation":[{"name":"Industrial Intelligence Laboratory, Advanced Institute of Convergence Technology , Suwon 16229 ,","place":["Republic of Korea"]},{"name":"Sungkyunkwan University Department of Industrial Engineering, , Suwon 16419 ,","place":["Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0887-6168","authenticated-orcid":false,"given":"Donghyun","family":"Park","sequence":"additional","affiliation":[{"name":"Autonomous Manufacturing Research Center, Korea Electronics Technology Institute , Seongnam 13449 ,","place":["Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-6338-0744","authenticated-orcid":false,"given":"Yong-Shin","family":"Kang","sequence":"additional","affiliation":[{"name":"Industrial Intelligence Laboratory, Advanced Institute of Convergence Technology , Suwon 16229 ,","place":["Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,2,9]]},"reference":[{"key":"2026031608162766500_bib1","article-title":"BoT-SORT: Robust associations multi-pedestrian tracking","author":"Aharon","year":"2022"},{"key":"2026031608162766500_bib2","doi-asserted-by":"publisher","first-page":"957","DOI":"10.1007\/s10514-017-9677-2","article-title":"Progress and prospects of the human\u2013robot collaboration","volume":"42","author":"Ajoudani","year":"2018","journal-title":"Autonomous Robots"},{"key":"2026031608162766500_bib3","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal machine learning: A survey and taxonomy","volume":"41","author":"Baltrusaitis","year":"2019","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2026031608162766500_bib4","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1016\/j.cirp.2017.04.009","article-title":"A new strategy for ensuring human safety during various levels of interaction with industrial robots","volume":"66","author":"Bdiwi","year":"2017","journal-title":"CIRP Annals"},{"key":"2026031608162766500_bib5","doi-asserted-by":"publisher","first-page":"30509","DOI":"10.1007\/s11042-020-09004-3","article-title":"Vision-based human activity recognition: A survey","volume":"79","author":"Beddiar","year":"2020","journal-title":"Multimedia Tools and Applications"},{"key":"2026031608162766500_bib6","volume-title":"Employer-reported Workplace Injuries and Illnesses","author":"Bureau of Labor Statistics (BLS)","year":"2023"},{"key":"2026031608162766500_bib7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3447744","article-title":"Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities","volume":"54","author":"Chen","year":"2022","journal-title":"ACM Computing Surveys"},{"key":"2026031608162766500_bib8","article-title":"Multimodal contrastive learning with hard negative sampling for human activity recognition","author":"Choi","year":"2023"},{"key":"2026031608162766500_bib9","doi-asserted-by":"publisher","first-page":"655","DOI":"10.3390\/s24020655","article-title":"Human digital twin in industry 5.0: A holistic approach to worker safety and well-being through advanced AI and emotional analytics","volume":"24","author":"Davila-Gonzalez","year":"2024","journal-title":"Sensors"},{"key":"2026031608162766500_bib10","doi-asserted-by":"publisher","first-page":"7758","DOI":"10.3390\/app10217758","article-title":"Digital twin for monitoring ergonomics during manufacturing production","volume":"10","author":"Greco","year":"2020","journal-title":"Applied Sciences"},{"key":"2026031608162766500_bib11","first-page":"1060","article-title":"Clothes-changing person re-identification with rgb modality only","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Gu","year":"2022"},{"key":"2026031608162766500_bib12","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1186\/s10033-024-00998-7","article-title":"From digital human modeling to human digital twin: Framework and perspectives in human factors","volume":"37","author":"He","year":"2024","journal-title":"Chinese Journal of Mechanical Engineering"},{"key":"2026031608162766500_bib13","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1007\/978-3-031-82170-7_3","article-title":"Human digital twin in human-centric smart manufacturing","volume-title":"Human-centric Smart Manufacturing Towards Industry 5.0","author":"He","year":"2025"},{"key":"2026031608162766500_bib14","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"key":"2026031608162766500_bib15","article-title":"A call for safer and healthier working environments","author":"International Labour Organization (ILO)","year":"2023","journal-title":"23rd World Congress on Safety and Health at Work: Shaping Change"},{"key":"2026031608162766500_bib16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3545572","article-title":"A review on methods and applications in multimodal deep learning","volume":"19","author":"Jabeen","year":"2023","journal-title":"ACM Transactions on Multimedia Computing, Communications, and Applications"},{"key":"2026031608162766500_bib17","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1007\/978-3-031-16411-8_61","article-title":"Human digital twin system for operator safety and work management","volume-title":"IFIP International Conference on Advances in Production Management Systems","author":"Kim","year":"2022"},{"key":"2026031608162766500_bib18","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1093\/jcde\/qwaf075","article-title":"Granular and explainable human activity recognition through sound segmentation and deep learning","volume":"12","author":"Kim","year":"2025","journal-title":"Journal of Computational Design and Engineering"},{"key":"2026031608162766500_bib19","doi-asserted-by":"publisher","first-page":"664","DOI":"10.1109\/TBME.2015.2468589","article-title":"Real-time patient-specific ECG classification by 1-D convolutional neural networks","volume":"63","author":"Kiranyaz","year":"2016","journal-title":"In IEEE Transactions on Bio-Medical Engineering"},{"key":"2026031608162766500_bib20","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1109\/34.667881","article-title":"On combining classifiers","volume":"20","author":"Kittler","year":"2002","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2026031608162766500_bib21","article-title":"Towards the Human digital Twin: Definition and design\u2014A survey","author":"Lauer-Schmaltz","year":"2024"},{"key":"2026031608162766500_bib22","doi-asserted-by":"publisher","first-page":"103984","DOI":"10.1016\/j.cviu.2024.103984","article-title":"Exploiting multimodal synthetic data for egocentric human-object interaction detection in an industrial scenario","volume":"242","author":"Leonardi","year":"2024","journal-title":"Computer Vision and Image Understanding"},{"key":"2026031608162766500_bib23","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1007\/978-3-319-10602-1_48","article-title":"Microsoft COCO: Common objects in context","volume-title":"Computer Vision\u2014ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6\u201312, 2014, Proceedings, Part V. Lecture Notes in Computer Science","author":"Lin","year":"2014"},{"key":"2026031608162766500_bib24","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1186\/s13677-024-00691-z","article-title":"Human digital twin: A survey","volume":"13","author":"Lin","year":"2024","journal-title":"Journal of Cloud Computing"},{"key":"2026031608162766500_bib25","doi-asserted-by":"publisher","first-page":"458","DOI":"10.1016\/j.procir.2021.11.077","article-title":"Architecture of a human-digital twin as common interface for operator 4.0 applications","volume":"104","author":"L\u00f6cklin","year":"2021","journal-title":"Procedia CIRP"},{"key":"2026031608162766500_bib26","first-page":"110","article-title":"Computer vision-based survey on human activity recognition system, challenges and applications","volume-title":"2021 3rd International Conference on Signal Processing and Communication (ICPSC)","author":"Manaf","year":"2021"},{"key":"2026031608162766500_bib27","doi-asserted-by":"crossref","unstructured":"Mehta N. K., Arvind H., Kumar H., Banerjee A., Saurav S., Singh S. (2025). A multimodal dataset for enhancing industrial task monitoring and engagement prediction. In 20th ACM\/IEEE International Conference on Human-Robot Interaction (HRI)(pp. 1047\u20131051.). New York: IEEE. 10.1109\/HRI61500.2025.10974158.","DOI":"10.1109\/HRI61500.2025.10974158"},{"key":"2026031608162766500_bib28","first-page":"529","article-title":"Self-supervised learning of pretext-invariant representations","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Misra","year":"2020"},{"key":"2026031608162766500_bib29","doi-asserted-by":"publisher","first-page":"42","DOI":"10.7232\/JKIIE.2020.46.1.042","article-title":"Digital twin-based application for design of human\u2013machine collaborative assembly production lines","volume":"46","author":"Nam","year":"2020","journal-title":"Journal Of Korean Institute of Industrial Engineering"},{"key":"2026031608162766500_bib30","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1016\/j.eswa.2018.03.056","article-title":"Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges","volume":"105","author":"Nweke","year":"2018","journal-title":"Expert Systems with Applications"},{"key":"2026031608162766500_bib31","doi-asserted-by":"publisher","first-page":"13029","DOI":"10.1109\/JSEN.2021.3069927","article-title":"Human activity recognition with smartphone and wearable sensors using deep learning techniques: A review","volume":"21","author":"Ramanujam","year":"2021","journal-title":"IEEE Sensors Journal"},{"key":"2026031608162766500_bib32","first-page":"6036","article-title":"Features for multi-target multi-camera tracking and re-identification","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Ristani","year":"2018"},{"key":"2026031608162766500_bib33","doi-asserted-by":"publisher","first-page":"1089","DOI":"10.1016\/j.procir.2021.11.183","article-title":"Towards the resilient operator 5.0: The future of work in smart resilient manufacturing systems","volume":"104","author":"Romero","year":"2021","journal-title":"Procedia CIRP"},{"key":"2026031608162766500_bib34","doi-asserted-by":"publisher","first-page":"815","DOI":"10.1109\/CVPR.2015.7298682","article-title":"Facenet: A unified embedding for face recognition and clustering","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Schroff","year":"2015"},{"key":"2026031608162766500_bib35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s43621-025-01201-0","article-title":"From automation to collaboration: Exploring the impact of industry 5.0 on sustainable manufacturing","volume":"6","author":"Shabur","year":"2025","journal-title":"Discover Sustainability"},{"key":"2026031608162766500_bib36","doi-asserted-by":"publisher","first-page":"2637","DOI":"10.1109\/WSC48552.2020.9384043","article-title":"A digital twin framework for real-time analysis and feedback of repetitive work in the manual material handling industry","volume-title":"Winter Simulation Conference (WSC)","author":"Sharotry","year":"2020"},{"key":"2026031608162766500_bib37","doi-asserted-by":"publisher","first-page":"101839","DOI":"10.1016\/j.rcim.2019.101839","article-title":"A digital twin-driven approach for the assembly-commissioning of high precision products","volume":"61","author":"Sun","year":"2020","journal-title":"Robotics and Computer-Integrated Manufacturing"},{"key":"2026031608162766500_bib38","article-title":"Modeling and mitigation of occupational safety risks in dynamic industrial environments","author":"Tewari","year":"2022"},{"key":"2026031608162766500_bib39","author":"Ultralytics","year":"2023"},{"key":"2026031608162766500_bib40","doi-asserted-by":"publisher","first-page":"826","DOI":"10.1016\/j.ifacol.2018.08.441","article-title":"Fatigue accumulation in the assignment of manual material handling activities to operators","volume":"51","author":"Valentina","year":"2018","journal-title":"IFAC-PapersOnLine"},{"key":"2026031608162766500_bib41","doi-asserted-by":"publisher","first-page":"102626","DOI":"10.1016\/j.rcim.2023.102626","article-title":"Human Digital Twin in the context of Industry 5.0","volume":"85","author":"Wang","year":"2024","journal-title":"Robotics and Computer-Integrated Manufacturing"},{"key":"2026031608162766500_bib42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.jmsy.2022.11.013","article-title":"A safety management approach for Industry 5.0\u2019s human-centered manufacturing based on digital twin","volume":"66","author":"Wang","year":"2023","journal-title":"Journal of Manufacturing Systems"},{"key":"2026031608162766500_bib43","doi-asserted-by":"publisher","first-page":"424","DOI":"10.1109\/ICISCE.2017.95","volume-title":"Joint Face Detection and Facial Expression Recognition with MTCNN 4th International Conference on Information Science and Control. Engineering (ICISCE)","author":"Xiang","year":"2017"},{"key":"2026031608162766500_bib44","doi-asserted-by":"publisher","first-page":"2162","DOI":"10.3390\/ijerph18042162","article-title":"Analysis of the characteristics of fatal accidents in the construction industry in China based on statistical data","volume":"18","author":"Xu","year":"2021","journal-title":"International Journal of Environmental Research and Public Health"},{"key":"2026031608162766500_bib45","doi-asserted-by":"publisher","first-page":"2872","DOI":"10.1109\/TPAMI.2021.3054775","article-title":"Deep learning for person re-identification: A survey and outlook","volume":"44","author":"Ye","year":"2021","journal-title":"IEEE transactions on pattern analysis and machine intelligence"},{"key":"2026031608162766500_bib46","doi-asserted-by":"publisher","first-page":"3061","DOI":"10.1109\/TCSVT.2021.3100128","article-title":"A cross view learning approach for skeleton-based action recognition","volume":"32","author":"Zheng","year":"2021","journal-title":"IEEE Transactions on Circuits and Systems for Video Technology"},{"key":"2026031608162766500_bib47","article-title":"Person re-identification: Past, present and future","author":"Zheng","year":"2016"}],"container-title":["Journal of Computational Design and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jcde\/advance-article-pdf\/doi\/10.1093\/jcde\/qwag011\/66820709\/qwag011.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/13\/3\/111\/66820709\/qwag011.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/13\/3\/111\/66820709\/qwag011.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T12:16:38Z","timestamp":1773663398000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jcde\/article\/13\/3\/111\/8469388"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,9]]},"references-count":47,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,2,28]]}},"URL":"https:\/\/doi.org\/10.1093\/jcde\/qwag011","relation":{},"ISSN":["2288-5048"],"issn-type":[{"value":"2288-5048","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,3]]},"published":{"date-parts":[[2026,2,9]]}}}