{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T15:28:21Z","timestamp":1789658901394,"version":"4.0.1"},"reference-count":42,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T00:00:00Z","timestamp":1789516800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Sojitz Corporation"},{"DOI":"10.13039\/100009619","name":"Japan Agency for Medical Research and Development","doi-asserted-by":"publisher","award":["AMED; grant number JP25he2822009"],"award-info":[{"award-number":["AMED; grant number JP25he2822009"]}],"id":[{"id":"10.13039\/100009619","id-type":"DOI","asserted-by":"publisher"}]},{"name":"JSPS KAKENHI","award":["grant number 24K13524"],"award-info":[{"award-number":["grant number 24K13524"]}]},{"name":"Japan Science and Technology Agency (JST) through the Strategic Basic Research Programs, CREST","award":["grant number JPMJCR19A2"],"award-info":[{"award-number":["grant number JPMJCR19A2"]}],"id":[{"id":"https:\/\/ror.org\/00097mb19","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"\n Presenteeism, attending work despite physical or mental health problems, is a major source of productivity loss worldwide, yet its early detection in everyday settings remains challenging. Conventional self-report instruments are time-consuming and require psychiatrist interpretation, which limits their scalability. We developed a multimodal machine-learning model that estimates work-function impairment as an indicator of presenteeism from brief smartphone-based dialogues. 38 employees recorded short self-report videos (10\u201330\u2009s) using front-facing smartphone cameras, which were independently rated on an ordinal three-level work-function scale (Healthy \/ Moderate \/ Unwell) by three psychiatrists; the consensus label served as supervision. We tested whether integrating multimodal behavioural signals (acoustic, facial, and linguistic) would provide additional cues beyond linguistic content alone, particularly when verbal cues are sparse. Acoustic, facial, and linguistic features were extracted from each clip and integrated using a three-stream Attention-based Multiple Instance Learning (Attention-MIL) framework with learned late fusion, an ordinal-regression head, and class-prior logit adjustment. Generalisation was assessed under a participant-disjoint protocol (25-fold repeated GroupKFold;\n n<\/jats:italic>\n \u2009=\u20091,768 out-of-fold clips from 29 participants). On the Full configuration, the proposed multimodal framework achieved macro-F1\u2009=\u20090.772 (95% CI [0.697, 0.817]), accuracy\u2009=\u20090.840, macro-AUROC\u2009=\u20090.940, and low expected calibration error (ECE\n \n \n \n \n \n \u2248<\/mml:mo>\n 0.024<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:alternatives>\n <\/jats:inline-formula>\n ). At the aggregate level, the four text-containing configurations (Full \/ Audio+Text \/ Face+Text \/ Text-only) were mutually indistinguishable on macro-F1 (Holm\u2013Bonferroni\n p<\/jats:italic>\n adj<\/jats:sub>\n \u2009>\u20090.40), whereas text-free configurations performed substantially worse (\n \n \n \n \n \n \u0394<\/mml:mo>\n <<\/mml:mo>\n \u2212<\/mml:mo>\n 0.33<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:alternatives>\n <\/jats:inline-formula>\n ,\n p<\/jats:italic>\n adj<\/jats:sub>\n \u2009<\u20090.001), confirming the necessity of the linguistic channel. In an exploratory subgroup analysis, Audio+Text outperformed Text-only in the clinically ambiguous subgroup\u2014short-speech responses from non-Healthy participants (\n n<\/jats:italic>\n \u2009=\u200977;\n \n \n \n \n \n \u0394<\/mml:mo>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:alternatives>\n <\/jats:inline-formula>\n macro-F1\u2009=+0.030, exploratory, requires replication). These findings provide localised but clinically meaningful support for the multimodal hypothesis under low-information conditions and motivate prospective replication.\n <\/jats:p>","DOI":"10.1371\/journal.pone.0354342","type":"journal-article","created":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T20:01:57Z","timestamp":1789588917000},"page":"e0354342","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Estimating presenteeism from repeated smartphone-based multimodal behavioral responses"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-3111-226X","authenticated-orcid":true,"given":"Taiga","family":"Noguchi","sequence":"first","affiliation":[{"name":"University of Tsukuba","place":["Tsukuba, Ibaraki, Japan"],"department":["Graduate School of Science and Technology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4026-9058","authenticated-orcid":true,"given":"Shotaro","family":"Doki","sequence":"additional","affiliation":[{"name":"University of Tsukuba","place":["Tsukuba, Ibaraki, Japan"],"department":["Institute of Medicine"]},{"name":"Minds in Co., Ltd.","place":["Chuo-ku, Tokyo, Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6129-3674","authenticated-orcid":true,"given":"Masakazu","family":"Hirokawa","sequence":"additional","affiliation":[{"name":"Minds in Co., Ltd.","place":["Chuo-ku, Tokyo, Japan"]},{"name":"NEC Corporation","place":["Meguro-ku, Tokyo, Japan"],"department":["Data Science Laboratories"]},{"name":"University of Tsukuba","place":["Tsukuba, Ibaraki, Japan"],"department":["Center for Cybernics Research"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Soma","family":"Nishimura","sequence":"additional","affiliation":[{"name":"University of Tsukuba","place":["Tsukuba, Ibaraki, Japan"],"department":["Graduate School of Comprehensive Human Sciences"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katsuya","family":"Hotta","sequence":"additional","affiliation":[{"name":"Sojitz Corporation","place":["Chiyoda-ku, Tokyo, Japan"],"department":["Retail & Consumer Service Division"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuya","family":"Iwata","sequence":"additional","affiliation":[{"name":"Sojitz Corporation","place":["Chiyoda-ku, Tokyo, Japan"],"department":["Human Capital Dept. 2"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Naoko","family":"Kouda","sequence":"additional","affiliation":[{"name":"Sojitz Corporation","place":["Chiyoda-ku, Tokyo, Japan"],"department":["Public Relations Dept."]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shota","family":"Matsumoto","sequence":"additional","affiliation":[{"name":"DG COMMUNICATIONS Co., Ltd.","place":["Shinjuku-ku, Tokyo, Japan"],"department":["Digital Business Division"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kenji","family":"Suzuki","sequence":"additional","affiliation":[{"name":"University of Tsukuba","place":["Tsukuba, Ibaraki, Japan"],"department":["Institute of Systems and Information Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,16]]},"reference":[{"issue":"7","key":"pone.0354342.ref001","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1136\/jech.54.7.502","article-title":"Sick but yet at work. An empirical study of sickness presenteeism","volume":"54","author":"G Aronsson","year":"2000","journal-title":"J Epidemiol Community Health"},{"key":"pone.0354342.ref002","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1016\/j.socscimed.2016.06.005","article-title":"The consequences of sickness presenteeism on health and wellbeing over time: a systematic review","volume":"161","author":"K Skagen","year":"2016","journal-title":"Soc Sci Med"},{"key":"pone.0354342.ref003","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1016\/j.jad.2019.06.041","article-title":"The relationship between depression symptoms, absenteeism and presenteeism","volume":"256","author":"DA Johnston","year":"2019","journal-title":"J Affect Disord"},{"issue":"2","key":"pone.0354342.ref004","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1007\/s40258-022-00761-w","article-title":"The role of mental health on workplace productivity: a critical review of the literature","volume":"21","author":"C de Oliveira","year":"2023","journal-title":"Appl Health Econ Health Policy"},{"key":"pone.0354342.ref005","unstructured":"Bryan ML, Bryce AM, Roberts J. Presenteeism in the UK: effects of physical and mental health on worker productivity. 2020."},{"key":"pone.0354342.ref006","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.vhri.2017.03.001","article-title":"Presenteeism, absenteeism, and lost work productivity among depressive patients from five cities of Colombia","volume":"14","author":"JM Uribe","year":"2017","journal-title":"Value Health Reg Issues"},{"issue":"5","key":"pone.0354342.ref007","doi-asserted-by":"crossref","DOI":"10.1097\/JOM.0000000000001291","article-title":"Total health-related costs due to absenteeism, presenteeism, and medical and pharmaceutical expenses in Japanese employers","volume":"60","author":"T Nagata","year":"2018","journal-title":"J Occup Environ Med"},{"issue":"1","key":"pone.0354342.ref008","doi-asserted-by":"crossref","DOI":"10.1002\/1348-9585.12346","article-title":"The differences of the economic losses due to presenteeism and treatment costs between high-stress workers and non-high-stress workers using the stress check survey in Japan","volume":"64","author":"T Nagata","year":"2022","journal-title":"J Occup Health"},{"issue":"9","key":"pone.0354342.ref009","doi-asserted-by":"crossref","first-page":"699","DOI":"10.1097\/JOM.0000000000003431","article-title":"The impact of productivity loss from presenteeism and absenteeism on mental health in Japan","volume":"67","author":"K Hara","year":"2025","journal-title":"J Occup Environ Med"},{"issue":"8","key":"pone.0354342.ref010","doi-asserted-by":"crossref","first-page":"1503","DOI":"10.1007\/s10198-025-01764-9","article-title":"Psychological distress and productivity loss: a longitudinal analysis of Australian working adults","volume":"26","author":"SA Keramat","year":"2025","journal-title":"Eur J Health Econ"},{"key":"pone.0354342.ref011","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1016\/j.jad.2011.07.028","article-title":"Factors associated with presenteeism among employed Australian adults reporting lifetime major depression with 12-month symptoms","volume":"135","author":"F Cocker","year":"2011","journal-title":"J Affect Disord"},{"issue":"2","key":"pone.0354342.ref012","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1108\/IJWHM-06-2023-0072","article-title":"When employees understate their stress: defensive biasing in work stress surveys","volume":"17","author":"B Greulich","year":"2024","journal-title":"IJWHM"},{"key":"pone.0354342.ref013","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.ypmed.2017.07.003","article-title":"The longitudinal relation between self-reported physical activity and presenteeism","volume":"102","author":"TJ Walker","year":"2017","journal-title":"Prev Med"},{"issue":"2","key":"pone.0354342.ref014","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1093\/occmed\/kqaa009","article-title":"Two definitions of presenteeism: sickness presenteeism and impaired work function","volume":"70","author":"T Ishimaru","year":"2020","journal-title":"Occup Med (Lond)"},{"key":"pone.0354342.ref015","doi-asserted-by":"crossref","first-page":"101761","DOI":"10.1016\/j.ssmph.2025.101761","article-title":"Gender disparities in job flexibility, job security, psychological distress, work absenteeism, and work presenteeism among U.S. adults","volume":"29","author":"ML Wang","year":"2025","journal-title":"SSM Popul Health"},{"issue":"5","key":"pone.0354342.ref016","doi-asserted-by":"crossref","first-page":"3192","DOI":"10.1109\/TCSVT.2023.3312858","article-title":"Transformer-based multimodal emotional perception for dynamic facial expression recognition in the wild","volume":"34","author":"X Zhang","year":"2024","journal-title":"IEEE Trans Circuits Syst Video Technol"},{"issue":"1","key":"pone.0354342.ref017","doi-asserted-by":"crossref","first-page":"1044","DOI":"10.1109\/TCSS.2025.3550419","article-title":"Deep learning-driven behavioral modeling in IoST for mental health monitoring and intervention","volume":"13","author":"J Li","year":"2026","journal-title":"IEEE Trans Comput Soc Syst"},{"key":"pone.0354342.ref018","article-title":"Quantification of stress and well-being using pulse, speech, and electrodermal data: study concept and design","author":"K Izumi","year":"2020","journal-title":"medRxiv"},{"key":"pone.0354342.ref019","doi-asserted-by":"crossref","unstructured":"Noguchi T, Hirokawa M, Doki S, Suzuki K. Computational modeling of mental health checkup with response-based characterization using a smart mirror. Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics; 2024; p. 2549\u201354. https:\/\/doi.org\/10.1109\/SMC54092.2024.10831537","DOI":"10.1109\/SMC54092.2024.10831537"},{"key":"pone.0354342.ref020","article-title":"Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns","author":"Z Jiang","year":"2023","journal-title":"medRxiv"},{"issue":"7","key":"pone.0354342.ref021","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pdig.0000413","article-title":"Evaluating and mitigating unfairness in multimodal remote mental health assessments","volume":"3","author":"Z Jiang","year":"2024","journal-title":"PLOS Digit Health"},{"key":"pone.0354342.ref022","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1146\/annurev.clinpsy.3.022806.091415","article-title":"Ecological momentary assessment","volume":"4","author":"S Shiffman","year":"2008","journal-title":"Annu Rev Clin Psychol"},{"issue":"6","key":"pone.0354342.ref023","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"JA Russell","year":"1980","journal-title":"J Pers Soc Psychol"},{"issue":"3","key":"pone.0354342.ref024","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1017\/S0954579405050340","article-title":"The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology","volume":"17","author":"J Posner","year":"2005","journal-title":"Dev Psychopathol"},{"issue":"6","key":"pone.0354342.ref025","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1016\/S0092-6566(03)00046-1","article-title":"A very brief measure of the Big-Five personality domains","volume":"37","author":"SD Gosling","year":"2003","journal-title":"J Res Pers"},{"key":"pone.0354342.ref026","first-page":"40","article-title":"Development, reliability, and validity of the Japanese version of Ten Item Personality Inventory (TIPI-J)","volume":"21","author":"A Oshio","year":"2012","journal-title":"Jpn J Pers"},{"key":"pone.0354342.ref027","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1016\/S0022-3999(00)00095-7","article-title":"Athens Insomnia Scale: validation of an instrument based on ICD-10 criteria","volume":"48","author":"CR Soldatos","year":"2000","journal-title":"J Psychosom Res"},{"issue":"6","key":"pone.0354342.ref028","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1111\/pcn.12073","article-title":"Development and validation of the Japanese version of the Athens Insomnia Scale","volume":"67","author":"I Okajima","year":"2013","journal-title":"Psychiatry Clin Neurosci"},{"issue":"6","key":"pone.0354342.ref029","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1017\/S0033291702006074","article-title":"Short screening scales to monitor population prevalences and trends in non-specific psychological distress","volume":"32","author":"RC Kessler","year":"2002","journal-title":"Psychol Med"},{"key":"pone.0354342.ref030","doi-asserted-by":"crossref","unstructured":"Baltrusaitis T, Robinson P, Morency LP. OpenFace: an open source facial behavior analysis toolkit. 2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016; 2016.","DOI":"10.1109\/WACV.2016.7477553"},{"key":"pone.0354342.ref031","doi-asserted-by":"crossref","unstructured":"Dutta S, Ganapathy S. Multimodal transformer with learnable frontend and self attention for emotion recognition. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; 2022. p. 5932\u20136. https:\/\/doi.org\/10.1109\/ICASSP43922.2022.9747723","DOI":"10.1109\/ICASSP43922.2022.9747723"},{"key":"pone.0354342.ref032","unstructured":"Radford A, Kim JW, Xu T, Brockman G, Mcleavey C, Sutskever I. Robust speech recognition via large-scale weak supervision; 2023. Available from: https:\/\/proceedings.mlr.press\/v202\/radford23a.html"},{"key":"pone.0354342.ref033","doi-asserted-by":"crossref","unstructured":"Reimers N, Gurevych I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference; 2019. p. 3982\u201392. https:\/\/doi.org\/10.18653\/V1\/D19-1410","DOI":"10.18653\/v1\/D19-1410"},{"key":"pone.0354342.ref034","doi-asserted-by":"crossref","first-page":"1669","DOI":"10.1016\/j.procs.2025.04.398","article-title":"Classification of mental illnesses from reddit posts using Sentence-BERT embeddings and neural networks","volume":"258","author":"FA Wagay","year":"2025","journal-title":"Procedia Comput Sci"},{"key":"pone.0354342.ref035","unstructured":"Ilse M, Tomczak J, Welling M. Attention-based deep multiple instance learning. In: Dy J, Krause A, editors. Proceedings of the 35th International Conference on Machine Learning. vol. 80 of Proceedings of Machine Learning Research. PMLR; 2018. p. 2127\u201336. Available from: https:\/\/proceedings.mlr.press\/v80\/ilse18a.html"},{"issue":"3","key":"pone.0354342.ref036","doi-asserted-by":"crossref","first-page":"941","DOI":"10.1007\/s10044-023-01181-9","article-title":"Deep neural networks for rank-consistent ordinal regression based on conditional probabilities","volume":"26","author":"X Shi","year":"2023","journal-title":"Pattern Anal Appl"},{"key":"pone.0354342.ref037","unstructured":"Menon AK, Jayasumana S, Rawat AS, Jain H, Veit A, Kumar S. Long-tail learning via logit adjustment; 2021. Available from: https:\/\/arxiv.org\/abs\/2007.07314"},{"key":"pone.0354342.ref038","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1016\/j.patrec.2020.11.008","article-title":"Rank consistent ordinal regression for neural networks with application to age estimation","volume":"140","author":"W Cao","year":"2020","journal-title":"Pattern Recognit Lett"},{"issue":"8","key":"pone.0354342.ref039","doi-asserted-by":"crossref","first-page":"1692","DOI":"10.1109\/TPAMI.2015.2461544","article-title":"ModDrop: adaptive multi-modal gesture recognition","volume":"38","author":"N Neverova","year":"2016","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"pone.0354342.ref040","doi-asserted-by":"crossref","unstructured":"Akiba T, Sano S, Yanase T, Ohta T, Koyama M. Optuna: a next-generation hyperparameter optimization framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD \u201919. New York, NY, USA: Association for Computing Machinery; 2019. p. 2623\u201331. Available from: https:\/\/doi.org\/10.1145\/3292500.3330701","DOI":"10.1145\/3292500.3330701"},{"issue":"2","key":"pone.0354342.ref041","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"S Holm","year":"1979","journal-title":"Scand J Stat"},{"key":"pone.0354342.ref042","unstructured":"Powers DMW, Ailab. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. 2020."}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0354342","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T20:02:07Z","timestamp":1789588927000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0354342"}},"subtitle":[],"editor":[{"given":"Iftikhar Ahmed","family":"Khan","sequence":"first","affiliation":[{"name":"University of Lahore - Raiwind Road Campus: The University of Lahore","place":["PAKISTAN"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,16]]},"references-count":42,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,16]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0354342","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,16]]}}}