{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,24]],"date-time":"2025-11-24T16:44:12Z","timestamp":1764002652303,"version":"3.41.2"},"reference-count":46,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,8,3]],"date-time":"2023-08-03T00:00:00Z","timestamp":1691020800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Biomarkers of mental effort may help to identify subtle cognitive impairments in the absence of task performance deficits. Here, we aim to detect mental effort on a verbal task, using automated voice analysis and machine learning.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Audio data from the digit span backwards task were recorded and scored with automated speech recognition using the online platform NeuroVocalix<jats:sup>TM<\/jats:sup>, yielding usable data from 2,764 healthy adults (1,022 male, 1,742 female; mean age 31.4 years). Acoustic features were aggregated across each trial and normalized within each subject. Cognitive load was dichotomized for each trial by categorizing trials at &amp;gt;0.6 of each participants' maximum span as \u201chigh load.\u201d Data were divided into training (60%), test (20%), and validate (20%) datasets, each containing different participants. Training and test data were used in model building and hyper-parameter tuning. Five classification models (Logistic Regression, Naive Bayes, Support Vector Machine, Random Forest, and Gradient Boosting) were trained to predict cognitive load (\u201chigh\u201d vs. \u201clow\u201d) based on acoustic features. Analyses were limited to correct responses. The model was evaluated using the validation dataset, across all span lengths and within the subset of trials with a four-digit span. Classifier discriminant power was examined with Receiver Operating Curve (ROC) analysis.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Participants reached a mean span of 6.34 out of 8 items (SD = 1.38). The Gradient Boosting classifier provided the best performing model on test data (AUC = 0.98) and showed excellent discriminant power for cognitive load on the validation dataset, across all span lengths (AUC = 0.99), and for four-digit only utterances (AUC = 0.95).<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>A sensitive biomarker of mental effort can be derived from vocal acoustic features in remotely administered verbal cognitive tests. The use-case of this biomarker for improving sensitivity of cognitive tests to subtle pathology now needs to be examined.<\/jats:p><\/jats:sec>","DOI":"10.3389\/frai.2023.1171652","type":"journal-article","created":{"date-parts":[[2023,8,3]],"date-time":"2023-08-03T23:01:20Z","timestamp":1691103680000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Prediction of mental effort derived from an automated vocal biomarker using machine learning in a large-scale remote sample"],"prefix":"10.3389","volume":"6","author":[{"given":"Nick","family":"Taptiklis","sequence":"first","affiliation":[]},{"given":"Merina","family":"Su","sequence":"additional","affiliation":[]},{"given":"Jennifer H.","family":"Barnett","sequence":"additional","affiliation":[]},{"given":"Caroline","family":"Skirrow","sequence":"additional","affiliation":[]},{"given":"Jasmin","family":"Kroll","sequence":"additional","affiliation":[]},{"given":"Francesca","family":"Cormack","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,8,3]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"694","DOI":"10.1016\/j.clinph.2013.08.033","article-title":"Complexity of functional connectivity networks in mild cognitive impairment subjects during a working memory task","volume":"125","author":"Ahmadlou","year":"2014","journal-title":"Clin. Neurophysiol."},{"key":"B2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/ICAICT.2016.7991664","article-title":"\u201cSpeech recognition using support vector machines,\u201d","author":"Aida-zade","year":"2016","journal-title":"IEEE 10th International Conference on Application of Information and Communication Technologies (AICT)"},{"key":"B3","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1016\/j.ijpsycho.2012.11.008","article-title":"Dysfunctional and compensatory duality in mild cognitive impairment during a continuous recognition memory task","volume":"87","author":"Aurtenetxe","year":"2013","journal-title":"Int. J. Psychophysiol."},{"key":"B4","doi-asserted-by":"publisher","first-page":"702538","DOI":"10.3389\/fpsyg.2021.702538","article-title":"The validity of physiological measures to identify differences in intrinsic cognitive load","volume":"12","author":"Ayres","year":"2021","journal-title":"Front. Psychol."},{"key":"B5","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1007\/s10772-016-9338-4","article-title":"Analysis of multiple types of voice recordings in cepstral domain using MFCC for discriminating between patients with Parkinson's disease and healthy people","volume":"19","author":"Benba","year":"2016","journal-title":"Int. J. Speech Technol."},{"key":"B6","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1007\/s10772-021-09808-0","article-title":"A review on speech processing using machine learning paradigm","volume":"24","author":"Bhangale","year":"2021","journal-title":"Int. J. Speech Technol."},{"key":"B7","doi-asserted-by":"publisher","first-page":"341","DOI":"10.4236\/jdaip.2020.84020","article-title":"Basic tenets of classification algorithms k-nearest-neighbor, support vector machine, random forest and neural network: a review","volume":"8","author":"Boateng","year":"2020","journal-title":"J. Data Anal. Inform. Process."},{"key":"B8","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1038\/s41598-017-00633-7","article-title":"EEG-based cognitive control behaviour assessment: an ecological study with professional air traffic controllers","volume":"7","author":"Borghini","year":"2017","journal-title":"Sci. Rep."},{"key":"B9","doi-asserted-by":"publisher","first-page":"895","DOI":"10.1111\/j.1553-2712.1996.tb03538.x","article-title":"Statistical methodology: I. Incorporating the prevalence of disease into the sample size calculation for sensitivity and specificity","volume":"3","author":"Buderer","year":"1996","journal-title":"Acad. Emerg. Med."},{"key":"B10","doi-asserted-by":"publisher","first-page":"2266","DOI":"10.1007\/s00034-018-0962-x","article-title":"A pre-classification-based language identification for northeast indian languages using prosody and spectral features","volume":"38","author":"China Bhanja","year":"2019","journal-title":"Circuits Syst. Signal Process."},{"key":"B11","doi-asserted-by":"publisher","first-page":"5364","DOI":"10.1109\/JBHI.2022.3197910","article-title":"Gradient boosting machine and efficient combination of features for speech-based detection of COVID-19","volume":"26","author":"Dash","year":"2022","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"B12","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1037\/0022-0663.100.1.223","article-title":"A comparison of three measures of cognitive load: evidence for separable measures of intrinsic, extraneous, and germane load","volume":"100","author":"DeLeeuw","year":"2008","journal-title":"J. Educ. Psychol."},{"key":"B13","doi-asserted-by":"publisher","first-page":"961","DOI":"10.1001\/jamaneurol.2014.803","article-title":"The preclinical Alzheimer cognitive composite: measuring amyloid-related decline","volume":"71","author":"Donohue","year":"2014","journal-title":"JAMA Neurol."},{"key":"B14","doi-asserted-by":"publisher","first-page":"819","DOI":"10.1016\/j.biopsych.2019.12.021","article-title":"Amyloid-\u03b2 positivity predicts cognitive decline but cognition predicts progression to amyloid-\u03b2 positivity","volume":"87","author":"Elman","year":"2020","journal-title":"Biol. Psychiatry"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1145\/2502081.2502224","article-title":"\u201cRecent developments in openSMILE, the munich open-source multimedia feature extractor,\u201d","author":"Eyben","year":"2013","journal-title":"Proceedings of the 21st ACM International Conference on Multimedia"},{"year":"2003","author":"Forsberg","key":"B16"},{"key":"B17","doi-asserted-by":"publisher","first-page":"191","DOI":"10.3389\/fnhum.2019.00191","article-title":"EEG-based prediction of cognitive load in intelligence tests","volume":"13","author":"Friedman","year":"2019","journal-title":"Front. Hum. Neurosci."},{"key":"B18","doi-asserted-by":"publisher","first-page":"380","DOI":"10.1097\/WCO.0000000000000469","article-title":"Measuring compensation in neurodegeneration using MRI","volume":"30","author":"Gregory","year":"2017","journal-title":"Curr. Opin. Neurol."},{"key":"B19","first-page":"627","article-title":"Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation","volume":"4","author":"Hajian-Tilaki","year":"2013","journal-title":"Caspian J. Intern. Med."},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1145\/3279810.3279851","article-title":"\u201cSymptoms of cognitive load in interactions with a dialogue system,\u201d","author":"Lopes","year":"2018","journal-title":"Paper Presented at the Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data"},{"key":"B21","doi-asserted-by":"publisher","first-page":"297","DOI":"10.3311\/PPee.10414","article-title":"Monitoring cognitive workload using vocal tract and voice source features","volume":"61","author":"Magn\u00fasd\u00f3ttir","year":"2017","journal-title":"Periodica Polytechnica Elect. Eng. Comput. Sci"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1109\/CogInfoCom.2016.7804532","article-title":"\u201cVocal tract and voice source features for monitoring cognitive workload,\u201d","author":"Meier","year":"2016","journal-title":"Paper Presented at the 2016 7th IEEE International Conference on Cognitive Infocommunications (CogInfoCom)"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1109\/CogInfoCom.2017.8268268","article-title":"\u201cClassification of cognitive load using voice features: a preliminary investigation,\u201d","author":"Miji\u0107","year":"2017","journal-title":"Paper Presented at the 2017 8th IEEE International Conference on Cognitive Infocommunications (CogInfoCom)."},{"key":"B24","doi-asserted-by":"publisher","first-page":"650","DOI":"10.1212\/WNL.0000000000004236","article-title":"Automatic measurement of prosody in behavioral variant FTD","volume":"89","author":"Nevler","year":"2017","journal-title":"Neurology"},{"key":"B25","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1207\/S15326985EP3801_8","article-title":"Cognitive load measurement as a means to advance cognitive load theory","volume":"38","author":"Paas","year":"2003","journal-title":"Educ. Psychol."},{"key":"B26","doi-asserted-by":"publisher","first-page":"122","DOI":"10.1037\/0022-0663.86.1.122","article-title":"Variability of worked examples and transfer of geometrical problem-solving skills: a cognitive-load approach","volume":"86","author":"Paas","year":"1994","journal-title":"J. Educ. Psychol."},{"key":"B27","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1016\/j.jbef.2017.12.004","article-title":"Prolific.ac\u2014A subject pool for online experiments","volume":"17","author":"Palan","year":"2018","journal-title":"J. Behav. Exp. Fin."},{"key":"B28","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1146\/annurev.psych.59.103006.093656","article-title":"The adaptive brain: aging and neurocognitive scaffolding","volume":"60","author":"Park","year":"2009","journal-title":"Annu. Rev. Psychol."},{"key":"B29","first-page":"2825","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"Pedregosa","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"B30","doi-asserted-by":"publisher","first-page":"1160","DOI":"10.1001\/archneur.62.7.1160","article-title":"Mild cognitive impairment as a clinical entity and treatment target","volume":"62","author":"Petersen","year":"2005","journal-title":"Arch. Neurol."},{"key":"B31","doi-asserted-by":"crossref","unstructured":"QuatieriT. F.\n            WilliamsonJ. R.\n            SmaltC. J.\n            PatelT.\n            PerriconeJ.\n            MehtaD. D.\n          10.21437\/Interspeech.2015-566Vocal Biomarkers to Discriminate Cognitive Load in a Working Memory Task2015","DOI":"10.21437\/Interspeech.2015-566"},{"key":"B32","doi-asserted-by":"publisher","first-page":"516","DOI":"10.1016\/j.neubiorev.2017.07.001","article-title":"Cognitive workload across the spectrum of cognitive impairments: a systematic review of physiological measures","volume":"80","author":"Ranchet","year":"2017","journal-title":"Neurosci. Biobehav. Rev."},{"year":"2011","author":"Rao","key":"B33"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2014-114","article-title":"\u201cClassification of cognitive load from speech using an i-vector framework,\u201d","author":"Segbroeck","year":"2014","journal-title":"Paper Presented at the Interspeech 2014"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1109\/ICCCE.2008.4580709","article-title":"\u201cAn overview of speech recognition system based on the support vector machines,\u201d","author":"Sonkamble","year":"2008","journal-title":"Paper Presented at the 2008 International Conference on Computer and Communication Engineering"},{"key":"B36","doi-asserted-by":"publisher","first-page":"280","DOI":"10.1016\/j.jalz.2011.03.003","article-title":"Toward defining the preclinical stages of Alzheimer's disease: recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease","volume":"7","author":"Sperling","year":"2011","journal-title":"Alzheimers Dement."},{"key":"B37","doi-asserted-by":"publisher","first-page":"S69","DOI":"10.1097\/01.wad.0000213815.20177.19","article-title":"Cognitive reserve and Alzheimer disease","volume":"20","author":"Stern","year":"2006","journal-title":"Alzheimer Dis. Assoc. Disord."},{"key":"B38","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1007\/978-1-4419-8126-4","article-title":"\u201cMeasuring cognitive load,\u201d","author":"Sweller","year":"2011","journal-title":"Cognitive Load Theory"},{"key":"B39","doi-asserted-by":"publisher","first-page":"P168","DOI":"10.1016\/j.jalz.2017.06.2619","volume":"13","author":"Taptiklis","year":"2017","journal-title":"Alzheimer's Dement."},{"key":"B40","first-page":"287","article-title":"On Naive Bayes in speech recognition","volume":"15","author":"T\u00f3th","year":"2005","journal-title":"Int. J. Appl. Math. Comput. Sci"},{"key":"B41","doi-asserted-by":"publisher","first-page":"2242","DOI":"10.1016\/j.procs.2022.09.534","article-title":"Evaluation of the efficiency of state-of-the-art speech recognition engines","volume":"207","author":"Trabelsi","year":"2022","journal-title":"Proc. Comput. Sci."},{"key":"B42","doi-asserted-by":"publisher","first-page":"1065","DOI":"10.1017\/S0033291706007744","article-title":"Brain reserve and cognitive decline: a non-parametric systematic review","volume":"36","author":"Valenzuela","year":"2006","journal-title":"Psychol. Med."},{"key":"B43","doi-asserted-by":"publisher","first-page":"375","DOI":"10.1007\/s10648-010-9145-4","article-title":"Cognitive load theory: advances in research on worked examples, animations, and cognitive load measurement","volume":"22","author":"van Gog","year":"2010","journal-title":"Educ. Psychol. Rev."},{"key":"B44","doi-asserted-by":"publisher","first-page":"1100","DOI":"10.1177\/1745691617693393","article-title":"Choosing prediction over explanation in psychology: lessons from machine learning","volume":"12","author":"Yarkoni","year":"2017","journal-title":"Perspect. Psychol. Sci."},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2008.4518041","article-title":"\u201cSpeech-based cognitive load monitoring system,\u201d","author":"Yin","year":"2008","journal-title":"Paper Presented at the 2008 IEEE International Conference on Acoustics, Speech and Signal Processing."},{"key":"B46","doi-asserted-by":"publisher","first-page":"1202","DOI":"10.3758\/s13428-020-01480-7","article-title":"Web-based and mixed-mode cognitive large-scale assessments in higher education: an evaluation of selection bias, measurement bias, and prediction bias","volume":"53","author":"Zinn","year":"2021","journal-title":"Behav. Res. Methods"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1171652\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,3]],"date-time":"2023-08-03T23:01:27Z","timestamp":1691103687000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1171652\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,3]]},"references-count":46,"alternative-id":["10.3389\/frai.2023.1171652"],"URL":"https:\/\/doi.org\/10.3389\/frai.2023.1171652","relation":{},"ISSN":["2624-8212"],"issn-type":[{"type":"electronic","value":"2624-8212"}],"subject":[],"published":{"date-parts":[[2023,8,3]]},"article-number":"1171652"}}