{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T21:12:00Z","timestamp":1785532320868,"version":"3.56.0"},"reference-count":44,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2024,1,2]],"date-time":"2024-01-02T00:00:00Z","timestamp":1704153600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAD"],"published-print":{"date-parts":[[2024,1,2]]},"abstract":"<jats:p>Background: Previous research has shown that verbal memory accurately measures cognitive decline in the early phases of neurocognitive impairment. Automatic speech recognition from the verbal learning task (VLT) can potentially be used to differentiate between people with and without cognitive impairment. Objective: Investigate whether automatic speech recognition (ASR) of the VLT is reliable and able to differentiate between subjective cognitive decline (SCD) and mild cognitive impairment (MCI). Methods: The VLT was recorded and processed via a mobile application. Following, verbal memory features were automatically extracted. The diagnostic performance of the automatically derived features was investigated by training machine learning classifiers to distinguish between participants with SCD versus MCI\/dementia. Results: The ICC for inter-rater reliability between the clinical and automatically derived features was 0.87 for the total immediate recall and 0.94 for the delayed recall. The full model including the total immediate recall, delayed recall, recognition count, and the novel verbal memory features had an AUC of 0.79 for distinguishing between participants with SCD versus MCI\/dementia. The ten best differentiating VLT features correlated low to moderate with other cognitive tests such as logical memory tasks, semantic verbal fluency, and executive functioning. Conclusions: The VLT with automatically derived verbal memory features showed in general high agreement with the clinical scoring and distinguished well between SCD and MCI\/dementia participants. This might be of added value in screening for cognitive impairment.<\/jats:p>","DOI":"10.3233\/jad-230608","type":"journal-article","created":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T11:36:51Z","timestamp":1702381011000},"page":"179-191","source":"Crossref","is-referenced-by-count":12,"title":["The Reliability and Clinical Validation of Automatically-Derived Verbal Memory Features of the Verbal Learning Test in Early Diagnostics of Cognitive Impairment"],"prefix":"10.1177","volume":"97","author":[{"given":"Nina","family":"Possemis","sequence":"first","affiliation":[{"name":"Alzheimer Centre Limburg, School for Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daphne","family":"ter Huurne","sequence":"additional","affiliation":[{"name":"Alzheimer Centre Limburg, School for Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leonie","family":"Banning","sequence":"additional","affiliation":[{"name":"Maastricht University Medical Centre+ (MUMC+), Department of Psychiatry & Psychology, Maastricht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angelique","family":"Gruters","sequence":"additional","affiliation":[{"name":"Catharina Hospital, Medical Psychology, Eindhoven, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephanie","family":"Van Asbroeck","sequence":"additional","affiliation":[{"name":"Alzheimer Centre Limburg, School for Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandra","family":"K\u00f6nig","sequence":"additional","affiliation":[{"name":"National Institute for Research in Computer Science and Automation (INRIA), Valbonne, Sophia Antipolis, France"},{"name":"ki:elements, Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicklas","family":"Linz","sequence":"additional","affiliation":[{"name":"ki:elements, Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johannes","family":"Tr\u00f6ger","sequence":"additional","affiliation":[{"name":"ki:elements, Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kai","family":"Langel","sequence":"additional","affiliation":[{"name":"Janssen Clinical Innovation, Beerse, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arjan","family":"Blokland","sequence":"additional","affiliation":[{"name":"Faculty of Psychology and Neuroscience, Department of Neuropsychology & Psychopharmacology, EURON, Maastricht University, Maastricht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jos","family":"Prickaerts","sequence":"additional","affiliation":[{"name":"School for Mental Health and Neuroscience, Department of Psychiatry and Neuropsychology, Maastricht University, Maastricht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marjolein","family":"de Vugt","sequence":"additional","affiliation":[{"name":"Alzheimer Centre Limburg, School for Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands"},{"name":"Maastricht University Medical Centre+ (MUMC+), Department of Psychiatry & Psychology, Maastricht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frans","family":"Verhey","sequence":"additional","affiliation":[{"name":"Alzheimer Centre Limburg, School for Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands"},{"name":"Maastricht University Medical Centre+ (MUMC+), Department of Psychiatry & Psychology, Maastricht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Inez","family":"Ramakers","sequence":"additional","affiliation":[{"name":"Alzheimer Centre Limburg, School for Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands"},{"name":"Maastricht University Medical Centre+ (MUMC+), Department of Psychiatry & Psychology, Maastricht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","reference":[{"key":"10.3233\/JAD-230608_ref1","first-page":"165","article-title":"Early detection of mild cognitive impairment (MCI) in primary care","volume":"7","author":"Sabbagh","year":"2020","journal-title":"J Prev Alzheimers Dis"},{"key":"10.3233\/JAD-230608_ref2","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1007\/BF01989627","article-title":"The clinical utility of the Rey Auditory-Verbal Learning Test in medical rehabilitation","volume":"1","author":"Callahan","year":"1994","journal-title":"J Clin Psychol Med Settings"},{"key":"10.3233\/JAD-230608_ref3","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1017\/S1355617708090176","article-title":"Diagnostic utility of the NAB List Learning test in Alzheimer\u2019s disease and amnestic mild cognitive impairment","volume":"15","author":"Gavett","year":"2009","journal-title":"J Int Neuropsychol Soc"},{"key":"10.3233\/JAD-230608_ref4","doi-asserted-by":"crossref","first-page":"1509","DOI":"10.1017\/S0033291714002645","article-title":"The trajectory of cognitive decline in the pre-dementia phase in memory clinic visitors: Findings from the 4C-MCI study","volume":"45","author":"Hamel","year":"2015","journal-title":"Psychol Med"},{"key":"10.3233\/JAD-230608_ref5","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1159\/000334525","article-title":"The diagnostic value of controlled oral word association test-FAS and category fluency in single-domain amnestic mild cognitive impairment","volume":"32","author":"Malek-Ahmadi","year":"2011","journal-title":"Dement Geriatr Cogn Disord"},{"key":"10.3233\/JAD-230608_ref6","doi-asserted-by":"crossref","first-page":"406","DOI":"10.1080\/13803395.2020.1741517","article-title":"Examining the factor structure of the Rey auditory verbal learning test in individuals across the life span","volume":"42","author":"Weitzner","year":"2020","journal-title":"J Clin Exp Neuropsychol"},{"key":"10.3233\/JAD-230608_ref7","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1093\/arclin\/acp093","article-title":"The predictive value of memory strategies for Alzheimer\u2019s disease in subjects with mild cognitive impairment","volume":"25","author":"Ramakers","year":"2010","journal-title":"Arch Clin Neuropsychol"},{"key":"10.3233\/JAD-230608_ref8","doi-asserted-by":"crossref","unstructured":"Satt A , Hoory R , K\u00f6nig A , Aalten P , Robert PH (2014) Speech-based automatic and robust detection of very early dementia. Fifteenth Annual Conference of the International Speech Communication Association.","DOI":"10.21437\/Interspeech.2014-544"},{"key":"10.3233\/JAD-230608_ref9","doi-asserted-by":"crossref","first-page":"e23777","DOI":"10.2196\/23777","article-title":"Paradigm shift toward digital neuropsychology and high-dimensional neuropsychological assessments: Review","volume":"22","author":"Parsons","year":"2020","journal-title":"J Med Internet Res"},{"key":"10.3233\/JAD-230608_ref10","doi-asserted-by":"crossref","first-page":"6929","DOI":"10.1038\/s41598-021-85981-1","article-title":"Comprehensive verbal fluency features predict executive function performance","volume":"11","author":"Amunts","year":"2021","journal-title":"Sci Rep"},{"key":"10.3233\/JAD-230608_ref11","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1159\/000487852","article-title":"Fully automatic speech-based analysis of the semantic verbal fluency task","volume":"45","author":"Konig","year":"2018","journal-title":"Dement Geriatr Cogn Disord"},{"key":"10.3233\/JAD-230608_ref12","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1016\/j.dadm.2014.11.012","article-title":"Automatic speech analysis for the assessment of patients with predementia and Alzheimer\u2019s disease","volume":"1","author":"Konig","year":"2015","journal-title":"Alzheimers Dement (Amst)"},{"key":"10.3233\/JAD-230608_ref13","doi-asserted-by":"crossref","first-page":"103","DOI":"10.18653\/v1\/W19-3012","article-title":"Temporal analysis of the semantic verbal fluency task in persons with subjective and mild cognitive impairment","author":"Linz","year":"2019","journal-title":"Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology"},{"key":"10.3233\/JAD-230608_ref14","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1007\/s11065-019-09423-6","article-title":"The factor structure of cognitive functioning in cognitively healthy participants: A meta-analysis and meta-analysis of individual participant data","volume":"30","author":"Agelink van Rentergem","year":"2020","journal-title":"Neuropsychol Rev"},{"key":"10.3233\/JAD-230608_ref15","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1016\/0022-3956(75)90026-6","article-title":"\u201cMini-mental state\u201d. A practical method for grading the cognitive state of patients for the clinician","volume":"12","author":"Folstein","year":"1975","journal-title":"J Psychiatr Res"},{"key":"10.3233\/JAD-230608_ref16","first-page":"1","article-title":"Gestandaardiseerde MMSE","author":"Kok","year":"2002","journal-title":"Zeist: Altrecht GGZ"},{"key":"10.3233\/JAD-230608_ref17","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1192\/bjp.140.6.566","article-title":"A new clinical scale for the staging of dementia","volume":"140","author":"Hughes","year":"1982","journal-title":"Br J Psychiatry"},{"key":"10.3233\/JAD-230608_ref18","doi-asserted-by":"crossref","first-page":"471","DOI":"10.5014\/ajot.53.5.471","article-title":"Development of a functional measure for persons with Alzheimer\u2019s disease: The disability assessment for dementia","volume":"53","author":"Gelinas","year":"1999","journal-title":"Am J Occup Ther"},{"key":"10.3233\/JAD-230608_ref19","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1017\/S1041610210000967","article-title":"Executive functioning mediates the link between other neuropsychological domains and daily functioning: A Project FRONTIER study","volume":"23","author":"O\u2019Bryant","year":"2011","journal-title":"Int Psychogeriatr"},{"key":"10.3233\/JAD-230608_ref20","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1300\/J018v05n01_09","article-title":"Geriatric Depression Scale (GDS): Recent evidence and development of a shorter version","volume":"5","author":"Sheikh","year":"1986","journal-title":"Clin Gerontol"},{"key":"10.3233\/JAD-230608_ref21","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1017\/S1355617705050344","article-title":"Rey\u2019s verbal learning test: Normative data for 1855 healthy participants aged 24-81 years and the influence of age, sex, education, and mode of presentation","volume":"11","author":"Van der Elst","year":"2005","journal-title":"J Int Neuropsychol Soc"},{"key":"10.3233\/JAD-230608_ref22","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1080\/00221309.1985.9711004","article-title":"Learning and retrieval rate of words presented auditorily and visually","volume":"112","author":"Brand","year":"1985","journal-title":"J Gen Psychol"},{"key":"10.3233\/JAD-230608_ref23","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1017\/S1355617706060115","article-title":"Normative data for the Animal, Profession and Letter M Naming verbal fluency tests for Dutch speaking participants and the effects of age, education, and sex","volume":"12","author":"Van der Elst","year":"2006","journal-title":"J Int Neuropsychol Soc"},{"key":"10.3233\/JAD-230608_ref24","doi-asserted-by":"crossref","first-page":"271","DOI":"10.2466\/pms.1958.8.3.271","article-title":"Validity of the Trail Making Test as an indicator of organic brain damage","volume":"8","author":"Reitan","year":"1958","journal-title":"Percept Mot Skills"},{"key":"10.3233\/JAD-230608_ref27","doi-asserted-by":"crossref","unstructured":"(2020) 2020 Alzheimer\u2019s disease facts and figures. Alzheimers Dement 16, 391\u2013460.","DOI":"10.1002\/alz.12068"},{"key":"10.3233\/JAD-230608_ref28","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1016\/j.jalz.2011.03.005","article-title":"The diagnosis of dementia due to Alzheimer\u2019s disease: Recommendations from the National Institute on Aging-Alzheimer\u2019s Association workgroups on diagnostic guidelines for Alzheimer\u2019s disease","volume":"7","author":"McKhann","year":"2011","journal-title":"Alzheimers Dement"},{"key":"10.3233\/JAD-230608_ref29","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/j.jalz.2012.09.017","article-title":"AD dementia risk in late MCI, in early MCI, and in subjective memory impairment","volume":"10","author":"Jessen","year":"2014","journal-title":"Alzheimers Dement"},{"key":"10.3233\/JAD-230608_ref30","unstructured":"Rey A (1958), L\u2019examen clinique en psychologie."},{"key":"10.3233\/JAD-230608_ref31","doi-asserted-by":"crossref","first-page":"667","DOI":"10.1093\/arclin\/acac105","article-title":"The accuracy of speech and linguistic analysis in early diagnostics of neurocognitive disorders in a memory clinic setting","volume":"38","author":"Ter Huurne","year":"2023","journal-title":"Arch Clin Neuropsychol"},{"key":"10.3233\/JAD-230608_ref33","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1108\/S0195-6310(2013)0000030017","article-title":"International standard classification of education: 2011","volume":"30","author":"Schneider","year":"2013","journal-title":"Comp Soc Res"},{"key":"10.3233\/JAD-230608_ref34","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.tjem.2018.08.001","article-title":"User\u2019s guide to correlation coefficients","volume":"18","author":"Akoglu","year":"2018","journal-title":"Turk J Emerg Med"},{"key":"10.3233\/JAD-230608_ref35","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"10.3233\/JAD-230608_ref36","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J Chiropr Med"},{"key":"10.3233\/JAD-230608_ref37","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1177\/1536867X1201200307","article-title":"A generalized Hosmer\u2013Lemeshow goodness-of-fit test for multinomial logistic regression models","volume":"12","author":"Fagerland","year":"2012","journal-title":"Stata J"},{"key":"10.3233\/JAD-230608_ref38","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1080\/13825585.2010.530221","article-title":"The usefulness of the story recall test in patients with mild cognitive impairment and Alzheimer\u2019s disease","volume":"18","author":"Baek","year":"2011","journal-title":"Aging Neuropsychol Cogn"},{"key":"10.3233\/JAD-230608_ref39","doi-asserted-by":"crossref","first-page":"1022","DOI":"10.1212\/WNL.59.7.1022","article-title":"Alzheimer\u2019s disease can be accurately diagnosed in very mildly impaired individuals","volume":"59","author":"Salmon","year":"2002","journal-title":"Neurology"},{"key":"10.3233\/JAD-230608_ref40","doi-asserted-by":"crossref","first-page":"1063","DOI":"10.1016\/S0010-9452(08)70878-5","article-title":"Acquisition, recall, and forgetting of verbal information in long-term memory by young, middle-aged, and elderly individuals","volume":"39","author":"Davis","year":"2003","journal-title":"Cortex"},{"key":"10.3233\/JAD-230608_ref41","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1159\/000214632","article-title":"A subtest of the MMSE as a valid test of episodic memory? Comparison with the Free and Cued Reminding Test","volume":"27","author":"Carcaillon","year":"2009","journal-title":"Dement Geriatr Cogn Disord"},{"key":"10.3233\/JAD-230608_ref42","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.jpsychires.2019.02.016","article-title":"Executive functioning in cognitively normal middle-aged offspring of late-onset Alzheimer\u2019s disease patients","volume":"112","author":"Abulafia","year":"2019","journal-title":"J Psychiatr Res"},{"key":"10.3233\/JAD-230608_ref43","first-page":"129","article-title":"Validity convergent and reliability test-retest of the rey auditory verbal learning test","volume":"9","author":"de Sousa Magalh\u00e3es","year":"2012","journal-title":"Clin Neuropsychiatry"},{"key":"10.3233\/JAD-230608_ref44","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/j.neuropsychologia.2017.05.021","article-title":"Is \u201cLearning\u201d episodic memory? Distinct cognitive and neuroanatomic correlates of immediate recall during learning trials in neurologically normal aging and neurodegenerative cohorts","volume":"102","author":"Casaletto","year":"2017","journal-title":"Neuropsychologia"},{"key":"10.3233\/JAD-230608_ref45","doi-asserted-by":"crossref","first-page":"e052250","DOI":"10.1136\/bmjopen-2021-052250","article-title":"Remote data collection speech analysis and prediction of the identification of Alzheimer\u2019s disease biomarkers in people at risk for Alzheimer\u2019s disease dementia: The Speech on the Phone Assessment (SPeAk) prospective observational study protocol","volume":"12","author":"Gregory","year":"2022","journal-title":"BMJ Open"},{"key":"10.3233\/JAD-230608_ref47","doi-asserted-by":"crossref","first-page":"1095","DOI":"10.3233\/JAD-150137","article-title":"The 12 years preceding mild cognitive impairment due to Alzheimer\u2019s disease: The temporal emergence of cognitive decline","volume":"48","author":"Mistridis","year":"2015","journal-title":"J Alzheimers Dis"},{"key":"10.3233\/JAD-230608_ref48","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1037\/neu0000413","article-title":"Word-list intrusion errors predict progression to mild cognitive impairment","volume":"32","author":"Thomas","year":"2018","journal-title":"Neuropsychology"}],"container-title":["Journal of Alzheimer's Disease"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/JAD-230608","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T10:09:00Z","timestamp":1777370940000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/JAD-230608"}},"subtitle":[],"editor":[{"given":"David","family":"Loewenstein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,1,2]]},"references-count":44,"journal-issue":{"issue":"1"},"URL":"https:\/\/doi.org\/10.3233\/jad-230608","relation":{},"ISSN":["1387-2877","1875-8908"],"issn-type":[{"value":"1387-2877","type":"print"},{"value":"1875-8908","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,2]]}}}