{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T03:27:11Z","timestamp":1793158031437,"version":"4.5.0"},"reference-count":41,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2022,1,21]],"date-time":"2022-01-21T00:00:00Z","timestamp":1642723200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"James M. Cox Foundation","id":[{"id":"https:\/\/ror.org\/05th1gp75","id-type":"ROR","asserted-by":"crossref"}]},{"DOI":"10.13039\/100008490","name":"Goizueta Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100008490","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006528","name":"Alzheimer\u2019s Disease Research Center, Emory University","doi-asserted-by":"publisher","award":["P50 AG025688"],"award-info":[{"award-number":["P50 AG025688"]}],"id":[{"id":"10.13039\/100006528","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["AG070937"],"award-info":[{"award-number":["AG070937"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Differences in expressing facial emotions are broadly observed in people with cognitive impairment. However, these differences have been difficult to objectively quantify and systematically evaluate among people with cognitive impairment across disease etiologies and severity. Therefore, a computer vision-based deep learning model for facial emotion recognition trained on 400.000 faces was utilized to analyze facial emotions expressed during a passive viewing memory test. In addition, this study was conducted on a large number of individuals (n = 493), including healthy controls and individuals with cognitive impairment due to diverse underlying etiologies and across different disease stages. Diagnoses included subjective cognitive impairment, Mild Cognitive Impairment (MCI) due to AD, MCI due to other etiologies, dementia due to Alzheimer\u2019s diseases (AD), and dementia due to other etiologies (e.g., Vascular Dementia, Frontotemporal Dementia, Lewy Body Dementia, etc.). The Montreal Cognitive Assessment (MoCA) was used to evaluate cognitive performance across all participants. A participant with a score of less than or equal to 24 was considered cognitively impaired (CI). Compared to cognitively unimpaired (CU) participants, CI participants expressed significantly less positive emotions, more negative emotions, and higher facial expressiveness during the test. In addition, classification analysis revealed that facial emotions expressed during the test allowed effective differentiation of CI from CU participants, largely independent of sex, race, age, education level, mood, and eye movements (derived from an eye-tracking-based digital biomarker for cognitive impairment). No screening methods reliably differentiated the underlying etiology of the cognitive impairment. The findings provide quantitative and comprehensive evidence that the expression of facial emotions is significantly different in people with cognitive impairment, and suggests this may be a useful tool for passive screening of cognitive impairment.<\/jats:p>","DOI":"10.1371\/journal.pone.0262527","type":"journal-article","created":{"date-parts":[[2022,1,21]],"date-time":"2022-01-21T13:59:59Z","timestamp":1642773599000},"page":"e0262527","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":52,"title":["Automated analysis of facial emotions in subjects with cognitive impairment"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3570-9461","authenticated-orcid":true,"given":"Zifan","family":"Jiang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salman","family":"Seyedi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rafi U.","family":"Haque","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alvince L.","family":"Pongos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kayci L.","family":"Vickers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5266-0631","authenticated-orcid":true,"given":"Cecelia M.","family":"Manzanares","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James J.","family":"Lah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Allan I.","family":"Levey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gari D.","family":"Clifford","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2022,1,21]]},"reference":[{"issue":"3","key":"pone.0262527.ref001","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1002\/alz.12068","article-title":"2020 Alzheimer\u2019s disease facts and figures","volume":"16","year":"2020","journal-title":"Alzheimer\u2019s & Dementia"},{"issue":"10","key":"pone.0262527.ref002","doi-asserted-by":"crossref","first-page":"1193","DOI":"10.1001\/archpsyc.65.10.1193","article-title":"Prevalence of Neuropsychiatric Symptoms in Mild Cognitive Impairment and Normal Cognitive Aging: Population-Based Study","volume":"65","author":"YE Geda","year":"2008","journal-title":"Archives of General Psychiatry"},{"issue":"4","key":"pone.0262527.ref003","doi-asserted-by":"crossref","first-page":"695","DOI":"10.1111\/j.1532-5415.2005.53221.x","article-title":"The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment","volume":"53","author":"ZS Nasreddine","year":"2005","journal-title":"Journal of the American Geriatrics Society"},{"issue":"4","key":"pone.0262527.ref004","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1016\/j.jpsychires.2008.04.014","article-title":"A meta-analysis of the accuracy of the mini-mental state examination in the detection of dementia and mild cognitive impairment","volume":"43","author":"AJ Mitchell","year":"2009","journal-title":"Journal of psychiatric research"},{"issue":"5","key":"pone.0262527.ref005","doi-asserted-by":"crossref","first-page":"1039","DOI":"10.12740\/PP\/45368","article-title":"Is the Montreal Cognitive Assessment (MoCA) test better suited than the Mini-Mental State Examination (MMSE) in mild cognitive impairment (MCI) detection among people aged over 60? Meta-analysis","volume":"50","author":"N Ciesielska","year":"2016","journal-title":"Psychiatr Pol"},{"key":"pone.0262527.ref006","first-page":"1","article-title":"Classifying Major Depressive Disorderand Response to Deep Brain Stimulation Over Time by Analyzing Facial Expressions","author":"Z Jiang","year":"2020","journal-title":"IEEE Trans Biomed Eng"},{"issue":"5","key":"pone.0262527.ref007","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1097\/JGP.0b013e318165dbce","article-title":"Facial Emotion Recognition Deficit in Amnestic Mild Cognitive Impairment and Alzheimer Disease","volume":"16","author":"I Spoletini","year":"2008","journal-title":"The American Journal of Geriatric Psychiatry"},{"issue":"2","key":"pone.0262527.ref008","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1177\/0891988709332939","article-title":"Impaired Social Cognition in Mild Alzheimer Disease","volume":"22","author":"B Bediou","year":"2009","journal-title":"J Geriatr Psychiatry Neurol"},{"issue":"4","key":"pone.0262527.ref009","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1159\/000100829","article-title":"Deficits in Facial Emotion Processing in Mild Cognitive Impairment","volume":"23","author":"E Teng","year":"2007","journal-title":"Dement Geriatr Cogn Disord"},{"issue":"1-2","key":"pone.0262527.ref010","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1016\/j.jns.2015.07.034","article-title":"Emotional processing in patients with mild cognitive impairment: The influence of the valence and intensity of emotional stimuli","volume":"357","author":"CM Sarabia-Cobo","year":"2015","journal-title":"Journal of the Neurological Sciences"},{"issue":"3-4","key":"pone.0262527.ref011","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1080\/13825580600735085","article-title":"Emotional experience and facial expression in Alzheimer\u2019s disease","volume":"13","author":"KW Burton","year":"2006","journal-title":"Aging, Neuropsychology, and Cognition"},{"issue":"1","key":"pone.0262527.ref012","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1037\/a0014001","article-title":"Emotion experience, expression, and regulation in Alzheimer\u2019s disease","volume":"24","author":"JD Henry","year":"2009","journal-title":"Psychology and Aging"},{"key":"pone.0262527.ref013","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.cobeha.2017.05.017","article-title":"Increased subjective experience of non-target emotions in patients with frontotemporal dementia and Alzheimer\u2019s disease","volume":"15","author":"KH Chen","year":"2017","journal-title":"Current Opinion in Behavioral Sciences"},{"issue":"2","key":"pone.0262527.ref014","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1177\/1533317512440495","article-title":"Facial Expression in Alzheimer\u2019s Disease: Impact of Cognitive Deficits and Neuropsychiatric Symptoms","volume":"27","author":"U Seidl","year":"2012","journal-title":"Am J Alzheimers Dis Other Demen"},{"issue":"1","key":"pone.0262527.ref015","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1002\/gps.3977","article-title":"Relationship between observable emotional expression and wandering behavior of people with dementia: Emotional expression and wandering","volume":"29","author":"KH Lee","year":"2014","journal-title":"Int J Geriatr Psychiatry"},{"issue":"3","key":"pone.0262527.ref016","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1017\/S104161029600275X","article-title":"Emotional Expression During Mid- to Late-Stage Dementia","volume":"8","author":"C Magai","year":"1996","journal-title":"Int Psychogeriatr"},{"issue":"11","key":"pone.0262527.ref017","doi-asserted-by":"crossref","first-page":"1163","DOI":"10.1002\/gps.4742","article-title":"Symptoms associated with Lewy body disease in mild cognitive impairment: Symptoms associated with Lewy body MCI","volume":"32","author":"PC Donaghy","year":"2017","journal-title":"Int J Geriatr Psychiatry"},{"issue":"1","key":"pone.0262527.ref018","doi-asserted-by":"crossref","first-page":"4367","DOI":"10.1038\/s41598-020-61310-w","article-title":"Impairments in face discrimination and emotion recognition are related to aging and cognitive dysfunctions in Parkinson\u2019s disease with dementia","volume":"10","author":"MWR Ho","year":"2020","journal-title":"Sci Rep"},{"issue":"2","key":"pone.0262527.ref019","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1017\/S1041610213001725","article-title":"A review of neuroimaging findings of apathy in Alzheimer\u2019s disease","volume":"26","author":"C Theleritis","year":"2014","journal-title":"International Psychogeriatrics"},{"issue":"3","key":"pone.0262527.ref020","doi-asserted-by":"crossref","first-page":"633","DOI":"10.3233\/JAD-131385","article-title":"Neurobiological Correlates of Apathy in Alzheimer\u2019s Disease and Mild Cognitive Impairment: A Critical Review","volume":"39","author":"F Stella","year":"2014","journal-title":"Journal of Alzheimer\u2019s Disease"},{"key":"pone.0262527.ref021","first-page":"82","volume-title":"Manual for the beck depression inventory-II","author":"AT Beck","year":"1996"},{"issue":"4","key":"pone.0262527.ref022","first-page":"709","article-title":"Geriatric depression scale","volume":"24","author":"JA Yesavage","year":"1988","journal-title":"Psychopharmacol Bull"},{"issue":"1-3","key":"pone.0262527.ref023","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1016\/j.jad.2008.06.026","article-title":"The PHQ-8 as a measure of current depression in the general population","volume":"114","author":"K Kroenke","year":"2009","journal-title":"Journal of affective disorders"},{"issue":"3","key":"pone.0262527.ref024","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1101\/lm.048124.118","article-title":"VisMET: a passive, efficient, and sensitive assessment of visuospatial memory in healthy aging, mild cognitive impairment, and Alzheimer\u2019s disease","volume":"26","author":"RU Haque","year":"2019","journal-title":"Learn Mem"},{"key":"pone.0262527.ref025","first-page":"1","article-title":"Deep convolutional neural networks and transfer learning for measuring cognitive impairment using eye-tracking in a distributed tablet-based environment","author":"RU Haque","year":"2020","journal-title":"IEEE Trans Biomed Eng"},{"key":"pone.0262527.ref026","doi-asserted-by":"crossref","unstructured":"Krafka K, Khosla A, Kellnhofer P, Kannan H, Bhandarkar S, Matusik W, et al. Eye tracking for everyone. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2016. p. 2176\u20132184.","DOI":"10.1109\/CVPR.2016.239"},{"issue":"6","key":"pone.0262527.ref027","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","author":"S Ren","year":"2017","journal-title":"IEEE Transactions on Pattern Analysis & Machine Intelligence"},{"key":"pone.0262527.ref028","unstructured":"Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556. 2014."},{"key":"pone.0262527.ref029","doi-asserted-by":"crossref","unstructured":"Yang S, Luo P, Loy CC, Tang X. Wider face: A face detection benchmark. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2016. p. 5525\u20135533.","DOI":"10.1109\/CVPR.2016.596"},{"issue":"1","key":"pone.0262527.ref030","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/TAFFC.2017.2740923","article-title":"Affectnet: A database for facial expression, valence, and arousal computing in the wild","volume":"10","author":"A Mollahosseini","year":"2017","journal-title":"IEEE Transactions on Affective Computing"},{"issue":"8","key":"pone.0262527.ref031","doi-asserted-by":"crossref","first-page":"1377","DOI":"10.1080\/02699930903485076","article-title":"Presentation and validation of the Radboud Faces Database","volume":"24","author":"O Langner","year":"2010","journal-title":"Cognition and emotion"},{"key":"pone.0262527.ref032","unstructured":"Wikipedia contributors. Von Luschan\u2019s chromatic scale\u2014Wikipedia, The Free Encyclopedia; 2020. Available from: https:\/\/en.wikipedia.org\/w\/index.php?title=Von_Luschan%27s_chromatic_scale&oldid=966384324."},{"issue":"6","key":"pone.0262527.ref033","doi-asserted-by":"crossref","first-page":"869","DOI":"10.1001\/archderm.1988.01670060015008","article-title":"The validity and practicality of sun-reactive skin types I through VI","volume":"124","author":"TB Fitzpatrick","year":"1988","journal-title":"Archives of dermatology"},{"key":"pone.0262527.ref034","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"F Pedregosa","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"pone.0262527.ref035","doi-asserted-by":"crossref","unstructured":"Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision; 2017. p. 618\u2013626.","DOI":"10.1109\/ICCV.2017.74"},{"issue":"3","key":"pone.0262527.ref036","doi-asserted-by":"crossref","first-page":"686","DOI":"10.1037\/0022-3514.74.3.686","article-title":"Sex differences in emotion: expression, experience, and physiology","volume":"74","author":"AM Kring","year":"1998","journal-title":"Journal of personality and social psychology"},{"issue":"3","key":"pone.0262527.ref037","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1111\/1467-9450.00193","article-title":"Sex differences in judgement of facial affect: A multivariate analysis of recognition errors","volume":"41","author":"J Thayer","year":"2000","journal-title":"Scandinavian Journal of Psychology"},{"issue":"1","key":"pone.0262527.ref038","doi-asserted-by":"crossref","DOI":"10.1080\/23311908.2017.1376425","article-title":"Mapping spontaneous facial expression in people with Parkinson\u2019s disease: A multiple case study design","volume":"4","author":"SD Gunnery","year":"2017","journal-title":"Cogent Psychology"},{"issue":"2","key":"pone.0262527.ref039","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1080\/026999399379339","article-title":"American-Japanese Cultural Differences in Judgements of Expression Intensity and Subjective Experience","volume":"13","author":"D Matsumoto","year":"1999","journal-title":"Cognition and Emotion"},{"issue":"2","key":"pone.0262527.ref040","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/BF00995188","article-title":"Ethnic differences in affect intensity, emotion judgments, display rule attitudes, and self-reported emotional expression in an American sample","volume":"17","author":"D Matsumoto","year":"1993","journal-title":"Motiv Emot"},{"key":"pone.0262527.ref041","unstructured":"Buolamwini J, Gebru T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In: Friedler SA, Wilson C, editors. Proceedings of the 1st Conference on Fairness, Accountability and Transparency. vol. 81 of Proceedings of Machine Learning Research. New York, NY, USA: PMLR; 2018. p. 77\u201391. Available from: http:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html."}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0262527","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,1,21]],"date-time":"2022-01-21T14:00:35Z","timestamp":1642773635000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0262527"}},"subtitle":[],"editor":[{"given":"Stephen D.","family":"Ginsberg","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,1,21]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1,21]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0262527","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,21]]}}}