{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T15:27:31Z","timestamp":1759937251219},"reference-count":70,"publisher":"MIT Press - Journals","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2015,12]]},"abstract":"<jats:p>Among the more recent applications for natural language processing algorithms has been the analysis of spoken language data for diagnostic and remedial purposes, fueled by the demand for simple, objective, and unobtrusive screening tools for neurological disorders such as dementia. The automated analysis of narrative retellings in particular shows potential as a component of such a screening tool since the ability to produce accurate and meaningful narratives is noticeably impaired in individuals with dementia and its frequent precursor, mild cognitive impairment, as well as other neurodegenerative and neurodevelopmental disorders. In this article, we present a method for extracting narrative recall scores automatically and highly accurately from a word-level alignment between a retelling and the source narrative. We propose improvements to existing machine translation\u2013based systems for word alignment, including a novel method of word alignment relying on random walks on a graph that achieves alignment accuracy superior to that of standard expectation maximization\u2013based techniques for word alignment in a fraction of the time required for expectation maximization. In addition, the narrative recall score features extracted from these high-quality word alignments yield diagnostic classification accuracy comparable to that achieved using manually assigned scores and significantly higher than that achieved with summary-level text similarity metrics used in other areas of NLP. These methods can be trivially adapted to spontaneous language samples elicited with non-linguistic stimuli, thereby demonstrating the flexibility and generalizability of these methods.<\/jats:p>","DOI":"10.1162\/coli_a_00232","type":"journal-article","created":{"date-parts":[[2015,12,10]],"date-time":"2015-12-10T19:06:29Z","timestamp":1449774389000},"page":"549-578","source":"Crossref","is-referenced-by-count":12,"title":["Graph-Based Word Alignment for Clinical Language Evaluation"],"prefix":"10.1162","volume":"41","author":[{"given":"Emily","family":"Prud'hommeaux","sequence":"first","affiliation":[{"name":"Rochester Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brian","family":"Roark","sequence":"additional","affiliation":[{"name":"Google, Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csda.2010.11.018"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1034\/j.1600-0447.2003.00081.x"},{"key":"R3","doi-asserted-by":"crossref","unstructured":"Ayan, Necip Fazil and Bonnie J. Dorr. 2006. Going beyond AER: An extensive analysis of word alignments and their impact on MT. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, pages 9\u201316, Sydney.","DOI":"10.3115\/1220175.1220177"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1212\/01.wnl.0000219668.47116.e6"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1348\/026151004X20685"},{"key":"R6","unstructured":"Brown, Peter, Vincent Della Pietra, Steven Della Pietra, and Robert Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263\u2013311."},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1212\/01.wnl.0000249117.23318.e1"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1145\/1961189.1961199"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1044\/1058-0360.0404.124"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1080\/02687039408248648"},{"key":"R11","unstructured":"Cortes, Corinna, Mehryar Mohri, and Ashish Rastogi. 2007. An alternative ranking problem for search engines. In Proceedings of the 6th Workshop on Experimental Algorithms, volume 4525 of Lecture Notes in Computer Science, pages 1\u201321."},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1037\/a0018107"},{"key":"R13","unstructured":"de la Rosa, Gabriela Ramirez, Thamar Solorio, Manuel Montes y Gomez, Aquiles Iglesias, Yang Liu, Lisa Bedore, and Elizabeth Pena. 2013. Exploring word class n-grams to measure language development in children. In Workshop on Biomedical Natural Language Processing (BIONLP 2013), pages 89\u201397, Sofia."},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1007\/s10802-005-9003-x"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1076\/jcen.24.1.26.965"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1016\/0021-9924(95)00053-4"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1523"},{"key":"R18","doi-asserted-by":"crossref","unstructured":"Fan, Jerome, Suneel Upadhye, and Andrew Worster. 2006. Understanding receiver operating characteristic (ROC) curves. Canadian Journal of Emergency Medicine, 8:19\u201320.","DOI":"10.1017\/S1481803500013336"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.1002\/sim.1228"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1016\/0022-3956(75)90026-6"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2007.33.3.293"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1016\/j.cortex.2012.12.006"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Gabani, Keyur, Melissa Sherman, Thamar Solorio, and Yang Liu. 2009. A corpus-based approach for the prediction of language impairment in monolingual English and Spanish-English bilingual children. In Proceedings of the Human Language Technology Conference of the NAACL, pages 46\u201355, Boulder, CO.","DOI":"10.3115\/1620754.1620762"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1093\/brain\/awq204"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1080\/02687039608248419"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1145\/1656274.1656278"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1148\/radiology.143.1.7063747"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1016\/0093-934X(85)90124-5"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.0b013e3181c34b47"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1037\/0894-4105.17.1.82"},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.1080\/13803390500409617"},{"key":"R34","doi-asserted-by":"crossref","unstructured":"Lehr, Maider, Emily Prud'hommeaux, Izhak Shafran, and Brian Roark. 2012. Fully automated neuropsychological assessment for detecting mild cognitive impairment. In Proceedings of the 13th Annual Conference of the International Speech Communication Association (Interspeech), pages 1039\u20131042, Portland, OR.","DOI":"10.21437\/Interspeech.2012-306"},{"key":"R35","unstructured":"Lehr, Maider, Izhak Shafran, Emily Prud'hommeaux, and Brian Roark. 2013. Discriminative joint modeling of lexical variation and acoustic confusion for automated narrative retelling assessment. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 211\u2013220, Atlanta, GA."},{"key":"R36","doi-asserted-by":"crossref","unstructured":"Liang, Percy, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In Proceedings of the Human Language Technology Conference of the NAACL, pages 104\u2013111, New York, NY.","DOI":"10.3115\/1220835.1220849"},{"key":"R37","unstructured":"Lin, Chin-Yiu. 2004. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out, pages 74\u201381, Barcelona."},{"key":"R38","unstructured":"Lopez, Adam and Philip Resnik. 2006. Word-based alignment, phrase-based translation: What's the link. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas, pages 90\u201399, Cambridge."},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1176\/appi.psychotherapy.2003.57.2.153"},{"key":"R41","doi-asserted-by":"publisher","DOI":"10.1002\/ana.21326"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1037\/0022-006X.55.6.914"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.43.11.2412-a"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.1001\/archneur.58.3.397"},{"key":"R45","doi-asserted-by":"publisher","DOI":"10.1080\/136820310000108133"},{"key":"R46","doi-asserted-by":"publisher","DOI":"10.1136\/jnnp.2004.050385"},{"key":"R47","doi-asserted-by":"publisher","DOI":"10.1162\/089120103321337421"},{"key":"R48","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2008.06.004"},{"key":"R51","doi-asserted-by":"crossref","unstructured":"Papineni, Kishore, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311\u2013318, Philadelphia, PA.","DOI":"10.3115\/1073083.1073135"},{"key":"R52","doi-asserted-by":"publisher","DOI":"10.1001\/archneur.56.3.303"},{"key":"R53","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMcp0910237"},{"key":"R54","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-148-6-200803180-00005"},{"key":"R55","doi-asserted-by":"publisher","DOI":"10.1016\/j.neurobiolaging.2009.04.002"},{"key":"R57","doi-asserted-by":"crossref","unstructured":"Prud'hommeaux, Emily and Brian Roark. 2011. Alignment of spoken narratives for automated neuropsychological assessment. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 484\u2013489, Kona, HI.","DOI":"10.1109\/ASRU.2011.6163979"},{"key":"R58","unstructured":"Prud'hommeaux, Emily and Brian Roark. 2012. Graph-based alignment of narratives for automated neuropsychological assessment. In Proceedings of the NAACL 2012 Workshop on Biomedical Natural Language Processing (BioNLP), pages 1\u201310, Montreal."},{"key":"R59","unstructured":"Prud'hommeaux, Emily and Masoud Rouhizadeh. 2012. Automatic detection of pragmatic deficits in children with autism. In Proceedings of the 3rd Workshop on Child, Computer and Interaction, pages 1\u20136, Portland, OR."},{"key":"R60","unstructured":"Prud'hommeaux, Emily Tucker. 2012. Alignment of Narrative Retellings for Automated Neuropsychological Assessment. Ph.D. thesis, Oregon Health and Science University."},{"key":"R61","doi-asserted-by":"publisher","DOI":"10.1007\/BF03324625"},{"key":"R62","unstructured":"Ridgway, James, Pierre Alquier, Nicolas Chopin, and Feng Liang. 2014. PAC-Bayesian AUC classification and scoring. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, 27:658\u2013666."},{"key":"R63","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.56.1.37"},{"key":"R64","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(99)06155-3"},{"key":"R65","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2011.2112351"},{"key":"R66","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.55.3.370"},{"key":"R67","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0501157102"},{"key":"R68","doi-asserted-by":"crossref","unstructured":"Solorio, Thamar and Yang Liu. 2008. Using language models to identify language impairment in Spanish-English bilingual children. In Proceedings of the ACL 2008 Workshop on Biomedical Natural Language Processing (BioNLP), pages 116\u2013117, Columbus, OH.","DOI":"10.3115\/1572306.1572337"},{"key":"R69","doi-asserted-by":"publisher","DOI":"10.1001\/archneur.1989.00520400037017"},{"key":"R70","doi-asserted-by":"publisher","DOI":"10.1111\/j.2044-835X.1995.tb00663.x"},{"key":"R71","doi-asserted-by":"publisher","DOI":"10.1007\/BF00910492"},{"key":"R72","doi-asserted-by":"publisher","DOI":"10.1212\/01.WNL.0000163773.21794.0B"},{"key":"R74","unstructured":"United Nations. 2002. World Population Ageing 1950\u20132050. United Nations, New York."},{"key":"R75","doi-asserted-by":"publisher","DOI":"10.1097\/00002093-200004000-00005"},{"key":"R76","doi-asserted-by":"publisher","DOI":"10.1016\/S0006-8993(01)03200-0"},{"key":"R78","doi-asserted-by":"crossref","unstructured":"Zweig, Mark H. and Gregory Campbell. 1993. Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39:561\u2013577.","DOI":"10.1093\/clinchem\/39.4.561"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00232","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,28]],"date-time":"2022-05-28T22:37:23Z","timestamp":1653777443000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/41\/4\/549-578\/1514"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,12]]},"references-count":70,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2015,12]]}},"alternative-id":["10.1162\/COLI_a_00232"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00232","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,12]]}}}