{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T11:37:52Z","timestamp":1782301072169,"version":"3.54.5"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1012117","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,10,7]],"date-time":"2024-10-07T00:00:00Z","timestamp":1728259200000}}],"reference-count":53,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2024,9,25]],"date-time":"2024-09-25T00:00:00Z","timestamp":1727222400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003246","name":"Nederlandse Organisatie voor Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["406.21.GO.019"],"award-info":[{"award-number":["406.21.GO.019"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Although word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely capture remains unclear. This study investigates whether large language models (LLMs) can fill this gap. Contextual predictions are implemented via a novel parallel-graded mechanism, where all predicted words at a given position are pre-activated as a function of contextual certainty, which varies dynamically as text processing unfolds. Through reading simulations with OB1-reader, a cognitive model of word recognition and eye-movement control in reading, we compare the model\u2019s fit to eye-movement data when using predictability values derived from a cloze task against those derived from LLMs (GPT-2 and LLaMA). Root Mean Square Error between simulated and human eye movements indicates that LLM predictability provides a better fit than cloze. This is the first study to use LLMs to augment a cognitive model of reading with higher-order language processing while proposing a mechanism on the interplay between word predictability and eye movements.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1012117","type":"journal-article","created":{"date-parts":[[2024,9,25]],"date-time":"2024-09-25T19:27:58Z","timestamp":1727292478000},"page":"e1012117","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":11,"title":["Language models outperform cloze predictability in a cognitive model of reading"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9378-8869","authenticated-orcid":true,"given":"Adrielli Tina","family":"Lopes Rego","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua","family":"Snell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martijn","family":"Meeter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,9,25]]},"reference":[{"issue":"3","key":"pcbi.1012117.ref001","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1037\/0278-7393.31.3.443","article-title":"Anticipating upcoming words in discourse: evidence from ERPs and reading times.","volume":"31","author":"JJA Van Berkum","year":"2005","journal-title":"J Exp Psychol Learn Mem Cogn"},{"issue":"2","key":"pcbi.1012117.ref002","doi-asserted-by":"crossref","first-page":"514","DOI":"10.1037\/a0020990","article-title":"Eye movements and word skipping during reading: effects of word length and predictability.","volume":"37","author":"K Rayner","year":"2011","journal-title":"J Exp Psychol Hum Percept Perform"},{"key":"pcbi.1012117.ref003","article-title":"Prediction during language comprehension: what is next?","author":"R Ryskin","year":"2023","journal-title":"Trends Cogn Sci."},{"key":"pcbi.1012117.ref004","author":"N Smith","year":"2011"},{"key":"pcbi.1012117.ref005","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3758\/PBR.16.1.1","article-title":"Using EZ Reader to model the effects of higher level language processing on eye movements during reading.","volume":"16","author":"ED Reichle","year":"2009","journal-title":"Psychon Bull Rev."},{"issue":"6","key":"pcbi.1012117.ref006","doi-asserted-by":"crossref","first-page":"969","DOI":"10.1037\/rev0000119","article-title":"OB1-reader: A model of word recognition and eye movements in text reading.","volume":"125","author":"J Snell","year":"2018","journal-title":"Psychol Rev"},{"issue":"4","key":"pcbi.1012117.ref007","doi-asserted-by":"crossref","first-page":"777","DOI":"10.1037\/0033-295X.112.4.777","article-title":"SWIFT: a dynamical model of saccade generation during reading.","volume":"112","author":"R Engbert","year":"2005","journal-title":"Psychol Rev"},{"issue":"6","key":"pcbi.1012117.ref008","doi-asserted-by":"crossref","first-page":"1139","DOI":"10.1037\/rev0000248","article-title":"An integrated model of word processing and eye-movement control during Chinese reading.","volume":"127","author":"X Li","year":"2020","journal-title":"Psychol Rev"},{"issue":"1","key":"pcbi.1012117.ref009","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.cogsys.2005.07.006","article-title":"Some empirical tests of an interactive activation model of eye movement control in reading.","volume":"7","author":"RG Reilly","year":"2006","journal-title":"Cogn Syst Res"},{"issue":"3","key":"pcbi.1012117.ref010","doi-asserted-by":"crossref","first-page":"302","DOI":"10.1016\/j.cognition.2013.02.013","article-title":"The effect of word predictability on reading time is logarithmic.","volume":"128","author":"N Smith","year":"2013","journal-title":"Cognition"},{"key":"pcbi.1012117.ref011","doi-asserted-by":"crossref","first-page":"730570","DOI":"10.3389\/frai.2021.730570","article-title":"Language models explain word reading times better than empirical predictability.","volume":"4","author":"MJ Hofmann","year":"2022","journal-title":"Front Artif Intell"},{"key":"pcbi.1012117.ref012","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780195370669.001.0001","volume-title":"Computational models of reading: A handbook","author":"ED Reichle","year":"2021"},{"issue":"1","key":"pcbi.1012117.ref013","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1037\/a0026473","article-title":"Using EZ Reader to simulate eye movements in nonreading tasks: A unified framework for understanding the eye\u2013mind link.","volume":"119","author":"ED Reichle","year":"2012","journal-title":"Psychol Rev."},{"issue":"4","key":"pcbi.1012117.ref014","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1177\/107769905303000401","article-title":"\u201cCloze procedure\u201d: A new tool for measuring readability.","volume":"30","author":"WL Taylor","year":"1953","journal-title":"Journalism quarterly."},{"issue":"3","key":"pcbi.1012117.ref015","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1037\/0033-2909.124.3.372","article-title":"Eye movements in reading and information processing: 20 years of research","volume":"124","author":"K. Rayner","year":"1998","journal-title":"Psychol Bull"},{"issue":"1","key":"pcbi.1012117.ref016","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.cognition.2011.07.012","article-title":"The processing of extraposed structures in English.","volume":"122","author":"R Levy","year":"2012","journal-title":"Cognition"},{"key":"pcbi.1012117.ref017","author":"D Jurafsky","year":"2023","journal-title":"Speech and Language Processing"},{"issue":"6","key":"pcbi.1012117.ref018","doi-asserted-by":"crossref","first-page":"211837","DOI":"10.1098\/rsos.211837","article-title":"Prediction as a basis for skilled reading: insights from modern language models.","volume":"9","author":"B Cevoli","year":"2022","journal-title":"R Soc Open Sci"},{"issue":"2\u20133","key":"pcbi.1012117.ref019","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1080\/00437956.1954.11659520","article-title":"Distributional structure.","volume":"10","author":"ZS Harris","year":"1954","journal-title":"Word"},{"key":"pcbi.1012117.ref020","author":"C Shain","year":"2022","journal-title":"Large-scale evidence for logarithmic effects of word predictability on reading time"},{"key":"pcbi.1012117.ref021","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jml.2015.02.004","article-title":"The influence of cloze probability and item constraint on cloze task response time.","volume":"82","author":"A Staub","year":"2015","journal-title":"J Mem Lang"},{"key":"pcbi.1012117.ref022","article-title":"So cloze yet so far: N400 amplitude is better predicted by distributional information than human predictability judgements.","author":"JA Michaelov","year":"2022","journal-title":"IEEE Trans Cogn Dev Syst"},{"issue":"3","key":"pcbi.1012117.ref023","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1038\/s41593-022-01026-4","article-title":"Shared computational principles for language processing in humans and deep language models","volume":"25","author":"A Goldstein","year":"2022","journal-title":"Nat Neurosci"},{"key":"pcbi.1012117.ref024","first-page":"1","article-title":"The neuroconnectionist research programme.","author":"A Doerig","year":"2023","journal-title":"Nat Rev Neurosci."},{"key":"pcbi.1012117.ref025","doi-asserted-by":"crossref","first-page":"104174","DOI":"10.1016\/j.jml.2020.104174","article-title":"Word predictability effects are linear, not logarithmic: Implications for probabilistic models of sentence comprehension.","volume":"116","author":"T Brothers","year":"2021","journal-title":"J Mem Lang"},{"issue":"1","key":"pcbi.1012117.ref026","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1080\/23273798.2015.1102299","article-title":"What do we mean by prediction in language comprehension?","volume":"31","author":"GR Kuperberg","year":"2016","journal-title":"Lang Cogn Neurosci."},{"issue":"8","key":"pcbi.1012117.ref027","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1111\/lnc3.12151","article-title":"The effect of lexical predictability on eye movements in reading: Critical review and theoretical interpretation.","volume":"9","author":"A. Staub","year":"2015","journal-title":"Lang Linguist Compass"},{"issue":"8","key":"pcbi.1012117.ref028","doi-asserted-by":"crossref","first-page":"902","DOI":"10.1080\/20445911.2017.1340303","article-title":"Predictability effects during reading in the absence of parafoveal preview","volume":"29","author":"AJ Parker","year":"2017","journal-title":"Journal of Cognitive Psychology"},{"key":"pcbi.1012117.ref029","doi-asserted-by":"crossref","first-page":"826","DOI":"10.3758\/s13428-017-0908-4","article-title":"The Provo Corpus: A large eye-tracking corpus with predictability norms.","volume":"50","author":"SG Luke","year":"2018","journal-title":"Behav Res Methods."},{"issue":"8","key":"pcbi.1012117.ref030","first-page":"9","article-title":"Language models are unsupervised multitask learners.","volume":"1","author":"A Radford","year":"2019","journal-title":"OpenAI blog"},{"key":"pcbi.1012117.ref031","volume-title":"Llama: Open and efficient foundation language models","author":"H Touvron","year":"2023"},{"key":"pcbi.1012117.ref032","first-page":"7503","volume-title":"Language Model Quality Correlates with Psychometric Predictive Power in Multiple Languages","author":"EG Wilcox","year":"2023"},{"key":"pcbi.1012117.ref033","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.cogpsych.2016.06.002","article-title":"Limits on lexical prediction during reading.","volume":"88","author":"SG Luke","year":"2016","journal-title":"Cogn Psychol."},{"issue":"5","key":"pcbi.1012117.ref034","doi-asserted-by":"crossref","first-page":"e1012119","DOI":"10.1371\/journal.pcbi.1012119","article-title":"Artificial neural networks for model identification and parameter estimation in computational cognitive models.","volume":"20","author":"M Rmus","year":"2024","journal-title":"PLoS Comput Biol."},{"key":"pcbi.1012117.ref035","first-page":"10","volume-title":"Predictive power of word surprisal for reading times is a linear function of language model quality","author":"A Goodkind","year":"2018"},{"key":"pcbi.1012117.ref036","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1162\/tacl_a_00548","article-title":"Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times?","volume":"11","author":"BD Oh","year":"2023","journal-title":"Trans Assoc Comput Linguist"},{"key":"pcbi.1012117.ref037","first-page":"139","author":"A De Varda","year":"2023"},{"key":"pcbi.1012117.ref038","first-page":"1","article-title":"On logical inference over brains, behaviour, and artificial neural networks.","author":"O Guest","year":"2023","journal-title":"Comput Brain Behav."},{"key":"pcbi.1012117.ref039","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1162\/nol_a_00087","article-title":"Predictive coding or just feature discovery? an alternative account of why language models fit brain data","author":"R Antonello","year":"2023","journal-title":"Neurobiology of Language"},{"issue":"1","key":"pcbi.1012117.ref040","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1038\/4580","article-title":"Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects","volume":"2","author":"RPN Rao","year":"1999","journal-title":"Nat Neurosci"},{"issue":"7","key":"pcbi.1012117.ref041","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1016\/j.tics.2009.04.005","article-title":"The free-energy principle: a rough guide to the brain?","volume":"13","author":"K. Friston","year":"2009","journal-title":"Trends Cogn Sci."},{"issue":"3","key":"pcbi.1012117.ref042","doi-asserted-by":"crossref","first-page":"e13256","DOI":"10.1111\/cogs.13256","article-title":"Large language models demonstrate the potential of statistical learning in language.","volume":"47","author":"P Contreras Kallens","year":"2023","journal-title":"Cogn Sci"},{"key":"pcbi.1012117.ref043","unstructured":"Chomsky N. Knowledge of language: Its nature, origin, and use. New York; 1986."},{"key":"pcbi.1012117.ref044","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.neubiorev.2016.12.023","article-title":"The growth of language: Universal Grammar, experience, and principles of computation.","volume":"81","author":"C Yang","year":"2017","journal-title":"Neurosci Biobehav Rev"},{"issue":"5294","key":"pcbi.1012117.ref045","first-page":"1926","article-title":"Statistical learning by 8-month-old infants","volume":"274","author":"JR Saffran","year":"1996","journal-title":"Science (1979)."},{"key":"pcbi.1012117.ref046","volume-title":"Explain me this: Creativity, competition, and the partial productivity of constructions.","author":"AE Goldberg","year":"2019"},{"issue":"1\u20132","key":"pcbi.1012117.ref047","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1080\/09541440340000213","article-title":"Length, frequency, and predictability effects of words on eye movements in reading","volume":"16","author":"R Kliegl","year":"2004","journal-title":"European journal of cognitive psychology"},{"key":"pcbi.1012117.ref048","doi-asserted-by":"crossref","first-page":"102","DOI":"10.3758\/BF03203644","article-title":"Completion norms collected from younger and older adults for 198 sentence contexts.","volume":"28","author":"MJ Hamberger","year":"1996","journal-title":"Behavior Research Methods, Instruments, & Computers."},{"issue":"3","key":"pcbi.1012117.ref049","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1207\/s1532799xssr1003_3","article-title":"Eye movements as reflections of comprehension processes in reading.","volume":"10","author":"K Rayner","year":"2006","journal-title":"Scientific studies of reading."},{"issue":"1","key":"pcbi.1012117.ref050","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1037\/0033-295X.105.1.125","article-title":"Toward a model of eye movement control in reading.","volume":"105","author":"ED Reichle","year":"1998","journal-title":"Psychol Rev."},{"key":"pcbi.1012117.ref051","volume-title":"Neural machine translation of rare words with subword units","author":"R Sennrich","year":"2015"},{"issue":"6","key":"pcbi.1012117.ref052","doi-asserted-by":"crossref","first-page":"1176","DOI":"10.1080\/17470218.2013.850521","article-title":"SUBTLEX-UK: A new and improved word frequency database for British English.","volume":"67","author":"WJB Van Heuven","year":"2014","journal-title":"Quarterly journal of experimental psychology"},{"key":"pcbi.1012117.ref053","author":"J Grainger","year":"2004","journal-title":"Modeling letter position coding in printed word perception"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1012117","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,10,7]],"date-time":"2024-10-07T00:00:00Z","timestamp":1728259200000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012117","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,7]],"date-time":"2024-10-07T17:40:14Z","timestamp":1728322814000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012117"}},"subtitle":[],"editor":[{"given":"Ronald","family":"van den Berg","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,9,25]]},"references-count":53,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,9,25]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1012117","relation":{"new_version":[{"id-type":"doi","id":"10.1371\/journal.pcbi.1012117","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,25]]}}}