{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T17:19:24Z","timestamp":1787764764625,"version":"build-2784847793"},"reference-count":100,"publisher":"National Academy of Sciences","issue":"35","license":[{"start":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T00:00:00Z","timestamp":1787702400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2026,9,1]]},"abstract":"<jats:p>\n                    Language is a fundamental human capacity. Large language models (LLMs) have presented the first viable model of language outside of humans, yet how these models learn and use language differs significantly from humans. Here, we compare LLMs and humans predicting language with varying levels of sensory information\u2014from disembodied written text to audiovisual videos of speakers\u2014to demonstrate that, in both humans and LLMs, sensory context is critical for optimal performance. We asked human participants to predict upcoming words within narratives presented as either audiovisual, audio-only, or written language (\n                    <jats:inline-formula>\n                      <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"inline\" overflow=\"scroll\">\n                        <mml:mrow>\n                          <mml:mi>N<\/mml:mi>\n                          <mml:mo>=<\/mml:mo>\n                          <mml:mn>1<\/mml:mn>\n                          <mml:mo>,<\/mml:mo>\n                          <mml:mn>500<\/mml:mn>\n                        <\/mml:mrow>\n                      <\/mml:math>\n                    <\/jats:inline-formula>\n                    total;\n                    <jats:inline-formula>\n                      <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"inline\" overflow=\"scroll\">\n                        <mml:mn>500<\/mml:mn>\n                      <\/mml:math>\n                    <\/jats:inline-formula>\n                    per modality), focusing on content words that convey essential meaning. We compared these predictions across modalities as well as to predictions generated by LLMs. Human predictions were overall more accurate than LLMs, regardless of the sensory modality of presentation. Compared to written language, both audiovisual and audio-only language increased the accuracy and consensus of human predictions and decreased alignment with LLMs. We identified that prosody, a signal that conveys important linguistic information via the auditory channel, partially drove both the observed advantage in accuracy within humans and divergence from LLMs. Integrating multimodal information\u2014i.e., prosody, auditory, or audiovisual information\u2014with the representation of language learned during LLM training improved models\u2019 next-word prediction performance and increased the efficiency of language learning. These findings demonstrate that sensory contexts are foundational to human-like language behavior, and that these contexts can enrich and accelerate language acquisition within LLMs similar to what is observed within human development.\n                  <\/jats:p>","DOI":"10.1073\/pnas.2600317123","type":"journal-article","created":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T16:33:41Z","timestamp":1787762021000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":0,"title":["Sensory context improves language prediction in humans and LLMs"],"prefix":"10.1073","volume":"123","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6615-2224","authenticated-orcid":false,"given":"Thomas L.","family":"Botch","sequence":"first","affiliation":[{"id":[{"id":"https:\/\/ror.org\/049s0rh22","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Psychological and Brain Sciences, Dartmouth College","place":["Hanover"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8591-3068","authenticated-orcid":false,"given":"Emily S.","family":"Finn","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/049s0rh22","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Psychological and Brain Sciences, Dartmouth College","place":["Hanover"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2026,8,26]]},"reference":[{"key":"e_1_3_4_1_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511817779"},{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.298.5598.1569"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/nrn1533"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.psych.59.103006.093659"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1095455"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2009.09.005"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.7146899"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0010-0277(99)00036-0"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1907367117"},{"key":"e_1_3_4_10_2","unstructured":"T. B. Brown et al. Language models are few-shot learners. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2005.14165 (Accessed 14 September 2022)."},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-linguistics-032020-051035"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2024.01.011"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-024-01882-z"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2405460121"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-023-01659-w"},{"key":"e_1_3_4_16_2","unstructured":"L. Ruis et al. Procedural knowledge in pretraining drives reasoning in large language models. arXiv [Preprint] (2024). https:\/\/arxiv.org\/abs\/2411.12580 (Accessed 13 December 2024)."},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-024-07522-w"},{"key":"e_1_3_4_18_2","unstructured":"M. Shanahan Talking about large language models. arXiv [Preprint] (2023).https:\/\/arxiv.org\/abs\/2212.03551 (Accessed 11 December 2024)."},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2023.08.007"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X00001515"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/11577.001.0001"},{"key":"e_1_3_4_22_2","doi-asserted-by":"crossref","unstructured":"A. Warstadt S. R. Bowman \u201cWhat artificial neural networks can tell us about human language acquisition\u201d in Algebraic Structures in Natural Language S. Lappin J.-P. Bernardy Eds. (CRC Press 2022) pp. 17\u201360.","DOI":"10.1201\/9781003205388-2"},{"key":"e_1_3_4_23_2","unstructured":"I. Sucholutsky et al. Getting aligned on representational alignment. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2310.13018 (Accessed 27 October 2023)."},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2215907120"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1162\/opmi_a_00160"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2022.0041"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1037\/rev0000297"},{"key":"e_1_3_4_28_2","volume-title":"Modern Language Models Refute Chomsky\u2019s Approach to Language","author":"Piantadosi S. T.","year":"2024","unstructured":"S. T. Piantadosi, Modern Language Models Refute Chomsky\u2019s Approach to Language (Language Science Press, 2024)."},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41593-022-01026-4"},{"key":"e_1_3_4_30_2","unstructured":"A. R. Vaidya J. Turek A. G. Huth Humans and language models diverge when predicting repeating text. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2310.06408 (Accessed 12 October 2023)."},{"key":"e_1_3_4_31_2","unstructured":"B. Shlegeris F. Roger L. Chan E. McLean Language models are better than humans at next-token prediction. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2212.11281 (Accessed 29 November 2023)."},{"key":"e_1_3_4_32_2","unstructured":"P. Bojanowski E. Grave A. Joulin T. Mikolov Enriching word vectors with subword information. arXiv [Preprint] (2017). https:\/\/arxiv.org\/abs\/1607.04606 (Accessed 1 February 2024)."},{"key":"e_1_3_4_33_2","doi-asserted-by":"crossref","unstructured":"H. H. Clark G. L. Murphy Audience design in meaning and reference. Adv. Psychol. 9 287\u2013299 (1982).","DOI":"10.1016\/S0166-4115(09)60059-5"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1008662107"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1093\/scan\/nsu168"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1093\/scan\/nsae059"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2016.11.001"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jecp.2018.12.011"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0065-2407(02)80041-6"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2018.02.004"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1332441100"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/0163-6383(86)90025-1"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1077066"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1749-818X.2008.00103.x"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2009.09.064"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.bandl.2016.08.002"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0305000914000725"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.1038\/nrn3924"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1038\/nn.4354"},{"key":"e_1_3_4_50_2","unstructured":"J. Kaplan et al. Scaling Laws for Neural Language. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2001.08361 (Accessed 27 March 2026)."},{"key":"e_1_3_4_51_2","unstructured":"A. Aghajanyan et al. Scaling laws for generative mixed-modal language models. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2301.03728 (Accessed 20 August 2025)."},{"key":"e_1_3_4_52_2","unstructured":"T. I. Regev et al. The time scale of redundancy between prosody and linguistic context. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/2503.11630 (Accessed 24 March 2025)."},{"key":"e_1_3_4_53_2","unstructured":"L. Wolf et al. Quantifying the redundancy between prosody and text. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2311.17233 (Accessed 1 December 2023)."},{"key":"e_1_3_4_54_2","unstructured":"M. Huh B. Cheung T. Wang P. Isola The platonic representation hypothesis. arXiv [Preprint] (2024). https:\/\/arxiv.org\/abs\/2405.07987 (Accessed 15 May 2024)."},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.adi1374"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1951.tb01366.x"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1016\/0960-0779(94)90079-5"},{"key":"e_1_3_4_58_2","unstructured":"P. F. Brown S. A. Della Pietra V. J. Della Pietra J. C. Lai R. L. Mercer An estimate of an upper bound for the entropy of English. Comput. Linguist. 18 31\u201340 (1992)."},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.7350657"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0301-0511(96)05215-5"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0022-5371(81)90220-6"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13423-010-0046-9"},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-linguistics-011724-121517"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2307876121"},{"key":"e_1_3_4_65_2","first-page":"415","article-title":"\u201cCloze Procedure\u2019\u2019: A new tool for measuring readability","volume":"30","author":"Taylor W. L.","year":"1953","unstructured":"W. L. Taylor, \u201cCloze Procedure\u2019\u2019: A new tool for measuring readability. Journal. Q. 30, 415\u2013433 (1953).","journal-title":"Journal. Q."},{"key":"e_1_3_4_66_2","unstructured":"T. Ullman Large language models fail on trivial alterations to theory-of-mind tasks. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2302.08399 (Accessed 25 February 2023)."},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X12000477"},{"key":"e_1_3_4_68_2","doi-asserted-by":"crossref","unstructured":"J. E. Kosie M. L. Nencheva J. A. Jung\u00e9 C. Lew-Williams Models of human learning should capture the multimodal complexity and communicative goals of the natural learning environment. Lang. Dev. Res. 5 188\u2013220 (2025).","DOI":"10.31234\/osf.io\/arqxp"},{"key":"e_1_3_4_69_2","doi-asserted-by":"publisher","DOI":"10.1167\/9.3.24"},{"key":"e_1_3_4_70_2","doi-asserted-by":"publisher","DOI":"10.1111\/cogs.12313"},{"key":"e_1_3_4_71_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog0000_29"},{"key":"e_1_3_4_72_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.0956-7976.2004.00694.x"},{"key":"e_1_3_4_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2023.08.003"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2105646118"},{"key":"e_1_3_4_75_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-025-02105-9"},{"key":"e_1_3_4_76_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-04270-z"},{"key":"e_1_3_4_77_2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2020.575566"},{"key":"e_1_3_4_78_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-021-01033-3"},{"key":"e_1_3_4_79_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-023-02437-z"},{"key":"e_1_3_4_80_2","first-page":"341","article-title":"PRAAT, a system for doing phonetics by computer","volume":"5","author":"Boersma P.","year":"2001","unstructured":"P. Boersma, PRAAT, a system for doing phonetics by computer. Glot Int. 5, 341\u2013345 (2001).","journal-title":"Glot Int."},{"key":"e_1_3_4_81_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-023-01732-4"},{"key":"e_1_3_4_82_2","unstructured":"T. Wolf et al. HuggingFace\u2019s transformers: State-of-the-art natural language processing. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/1910.03771 (Accessed 1 February 2024)."},{"key":"e_1_3_4_83_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-024-02096-z"},{"key":"e_1_3_4_84_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1701652114"},{"key":"e_1_3_4_85_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-024-01944-2"},{"key":"e_1_3_4_86_2","doi-asserted-by":"publisher","DOI":"10.1111\/cogs.13343"},{"key":"e_1_3_4_87_2","unstructured":"R. Schaeffer B. Miranda S. Koyejo Are Emergent Abilities of Large Language Models a Mirage? arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2304.15004 (Accessed 1 February 2024)."},{"key":"e_1_3_4_88_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2013.02.013"},{"key":"e_1_3_4_89_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026096204727"},{"key":"e_1_3_4_90_2","unstructured":"I. Nemenman F. Shafee W. Bialek Entropy and inference revisited. arXiv [Preprint] (2002). https:\/\/arxiv.org\/abs\/physics\/0108025 (Accessed 6 April 2026)."},{"key":"e_1_3_4_91_2","unstructured":"A. Radford et al. Robust speech recognition via large-scale weak supervision. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2212.04356 (Accessed 7 February 2025)."},{"key":"e_1_3_4_92_2","unstructured":"T. Afouras J. S. Chung A. Zisserman LRS3-TED: A large-scale dataset for visual speech recognition. arXiv [Preprint] (2018). https:\/\/arxiv.org\/abs\/1809.00496 (Accessed 26 March 2025)."},{"key":"e_1_3_4_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201357"},{"key":"e_1_3_4_94_2","doi-asserted-by":"crossref","unstructured":"J. S. Chung A. Nagrani A. Zisserman VoxCeleb2: Deep speaker recognition. arXiv [Preprint] (2018). https:\/\/arxiv.org\/abs\/1806.05622 (Accessed 26 March 2025).","DOI":"10.21437\/Interspeech.2018-1929"},{"key":"e_1_3_4_95_2","unstructured":"V. Pratap et al. Scaling Speech Technology to 1 000+ Languages. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2305.13516 (Accessed 9 April 2025)."},{"key":"e_1_3_4_96_2","doi-asserted-by":"crossref","unstructured":"M. McAuliffe M. Socolof S. Mihuc M. Wagner M. Sonderegger \u201cMontreal forced aligner: Trainable text-speech alignment using Kaldi\u201d in Interspeech 2017 (ISCA 2017) pp. 498\u2013502.","DOI":"10.21437\/Interspeech.2017-1386"},{"key":"e_1_3_4_97_2","unstructured":"A. Baevski et al. data2vec: A general framework for self-supervised learning in speech. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2202.03555 (Accessed 26 March 2025)."},{"key":"e_1_3_4_98_2","unstructured":"I. Loshchilov F. Hutter Decoupled weight decay regularization. arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1711.05101 (Accessed 18 November 2024)."},{"key":"e_1_3_4_99_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jml.2012.11.001"},{"key":"e_1_3_4_100_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.05351"}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2600317123","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T16:34:11Z","timestamp":1787762051000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/10.1073\/pnas.2600317123"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,26]]},"references-count":100,"journal-issue":{"issue":"35","published-print":{"date-parts":[[2026,9,1]]}},"alternative-id":["10.1073\/pnas.2600317123"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2600317123","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,26]]},"assertion":[{"value":"2026-01-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-29","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2600317123"}}