{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T17:52:14Z","timestamp":1775065934767,"version":"3.50.1"},"reference-count":56,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T00:00:00Z","timestamp":1719792000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>Large language models (LLMs) have demonstrated remarkable generalization and across diverse tasks, leading individuals to increasingly use them as personal assistants due to their emerging reasoning capabilities. Nevertheless, a notable obstacle emerges when including numerical\/temporal data into these prompts, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. This article discusses the challenges of representing and tokenizing temporal data. It argues that naively passing timeseries to LLMs can be ineffective due to the modality gap between numbers and text.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and methods<\/jats:title>\n                  <jats:p>We conduct a case study by tokenizing a sample mobile sensing dataset using the OpenAI tokenizer. We also review recent works that feed timeseries data into LLMs for human-centric tasks, outlining common experimental setups like zero-shot prompting and few-shot learning.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The case study shows that popular LLMs split timestamps and sensor values into multiple nonmeaningful tokens, indicating they struggle with temporal data. We find that preliminary works rely heavily on prompt engineering and timeseries aggregation to \u201cground\u201d LLMs, hinting that the \u201cmodality gap\u201d hampers progress. The literature was critically analyzed through the lens of models optimizing for expressiveness versus parameter efficiency. On one end of the spectrum, training large domain-specific models from scratch is expressive but not parameter-efficient. On the other end, zero-shot prompting of LLMs is parameter-efficient but lacks expressiveness for temporal data.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>We argue tokenizers are not optimized for numerical data, while the scarcity of timeseries examples in training corpora exacerbates difficulties. We advocate balancing model expressiveness and computational efficiency when integrating temporal data. Prompt tuning, model grafting, and improved tokenizers are highlighted as promising directions.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>We underscore that despite promising capabilities, LLMs cannot meaningfully process temporal data unless the input representation is addressed. We argue that this paradigm shift in how we leverage pretrained models will particularly affect the area of biomedical signals, given the lack of modality-specific foundation models.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae090","type":"journal-article","created":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T21:21:04Z","timestamp":1719868864000},"page":"2151-2158","source":"Crossref","is-referenced-by-count":22,"title":["<i>The first step is the hardest<\/i>: pitfalls of representing and tokenizing temporal data for large language models"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9761-951X","authenticated-orcid":false,"given":"Dimitris","family":"Spathis","sequence":"first","affiliation":[{"name":"Nokia Bell Labs , Cambridge, CB3 0FA, United Kingdom"},{"name":"Department of Computer Science and Technology, University of Cambridge , Cambridge, CB3 0FD, United Kingdom"}]},{"given":"Fahim","family":"Kawsar","sequence":"additional","affiliation":[{"name":"Nokia Bell Labs , Cambridge, CB3 0FA, United Kingdom"},{"name":"School of Computing Science, University of Glasgow , Glasgow, G12 8RZ, United Kingdom"}]}],"member":"286","published-online":{"date-parts":[[2024,7,1]]},"reference":[{"key":"2024082207522604900_ocae090-B1","author":"Bommasani","year":"2021"},{"key":"2024082207522604900_ocae090-B2","author":"OpenAI","year":"2023"},{"key":"2024082207522604900_ocae090-B3","author":"Mukherjee","year":"2024"},{"key":"2024082207522604900_ocae090-B4","first-page":"17612","article-title":"Mind the gap: understanding the modality gap in multi-modal contrastive representation learning","volume":"35","author":"Liang","year":"2022","journal-title":"Adv Neural Inform Process Syst"},{"key":"2024082207522604900_ocae090-B5","first-page":"1715","author":"Sennrich","year":"2016"},{"key":"2024082207522604900_ocae090-B6","first-page":"5149","author":"Schuster","year":"2012"},{"key":"2024082207522604900_ocae090-B7","first-page":"66","author":"Kudo"},{"key":"2024082207522604900_ocae090-B8","author":"Touvron","year":"2023"},{"key":"2024082207522604900_ocae090-B9","author":"Millidge","year":"2023"},{"key":"2024082207522604900_ocae090-B10","author":"Nogueira","year":"2021"},{"key":"2024082207522604900_ocae090-B11","doi-asserted-by":"crossref","first-page":"133190","DOI":"10.1109\/ACCESS.2019.2940729","article-title":"Smartphone and smartwatch-based biometrics using activities of daily living","volume":"7","author":"Weiss","year":"2019","journal-title":"IEEE Access"},{"key":"2024082207522604900_ocae090-B12","first-page":"7628","author":"Lu"},{"issue":"1","key":"2024082207522604900_ocae090-B13","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1038\/s41746-023-00840-9","article-title":"A foundational vision transformer improves diagnostic performance for electrocardiograms","volume":"6","author":"Vaid","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2024082207522604900_ocae090-B14","first-page":"1","author":"Louis Gaudilliere","year":"2021"},{"key":"2024082207522604900_ocae090-B15","author":"Gao","year":"2020"},{"key":"2024082207522604900_ocae090-B16","first-page":"4400","author":"Yeh","year":"2023"},{"key":"2024082207522604900_ocae090-B17","author":"Abbaspourazad"},{"key":"2024082207522604900_ocae090-B18","first-page":"3109","author":"Ma","year":"2019"},{"issue":"2","key":"2024082207522604900_ocae090-B19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3328932","article-title":"Multi-task self-supervised learning for human activity detection","volume":"3","author":"Saeed","year":"2019","journal-title":"Proc ACM Interact Mob Wearable Ubiquitous Technol"},{"issue":"3","key":"2024082207522604900_ocae090-B20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3550299","article-title":"Assessing the state of self-supervised human activity recognition using wearables","volume":"6","author":"Haresamudram","year":"2022","journal-title":"Proc ACM Interact Mob Wearable Ubiquitous Technol"},{"issue":"1","key":"2024082207522604900_ocae090-B21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3448112","article-title":"SelfHAR: improving human activity recognition through self-training with unlabeled data","volume":"5","author":"Tang","year":"2021","journal-title":"Proc ACM Interact Mob Wearable Ubiquitous Technol"},{"issue":"2","key":"2024082207522604900_ocae090-B22","doi-asserted-by":"crossref","first-page":"100410","DOI":"10.1016\/j.patter.2021.100410","article-title":"Breaking away from labels: the promise of self-supervised machine learning in intelligent health","volume":"3","author":"Spathis","year":"2022","journal-title":"Patterns"},{"issue":"1","key":"2024082207522604900_ocae090-B23","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1038\/s41746-024-01062-3","article-title":"Self-supervised learning for human activity recognition using 700,000 person-days of wearable data","volume":"7","author":"Yuan","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2024082207522604900_ocae090-B24","first-page":"69","author":"Spathis","year":"2021"},{"key":"2024082207522604900_ocae090-B25","author":"Wei"},{"issue":"9","key":"2024082207522604900_ocae090-B26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3560815","article-title":"Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing","volume":"55","author":"Liu","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2024082207522604900_ocae090-B27","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inform Process Syst"},{"issue":"3","key":"2024082207522604900_ocae090-B28","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1038\/s42256-023-00626-4","article-title":"Parameter-efficient fine-tuning of large-scale pre-trained language models","volume":"5","author":"Ding","year":"2023","journal-title":"Nat Mach Intell"},{"key":"2024082207522604900_ocae090-B29","author":"Liu","year":"2023"},{"key":"2024082207522604900_ocae090-B30","first-page":"1","author":"Chowdhery"},{"key":"2024082207522604900_ocae090-B31","author":"Kim","year":"2024"},{"key":"2024082207522604900_ocae090-B32","first-page":"561","author":"Sooriya Patabandige","year":"2023"},{"key":"2024082207522604900_ocae090-B33","author":"Xue"},{"key":"2024082207522604900_ocae090-B34","author":"Shi","year":"2023"},{"key":"2024082207522604900_ocae090-B35","author":"Jin","year":"2024"},{"key":"2024082207522604900_ocae090-B36","first-page":"3045","author":"Lester"},{"key":"2024082207522604900_ocae090-B37","author":"Hu"},{"key":"2024082207522604900_ocae090-B38","author":"He"},{"key":"2024082207522604900_ocae090-B39","first-page":"2735","author":"Sun"},{"key":"2024082207522604900_ocae090-B40","author":"Belyaeva","year":"2023"},{"key":"2024082207522604900_ocae090-B41","first-page":"15180","author":"Girdhar","year":"2023"},{"key":"2024082207522604900_ocae090-B42","first-page":"13246","author":"Moon"},{"key":"2024082207522604900_ocae090-B43","author":"Moon","year":"2023"},{"key":"2024082207522604900_ocae090-B44","author":"Zhang","year":"2023"},{"key":"2024082207522604900_ocae090-B45","author":"Xu"},{"key":"2024082207522604900_ocae090-B46","author":"Liu"},{"key":"2024082207522604900_ocae090-B47","first-page":"19730","author":"Li"},{"key":"2024082207522604900_ocae090-B48","author":"Corrado","year":"2023"},{"key":"2024082207522604900_ocae090-B49","author":"Liu","year":"2023"},{"key":"2024082207522604900_ocae090-B50","author":"Taylor","year":"2022"},{"key":"2024082207522604900_ocae090-B51","author":"Gruver"},{"key":"2024082207522604900_ocae090-B52","author":"Golkar","year":"2023"},{"key":"2024082207522604900_ocae090-B53","author":"Jin"},{"key":"2024082207522604900_ocae090-B54","author":"Elsayed"},{"key":"2024082207522604900_ocae090-B55","author":"Chang","year":"2023"},{"key":"2024082207522604900_ocae090-B56","first-page":"11763","article-title":"LIFT: language-interfaced fine-tuning for non-language machine learning tasks","volume":"35","author":"Dinh","year":"2022","journal-title":"Adv Neural Inform Process Syst"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2151\/58868315\/ocae090.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2151\/58868315\/ocae090.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T12:44:43Z","timestamp":1724330683000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/2151\/7702405"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,1]]},"references-count":56,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,7,1]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae090","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,7,1]]}}}