{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T18:56:03Z","timestamp":1790794563029,"version":"4.1.0"},"reference-count":74,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Temporal relation extraction (TRE) in clinical narratives is crucial for understanding patient history, disease progression, and treatment pathways. However, it remains challenging due to limited annotated data and to the complexity of clinical text, including domain-specific terminology, inconsistent information, and implicit temporal reasoning. These challenges are particularly acute for rare diseases, where information such as the date of diagnosis or the onset of key phenotypes is rarely available in structured data, despite being essential for estimating diagnostic delay and reconstructing the natural history of the disease.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>We explore the potential of open-weight and on-premises large language models (LLMs) on clinical TRE and normalization using zero- and few-shot prompting, aiming to reduce the time-consuming and costly annotation process. Based on recent promising capabilities of LLMs in understanding and reasoning over text, we evaluate whether these models can efficiently perform TRE and normalization in real-world clinical settings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We cast the temporal relation task as a question-answering task, in which the models extract the temporal expressions associated with a given clinical event. We propose a prompt-chaining strategy that sequentially performs normalization on the extracted temporal expressions. Our LLM-based approach is evaluated on constructed and annotated French real-world clinical narratives, covering temporal relations involving 2 types of clinical events: phenotypes and rare disease diagnoses. Four open-weight LLMs were evaluated across multiple prompt configurations and compared with rule-based and neural baselines using exact-match metrics, an LLM-as-a-judge evaluation, and human validation. We further evaluate our approach by applying it to the English 2012 i2b2 corpus, involving other types of clinical events.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      The proposed approach achieves stable extraction performance with only a small number of in-context examples across both clinical event types, reaching a maximum\n                      <jats:italic>F<\/jats:italic>\n                      <jats:sub>1<\/jats:sub>\n                      -score of 0.72 for rare disease diagnoses and 0.61 for phenotype events. LLM-as-a-judge evaluation supports these findings by capturing minor temporal variations penalized by exact-match metrics, while human validation further confirms the clinical validity of the extracted relations. On the 2012 i2b2 corpus, the approach remains robust despite increased complexity and minimal supervision. Adding normalization using the prompt-chaining strategy maintains stable overall performance, indicating minimal error propagation and efficient temporal normalization.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Prompting LLMs with a minimal set of in-context examples can achieve strong performance for TRE and normalization in low-resource settings with limited domain-specific annotations. Relying on open-weight, on-premises models, the proposed approach therefore offers a practical path toward deployment in real-world clinical workflows and could further be extended to extract key clinical information, such as age at diagnosis or age at onset, particularly in the context of rare diseases, where it could help prefill structured reporting forms, reducing the manual effort needed to retrieve such information directly from clinical narratives.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/95198","type":"journal-article","created":{"date-parts":[[2026,8,10]],"date-time":"2026-08-10T18:30:08Z","timestamp":1786386608000},"page":"e95198-e95198","source":"Crossref","is-referenced-by-count":0,"title":["Leveraging Generative Large Language Models for Temporal Relation Extraction From French Clinical Narratives: Prompt-Chaining Approach"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2348-6691","authenticated-orcid":false,"given":"Nesrine","family":"Bannour","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9590-0906","authenticated-orcid":false,"given":"Guillaume","family":"Assi\u00e9","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8001-8539","authenticated-orcid":false,"given":"Anne Sophie","family":"Jannot","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3957-1359","authenticated-orcid":false,"given":"Olivia","family":"Boyer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3326-2811","authenticated-orcid":false,"given":"Nicolas","family":"Garcelon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2452-8868","authenticated-orcid":false,"given":"Xavier","family":"Tannier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6921-8161","authenticated-orcid":false,"given":"Marc","family":"Vincent","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,30]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-017-0537-y","article-title":"A novel data-driven workflow combining literature and electronic health records to estimate comorbidities burden for a specific disease: a case study on autoimmune comorbidities in patients with celiac disease","volume":"17","author":"Escudi\u00e9","journal-title":"BMC Med Inform Decis Mak"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2020.103526","article-title":"Clinical concept extraction: a methodology review","volume":"109","author":"Fu","journal-title":"J Biomed Inform"},{"issue":"3","key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/17934","article-title":"Hybrid deep learning for medication-related information extraction from clinical texts in French: MedExt algorithm development study","volume":"9","author":"Jouffroy","journal-title":"JMIR Med Inform"},{"issue":"5","key":"R4","doi-asserted-by":"publisher","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","article-title":"Evaluating temporal relations in clinical text: 2012 i2b2 Challenge","volume":"20","author":"Sun","journal-title":"J Am Med Inform Assoc"},{"key":"R5","doi-asserted-by":"crossref","unstructured":"Bethard S Derczynski L Savova G Pustejovsky J Verhagen M . SemEval-2015 Task 6: Clinical TempEval. Presented at: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015); Jun 4-5, 2015; Denver, CO, USA. [doi: 10.18653\/v1\/S15-2136]","DOI":"10.18653\/v1\/S15-2136"},{"key":"R6","doi-asserted-by":"crossref","unstructured":"Bethard S Savova G Chen WT Derczynski L Pustejovsky J Verhagen M . Task 12: clinical TempEval. Presented at: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016); Jun 16-17, 2016; San Diego, CA, USA. [doi: 10.18653\/v1\/S16-1165]","DOI":"10.18653\/v1\/S16-1165"},{"key":"R7","doi-asserted-by":"crossref","unstructured":"Bethard S Savova G Palmer M Pustejovsky J . Task 12: clinical TempEval. Presented at: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017); Aug 3-4, 2017; Vancouver, Canada. [doi: 10.18653\/v1\/S17-2093]","DOI":"10.18653\/v1\/S17-2093"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2013.09.007","article-title":"TEMPTING system: a hybrid method of rule and machine learning for temporal relation extraction in patient discharge summaries","volume":"46 Suppl","author":"Chang","journal-title":"J Biomed Inform"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2016.06.006","article-title":"A new algorithmic approach for the extraction of temporal associations from clinical narratives with an application to medical product safety surveillance reports","volume":"62","author":"Wang","journal-title":"J Biomed Inform"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2020.101860","article-title":"Reconstructing the patient\u2019s natural history from electronic health records","volume":"105","author":"Najafabadipour","journal-title":"Artif Intell Med"},{"issue":"2","key":"R11","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1093\/jamia\/ocv113","article-title":"Multilayered temporal modeling for the clinical domain","volume":"23","author":"Lin","journal-title":"J Am Med Inform Assoc"},{"key":"R12","doi-asserted-by":"crossref","unstructured":"Tourille J Ferret O N\u00e9v\u00e9ol A Tannier X . LIMSI-COT at SemEval-2016 Task 12: temporal relation identification using a pipeline of classifiers. Presented at: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016); Jun 16-17, 2016; San Diego, CA, USA. [doi: 10.18653\/v1\/S16-1175]","DOI":"10.18653\/v1\/S16-1175"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2019.103219","article-title":"Supervised methods to extract clinical events from cardiology reports in Italian","volume":"95","author":"Viani","journal-title":"J Biomed Inform"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2021.103915","article-title":"Attention-based bidirectional long short-term memory networks for extracting temporal relationships from clinical discharge summaries","volume":"123","author":"Alfattni","journal-title":"J Biomed Inform"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2021.103784","article-title":"Review of temporal reasoning in the clinical domain for timeline extraction: where we are and where we need to be","volume":"118","author":"Olex","journal-title":"J Biomed Inform"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Han X Gao T Yao Y Ye D Liu Z Sun M . OpenNRE: an open and extensible toolkit for neural relation extraction. Presented at: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); Nov 3-7, 2019; Hong Kong, China. [doi: 10.18653\/v1\/D19-3029]","DOI":"10.18653\/v1\/D19-3029"},{"key":"R17","doi-asserted-by":"crossref","unstructured":"Tourille J Ferret O Tannier X N\u00e9v\u00e9ol A . LIMSI-COT at SemEval-2017 Task 12: neural architecture for temporal information extraction from clinical narratives. Presented at: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017); Aug 3-4, 2017; Vancouver, Canada. [doi: 10.18653\/v1\/S17-2098]","DOI":"10.18653\/v1\/S17-2098"},{"issue":"16","key":"R18","doi-asserted-by":"publisher","first-page":"14647","DOI":"10.1609\/aaai.v35i16.17721","article-title":"Clinical temporal relation extraction with probabilistic soft logic regularization and global inference","volume":"35","author":"Zhou","journal-title":"AAAI"},{"key":"R19","doi-asserted-by":"crossref","unstructured":"Dligach D Miller T Lin C Bethard S Savova G . Neural temporal relation extraction. Presented at: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers; Apr 3-7, 2017; Valencia, Spain. [doi: 10.18653\/v1\/E17-2118]","DOI":"10.18653\/v1\/E17-2118"},{"key":"R20","doi-asserted-by":"crossref","unstructured":"Gaizauskas R Harkema H Hepple M Setzer A . Task-oriented extraction of temporal information: the case of clinical narratives. Presented at: Proceedings of the International Workshop on Temporal Representation and Reasoning; Jun 15-17, 2006; Budapest, Hungary. [doi: 10.1109\/TIME.2006.27]","DOI":"10.1109\/TIME.2006.27"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Chikka VR . CDE-IIITH at SemEval-2016 Task 12: extraction of temporal information from clinical documents using machine learning techniques. Presented at: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016); Jun 16-17, 2016; San Diego, CA, USA. [doi: 10.18653\/v1\/S16-1192]","DOI":"10.18653\/v1\/S16-1192"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Li P Huang H . UTA DLNLP at SemEval-2016 Task 12: deep learning based natural language processing system for clinical information identification from clinical notes and pathology reports. Presented at: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016); Jun 16-17, 2016; San Diego, CA, USA. [doi: 10.18653\/v1\/S16-1197]","DOI":"10.18653\/v1\/S16-1197"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Tourille J Ferret O Tannier X Neveol A . Temporal information extraction from clinical text. Presented at: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers; Apr 3-7, 2017; Valencia, Spain. [doi: 10.18653\/v1\/E17-2117]","DOI":"10.18653\/v1\/E17-2117"},{"key":"R24","doi-asserted-by":"crossref","unstructured":"Galvan D Okazaki N Matsuda K Inui K . Investigating the challenges of temporal relation extraction from clinical text. Presented at: Proceedings of the Ninth International Workshop on Health Text Mining and Information Analysis; Oct 31, 2018; Brussels, Belgium. [doi: 10.18653\/v1\/W18-5607]","DOI":"10.18653\/v1\/W18-5607"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Wolf T Debut L Sanh V . Transformers: state-of-the-art natural language processing. Presented at: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing; Nov 16-20, 2020. [doi: 10.18653\/v1\/2020.emnlp-demos.6]","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"R26","doi-asserted-by":"crossref","unstructured":"Lin C Miller T Dligach D Bethard S Savova G . A BERT-based universal model for both within- and cross-sentence clinical temporal relation extraction. Presented at: Proceedings of the 2nd Clinical Natural Language Processing Workshop; Jun 7, 2019; Minneapolis, MN, USA. [doi: 10.18653\/v1\/W19-1908]","DOI":"10.18653\/v1\/W19-1908"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Miller T Bethard S Dligach D Savova G . End-to-end clinical temporal information extraction with multi-head attention. In: Demner-fushman D Ananiadou S Cohen K , editors. Presented at: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks; Jul 13, 2023; Toronto, Canada. [doi: 10.18653\/v1\/2023.bionlp-1.28]","DOI":"10.18653\/v1\/2023.bionlp-1.28"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00172","article-title":"Temporal annotation in the clinical domain","volume":"2","author":"Styler","journal-title":"Trans Assoc Comput Linguist"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Zeng D Liu K Chen Y Zhao J . Distant supervision for relation extraction via piecewise convolutional neural networks. In: M\u00e0rquez L Callison-Burch C Su J , editors. Presented at: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing; Sep 17-21, 2015; Lisbon, Portugal. [doi: 10.18653\/v1\/D15-1203]","DOI":"10.18653\/v1\/D15-1203"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Chaturvedi R Baghershahi P Medya S Di Eugenio B . Temporal relation extraction in clinical texts: a span-based graph transformer approach. In: Che W Nabende J Shutova E Pilehvar MT , editors. Presented at: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); Jul 27 to Aug 1, 2025; Vienna, Austria. [doi: 10.18653\/v1\/2025.acl-long.1251]","DOI":"10.18653\/v1\/2025.acl-long.1251"},{"key":"R31","doi-asserted-by":"crossref","unstructured":"Ning Q Wu H Roth D . A multi-axis annotation scheme for event temporal relations. Presented at: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); Jul 15-20, 2018; Melbourne, Australia. [doi: 10.18653\/v1\/P18-1122]","DOI":"10.18653\/v1\/P18-1122"},{"key":"R32","doi-asserted-by":"crossref","unstructured":"Bannour N Rance B Tannier X Neveol A . Event-independent temporal positioning: application to french clinical text. In: Demner-fushman D Ananiadou S Cohen K , editors. Presented at: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks; Jul 13, 2023; Toronto, Canada. [doi: 10.18653\/v1\/2023.bionlp-1.16]","DOI":"10.18653\/v1\/2023.bionlp-1.16"},{"issue":"6","key":"R33","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooaf121","article-title":"Using large language models for temporal relation extraction from pediatric clinical reports","volume":"8","author":"Andrew","journal-title":"JAMIA Open"},{"issue":"7","key":"R34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3462475","article-title":"Temporal relation extraction in clinical texts","volume":"54","author":"Gumiel","journal-title":"ACM Comput Surv"},{"key":"R35","doi-asserted-by":"crossref","unstructured":"Lewis M Liu Y Goyal N et al. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Jurafsky D Chai J Schluter N Tetreault J , editors. Presented at: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; Jul 5-10, 2020. [doi: 10.18653\/v1\/2020.acl-main.703]","DOI":"10.18653\/v1\/2020.acl-main.703"},{"issue":"1","key":"R36","volume":"21","author":"Raffel","journal-title":"J Mach Learn Res"},{"key":"R37","unstructured":"Radford A Narasimhan K . Improving language understanding by generative pre-training. OpenAI. Preprint posted online on 2018. URL: https:\/\/cdn.openai.com\/research-covers\/language-unsupervised\/language_understanding_paper.pdf [Accessed 13-06-2026]"},{"key":"R38","unstructured":"Touvron H Lavril T Izacard G et al. LLaMA: open and efficient foundation language models. arXiv. Preprint posted online on Feb 27, 2023. [doi: 10.48550\/arXiv.2302.13971]"},{"key":"R39","doi-asserted-by":"crossref","unstructured":"Dligach D Bethard S Miller T Savova G . Exploring text representations for generative temporal relation extraction. In: Naumann T Bethard S Roberts K Rumshisky A , editors. Presented at: Proceedings of the 4th Clinical Natural Language Processing Workshop; Jul 14, 2022; Seattle, WA, USA. [doi: 10.18653\/v1\/2022.clinicalnlp-1.12]","DOI":"10.18653\/v1\/2022.clinicalnlp-1.12"},{"key":"R40","unstructured":"Saiz JJ Altuna B . End-to-end temporal relation extraction in the clinical domain. Presented at: Proceedings of the Text2Story\u201923 Workshop; Apr 2, 2023; Dublin, Ireland. URL: https:\/\/text2story23.inesctec.pt\/ [Accessed 13-09-2026]"},{"key":"R41","doi-asserted-by":"crossref","unstructured":"Huguet Cabot PL Navigli R . REBEL: relation extraction by end-to-end language generation. In: Moens MF Huang X Specia L Tau YS , editors. Presented at: Findings of the Association for Computational Linguistics; Nov 7-11, 2021; Punta Cana, Dominican Republic. [doi: 10.18653\/v1\/2021.findings-emnlp.204]","DOI":"10.18653\/v1\/2021.findings-emnlp.204"},{"key":"R42","doi-asserted-by":"crossref","unstructured":"Yuan C Xie Q Ananiadou S . Zero-shot temporal relation extraction with ChatGPT. In: Demner-fushman D Ananiadou S Cohen K , editors. Presented at: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks; Jul 2023; Toronto, Canada. [doi: 10.18653\/v1\/2023.bionlp-1.7]","DOI":"10.18653\/v1\/2023.bionlp-1.7"},{"key":"R43","unstructured":"Tang R Han X Jiang X Hu X . Does synthetic data generation of LLMs help clinical text mining. arXiv. Preprint posted online on Apr 10, 2023. [doi: 10.48550\/arXiv.2303.04360]"},{"key":"R44","doi-asserted-by":"crossref","unstructured":"Su X Howard P Bethard S . Transformer-based temporal information extraction and application: a review. Presented at: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing; Nov 4-9, 2025; Suzhou, China. [doi: 10.18653\/v1\/2025.emnlp-main.1467]","DOI":"10.18653\/v1\/2025.emnlp-main.1467"},{"issue":"8081","key":"R45","doi-asserted-by":"publisher","first-page":"633","DOI":"10.1038\/s41586-025-09422-z","article-title":"DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning","volume":"645","author":"Guo","journal-title":"Nature"},{"key":"R46","unstructured":"Yang A Li A Yang B et al. Qwen3 technical report. arXiv. Preprint posted online on May 14, 2025. [doi: 10.48550\/arXiv.2505.09388]"},{"key":"R47","doi-asserted-by":"crossref","unstructured":"Kougia V Sedova A Stephan AJ Zaporojets K Roth B . Analysing zero-shot temporal relation extraction on clinical notes using temporal consistency. Presented at: Proceedings of the 23rd Workshop on Biomedical Natural Language Processing; Aug 16, 2024; Bangkok, Thailand. [doi: 10.18653\/v1\/2024.bionlp-1.6]","DOI":"10.18653\/v1\/2024.bionlp-1.6"},{"key":"R48","doi-asserted-by":"crossref","unstructured":"Bannour N Andrew JJ Vincent M . Team NLPeers at Chemotimelines 2024: evaluation of two timeline extraction methods, can generative LLM do it all or is smaller model fine-tuning still relevant? In: Naumann T Ben Abacha A Bethard S Roberts K Bitterman D , editors. Presented at: Proceedings of the 6th Clinical Natural Language Processing Workshop; Jun 21, 2024; Mexico City, Mexico. [doi: 10.18653\/v1\/2024.clinicalnlp-1.39]","DOI":"10.18653\/v1\/2024.clinicalnlp-1.39"},{"key":"R49","doi-asserted-by":"crossref","unstructured":"Yao J Hochheiser H Yoon WJ Goldner E Savova G . Overview of the 2024 shared task on chemotherapy treatment timeline extraction. Presented at: Proceedings of the 6th Clinical Natural Language Processing Workshop; Jun 21, 2024; Mexico City, Mexico. [doi: 10.18653\/v1\/2024.clinicalnlp-1.53]","DOI":"10.18653\/v1\/2024.clinicalnlp-1.53"},{"key":"R50","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2024.104630","article-title":"Model tuning or prompt tuning? A study of large language models for clinical concept and relation extraction","volume":"153","author":"Peng","journal-title":"J Biomed Inform"},{"issue":"9","key":"R51","doi-asserted-by":"publisher","first-page":"1892","DOI":"10.1093\/jamia\/ocae078","article-title":"Generative large language models are all-purpose text analytics engines: text-to-text learning is all your need","volume":"31","author":"Peng","journal-title":"J Am Med Inform Assoc"},{"key":"R52","doi-asserted-by":"crossref","unstructured":"Yeh HS Lavergne T Zweigenbaum P . Decorate the examples: a simple method of prompt design for biomedical relation extraction. In: Calzolari N B\u00e9chet F Blache P , editors. Presented at: Thirteenth Language Resources and Evaluation Conference; Marseille, France. [doi: 10.63317\/2x38wgwt3eai]","DOI":"10.63317\/2x38wgwt3eai"},{"key":"R53","unstructured":"He J Rasmy L Li H et al. Prompting large language models for clinical temporal relation extraction. arXiv. Preprint posted online on Dec 4, 2024. [doi: 10.48550\/arXiv.2412.04512]"},{"key":"R54","doi-asserted-by":"crossref","unstructured":"Haddadan S Le TD Duong T Thieu T . LAILab at Chemotimelines 2024: finetuning sequence-to-sequence language models for temporal relation extraction towards cancer patient undergoing chemotherapy treatment. Presented at: Proceedings of the 6th Clinical Natural Language Processing Workshop; Jun 21, 2024; Mexico City, Mexico. [doi: 10.18653\/v1\/2024.clinicalnlp-1.37]","DOI":"10.18653\/v1\/2024.clinicalnlp-1.37"},{"issue":"70","key":"R55","first-page":"1","volume":"25","author":"Chung","journal-title":"J Mach Learn Res"},{"key":"R56","unstructured":"Yao J Hochheiser H Yoon W Goldner ET Savova GK . Overview of the 2025 shared task on chemotherapy treatment timeline extraction. Presented at: Proceedings of the 7th Clinical Natural Language Processing Workshop; Oct 30, 2025. URL: https:\/\/aclanthology.org\/2025.clinicalnlp-1.1\/ [Accessed 02-02-2026]"},{"key":"R57","unstructured":"Zhao Z Vydiswaran VGV . Team NLP4Health at ChemoTimelines 2025: finetuning large language models for temporal relation extractions from clinical notes. Presented at: Proceedings of the 7th Clinical Natural Language Processing Workshop; Oct 30, 2025. URL: https:\/\/aclanthology.org\/2025.clinicalnlp-1.4\/ [Accessed 02-02-2026]"},{"key":"R58","doi-asserted-by":"publisher","DOI":"10.2196\/60501","article-title":"Prompt engineering paradigms for medical applications: scoping review","volume":"26","author":"Zaghir","journal-title":"J Med Internet Res"},{"key":"R59","unstructured":"Brown T Mann B Ryder N et al. Language models are few-shot learners. In: Larochelle H Ranzato M Hadsell R Balcan MF Lin H , editors. Presented at: Advances in Neural Information Processing Systems; Dec 6-12, 2020; Vancouver, Canada. URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf [Accessed 13-09-2026]"},{"key":"R60","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2018.02.019","article-title":"A clinician friendly data warehouse oriented toward narrative reports: Dr. Warehouse","volume":"80","author":"Garcelon","journal-title":"J Biomed Inform"},{"key":"R61","unstructured":"Orphanet nomenclature pack. ORPHAcodes. URL: https:\/\/www.orphacode.org\/pack-nomenclature\/ [Accessed 02-02-2026]"},{"key":"R62","unstructured":"French language translation. Human Phenotype Ontology Internationalisation Effort. URL: https:\/\/obophenotype.github.io\/hpo-translations\/translations\/fr\/ [Accessed 02-02-2026]"},{"key":"R63","unstructured":"UMLS\u00ae 2023AA Release Available. National Library of Medicine. 2023. URL: https:\/\/www.nlm.nih.gov\/pubs\/techbull\/mj23\/mj23_umls_2023aa_release.html [Accessed 02-02-2026]"},{"key":"R64","unstructured":"Wajsburt P Petit-Jean T Dura B Cohen A Jean C Bey R . EDS-NLP: efficient information extraction from French clinical notes. Zenodo. URL: https:\/\/zenodo.org\/records\/17273431 [Accessed 13-09-2026]"},{"key":"R65","unstructured":"Dates. EDS-NLP. URL: https:\/\/aphp.github.io\/edsnlp\/latest\/pipes\/misc\/dates\/ [Accessed 03-02-2026]"},{"key":"R66","unstructured":"Stenetorp P Pyysalo S Topi\u0107 G Ohta T Ananiadou S Tsujii J . BRAT: a web-based tool for NLP-assisted text annotation. Presented at: Proceedings of the Demonstrations Session at EACL 2012; Apr 23-27, 2012; Avignon, France. [doi: 10.5555\/2380921.2380942]"},{"key":"R67","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2013.07.004","article-title":"Annotating temporal information in clinical narratives","volume":"46","author":"Sun","journal-title":"J Biomed Inform"},{"key":"R68","unstructured":"aphp\/edsnlp at rule_based_relation. GitHub. URL: https:\/\/github.com\/aphp\/edsnlp\/tree\/rule_based_relation [Accessed 03-02-2026]"},{"key":"R69","doi-asserted-by":"crossref","unstructured":"Dekking FM Kraaikamp C Lopuha\u00e4 HP Meester LE . A Modern Introduction to Probability and Statistics. Springer; 2005. [doi: 10.1007\/1-84628-168-7]","DOI":"10.1007\/1-84628-168-7"},{"key":"R70","doi-asserted-by":"crossref","unstructured":"Fan Y Liu Y Yao Z Yu J Hou L Li J . Evaluating generative language models in information extraction as subjective question correction. In: Calzolari N Kan MY Hoste V Lenci A Sakti S Xue N , editors. Presented at: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024); May 20-25, 2024; Turin, Italy. [doi: 10.63317\/2rbzu6ev8x4m]","DOI":"10.63317\/2rbzu6ev8x4m"},{"issue":"3","key":"R71","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3641289","article-title":"A survey on evaluation of large language models","volume":"15","author":"Chang","journal-title":"ACM Trans Intell Syst Technol"},{"key":"R72","unstructured":"gpt-oss:20b. URL: https:\/\/ollama.com\/library\/gpt-oss:20b [Accessed 03-02-2026]"},{"issue":"12","key":"R73","doi-asserted-by":"publisher","DOI":"10.1002\/advs.202100707","article-title":"Green Algorithms: quantifying the carbon footprint of computation","volume":"8","author":"Lannelongue","journal-title":"Adv Sci (Weinh)"},{"key":"R74","doi-asserted-by":"crossref","unstructured":"Laskar MTR Jahan I Dolatabadi E Peng C Hoque E Huang J . Improving automatic evaluation of large language models (LLMs) in biomedical relation extraction via LLMs-as-the-Judge. In: Che W Nabende J Shutova E Pilehvar MT , editors. Presented at: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1); Jul 27 to Aug 1, 2025; Vienna, Austria. [doi: 10.18653\/v1\/2025.acl-long.1238]","DOI":"10.18653\/v1\/2025.acl-long.1238"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T18:13:03Z","timestamp":1790791983000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e95198"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,30]]},"references-count":74,"URL":"https:\/\/doi.org\/10.2196\/95198","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,30]]},"article-number":"v28i1e95198"}}