{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T21:41:37Z","timestamp":1792705297623,"version":"4.5.0"},"reference-count":43,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Negation in radiology reports (eg, \u201cno evidence of dissection\u201d) is essential for accurately conveying diagnostic information. Clinical natural language processing systems, however, often fail to determine whether findings occur within negated or speculative contexts, leading to excessive false keyword alerts, noise in information extraction, and reduced precision in downstream clinical or research applications. Transformer models have advanced detection capabilities but require costly expert labeling. Large language models (LLMs) may provide label-efficient alternatives through few-shot prompting or knowledge distillation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aims to systematically evaluate LLMs and transformer models under different supervision strategies, including comparisons of expert-labeled and LLM-distilled transformers, teacher LLMs (GPT-5.3 Chat, Gemma 4, and Qwen3.6-Plus), sampling approaches, and direct few-shot LLM inference, to determine whether LLM inference or LLM-based distillation can substitute for human labeling in radiology-report negation detection while reducing annotation costs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We reused and analyzed 6000 deidentified English-language radiology reports from 3 hospital branches. Three LLMs\u2014GPT-5.3 Chat, Gemma 4, and Qwen3.6-Plus\u2014performed 0-, 5-, and 10-shot inference to identify negations and speculations unrelated to abnormal findings and generate pseudo-labels for transformer fine-tuning (Bio_ClinicalBERT [Bio + Clinical BERT model], BERT [Bidirectional Encoder Representations from Transformers], and ALBERT [A Lite Bidirectional Encoder Representations from Transformers]). We trained models using human-labeled or LLM-distilled supervision across 28 training data scales (n=42-4200) and 3 data sampling seeds and compared them across multiple supervision and training conditions encompassing backbone, label source, and sampling strategy. We assessed performance using keyword-level precision, recall, and F1-score with paired bootstrap tests (10,000 replicates).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We refined 1000 report annotations (interrater \u03ba=0.957) and fine-tuned 1260 transformer models. Human-labeled transformers consistently outperformed distillation-labeled ones, with the largest gap at small training scales (\u0394F1-score \u22480.042; adjusted P&lt;.001), which narrowed at larger scales (\u0394F1-score \u22480.022; adjusted P&lt;.001). Teacher model choice had minimal effect on distilled performance. All 3 human-labeled transformer backbones (BERT, ALBERT, and Bio_ClinicalBERT) achieved high keyword-level F1-scores (0.982-0.985 under their best conditions; n=4200). In the 10-shot setting, GPT-5.3 Chat (F1-score 0.974) modestly underperformed human-labeled transformers in the large-sample range (\u0394F1-score=0.009; adjusted P=.047) and significantly outperformed Gemma 4 (F1-score 0.957; adjusted P=.01) and Qwen3.6-Plus (F1-score 0.950; adjusted P&lt;.001) in the bootstrap analysis. GPT-5.3 Chat labeled reports at approximately US $0.018 per report and became less expensive than manual annotation when expert hourly wages exceeded US $0.64-1.50.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Human-supervised transformers remain the strongest baseline, but frontier LLMs such as GPT-5.3 Chat offer comparable performance with minimal supervision. LLM-based distillation offers a cost-effective alternative with only minor performance loss. Dataset scale proved more influential than sampling strategy. Additionally, superior teacher performance yielded no proportional gains through distillation, as open-weight or low-cost teachers achieved comparable results. If supported by external validation, the proposed label-efficient framework could enable accurate and affordable negation detection and rapid domain adaptation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/103912","type":"journal-article","created":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T19:05:06Z","timestamp":1791399906000},"page":"e103912","source":"Crossref","is-referenced-by-count":0,"title":["Reducing Annotation Costs Using Knowledge Distillation for Label\u2011Efficient Negation Detection in Radiology Reports: Multisite Internal Validation Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4207-1512","authenticated-orcid":false,"given":"Kung-Hsun","family":"Weng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-6728-0010","authenticated-orcid":false,"given":"Bo-Lin","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8704-0739","authenticated-orcid":false,"given":"Yu-Ting","family":"Kuo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2630-9773","authenticated-orcid":false,"given":"Yi-Chen","family":"Chou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9076-6584","authenticated-orcid":false,"given":"Tsyh-Jyi","family":"Hsieh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6698-0273","authenticated-orcid":false,"given":"Chung-Feng","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,22]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.2196\/46348"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-009-9250-4"},{"key":"ref3","unstructured":"KhandelwalASawantSNegBERT: a transfer learning approach for negation detection and scope resolution2020The Twelfth Language Resources and Evaluation ConferenceMay 11-16, 2020Marseille, FranceParis, FranceEuropean Language Resources Association57395748"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-024-01274-9"},{"key":"ref5","first-page":"e1","volume":"2024","author":"He, Y","year":"2024","journal-title":"Transact Mach Learn Res"},{"key":"ref6","unstructured":"BrownTMannBRyderNSubbiahMKaplanJDDhariwalPNeelakantanAShyamPSastryGAskellAAgarwalSHerbert-VossAKruegerGHenighanTChildRRameshAZieglerDWuJWinterCHesseCChenMSiglerELitwinMGraySChessBClarkJBernerCMcCandlishSRadfordASutskeverIAmodeiDLanguage models are few-shot learners2020Conference on Neural Information Processing SystemsDec 6-12, 2020Virtual Conference18771901"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.354"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1006\/jbin.2001.1029"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.2001.0080598"},{"key":"ref10","first-page":"187","volume":"136","author":"Gindl, S","year":"2008","journal-title":"Stud Health Technol Inform"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.1999.0060393"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2015.02.010"},{"key":"ref13","first-page":"188","volume":"2017","author":"Peng, Y","year":"2018","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"ref14","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1007\/978-981-10-2053-7_44","author":"Lazib, L","year":"2016","journal-title":"Social Computing"},{"key":"ref15","first-page":"1018","volume":"2018","author":"Taylor, SJ","year":"2018","journal-title":"AMIA Annu Symp Proc"},{"key":"ref16","first-page":"212","volume":"2019","author":"Sergeeva, E","year":"2019","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1017\/s1351324920000509"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.louhi-1.9"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1047"},{"key":"ref20","unstructured":"GautamDMaharjanNBanjadeRTamangLJRusVLong short term memory based models for negation handling in tutorial dialoguesProceedings of the Thirty-First International Florida Artificial Intelligence Research Society Conference (FLAIRS-31)2018052131st International Florida Artificial Intelligence Research Society ConferenceMay 21-23, 2018Melbourne, FL1419"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/d16-1078"},{"key":"ref22","unstructured":"MedlockBBriscoeTWeakly supervised learning for Hedge classification in scientific literatureProceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL 2007)20070645th Annual Meeting of the Association for Computational LinguisticsJune 23-30, 2007Prague, Czech Republic992999"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9061-0"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613805"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.3115\/1596374.1596381"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2015.03.001"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1503.02531"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1017\/s1351324923000542"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1910.01108"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.626"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.130"},{"key":"ref32","unstructured":"GuYZhangSUsuyamaNWoldesenbetYWongCSanapathiPWeiMValluriNStrandbergENaumannTPoonHDistilling large language models for biomedical knowledge extraction: a case study on adverse drug eventsarXiv. Preprint published July 12, 2023https:\/\/arxiv.org\/abs\/2307.06439"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0001275"},{"key":"ref34","unstructured":"\u6cd5\u52d9\u90e8\u6c11\u6cd5\u90e8\u5206\u689d\u6587\u4fee\u6b63\u5c0d\u7167\u8868\u6cd5\u52d9\u90e8\u5168\u7403\u8cc7\u8a0a\u7db220212026-08-09\u81fa\u5317\u5e02\u6cd5\u52d9\u90e8https:\/\/tinyurl.com\/66hztvkm"},{"key":"ref35","unstructured":"KokhanO7 proven ways to increase annotation throughput at scale (freelancers vs. teams)Tinko Group website2026-05-17https:\/\/tinkogroup.com\/annotation-throughput-freelancers-vs-teams-guide\/"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.100"},{"key":"ref37","unstructured":"HonnibalMMontaniIVan LandeghemSBoydAspaCy - Industrial-strength Natural Language Processing in PythonspaCy2026-07-21https:\/\/spacy.io\/"},{"key":"ref38","unstructured":"DuMTiered Super-Moore's law: price evolution, production frontiers, and market competition in large language model inference servicesarXiv. Preprint published March 30, 2026202603202026-05-28https:\/\/arxiv.org\/abs\/2603.28576"},{"key":"ref39","unstructured":"MillerTLaparraEBethardSDomain adaptation in practice: lessons from a real-world information extraction pipelineProceedings of the Second Workshop on Domain Adaptation for NLP (AdaptNLP 2021)202104Second Workshop on Domain Adaptation for NLP (AdaptNLP 2021)April 20, 2021Kyiv, Ukraine105110"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02600-5"},{"key":"ref41","unstructured":"HuEJShenYWallisPAllen-ZhuZLiYWangSWangLChenWLoRA: low-rank adaptation of large language modelsInternational Conference on Learning Representations (ICLR 2022)20220425Tenth International Conference on Learning Representations (ICLR 2022)April 25-29, 2022Virtual Conference"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.wmt-1.43"},{"key":"ref43","unstructured":"Kung-HsunWd8888\/llm_negation_study_2025_publicGitHub2026-08-28https:\/\/github.com\/d8888\/llm_negation_study_2025_public"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e103912","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T21:25:09Z","timestamp":1792704309000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e103912"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,22]]},"references-count":43,"URL":"https:\/\/doi.org\/10.2196\/103912","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.103912","asserted-by":"object"}]},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,22]]}}}