{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:44:38Z","timestamp":1785545078815,"version":"3.56.0"},"reference-count":40,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Ultrasound examinations, while valuable, are time-consuming and often limited in availability. Consequently, many hospitals implement reservation systems; however, these systems typically lack prioritization for examination purposes. Hence, our hospital uses a waitlist system that prioritizes examination requests based on their clinical value when slots become available due to cancellations. This system, however, requires a manual review of examination purposes, which are recorded in free-form text. We hypothesized that artificial intelligence language models could preliminarily estimate the priority of requests before manual reviews.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to investigate potential challenges associated with using language models for estimating the priority of medical examination requests and to evaluate the performance of language models in processing Japanese medical texts.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We retrospectively collected ultrasound examination requests from the waitlist system at Keio University Hospital, spanning January 2020 to March 2023. Each request comprised an examination purpose documented by the requesting physician and a 6-tier priority level assigned by a radiologist during the clinical workflow. We fine-tuned JMedRoBERTa, Luke, OpenCalm, and LLaMA2 under two conditions: (1) tuning only the final layer and (2) tuning all layers using either standard backpropagation or low-rank adaptation.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n We had 2335 and 204 requests in the training and test datasets post cleaning. When only the final layers were tuned, JMedRoBERTa outperformed the other models (Kendall coefficient=0.225). With full fine-tuning, JMedRoBERTa continued to perform best (Kendall coefficient=0.254), though with reduced margins compared with the other models. The radiologist\u2019s retrospective re-evaluation yielded a Kendall coefficient of 0.221.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Language models can estimate the priority of examination requests with accuracy comparable with that of human radiologists. The fine-tuning results indicate that general-purpose language models can be adapted to domain-specific texts (ie, Japanese medical texts) with sufficient fine-tuning. Further research is required to address priority rank ambiguity, expand the dataset across multiple institutions, and explore more recent language models with potentially higher performance or better suitability for this task.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/68020","type":"journal-article","created":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T21:21:36Z","timestamp":1753219296000},"page":"e68020-e68020","source":"Crossref","is-referenced-by-count":1,"title":["Training Language Models for Estimating Priority Levels in Ultrasound Examination Waitlists: Algorithm Development and Validation"],"prefix":"10.2196","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9209-4515","authenticated-orcid":false,"given":"Kanato","family":"Masayoshi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0162-5312","authenticated-orcid":false,"given":"Masahiro","family":"Hashimoto","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9624-8400","authenticated-orcid":false,"given":"Naoki","family":"Toda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-2869-5574","authenticated-orcid":false,"given":"Hirozumi","family":"Mori","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2778-9163","authenticated-orcid":false,"given":"Goh","family":"Kobayashi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4714-2810","authenticated-orcid":false,"given":"Hasnine","family":"Haque","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-8885-4511","authenticated-orcid":false,"given":"Mizuki","family":"So","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8241-6565","authenticated-orcid":false,"given":"Masahiro","family":"Jinzaki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,7,22]]},"reference":[{"key":"R1","unstructured":"Vaswani A Shazeer N Parmar N et al. Attention is all you need. Presented at: Proceedings of the 31st International Conference on Neural Information Processing Systems; Dec 4-9, 2017:6000-6010; Long Beach, CA. URL: https:\/\/dl.acm.org\/doi\/10.5555\/3295222.3295349"},{"key":"R2","unstructured":"Lee J Yoon W Kim S et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. arXiv. Preprint posted online on Oct 18, 2019. URL: http:\/\/arxiv.org\/abs\/1901.08746 [Accessed 14-07-2025]"},{"key":"R3","unstructured":"Huang K Altosaar J Ranganath R . ClinicalBERT: modeling clinical notes and predicting hospital readmission. arXiv. Preprint posted online on Nov 29, 2020. URL: http:\/\/arxiv.org\/abs\/1904.05342 [Accessed 14-07-2025]"},{"key":"R4","unstructured":"Gu Y Tinn R Cheng H et al. Domain-specific language model pretraining for biomedical natural language processing. arXiv. Preprint posted online on Sep 16, 2021. URL: http:\/\/arxiv.org\/abs\/2007.15779 [Accessed 14-07-2025]"},{"key":"R5","doi-asserted-by":"crossref","unstructured":"Peng Y Yan S Lu Z . Transfer learning in biomedical natural language processing: an evaluation of BERT and elmo on ten benchmarking datasets. arXiv. Preprint posted online on Jun 18, 2019. [doi: 10.18653\/v1\/W19-5006]","DOI":"10.18653\/v1\/W19-5006"},{"key":"R6","unstructured":"Sugimoto K Iki T Chida Y Kanazawa T Aizawa A . JMedRoBERTa: a japanese pre-trained language model on academic articles in medical sciences (in japanese). Presented at: Proceedings of the 29th Annual Meeting of the Association for Natural Language Processing 2023; Mar 13-17, 2023; Okinawa, Japan. URL: https:\/\/www.anlp.jp\/proceedings\/annual_meeting\/2023\/pdf_dir\/P3-1.pdf [Accessed 14-07-2025]"},{"key":"R7","unstructured":"Brown TB Mann B Ryder N et al. Language models are few-shot learners. Presented at: Proceedings of the 34th International Conference on Neural Information Processing Systems; Dec 6-12, 2020:1877-1901; Virtual event. URL: https:\/\/dl.acm.org\/doi\/abs\/10.5555\/3495724.3495883 [Accessed 14-07-2025]"},{"key":"R8","unstructured":"Ouyang L Wu J Jiang X et al. Training language models to follow instructions with human feedback. arXiv. Preprint posted online on Mar 4, 2022. URL: http:\/\/arxiv.org\/abs\/2203.02155 [Accessed 14-07-2025]"},{"key":"R9","unstructured":"Zhou H Liu F Gu B et al. A survey of large language models in medicine: progress, application, and challenges. arXiv. Preprint posted online on Jul 22, 2024. URL: http:\/\/arxiv.org\/abs\/2311.05112 [Accessed 14-07-2025]"},{"issue":"7972","key":"R10","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature New Biol"},{"key":"R11","unstructured":"Singhal K Tu T Gottweis J et al. Towards expert-level medical question answering with large language models. arXiv. Preprint posted online on May 16, 2023. URL: http:\/\/arxiv.org\/abs\/2305.09617 [Accessed 14-07-2025]"},{"key":"R12","unstructured":"Touvron H Lavril T Izacard G et al. LLaMA: open and efficient foundation language models. arXiv. Preprint posted online on Feb 27, 2023. URL: http:\/\/arxiv.org\/abs\/2302.13971 [Accessed 14-07-2025]"},{"key":"R13","unstructured":"Touvron H Martin L Stone K et al. Llama 2: open foundation and fine-tuned chat models. arXiv. Preprint posted online on Jul 19, 2023. URL: http:\/\/arxiv.org\/abs\/2307.09288 [Accessed 14-07-2025]"},{"key":"R14","unstructured":"CyberAgent, Inc. cyberagent\/open-calm-7b. Hugging Face; 2023. URL: https:\/\/huggingface.co\/cyberagent\/open-calm-7b [Accessed 25-08-2023]"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/43803","article-title":"Triage and diagnostic accuracy of online symptom checkers: systematic review","volume":"25","author":"Riboli-Sasco","journal-title":"J Med Internet Res"},{"issue":"5","key":"R16","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.8895","article-title":"Use of a large language model to assess clinical acuity of adults in the emergency department","volume":"7","author":"Williams","journal-title":"JAMA Netw Open"},{"issue":"12","key":"R17","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0279953","article-title":"Applications of natural language processing at emergency department triage: a narrative review","volume":"18","author":"Stewart","journal-title":"PLoS ONE"},{"issue":"11","key":"R18","doi-asserted-by":"publisher","DOI":"10.2196\/21252","article-title":"Patient triage by topic modeling of referral letters: feasibility study","volume":"8","author":"Spasic","journal-title":"JMIR Med Inform"},{"issue":"12","key":"R19","doi-asserted-by":"publisher","DOI":"10.2196\/27008","article-title":"A novel deep learning-based system for triage in the emergency department using electronic medical records: retrospective cohort study","volume":"23","author":"Yao","journal-title":"J Med Internet Res"},{"key":"R20","doi-asserted-by":"crossref","unstructured":"Ali M Fromm M Thellmann K et al. Tokenizer choice for LLM training: negligible or crucial? arXiv. Preprint posted online on Mar 17, 2024. URL: http:\/\/arxiv.org\/abs\/2310.08754 [Accessed 14-07-2025]","DOI":"10.18653\/v1\/2024.findings-naacl.247"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Yamada I Asai A Shindo H Takeda H Matsumoto Y . LUKE: deep contextualized entity representations with entity-aware self-attention. Presented at: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP); Nov 16-12, 2020:6442-6454; Online. URL: https:\/\/www.aclweb.org\/anthology\/2020.emnlp-main [doi: 10.18653\/v1\/2020.emnlp-main.523]","DOI":"10.18653\/v1\/2020.emnlp-main.523"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Wolf T Debut L Sanh V et al. HuggingFace\u2019s transformers: state-of-the-art natural language processing. arXiv. Preprint posted online on Jul 14, 2020. [doi: 10.48550\/arXiv.1910.03771]","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Chen T Guestrin C . XGBoost: a scalable tree boosting system. Presented at: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Aug 13-17, 2016:785-794; San Francisco, CA. [doi: 10.1145\/2939672.2939785]","DOI":"10.1145\/2939672.2939785"},{"key":"R24","unstructured":"Kudo T Yamamoto K Matsumoto Y . Applying conditional random fields to japanese morphological analysis. Presented at: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing; 2004:230-237; Barcelona, Spain."},{"key":"R25","unstructured":"Sato T Okumura M . Operation of a word segmentation dictionary generation system called neologd (in japanese). Presented at: Information Processing Society of Japan, Special Interest Group on Natural Language Processing (IPSJ-SIGNL) Information Processing Society of Japan; Dec 20-22, 2016; Tokyo."},{"key":"R26","unstructured":"Hu EJ Shen Y Wallis P et al. LoRA: low-rank adaptation of large language models. arXiv. Preprint posted online on Oct 16, 2021. URL: http:\/\/arxiv.org\/abs\/2106.09685 [Accessed 14-07-2025]"},{"key":"R27","unstructured":"Mangrulkar S Gugger S Debut L Belkada Y Paul S Bossan B . PEFT: state-of-the-art parameter-efficient fine-tuning methods. GitHub. 2022. URL: https:\/\/github.com\/huggingface\/peft [Accessed 12-07-2025]"},{"key":"R28","unstructured":"Lundberg SM Lee SI . A unified approach to interpreting model predictions. Presented at: Advances in Neural Information Processing Systems; Dec 4-9, 2017; Long Beach, CA. URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2017\/file\/8a20a8621978632d76c43dfd28b67767-Paper.pdf [Accessed 14-07-2025]"},{"key":"R29","unstructured":"Christophe C Kanithi PK Munjal P et al. Med42 -- evaluating fine-tuning strategies for medical llms: full-parameter vs. parameter-efficient approaches. arXiv. Preprint posted online on Apr 23, 2024. URL: http:\/\/arxiv.org\/abs\/2404.14779 [Accessed 14-07-2025]"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Sukeda I Suzuki M Sakaji H Kodera S . JMedLoRA: medical domain adaptation on japanese large language models using instruction-tuning. arXiv. Preprint posted online on Dec 1, 2023. URL: http:\/\/arxiv.org\/abs\/2310.10083 [Accessed 14-07-2025] [doi: 10.36922\/aih.2695]","DOI":"10.36922\/aih.2695"},{"key":"R31","unstructured":"Dettmers T Pagnoni A Holtzman A Zettlemoyer L . QLoRA: efficient finetuning of quantized llms. arXiv. Preprint posted online on May 23, 2023. URL: http:\/\/arxiv.org\/abs\/2305.14314 [Accessed 14-07-2025]"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1016\/j.pec.2023.108076","article-title":"AI triage or manual triage? Exploring medical staffs\u2019 preference for AI triage in China","volume":"119","author":"Cao","journal-title":"Patient Educ Couns"},{"issue":"2","key":"R33","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1111\/1742-6723.14345","article-title":"Attitudes towards artificial intelligence in emergency medicine","volume":"36","author":"Stewart","journal-title":"Emerg Med Australas"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2023.1229308","article-title":"Perspectives on artificial intelligence in healthcare from a Patient and Public Involvement Panel in Japan: an exploratory study","volume":"5","author":"Katirai","journal-title":"Front Digit Health"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","DOI":"10.2196\/50525","article-title":"Risk perception, acceptance, and trust of using AI in gastroenterology practice in the Asia-Pacific region: web-based survey study","volume":"3","author":"Goh","journal-title":"JMIR AI"},{"issue":"3","key":"R36","doi-asserted-by":"publisher","DOI":"10.2196\/19713","article-title":"Patients\u2019 utilization and perception of an artificial intelligence-based symptom assessment and advice technology in a British primary care waiting room: exploratory pilot study","volume":"7","author":"Miller","journal-title":"JMIR Hum Factors"},{"issue":"6","key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/37209","article-title":"Triage errors in primary and pre-primary care","volume":"24","author":"Nguyen","journal-title":"J Med Internet Res"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.2196\/52190","article-title":"Traditional machine learning, deep learning, and BERT (Large Language Model) approaches for predicting hospitalizations from nurse triage notes: comparative evaluation of resource management","volume":"3","author":"Patel","journal-title":"JMIR AI"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","DOI":"10.2196\/53297","article-title":"Triage performance across large language models, ChatGPT, and untrained doctors in emergency medicine: comparative study","volume":"26","author":"Masanneck","journal-title":"J Med Internet Res"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","DOI":"10.2196\/56306","article-title":"Finding consensus on trust in AI in health care: recommendations from a panel of international experts","volume":"27","author":"Starke","journal-title":"J Med Internet Res"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e68020","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:27:25Z","timestamp":1785536845000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e68020"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,22]]},"references-count":40,"URL":"https:\/\/doi.org\/10.2196\/68020","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,22]]},"article-number":"v4i7972e68020"}}