{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T07:54:18Z","timestamp":1790927658757,"version":"4.1.0"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T00:00:00Z","timestamp":1790899200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T00:00:00Z","timestamp":1790899200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003710","name":"Korea Health Industry Development Institute","doi-asserted-by":"publisher","award":["RS-2024-00403047"],"award-info":[{"award-number":["RS-2024-00403047"]}],"id":[{"id":"10.13039\/501100003710","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Med Syst"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Systematized Nomenclature of Medicine\u2013Clinical Terminology (SNOMED CT) is the principal international standard for semantic interoperability of clinical information, but mapping free-text clinical narratives to SNOMED CT concepts remains labor-intensive. We developed a large language model agent system for mapping bilingual clinical text to SNOMED CT concepts and evaluated its effect on mapping accuracy and efficiency within a human-AI collaborative workflow. We designed a three-module agent system comprising translation, abbreviation expansion, and vector-based retrieval components, integrated with a pre-embedded SNOMED CT vector database. Three health information managers independently mapped bilingual clinical text segments using three approaches: human-only, Agent-only, and Agent-assisted human mapping. Performance was evaluated by using hit rate, precision, recall, and F1 score at k\u2009=\u20091 and 5, and R-precision. Mapping time was compared between human-only and human-AI collaborative approaches. A total of 2,261 de-identified clinical text segments across nine clinical categories were collected at a tertiary academic hospital in South Korea. The human-AI collaborative workflow, which expanded the set of valid SNOMED CT candidates presented at each mapping decision, raised pooled hit rate@1 from 0.837 to 0.868 (difference 0.031, 95% confidence interval [CI] 0.021 to 0.042;\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u20090.001), raised R-precision from 0.632 to 0.674 (difference 0.042, 95% CI 0.034 to 0.051;\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u20090.001), and reduced total mapping time by 53.9% (from 1.57 to 0.72\u00a0min per segment, including agent processing). By expanding the space of valid SNOMED CT candidates available to expert mappers, the human-AI collaborative approach improved SNOMED CT mapping accuracy while reducing time by about half. Its modular architecture, supporting periodic vector database updates without retraining, offers a sustainable and efficient solution for bilingual clinical terminology standardization.\n                  <\/jats:p>","DOI":"10.1007\/s10916-026-02465-3","type":"journal-article","created":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T07:14:22Z","timestamp":1790925262000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Development and Validation of Human-AI Collaborative Workflow in SNOMED CT Mapping of Bilingual Clinical Text"],"prefix":"10.1007","volume":"50","author":[{"given":"Hyeonhoon","family":"Lee","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seonhye","family":"Choi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Duyeon","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyunglan","family":"Hong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyeonsik","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang Wook","family":"Jeong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyung-Chul","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,10,2]]},"reference":[{"key":"2465_CR1","doi-asserted-by":"publisher","unstructured":"Chang E, Mostafa J. The use of SNOMED CT, 2013\u20132020: a literature review. J Am Med Inform Assoc. 2021;28(9):2017\u20132026. https:\/\/doi.org\/10.1093\/jamia\/ocab084. PMID: 34151978.","DOI":"10.1093\/jamia\/ocab084"},{"key":"2465_CR2","doi-asserted-by":"publisher","unstructured":"Cornet R, de Keizer N. Forty years of SNOMED: a literature review. BMC Med Inform Decis Mak. 2008;8(Suppl 1):S2. https:\/\/doi.org\/10.1186\/1472-6947-8-S1-S2. PMID: 19007439.","DOI":"10.1186\/1472-6947-8-S1-S2"},{"key":"2465_CR3","doi-asserted-by":"publisher","unstructured":"Sung S, Park HA, Jung H, Kang H. A SNOMED CT mapping guideline for the local terms used to document clinical findings and procedures in electronic medical records in South Korea: methodological study. JMIR Med Inform. 2023;11:e46127. https:\/\/doi.org\/10.2196\/46127. PMID: 37071456.","DOI":"10.2196\/46127"},{"key":"2465_CR4","doi-asserted-by":"publisher","unstructured":"Park HA. Why terminology standards matter for data-driven artificial intelligence in healthcare. Ann Lab Med. 2024;44(6):467\u2013471. https:\/\/doi.org\/10.3343\/alm.2024.0105. PMID: 38955364.","DOI":"10.3343\/alm.2024.0105"},{"key":"2465_CR5","doi-asserted-by":"publisher","unstructured":"Wang L, Ma Y, Bi W, Lv H, Li Y. An entity extraction pipeline for medical text records using large language models: analytical study. J Med Internet Res. 2024;26:e54580. https:\/\/doi.org\/10.2196\/54580. PMID: 38551633.","DOI":"10.2196\/54580"},{"key":"2465_CR6","doi-asserted-by":"publisher","unstructured":"Richesson RL, Andrews JE, Krischer JP. Use of SNOMED CT to represent clinical research data: a semantic characterization of data items on case report forms in vasculitis research. J Am Med Inform Assoc. 2006;13(5):536\u2013546. https:\/\/doi.org\/10.1197\/jamia.M2093. PMID: 16799121.","DOI":"10.1197\/jamia.M"},{"key":"2465_CR7","doi-asserted-by":"publisher","unstructured":"Balch JA, Ruppert MM, Loftus TJ, et al. Machine learning-enabled clinical information systems using fast healthcare interoperability resources data standards: scoping review. JMIR Med Inform. 2023;11:e48297. https:\/\/doi.org\/10.2196\/48297. PMID: 37646309.","DOI":"10.2196\/48297"},{"key":"2465_CR8","doi-asserted-by":"publisher","unstructured":"Minarro-Gimenez JA, Martinez-Costa C, Karlsson D, Schulz S, Goeg KR. Qualitative analysis of manual annotations of clinical text with SNOMED CT. PLoS One. 2018;13(12):e0209547. https:\/\/doi.org\/10.1371\/journal.pone.0209547. PMID: 30589855.","DOI":"10.1371\/journal.pone.0209547"},{"key":"2465_CR9","doi-asserted-by":"publisher","unstructured":"Andrews JE, Richesson RL, Krischer J. Variation of SNOMED CT coding of clinical research concepts among coding experts. J Am Med Inform Assoc. 2007;14(4):497\u2013506. https:\/\/doi.org\/10.1197\/jamia.M2372. PMID: 17460128.","DOI":"10.1197\/jamia.M2372"},{"key":"2465_CR10","doi-asserted-by":"publisher","unstructured":"Schwab JD, Werle SD, Huhne R, Spohn H, Kaisers UX, Kestler HA. The necessity of interoperability to uncover the full potential of digital health devices. JMIR Med Inform. 2023;11:e49301. https:\/\/doi.org\/10.2196\/49301. PMID: 38133917.","DOI":"10.2196\/49301"},{"key":"2465_CR11","doi-asserted-by":"publisher","unstructured":"Ochs C, Geller J, Perl Y, et al. Scalable quality assurance for large SNOMED CT hierarchies using subject-based subtaxonomies. J Am Med Inform Assoc. 2015;22(3):507\u2013518. https:\/\/doi.org\/10.1136\/amiajnl-2014-003151. PMID: 25336594.","DOI":"10.1136\/amiajnl-2014-003151"},{"key":"2465_CR12","doi-asserted-by":"publisher","unstructured":"Kang B, Yoon J, Kim HY, Jo SJ, Lee Y, Kam HJ. Deep-learning-based automated terminology mapping in OMOP-CDM. J Am Med Inform Assoc. 2021;28(7):1489\u20131496. https:\/\/doi.org\/10.1093\/jamia\/ocab030. PMID: 33987667.","DOI":"10.1093\/jamia\/ocab030"},{"key":"2465_CR13","doi-asserted-by":"publisher","unstructured":"Davidson R, Hardman W, Amit G, et al. SNOMED CT entity linking challenge. J Am Med Inform Assoc. 2025;32(9):1397\u20131406. https:\/\/doi.org\/10.1093\/jamia\/ocaf104. PMID: 40657868.","DOI":"10.1093\/jamia\/ocaf104"},{"key":"2465_CR14","doi-asserted-by":"publisher","unstructured":"Fan H, Rossetti SC, Thate J, Mugoya R, Lai AM, Yen PY. Semi-automated pipeline to accelerate multi-site flowsheet alignment and concept mapping in electronic health records. J Am Med Inform Assoc. 2025;32(7):1140\u20131148. https:\/\/doi.org\/10.1093\/jamia\/ocaf076. PMID: 40378254.","DOI":"10.1093\/jamia\/ocaf076"},{"key":"2465_CR15","doi-asserted-by":"publisher","unstructured":"Puts S, Zegers CML, Dekker A, Bermejo I. Developing an ICD-10 coding assistant: pilot study using RoBERTa and GPT-4 for term extraction and description-based code selection. JMIR Form Res. 2025;9:e60095. https:\/\/doi.org\/10.2196\/60095. PMID: 39935026.","DOI":"10.2196\/60095"},{"key":"2465_CR16","doi-asserted-by":"publisher","unstructured":"Abeysinghe R, Zheng F, Bernstam EV, Shi J, Bodenreider O, Cui L. A deep learning approach to identify missing is-a relations in SNOMED CT. J Am Med Inform Assoc. 2023;30(3):475\u2013484. https:\/\/doi.org\/10.1093\/jamia\/ocac248. PMID: 36539234.","DOI":"10.1093\/jamia\/ocac248"},{"key":"2465_CR17","doi-asserted-by":"publisher","unstructured":"Kim S, Choi B, Lee K, Lee S, Kim S. Assessing the performance of a method for case-mix adjustment in the Korean Diagnosis-Related Groups (KDRG) system and its policy implications. Health Res Policy Syst. 2021;19(1):98. https:\/\/doi.org\/10.1186\/s12961-021-00739-5. PMID: 34187515.","DOI":"10.1186\/s12961-021-00739-5"},{"key":"2465_CR18","volume-title":"SNOMED CT International Edition, release 2024-08-01","author":"SNOMED International","year":"2024","unstructured":"SNOMED International. SNOMED CT International Edition, release 2024-08-01. London: SNOMED International; 2024."},{"key":"2465_CR19","unstructured":"Chroma. Chroma: the open-source AI application database. https:\/\/www.trychroma.com (accessed 13 August 2026)."},{"key":"2465_CR20","doi-asserted-by":"publisher","unstructured":"Jin Q, Kim W, Chen Q, et al. MedCPT: contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval. Bioinformatics. 2023;39(11):btad651. https:\/\/doi.org\/10.1093\/bioinformatics\/btad651. PMID: 37930897.","DOI":"10.1093\/bioinformatics\/btad651"},{"key":"2465_CR21","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809071","volume-title":"Introduction to Information Retrieval","author":"CD Manning","year":"2008","unstructured":"Manning CD, Raghavan P, Sch\u00fctze H. Introduction to Information Retrieval. Cambridge: Cambridge University Press; 2008."},{"key":"2465_CR22","doi-asserted-by":"crossref","unstructured":"Buckley C, Voorhees EM. Evaluating evaluation measure stability. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM; 2000. p. 33\u201340.","DOI":"10.1145\/345508.345543"},{"key":"2465_CR23","doi-asserted-by":"publisher","unstructured":"Yuan X, Wang T, Meng R, et al. One size does not fit all: generating and evaluating variable number of keyphrases. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; 2020. https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.710.","DOI":"10.18653\/v1\/2020.acl-main.710"},{"key":"2465_CR24","doi-asserted-by":"publisher","unstructured":"Chang E, Sung S. Use of SNOMED CT in large language models: scoping review. JMIR Med Inform. 2024;12:e62924. https:\/\/doi.org\/10.2196\/62924. PMID: 39374057.","DOI":"10.2196\/62924"},{"key":"2465_CR25","doi-asserted-by":"publisher","unstructured":"Gao Y, Chen Y, Wang M, et al. Optimising the paradigms of human AI collaborative clinical coding. NPJ Digit Med. 2024;7(1):368. https:\/\/doi.org\/10.1038\/s41746-024-01363-7. PMID: 39702575.","DOI":"10.1038\/s41746-024-01363-7"},{"key":"2465_CR26","doi-asserted-by":"publisher","unstructured":"Adams MCB, Perkins ML, Hudson C, et al. Breaking digital health barriers through a large language model-based tool for automated Observational Medical Outcomes Partnership mapping: development and validation study. J Med Internet Res. 2025;27:e69004. https:\/\/doi.org\/10.2196\/69004. PMID: 40146872.","DOI":"10.2196\/69004"},{"key":"2465_CR27","doi-asserted-by":"publisher","unstructured":"Kimura E, Kawakami Y, Inoue S, Okajima A. Mapping drug terms via integration of a retrieval-augmented generation algorithm with a large language model. Healthc Inform Res. 2024;30(4):355\u2013363. https:\/\/doi.org\/10.4258\/hir.2024.30.4.355. PMID: 39551922.","DOI":"10.4258\/hir.2024.30.4.355"},{"key":"2465_CR28","doi-asserted-by":"publisher","unstructured":"Berkowitz JS, Srinivasan A, Acitores Cortina JM, Fatapour Y, Tatonetti NP. Biomedical text normalization through generative modeling. J Biomed Inform. 2025;167:104850. https:\/\/doi.org\/10.1016\/j.jbi.2025.104850. PMID: 40381869.","DOI":"10.1016\/j.jbi.2025.104850"},{"key":"2465_CR29","doi-asserted-by":"publisher","unstructured":"Liu S, McCoy AB, Wright A. Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. J Am Med Inform Assoc. 2025;32(4):605\u2013615. doi:https:\/\/doi.org\/10.1093\/jamia\/ocaf008. PMID: 39812777.","DOI":"10.1093\/jamia\/ocaf008"},{"key":"2465_CR30","doi-asserted-by":"publisher","unstructured":"Bedi S, Liu Y, Orr-Ewing L, et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA. 2025;333(4):319\u2013328. https:\/\/doi.org\/10.1001\/jama.2024.21700. PMID: 39405325.","DOI":"10.1001\/jama.2024.21700"},{"key":"2465_CR31","doi-asserted-by":"publisher","unstructured":"Duggan MJ, Gervase J, Schoenbaum A, et al. Clinician experiences with ambient scribe technology to assist with documentation burden and efficiency. JAMA Netw Open. 2025;8(2):e2460637. https:\/\/doi.org\/10.1001\/jamanetworkopen.2024.60637. PMID: 39969880.","DOI":"10.1001\/jamanetworkopen.2024.60637"},{"key":"2465_CR32","doi-asserted-by":"publisher","unstructured":"Dai HJ, Wang CK, Chen CC, et al. Evaluating a natural language processing-driven, AI-assisted International Classification of Diseases, 10th Revision, Clinical Modification coding system for diagnosis related groups in a real hospital environment: algorithm development and validation study. J Med Internet Res. 2024;26:e58278. https:\/\/doi.org\/10.2196\/58278. PMID: 39302714.","DOI":"10.2196\/58278"}],"container-title":["Journal of Medical Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10916-026-02465-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10916-026-02465-3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10916-026-02465-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T07:14:24Z","timestamp":1790925264000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10916-026-02465-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,2]]},"references-count":32,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["2465"],"URL":"https:\/\/doi.org\/10.1007\/s10916-026-02465-3","relation":{},"ISSN":["1573-689X"],"issn-type":[{"value":"1573-689X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,2]]},"assertion":[{"value":"2 June 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 September 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 October 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This was a retrospective comparative study evaluating three approaches for mapping clinical text to SNOMED CT concepts. The study protocol was approved by the Institutional Review Board of Seoul National University Hospital (SNUH) (IRB No. 2408-106-1562), with a waiver of informed consent granted given the retrospective, de-identified nature of the data.","order":1,"name":"Ethics","label":"Human Ethics and Consent to Participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","label":"Clinical trial number","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"141"}}