{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:47:43Z","timestamp":1782841663756,"version":"3.54.5"},"reference-count":37,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Translation of medical consultation summaries is essential for equitable health care communication in culturally and linguistically diverse populations. While machine translation (MT) tools and large language models (LLMs) are widely accessible, their feasibility and safety for health care contexts remain underexplored.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This pilot study investigates the feasibility and limitations of using LLMs and traditional MT tools to translate medical consultation summaries from English into the most common languages other than English spoken in Australia\u2014Arabic, Chinese (simplified written form), and Vietnamese.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Two simulated summaries\u2014a simple patient-facing summary and a complex clinician-oriented interprofessional letter\u2014were translated using 3 LLMs (GPT-4o, Llama-3.1, and Gemma-2) and 3 MT tools (Google Translate, Microsoft Bing Translator, and DeepL). Translations were benchmarked against professional third-party interpreter translations using Bilingual Evaluation Understudy, Character-level F-score, and Metric for Evaluation of Translation with Explicit Ordering metrics.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The translation performance varied across languages, tools, and summary complexity when assessed using automatic evaluation metrics. Traditional MT tools outperformed LLMs on surface-level metrics, while LLMs showed relative strengths in semantic similarity for Vietnamese and Chinese. Arabic translations improved with complex input, suggesting morphological advantages. The metric-based evaluation highlighted feasibility but also risks, particularly in Chinese clinical contexts.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>This pilot study provides formative evidence of opportunities and limitations in applying artificial intelligence translation for health care communication. Findings underscore the importance of human oversight; domain-specific evaluation metrics; and further formative and clinical research to guide the safe, equitable use of artificial intelligence translation tools.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/85169","type":"journal-article","created":{"date-parts":[[2026,2,16]],"date-time":"2026-02-16T20:05:05Z","timestamp":1771272305000},"page":"e85169-e85169","source":"Crossref","is-referenced-by-count":1,"title":["Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: Quantitative Pilot Feasibility Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-1990-928X","authenticated-orcid":false,"given":"Andy","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2548-6348","authenticated-orcid":false,"given":"Wei","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5147-8096","authenticated-orcid":false,"given":"Rashina","family":"Hoda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5281-2364","authenticated-orcid":false,"given":"Chris","family":"Bain","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5175-3430","authenticated-orcid":false,"given":"Peter","family":"Poon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,4,13]]},"reference":[{"key":"R1","unstructured":"Google Translate. Google. URL: https:\/\/translate.google.com\/?hl=bn&sl=auto&tl=bn&op=translate [Accessed 27-02-2026]"},{"key":"R2","unstructured":"Bing Translator. Microsoft. URL: https:\/\/www.microsoft.com\/en-us\/bing\/features\/translator\/?form=MA13FV [Accessed 27-02-2026]"},{"key":"R3","unstructured":"DeepL Translate. DeepL. URL: https:\/\/www.deepl.com\/en\/translator [Accessed 27-02-2026]"},{"key":"R4","unstructured":"Costa-Juss\u00e0 MR Farr\u00fas M Serrano Pons J . Machine translation in medicine: a quality analysis of statistical machine translation in the medical domain. Presented at: 1st Virtual International Conference on Advanced Research in Scientific Fields (ARSA-2012); Dec 3-7, 2012; Slovakia. URL: https:\/\/repositori.upf.edu\/items\/d591266b-d0aa-4bcf-817b-f34391b07fab [Accessed 27-02-2026]"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.2196\/43255","article-title":"Overcoming language barriers in paramedic care with an app designed to improve communication with foreign-language patients: nonrandomized controlled pilot study","volume":"7","author":"M\u00fcller","journal-title":"JMIR Form Res"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.2196\/63095","article-title":"Digital translation platform (translatly) to overcome communication barriers in clinical care: pilot study","volume":"9","author":"Olsavszky","journal-title":"JMIR Form Res"},{"key":"R7","unstructured":"Hello GPT\u20114o. OpenAI. URL: https:\/\/openai.com\/index\/hello-gpt-4o\/ [Accessed 27-02-2026]"},{"key":"R8","unstructured":"Gemma. Google DeepMind. URL: https:\/\/deepmind.google\/models\/gemma\/ [Accessed 27-02-2026]"},{"key":"R9","unstructured":"Open-source AI models for any application | Llama 3. Meta. URL: https:\/\/www.llama.com\/models\/llama-3\/ [Accessed 18-03-2026]"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1177\/20552076231224603","article-title":"Evaluation of information provided to patients by ChatGPT about chronic diseases in Spanish language","volume":"10","author":"Soto-Ch\u00e1vez","journal-title":"Digit Health"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1177\/20552076241253523","article-title":"Safety and quality of AI chatbots for drug-related inquiries: a real-world comparison with licensed pharmacists","volume":"10","author":"Albogami","journal-title":"Digit Health"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1177\/20552076241236635","article-title":"Performance of ChatGPT on the National Korean Occupational Therapy Licensing Examination","volume":"10","author":"Lee","journal-title":"Digit Health"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1177\/20552076241227132","article-title":"Reliability of ChatGPT for performing triage task in the emergency department using the Korean Triage and Acuity Scale","volume":"10","author":"Kim","journal-title":"Digit Health"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1177\/20552076231224074","article-title":"Performance of ChatGPT incorporated chain-of-thought method in bilingual nuclear medicine physician board examinations","volume":"10","author":"Ting","journal-title":"Digit Health"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1177\/20552076241233144","article-title":"Performance of ChatGPT on Stage 1 of the Taiwanese medical licensing exam","volume":"10","author":"Huang","journal-title":"Digit Health"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1177\/20552076241295565","article-title":"Integrating ChatGPT: enhancing postpartum mental healthcare with artificial intelligence (AI) support","volume":"10","author":"Alam","journal-title":"Digit Health"},{"key":"R17","doi-asserted-by":"crossref","unstructured":"Wang L Lyu C Ji T et al. Document-level machine translation with large language models. arXiv. Preprint posted online on Apr 5, 2023. [doi: 10.48550\/arXiv.2304.02210]","DOI":"10.18653\/v1\/2023.emnlp-main.1036"},{"key":"R18","unstructured":"Kocmi T Federmann C . Large language models are state-of-the-art evaluators of translation quality. arXiv. Preprint posted online on Feb 28, 2023. [doi: 10.48550\/arXiv.2302.14520]"},{"key":"R19","unstructured":"Wassie AK Molaei M Moslem Y . Domain-specific translation with open-source large language models: resource-oriented analysis. arXiv. Preprint posted online on Dec 8, 2024. [doi: 10.48550\/arXiv.2412.05862]"},{"key":"R20","unstructured":"Rios M . Instruction-tuned large language models for machine translation in the medical domain. arXiv. Preprint posted online on Aug 29, 2024. [doi: 10.48550\/arXiv.2408.16440]"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Cardey S Greenfield P Wu X . Designing a controlled language for the machine translation of medical protocols: the case of English to Chinese. Presented at: 6th Conference of the Association for Machine Translation in the Americas; Sep 28 to Oct 2, 2004; Washington, DC. URL: https:\/\/link.springer.com\/chapter\/10.1007\/978-3-540-30194-3_5 [Accessed 27-02-2026]","DOI":"10.1007\/978-3-540-30194-3_5"},{"key":"R22","unstructured":"Cultural diversity of Australia. Australian Bureau of Statistics. 2022. URL: https:\/\/www.abs.gov.au\/articles\/cultural-diversity-australia [Accessed 27-02-2026]"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1177\/20552076241293932","article-title":"Exploring the opportunities of large language models for summarizing palliative care consultations: a pilot comparative study","volume":"10","author":"Chen","journal-title":"Digit Health"},{"key":"R24","doi-asserted-by":"crossref","unstructured":"Papineni K Roukos S Ward T Zhu WJ . BLEU: a method for automatic evaluation of machine translation. Presented at: ACL \u201902: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics; Jul 7-12, 2002; Philadelphia, PA. [doi: 10.3115\/1073083.1073135]","DOI":"10.3115\/1073083.1073135"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Popovi\u0107 M . chrF: character n-gram F-score for automatic MT evaluation. Presented at: Proceedings of the Tenth Workshop on Statistical Machine Translation; Sep 17-18, 2017; Lisbon, Portugal. [doi: 10.18653\/v1\/W15-3049]","DOI":"10.18653\/v1\/W15-3049"},{"key":"R26","unstructured":"Banerjee S Lavie A . METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. Presented at: ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization; Jun 29, 2005; Ann Arbor, MI. URL: https:\/\/aclanthology.org\/W05-0909\/ [Accessed 28-02-2026]"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Baek Y Lee K Ki D Park C Lee HG Choo J . Towards accurate translation via semantically appropriate application of lexical constraints. arXiv. Preprint posted online on Jun 21, 2023. [doi: 10.48550\/arXiv.2306.12089]","DOI":"10.18653\/v1\/2023.findings-acl.298"},{"key":"R28","doi-asserted-by":"crossref","unstructured":"Du\u0161ek O Haji\u010d J Hlav\u00e1\u010dov\u00e1 J et al. Machine translation of medical texts in the Khresmoi project. Presented at: Proceedings of the Ninth Workshop on Statistical Machine Translation; Jun 26-27, 2014; Baltimore, MD. URL: https:\/\/aclanthology.org\/W14-3326.pdf [Accessed 28-02-2026]","DOI":"10.3115\/v1\/W14-3326"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Elshin D Karpachev N Gruzdev B et al. From general LLM to translation: how we dramatically improve translation quality using human evaluation data for LLM finetuning. Presented at: Proceedings of the Ninth Conference on Machine Translation; Nov 15-16, 2024; Miami, FL. [doi: 10.18653\/v1\/2024.wmt-1.17]","DOI":"10.18653\/v1\/2024.wmt-1.17"},{"issue":"11","key":"R30","doi-asserted-by":"publisher","first-page":"1515","DOI":"10.1080\/1369118X.2020.1776370","article-title":"Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases","volume":"24","author":"Vieira","journal-title":"Inf Commun Soc"},{"issue":"2","key":"R31","doi-asserted-by":"publisher","first-page":"170","DOI":"10.69760\/aghel.01024080","article-title":"Navigating complexities in medical text translation: challenges, strategies, and solutions","volume":"1","author":"Ismayilli","journal-title":"Acta Glob Humanit Linguar"},{"key":"R32","unstructured":"Zhang B Haddow B Birch A . Prompting large language model for machine translation: a case study. Presented at: ICML\u201923: Proceedings of the 40th International Conference on Machine Learning; Jul 23-29, 2023; Honolulu, HI. [doi: 10.5555\/3618408.3620130]"},{"key":"R33","unstructured":"Garcia X Bansal Y Cherry C et al. The unreasonable effectiveness of few-shot learning for machine translation. Presented at: ICML\u201923: Proceedings of the 40th International Conference on Machine Learning; Jul 23-29, 2023; Honolulu, HI. [doi: 10.5555\/3618408.3618846]"},{"key":"R34","volume":"2","author":"Eshbo\u2019riyevich","journal-title":"International Conference on Linguistics, Literature and Translation (London)"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-02005-2","article-title":"Evaluating clinical AI summaries with large language models as judges","volume":"8","author":"Croxford","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R36","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01074-z","article-title":"Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI","volume":"7","author":"Abbasian","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R37","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation","volume":"8","author":"Asgari","journal-title":"NPJ Digit Med"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e85169","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:13:35Z","timestamp":1782839615000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e85169"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,13]]},"references-count":37,"URL":"https:\/\/doi.org\/10.2196\/85169","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,13]]},"article-number":"v10i11e85169"}}