{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T20:58:34Z","timestamp":1791406714002,"version":"4.3.3"},"reference-count":26,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T00:00:00Z","timestamp":1770940800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The absence of standardization in the documentation of routine clinical data complicates research usage of retrospective data on a large-scale basis. Medically trained personnel is required for interpretation and conversion into a structured format making it time and cost intensive and creating a potential bias of such data. To address these challenges, we have developed a semi-automated approach for evaluating Multiple Sclerosis (MS) outpatients reports that utilizes different large-language models (LLM) and their consensus in comparison to manual evaluation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We used several commercially available LLMs by OpenAI, Anthropic and Google to create a structured output of several variables with differing complexity of 30 anonymized outpatient reports with zero-shot-learning. We added a consensus output by combining the results of three different LLMs. Over several runs, we adapted the prompt, compared the results with a reference and assessed the error rate. Any deviation from the reference was considered an error. A true-error rate was determined for the LLM consensus output and the neurology specialist output, where only content deviations are counted as errors.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Through 9 iterations of improving the structure and content of the prompt, we have seen a clear reduction in the error rate of the various LLMs. By creating an LLM consensus with the final prompt design, we were able to overcome a ceiling effect in reducing the error rate. With a true-error rate of 1.48%, the LLM consensus shows a similar error rate as neurologists (around 2%) in the creation of structured data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Our method enables fast and reliable LLM-based analysis of large clinical routine data sets of varying complexity with a low technical barrier to entry. By generating an LLM consensus, we were able to considerably improve the quality of the output making it comparable to data created by neurology specialists. This approach allows large amounts of unstructured data to be analyzed in a time and cost-efficient manner. Nevertheless, the evaluation of errors in results produced by LLM remains difficult. Scientific work using such methods must continue to be subject to strict testing of the validity of the method in the future.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frai.2026.1658575","type":"journal-article","created":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T06:38:10Z","timestamp":1770964690000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Improving reliability and accuracy of structured data extraction using a consensus large-language model approach\u2013a use case description in multiple sclerosis"],"prefix":"10.3389","volume":"9","author":[{"given":"Philip Lennart","family":"Poser","sequence":"first","affiliation":[{"name":"Department of Neurology, St. Josef-Hospital, Ruhr-University Bochum","place":["Bochum, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rafael","family":"Klimas","sequence":"additional","affiliation":[{"name":"Department of Neurology, St. Josef-Hospital, Ruhr-University Bochum","place":["Bochum, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Justus","family":"Luerweg","sequence":"additional","affiliation":[{"name":"Department of Neurology, St. Josef-Hospital, Ruhr-University Bochum","place":["Bochum, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emilie","family":"Reuter","sequence":"additional","affiliation":[{"name":"Department of Neurology, St. Josef-Hospital, Ruhr-University Bochum","place":["Bochum, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christoph","family":"Hanefeld","sequence":"additional","affiliation":[{"name":"Department of Internal Medicine, Katholisches Klinikum Bochum, Ruhr-University Bochum","place":["Bochum, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ralf","family":"Gold","sequence":"additional","affiliation":[{"name":"Department of Neurology, St. Josef-Hospital, Ruhr-University Bochum","place":["Bochum, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anke","family":"Salmen","sequence":"additional","affiliation":[{"name":"Department of Neurology, St. Josef-Hospital, Ruhr-University Bochum","place":["Bochum, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeremias","family":"Motte","sequence":"additional","affiliation":[{"name":"Department of Neurology, St. Josef-Hospital, Ruhr-University Bochum","place":["Bochum, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,2,13]]},"reference":[{"key":"ref1","first-page":"301","article-title":"Role and challenges of unstructured big data in healthcare","author":"Adnan","year":"2020"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"104662","DOI":"10.1016\/j.jbi.2024.104662","article-title":"Applying generative AI with retrieval augmented generation to summarize and extract key clinical information from electronic health records","volume":"156","author":"Alkhalaf","year":"2024","journal-title":"J. Biomed. Inform."},{"key":"ref3","doi-asserted-by":"publisher","first-page":"108655","DOI":"10.1016\/j.cmpb.2025.108655","article-title":"Fine-tuning large language models for improved health communication in low-resource languages","volume":"263","author":"Bui","year":"2025","journal-title":"Comput. Methods Prog. Biomed."},{"key":"ref4","doi-asserted-by":"publisher","first-page":"e2425981","DOI":"10.1001\/jamanetworkopen.2024.25981","article-title":"Use of generative AI to identify helmet status among patients with micromobility-related injuries from unstructured clinical notes","volume":"7","author":"Burford","year":"2024","journal-title":"JAMA Netw. Open"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"1079","DOI":"10.13063\/2327-9214.1079","article-title":"Availability of structured and unstructured clinical data for comparative effectiveness research and quality improvement: a multisite assessment","volume":"2","author":"Capurro","year":"2014","journal-title":"EGEMS (Wash. DC)"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"Clusmann","year":"2023","journal-title":"Commun Med (Lond)."},{"key":"ref7","doi-asserted-by":"publisher","first-page":"1418","DOI":"10.1038\/s41467-024-45563-x","article-title":"Structured information extraction from scientific text with large language models","volume":"15","author":"Dagdelen","year":"2024","journal-title":"Nat. Commun."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"1612","DOI":"10.1093\/aje\/kwae071","article-title":"Validation of algorithms in studies based on routinely collected health data: general principles","volume":"193","author":"Ehrenstein","year":"2024","journal-title":"Am. J. Epidemiol."},{"key":"ref9","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models","volume":"31","author":"Gallifant","year":"2025","journal-title":"Nat. Med."},{"key":"ref10","doi-asserted-by":"publisher","first-page":"223","DOI":"10.2147\/JMDH.S502351","article-title":"Large language models in healthcare: a bibliometric analysis and examination of research trends","volume":"18","author":"Gencer","year":"2025","journal-title":"J. Multidiscip. Healthc."},{"key":"ref11","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1038\/s41746-024-01079-8","article-title":"A critical assessment of using ChatGPT for extracting structured data from clinical notes","volume":"7","author":"Huang","year":"2024","journal-title":"NPJ Digit. Med."},{"key":"ref12","doi-asserted-by":"publisher","first-page":"103809","DOI":"10.1016\/j.ipm.2024.103809","article-title":"Are LLMs good at structured outputs? A benchmark for evaluating structured output capabilities in LLMs","volume":"61","author":"Liu","year":"2024","journal-title":"Inf. Process. Manag."},{"key":"ref13","doi-asserted-by":"publisher","first-page":"1308","DOI":"10.1016\/j.bja.2025.01.028","article-title":"Automated structured data extraction from intraoperative echocardiography reports using large language models","volume":"134","author":"MacKay","year":"2025","journal-title":"Br. J. Anaesth."},{"key":"ref14","doi-asserted-by":"publisher","first-page":"109713","DOI":"10.1016\/j.isci.2024.109713","article-title":"The application of large language models in medicine: a scoping review","volume":"27","author":"Meng","year":"2024","journal-title":"iScience"},{"key":"ref15","doi-asserted-by":"publisher","first-page":"57","DOI":"10.3390\/informatics11030057","article-title":"Large language models in healthcare and medical domain: a review","volume":"11","author":"Nazi","year":"2024","journal-title":"Informatics"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"1139","DOI":"10.1136\/bmjhci-2024-101139","article-title":"Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation","volume":"32","author":"Ntinopoulos","year":"2025","journal-title":"BMJ Health Care Inform."},{"key":"ref17","doi-asserted-by":"publisher","first-page":"110731","DOI":"10.1016\/j.compbiomed.2025.110731","article-title":"Refining LLMs outputs with iterative consensus ensemble (ICE)","volume":"196","author":"Omar","year":"2025","journal-title":"Comput. Biol. Med."},{"key":"ref18","doi-asserted-by":"publisher","first-page":"101048","DOI":"10.1016\/j.lanwpc.2024.101048","article-title":"The application of multimodal large language models in medicine","volume":"45","author":"Qiu","year":"2024","journal-title":"Lancet Reg Health West Pac."},{"key":"ref19","doi-asserted-by":"publisher","first-page":"e2425953","DOI":"10.1001\/jamanetworkopen.2024.25953","article-title":"Accuracy, consistency, and hallucination of large language models when analyzing unstructured clinical notes in electronic medical records","volume":"7","author":"Shah","year":"2024","journal-title":"JAMA Netw. Open"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"2293","DOI":"10.1056\/NEJMsb1609216","article-title":"Real-world evidence - what is it and what can it tell us?","volume":"375","author":"Sherman","year":"2016","journal-title":"N. Engl. J. Med."},{"key":"ref21","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","year":"2025","journal-title":"Nat. Med."},{"key":"ref22","doi-asserted-by":"publisher","first-page":"e1549","DOI":"10.1002\/wics.1549","article-title":"Challenges and opportunities beyond structured data in analysis of electronic health records","volume":"13","author":"Tayefi","year":"2021","journal-title":"WIREs Comput. Stat."},{"key":"ref23","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1186\/s13000-024-01464-7","article-title":"Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic medicine with a focus on digital pathology - a recent scoping review","volume":"19","author":"Ullah","year":"2024","journal-title":"Diagn. Pathol."},{"key":"ref24","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1038\/s41746-024-01233-2","article-title":"Privacy-preserving large language models for structured medical information retrieval","volume":"7","author":"Wiest","year":"2024","journal-title":"NPJ Digit. Med."},{"key":"ref25","doi-asserted-by":"publisher","first-page":"2917","DOI":"10.1101\/2024.09.02.24312917","article-title":"LLM-AIx: an open source pipeline for information extraction from unstructured medical text based on privacy preserving large language models","volume":"3","author":"Wiest","year":"2024","journal-title":"medRxiv"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"2018","DOI":"10.1007\/s00330-024-11074-y","article-title":"Automatic structuring of radiology reports with on-premise open-source large language models","volume":"35","author":"Woznicki","year":"2025","journal-title":"Eur. Radiol."}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1658575\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T06:38:11Z","timestamp":1770964691000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1658575\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,13]]},"references-count":26,"alternative-id":["10.3389\/frai.2026.1658575"],"URL":"https:\/\/doi.org\/10.3389\/frai.2026.1658575","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,13]]},"article-number":"1658575"}}