{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T12:49:20Z","timestamp":1785761360635,"version":"3.56.0"},"reference-count":26,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2026,8,2]],"date-time":"2026-08-02T00:00:00Z","timestamp":1785628800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Clin. Med.","JCM"],"abstract":"<jats:p>Background: The reliable integration of large language models (LLMs) into neuroimaging data extraction workflows remains unresolved. Prior benchmarking shows that exact-match accuracy underestimates LLM extraction performance, but whether inter-model consensus and variable complexity can guide automation remains unclear. We evaluated whether inter-model consensus can serve as a confidence signal for human\u2013artificial intelligence (AI) extraction and can guide complexity-stratified workflow triage. Methods: Four frontier LLMs were queried via OpenRouter with an identical zero-shot structured prompt to extract 22 predefined variables from 91 peer-reviewed neuroimaging AI articles, yielding 2002 article\u2013variable items per model. Variables were stratified a priori into low- (n = 7), medium- (n = 8), and high-complexity (n = 7). Performance was compared with an expert reference using exact-match and semantic-equivalence accuracy. Item-level consensus and five triage strategies characterized the efficiency\u2013accuracy trade-off. Results: Semantic-equivalence accuracy converged to 80.5\u201383.4% across models despite approximately ten percentage-point exact-match differences. Unanimous 4\/4 consensus occurred in 45.6% (910\/1994) of items, with exact-match accuracy of 85.8%, rising to 95.3% after semantic normalization; however, 14.2% still failed to match the reference. Reliability was complexity-dependent: 96.6% for low-complexity variables, 73.2% for medium-complexity variables, and 38.1% for high-complexity variables. A hybrid strategy auto-accepting 4\/4 items and routing 3\/4 items to rapid verification reduced estimated review effort by approximately 59%. Conclusions: Inter-model consensus is useful, but it is incomplete and depends on variable complexity. We show that LLM-assisted extraction in neuroimaging AI is a complexity-stratified workflow design problem: low-complexity neuroimaging variables may be selectively automated, while medium-complexity variables require rapid verification, and high-complexity methodological variables should remain human-led.<\/jats:p>","DOI":"10.3390\/jcm15156005","type":"journal-article","created":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T11:50:21Z","timestamp":1785757821000},"page":"6005","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["How Often Do Large Language Models Agree with Each Other\u2014And with the Truth? A Consensus- and Complexity-Stratified Analysis of Data Extraction for Neuroimaging AI"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4192-9268","authenticated-orcid":false,"given":"Nafiye","family":"Sanlier","sequence":"first","affiliation":[{"name":"Department of Neurological Surgery, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4839-3347","authenticated-orcid":false,"given":"Umid","family":"Sulaimanov","sequence":"additional","affiliation":[{"name":"Department of Neurological Surgery, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5171-3532","authenticated-orcid":false,"given":"Ariorad","family":"Moniri","sequence":"additional","affiliation":[{"name":"School of Medicine, Acibadem University, Istanbul 34752, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3474-2232","authenticated-orcid":false,"given":"Behman","family":"Demir","sequence":"additional","affiliation":[{"name":"Department of Neurological Surgery, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-0244-6623","authenticated-orcid":false,"given":"Gular","family":"Ismayilova","sequence":"additional","affiliation":[{"name":"Department of Neurosurgery, University of Health Sciences, Istanbul Training and Research Hospital, Istanbul 34098, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5757-4502","authenticated-orcid":false,"given":"Melih Yucel","family":"Sanlier","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Istanbul Medeniyet University, Prof. Dr. Suleyman Yalcin City Hospital, Istanbul 34722, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ugur","family":"Erginoglu","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology-Head and Neck Surgery, Basaksehir State Hospital, Istanbul 34480, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9668-7246","authenticated-orcid":false,"given":"Ahmed Rasim","family":"Bayramoglu","sequence":"additional","affiliation":[{"name":"Department of Neurological Surgery, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maryam Sabah","family":"Al-Jebur","sequence":"additional","affiliation":[{"name":"Department of Neurological Surgery, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simon Gashaw","family":"Ammanuel","sequence":"additional","affiliation":[{"name":"Department of Neurological Surgery, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3169-6832","authenticated-orcid":false,"given":"Erkin","family":"Otles","sequence":"additional","affiliation":[{"name":"BerbeeWalsh Department of Emergency Medicine, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abdullah","family":"Keles","sequence":"additional","affiliation":[{"name":"Department of Neurological Surgery, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5387-4698","authenticated-orcid":false,"given":"Ufuk","family":"Erginoglu","sequence":"additional","affiliation":[{"name":"Department of Neurological Surgery, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mustafa K.","family":"Baskaya","sequence":"additional","affiliation":[{"name":"Department of Neurological Surgery, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI 53792, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,8,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"e069155","DOI":"10.1136\/bmj-2021-069155","article-title":"Validity of Data Extraction in Evidence Synthesis Practice of Adverse Events: Reproducibility Study","volume":"377","author":"Xu","year":"2022","journal-title":"BMJ"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"818","DOI":"10.1002\/jrsm.1732","article-title":"Performance of Two Large Language Models for Data Extraction in Evidence Synthesis","volume":"15","author":"Konet","year":"2024","journal-title":"Res. Synth. Methods"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Mutinda, F.W., Liew, K., Yada, S., Wakamiya, S., and Aramaki, E. (2022). Automatic Data Extraction to Support Meta-Analysis Statistical Analysis: A Case Study on Breast Cancer. BMC Med. Inform. Decis. Mak., 22.","DOI":"10.1186\/s12911-022-01897-4"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1002\/jrsm.1710","article-title":"Data Extraction for Evidence Synthesis Using a Large Language Model: A Proof-of-Concept Study","volume":"15","author":"Gartlehner","year":"2024","journal-title":"Res. Synth. Methods"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"111746","DOI":"10.1016\/j.jclinepi.2025.111746","article-title":"Large Language Models for Conducting Systematic Reviews: On the Rise, but Not yet Ready for Use-a Scoping Review","volume":"181","author":"Lieberum","year":"2025","journal-title":"J. Clin. Epidemiol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1136\/bmjebm-2024-113199","article-title":"From Promise to Practice: Challenges and Pitfalls in the Evaluation of Large Language Models for Data Extraction in Evidence Synthesis","volume":"30","author":"Gartlehner","year":"2025","journal-title":"BMJ Evid.-Based Med."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e70058","DOI":"10.1002\/cesm.70058","article-title":"Harnessing Large-Language Models for Efficient Data Extraction in Systematic Reviews: The Role of Prompt Engineering","volume":"3","author":"Murton","year":"2025","journal-title":"Cochrane Evid. Synth. Methods"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Sulaimanov, U., Sanlier, N., Moniri, A., Demir, B., Serikkanov, Y., Bayramoglu, A.R., Al-Jebur, M.S., Uslu, I., Ozturk, O., and Nizzola, M. (2026). Are AI Neuroimaging Models Ready for Clinical Use? A Systematic Methodological Review. J. Clin. Med., 15.","DOI":"10.3390\/jcm15093441"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e200029","DOI":"10.1148\/ryai.2020200029","article-title":"Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers","volume":"2","author":"Mongan","year":"2020","journal-title":"Radiol. Artif. Intell."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e078378","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD+AI Statement: Updated Guidance for Reporting Clinical Prediction Models That Use Regression or Machine Learning Methods","volume":"385","author":"Collins","year":"2024","journal-title":"BMJ"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1763","DOI":"10.7326\/ANNALS-25-00739","article-title":"Artificial Intelligence-Assisted Data Extraction With a Large Language Model: A Study Within Reviews","volume":"178","author":"Gartlehner","year":"2025","journal-title":"Ann. Intern. Med."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Sulaimanov, U., Sanlier, N., Moniri, A., Demir, B., Serikkanov, Y., Bayramoglu, A.R., Al-Jebur, M.S., Sanlier, M.Y., Erginoglu, U., and Otles, E. (2026). Evaluating Large Language Models for Automated Evidence Synthesis in Neuroimaging AI: A Multi-Model Benchmark. J. Clin. Med., 15.","DOI":"10.3390\/jcm15114230"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"766","DOI":"10.1097\/XCS.0000000000001478","article-title":"Data Extraction and Curation from Radiology Reports for Pancreatic Cyst Surveillance Using Large Language Models","volume":"241","author":"Choubey","year":"2025","journal-title":"J. Am. Coll. Surg."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Satheakeerthy, S., Jesudason, D., Bahrami, B., Bacchi, S., Lee, Y.M., Casson, R., Sun, M., and Chan, W. (2025). Zero-Shot LLM-Based Visual Acuity Extraction: A Pilot Study. BMC Ophthalmol., 25.","DOI":"10.1186\/s12886-025-04193-7"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Motzfeldt Jensen, M., Brix Danielsen, M., Riis, J., Assifuah Kristjansen, K., Andersen, S., Okubo, Y., and J\u00f8rgensen, M.G. (2025). ChatGPT-4o Can Serve as the Second Rater for Data Extraction in Systematic Reviews. PLoS ONE, 20.","DOI":"10.1371\/journal.pone.0313401"},{"key":"ref_16","first-page":"273","article-title":"Impact of an Evidence-Based Large Language Model (LLM) Diagnostic Decision Support System: A Randomised Controlled Trial","volume":"329","author":"Ahn","year":"2025","journal-title":"Stud. Health Technol. Inform."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1097\/WNO.0000000000002145","article-title":"A Comparative Study of Large Language Models, Human Experts, and Expert-Edited Large Language Models to Neuro-Ophthalmology Questions","volume":"45","author":"Tailor","year":"2025","journal-title":"J. Neuro-Ophthalmol. Off. J. North Am. Neuro-Ophthalmol. Soc."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Fattah, F.H., Salih, A.M., Salih, A.M., Asaad, S.K., Ghafour, A.K., Bapir, R., Abdalla, B.A., Othman, S., Ahmed, S.M., and Hasan, S.J. (2025). Comparative Analysis of ChatGPT and Gemini (Bard) in Medical Inquiry: A Scoping Review. Front. Digit. Health, 7.","DOI":"10.3389\/fdgth.2025.1482712"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"43773","DOI":"10.1038\/s41598-025-27593-7","article-title":"Evaluating the Reliability of Large Language Models for Clinical Data Extraction in Bladder Cancer Prognosis","volume":"15","author":"Sun","year":"2025","journal-title":"Sci. Rep."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"105733","DOI":"10.1016\/j.jdent.2025.105733","article-title":"Chatbots for Conducting Systematic Reviews in Pediatric Dentistry","volume":"158","author":"Rokhshad","year":"2025","journal-title":"J. Dent."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"ooaf097","DOI":"10.1093\/jamiaopen\/ooaf097","article-title":"Generative Artificial Intelligence for Automated Data Extraction from Unstructured Medical Text","volume":"8","author":"Dao","year":"2025","journal-title":"JAMIA Open"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1093\/jamia\/ocaf030","article-title":"Enhancing Systematic Literature Reviews with Generative Artificial Intelligence: Development, Applications, and Performance Evaluation","volume":"32","author":"Li","year":"2025","journal-title":"J. Am. Med. Inform. Assoc. JAMIA"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Br\u00fcgge, E., Ricchizzi, S., Arenbeck, M., Keller, M.N., Schur, L., Stummer, W., Holling, M., Lu, M.H., and Darici, D. (2024). Large Language Models Improve Clinical Decision Making of Medical Students through Patient Simulation and Structured Feedback: A Randomized Controlled Trial. BMC Med. Educ., 24.","DOI":"10.1186\/s12909-024-06399-7"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1016\/j.ajem.2024.12.024","article-title":"Evaluating LLM-Based Generative AI Tools in Emergency Triage: A Comparative Study of ChatGPT Plus, Copilot Pro, and Triage Nurses","volume":"89","author":"Arslan","year":"2025","journal-title":"Am. J. Emerg. Med."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"e68097","DOI":"10.2196\/68097","article-title":"Critical Assessment of Large Language Models\u2019 (ChatGPT) Performance in Data Extraction for Systematic Reviews: Exploratory Study","volume":"4","author":"Mahmoudi","year":"2025","journal-title":"Jmir Ai"}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/15\/6005\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T12:06:17Z","timestamp":1785758777000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/15\/6005"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,2]]},"references-count":26,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2026,8]]}},"alternative-id":["jcm15156005"],"URL":"https:\/\/doi.org\/10.3390\/jcm15156005","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,2]]}}}