{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T18:15:54Z","timestamp":1783016154142,"version":"3.54.6"},"reference-count":115,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T00:00:00Z","timestamp":1782950400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Bioengineering"],"abstract":"<jats:p>(1) Background: Clinical trial data extraction from registries such as ClinicalTrials.gov remains labor-intensive and error-prone, often missing critical details hidden in unstructured protocol descriptions. Large Language Models (LLMs) offer potential to automate this process, yet systematic multi-model comparisons on real clinical trial data remain scarce. (2) Methods: Four LLMs (OpenAI o4-mini-high, Anthropic Claude-Sonnet-4, Google Gemini 2.5-Pro, and Meta Llama-4-Maverick) extracted brain stimulation parameters from 67 transcranial direct current stimulation (tDCS) trials in Parkinson\u2019s disease via a structured JSON schema. Pairwise inter-model agreement was quantified with Cohen\u2019s Kappa and percentage agreement across binary, categorical, and multi-component task tiers. (3) Results: Under exact-string matching, agreement was near-perfect for binary classifications (non-invasive classification: 100%; brain stimulation presence: 99.3%, \u03ba = 0.50) and substantial for categorical extractions (primary stimulation type: 96.4%, \u03ba = 0.70), but fell to 48.6% (\u03ba = 0.43) for complex anatomical targets. Numeric parameters revealed model-specific strengths: o4-mini-high and Claude-Sonnet-4 achieved perfect duration agreement (r = 1.000, n = 19) while Llama-4-Maverick diverged substantially (r &lt; 0.12). Validation against an expert gold standard (100% inter-annotator agreement on a 20-trial overlap) confirmed high extraction accuracy across all features (mean 93.7\u201398.9%). Crucially, the low agreement on anatomical targets proved to be an artifact of exact-string scoring: under the same semantic matching used to measure accuracy, inter-model agreement rose to 97.0%, coinciding with the 95.5% expert accuracy. Inter-model agreement therefore tracks accuracy once both are measured on a common basis. (4) Conclusions: Exact-string inter-model agreement decreases with task complexity, but this decline largely reflects interchangeable free-text wording rather than reduced accuracy. Evaluated semantically, agreement and expert accuracy are both high and closely aligned. A residual risk is not low accuracy but the rare error shared across all models, which agreement cannot detect, and which overall accuracy can itself mask when one class dominates. These findings inform hybrid human\u2013AI systematic review pipelines in which targeted expert oversight focuses on shared-error and minority-class detection.<\/jats:p>","DOI":"10.3390\/bioengineering13070773","type":"journal-article","created":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T15:52:24Z","timestamp":1783007544000},"page":"773","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["From API to Action: A Multi-Model Comparison of OpenAI, Anthropic, Google, and Meta LLMs for Clinical Trial Data Extraction"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1109-7552","authenticated-orcid":false,"given":"Richard J.","family":"Young","sequence":"first","affiliation":[{"name":"Interdisciplinary Neuroscience, University of Nevada, Las Vegas, NV 89154, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7917-5169","authenticated-orcid":false,"given":"Jorge","family":"Fonseca","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Howard R. Hughes College of Engineering, University of Nevada, Las Vegas, NV 89154, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brach","family":"Poston","sequence":"additional","affiliation":[{"name":"Interdisciplinary Neuroscience, University of Nevada, Las Vegas, NV 89154, USA"},{"name":"Department of Kinesiology and Nutrition Sciences, University of Nevada, Las Vegas, NV 89154, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"868","DOI":"10.1016\/j.ejvs.2020.01.019","article-title":"Learning from Artificial Intelligence and Big Data in Health Care","volume":"59","author":"Ricco","year":"2020","journal-title":"Eur. J. Vasc. Endovasc. Surg."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1998","DOI":"10.1056\/NEJMsr1611785","article-title":"Trial Reporting in ClinicalTrials.gov\u2014The Final Rule","volume":"375","author":"Zarin","year":"2016","journal-title":"N. Engl. J. Med."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1966","DOI":"10.1056\/NEJMsr1907644","article-title":"10-Year Update on Study Results Submitted to ClinicalTrials.gov","volume":"381","author":"Zarin","year":"2019","journal-title":"N. Engl. J. Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1186\/s13063-019-3408-2","article-title":"Some data quality issues at ClinicalTrials.gov","volume":"20","author":"Chaturvedi","year":"2019","journal-title":"Trials"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Young, R.J., Matthews, A.M., and Poston, B. (2025). Benchmarking Multiple Large Language Models for Automated Clinical Trial Data Extraction in Aging Research. Algorithms, 18.","DOI":"10.3390\/a18050296"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1002\/jrsm.1287","article-title":"Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner\u2019s guide","volume":"9","author":"Marshall","year":"2018","journal-title":"Res. Synth. Methods"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1186\/2046-4053-3-74","article-title":"Systematic review automation technologies","volume":"3","author":"Tsafnat","year":"2014","journal-title":"Syst. Rev."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1504\/IJCBDD.2013.052198","article-title":"A new iterative method to reduce workload in systematic review process","volume":"6","author":"Jonnalagadda","year":"2013","journal-title":"Int. J. Comput. Biol. Drug Des."},{"key":"ref_9","unstructured":"OpenAI (2026, June 19). Introducing OpenAI o3 and o4-mini. Available online: https:\/\/openai.com\/index\/introducing-o3-and-o4-mini\/."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1399","DOI":"10.1007\/s11604-024-01634-z","article-title":"Diagnostic performances of Claude 3 Opus and Claude 3.5 Sonnet from patient history and key images in Radiology\u2019s \u201cDiagnosis Please\u201d cases","volume":"42","author":"Kurokawa","year":"2024","journal-title":"Jpn. J. Radiol."},{"key":"ref_11","unstructured":"Google Team (2025). Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv."},{"key":"ref_12","unstructured":"(2026, June 19). Llama-Models. Official Release Announcement; No Peer-Reviewed Specification Paper Is Currently Available. Available online: https:\/\/github.com\/meta-llama\/llama-models."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"AIe2300128","DOI":"10.1056\/AIe2300128","article-title":"Why We Support and Encourage the Use of Large Language Models in NEJM AI Submissions","volume":"1","author":"Koller","year":"2024","journal-title":"NEJM AI"},{"key":"ref_14","first-page":"e35179","article-title":"Artificial Hallucinations in ChatGPT: Implications in Scientific Writing","volume":"15","author":"Alkaissi","year":"2023","journal-title":"Cureus"},{"key":"ref_15","unstructured":"Tonmoy, S.M.T.I., Zaman, S.M.M., Jain, V., Rani, A., Rawte, V., Chadha, A., and Das, A. (2024). A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1002\/jrsm.1710","article-title":"Data Extraction for Evidence Synthesis Using a Large Language Model: A Proof-of-Concept Study","volume":"15","author":"Gartlehner","year":"2024","journal-title":"Res. Synth. Methods"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"e12239","DOI":"10.2196\/12239","article-title":"Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Review","volume":"7","author":"Sheikhalishahi","year":"2019","journal-title":"JMIR Med. Inform."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Stuhlmiller, T.J., Rabe, A., Rapp, J., Awawda, A., Kouser, H., Lui, K., Salamon, H., Chuyka, D., Mahoney, W., and Furgason, J.M. (2025). A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models. medRxiv.","DOI":"10.1101\/2025.02.25.25322898"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"818","DOI":"10.1002\/jrsm.1732","article-title":"Performance of two large language models for data extraction in evidence synthesis","volume":"15","author":"Konet","year":"2024","journal-title":"Res. Synth. Methods"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"e101139","DOI":"10.1136\/bmjhci-2024-101139","article-title":"Large language models for data extraction from unstructured and semi-structured electronic health records: A multiple model performance evaluation","volume":"32","author":"Ntinopoulos","year":"2025","journal-title":"BMJ Health Care Inform."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"638","DOI":"10.1093\/jamia\/ocae325","article-title":"Collaborative large language models for automated data extraction in living systematic reviews","volume":"32","author":"Khan","year":"2025","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1038\/s41746-025-01457-w","article-title":"Language models for data extraction and risk of bias assessment in complementary medicine","volume":"8","author":"Lai","year":"2025","journal-title":"npj Digit. Med."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2722","DOI":"10.1097\/JS9.0000000000002215","article-title":"AI-driven evidence synthesis: Data extraction of randomized controlled trials with large language models","volume":"111","author":"Liu","year":"2025","journal-title":"Int. J. Surg."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1017\/rsm.2025.10054","article-title":"Beyond Human Gold Standards: A Multimodel Framework for Automated Abstract Classification and Information Extraction","volume":"17","author":"Courvoisier","year":"2025","journal-title":"Res. Synth. Methods"},{"key":"ref_25","first-page":"ztaf127","article-title":"Large Language Models for Structured Cardiovascular Data Extraction: A Foundation for Scalable Research and Clinical Applications","volume":"7","author":"Atsma","year":"2025","journal-title":"Eur. Heart J. Digit. Health"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Poser, P.L., Klimas, R., Luerweg, J., Reuter, E., Hanefeld, C., Gold, R., Salmen, A., and Motte, J. (2026). Improving Reliability and Accuracy of Structured Data Extraction Using a Consensus Large-Language Model Approach: A Use Case Description in Multiple Sclerosis. Front. Artif. Intell., 9.","DOI":"10.3389\/frai.2026.1658575"},{"key":"ref_27","unstructured":"Polzak, C., Lozano, A., Sun, M.W., Burgess, J., Zhang, Y., Wu, K., and Yeung-Levy, S. (2025). Can Large Language Models Match the Conclusions of Systematic Reviews?. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"15606","DOI":"10.1038\/s41598-026-45326-2","article-title":"Benchmarking Agreement Between Large Language Models and Published Clinical Trial Conclusions Across Four Artificial Intelligence Platforms","volume":"16","author":"Mao","year":"2026","journal-title":"Sci. Rep."},{"key":"ref_29","unstructured":"Murin, M. (2026). Measuring the Sensitivity of LLM-Based Structured Extraction to Prompt, Model, and Schema Choices in Clinical Discharge Summaries. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"102192","DOI":"10.1016\/j.smrv.2025.102192","article-title":"Accuracy of Large Language Models in Data Extraction from Randomized Controlled Trials in Sleep Medicine: A Proof-of-Concept Study","volume":"84","author":"Peng","year":"2025","journal-title":"Sleep Med. Rev."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Passweg, L.P., Schwenke, J.M., Sch\u00f6nenberger, C.M., Locher, F., Picker, J., Dieterle, M., Thiele, B., Hasler, D., Danelli, A., and Schmitt, A.M. (2025). Data Extraction from Oncology Imaging Reports by Large Language Models: A Comparative Accuracy Study. medRxiv.","DOI":"10.64898\/2025.12.30.25343206"},{"key":"ref_32","unstructured":"Jain, N., Suh, H., Adeyinka, S., Roseman, L., and Allsop, A. (2025). Multi-LLM Thematic Analysis with Dual Reliability Metrics: Combining Cohen\u2019s Kappa and Semantic Similarity for Qualitative Research Validation. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Haldar, R., and Hockenmaier, J. (2025). Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks. arXiv.","DOI":"10.18653\/v1\/2025.findings-emnlp.1361"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"James, J. (2026, January 11\u201316). Counting on Consensus: Selecting the Right Inter-Annotator Agreement Metric for NLP Annotation and Evaluation. Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), Palma, Spain.","DOI":"10.63317\/2wuxteb6uvcj"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Bukhari, K., Rodriguez-Monguio, R., Lopez-Bermudez, B., Yamaki, J., Brown, L.M., Beuttler, R., Ong, J.C.L., and Seoane-Vazquez, E. (2025). When AI Meets the FDA: An Evaluation of Large Language Models Performance in Regulatory and Clinical Trial Data Extraction, Synthesis, and Analysis. medRxiv.","DOI":"10.64898\/2025.12.22.25342875"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"43773","DOI":"10.1038\/s41598-025-27593-7","article-title":"Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis","volume":"15","author":"Sun","year":"2025","journal-title":"Sci. Rep."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"e73486","DOI":"10.2196\/73486","article-title":"Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study","volume":"27","author":"Hsu","year":"2025","journal-title":"J. Med. Internet Res."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1038\/s41537-025-00601-5","article-title":"Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: A proof of concept study","volume":"11","author":"Volkmer","year":"2025","journal-title":"Schizophr"},{"key":"ref_39","unstructured":"Mahbub, M., Dams, G.M., Arnold, J., Rizy, C., Srinivasan, S., Fielstein, E.M., Aghevli, M.A., Craig, K.L., Oliva, E.M., and Erdos, J. (2026). A Multi-Stage Validation Framework for Trustworthy Large-scale Clinical Information Extraction using Large Language Models. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Borse, N.S., Chatta Subramaniam, R., and Rebello, N.S. (2025). Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis. Physics Education Research Conference Proceedings, The American Association of Physics Teachers.","DOI":"10.1119\/perc.2025.pr.Borse"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1016\/j.clinph.2017.01.004","article-title":"Effects of tDCS on motor learning and memory formation: A consensus and critical position paper","volume":"128","author":"Buch","year":"2017","journal-title":"Clin. Neurophysiol."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Pantovic, M., Albuquerque, L.L.d., Mastrantonio, S., Pomerantz, A.S., Wilkins, E.W., Riley, Z.A., Guadagnoli, M.A., and Poston, B. (2023). Transcranial Direct Current Stimulation of Primary Motor Cortex over Multiple Days Improves Motor Learning of a Complex Overhand Throwing Task. Brain Sci., 13.","DOI":"10.3390\/brainsci13101441"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1693","DOI":"10.1002\/mds.21012","article-title":"Noninvasive cortical stimulation with transcranial direct current stimulation in Parkinson\u2019s disease","volume":"21","author":"Fregni","year":"2006","journal-title":"Mov. Disord."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1064","DOI":"10.1002\/mds.25897","article-title":"Transcranial direct current stimulation for treatment of freezing of gait: A cross-over study","volume":"29","author":"Valentino","year":"2014","journal-title":"Mov. Disord."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Lee, S.a., and Kim, M.K. (2021). The Effect of Transcranial Direct Current Stimulation Combined with Visual Cueing Training on Motor Function, Balance, and Gait Ability of Patients with Parkinson\u2019s Disease. Medicina, 57.","DOI":"10.3390\/medicina57111146"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1097\/TGR.0000000000000203","article-title":"Effects of Acute Transcranial Direct Current Stimulation on Gait Kinematics of Individuals with Parkinson Disease","volume":"34","author":"Lemos","year":"2018","journal-title":"Top. Geriatr. Rehabil."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1972","DOI":"10.1002\/mds.27533","article-title":"Anodal tDCS over prefrontal cortex improves dual-task walking in Parkinsonian patients with freezing","volume":"33","author":"Putzolu","year":"2018","journal-title":"Mov. Disord."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Wong, P.L., Yang, Y.R., Huang, S.F., Fuh, J.L., Chiang, H.L., and Wang, R.Y. (2022). Transcranial Direct Current Stimulation on Different Targets to Modulate Cortical Activity and Dual-Task Walking in Individuals with Parkinson\u2019s Disease: A Double Blinded Randomized Controlled Trial. Front. Aging Neurosci., 14.","DOI":"10.3389\/fnagi.2022.807151"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Lima de Albuquerque, L., Pantovic, M., Clingo, M., Fischer, K., Jalene, S., Landers, M., Mari, Z., and Poston, B. (2020). An Acute Application of Cerebellar Transcranial Direct Current Stimulation Does Not Improve Motor Performance in Parkinson\u2019s Disease. Brain Sci., 10.","DOI":"10.3390\/brainsci10100735"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"de Albuquerque, L.L., Pantovic, M., Clingo, M., Fischer, K., Jalene, S., Landers, M., Mari, Z., and Poston, B. (2023). A Single Application of Cerebellar Transcranial Direct Current Stimulation Fails to Enhance Motor Skill Acquisition in Parkinson\u2019s Disease: A Pilot Study. Biomedicines, 11.","DOI":"10.3390\/biomedicines11082219"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1038\/s41531-024-00821-z","article-title":"Transcranial direct current stimulation for Parkinson\u2019s disease: Systematic review and meta-analysis of motor and cognitive effects","volume":"10","author":"Duan","year":"2024","journal-title":"npj Park. Dis."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1177\/15459683241310984","article-title":"Exploring Non-invasive Brain Stimulation Effects on Physical Outcomes in People with Parkinson\u2019s Disease: An Umbrella Evidence Mapping Review with Meta-analyses","volume":"39","author":"Harris","year":"2025","journal-title":"Neurorehabil. Neural Repair"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"548","DOI":"10.1001\/jama.2019.22360","article-title":"Diagnosis and Treatment of Parkinson Disease: A Review","volume":"323","author":"Armstrong","year":"2020","journal-title":"JAMA"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"de Oliveira, P.C.A., de Ara\u00fajo, T.A.B., de Silva Machado, D.G., Rodrigues, A.C., Bikson, M., Andrade, S.M., Okano, A.H., Simplicio, H., Pegado, R., and Morya, E. (2021). Transcranial Direct Current Stimulation on Parkinson\u2019s Disease: Systematic Review and Meta-Analysis. Front. Neurol., 12.","DOI":"10.3389\/fneur.2021.794784"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2018, January 1). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. Proceedings of the BlackboxNLP@EMNLP, Brussels, Belgium.","DOI":"10.18653\/v1\/W18-5446"},{"key":"ref_56","unstructured":"Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2019). SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. arXiv."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., and Jiang, Z. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv.","DOI":"10.52202\/079017-3018"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Kweon, S., Kim, J., Kwak, H., Cha, D., Yoon, H., Kim, K., Yang, J., Won, S., and Choi, E. (2024, January 10\u201315). EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries. Proceedings of the Advances in Neural Information Processing Systems 37, Vancouver, BC, Canada.","DOI":"10.52202\/079017-3958"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1145\/3308774.3308807","article-title":"The Knowledge and Language Gap in Medical Information Seeking","volume":"52","author":"Soldaini","year":"2019","journal-title":"ACM SIGIR Forum"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1186\/s40537-022-00600-8","article-title":"Transforming OpenAPI Specification 3.0 documents into RDF-based semantic web services","volume":"9","author":"Muhamad","year":"2022","journal-title":"J. Big Data"},{"key":"ref_61","first-page":"2079","article-title":"Streamlining Access to ClinicalTrials.gov APIs Retrieval of Bulk Data Automation","volume":"4","author":"S","year":"2024","journal-title":"Int. J. Progress. Res. Eng. Manag. Sci."},{"key":"ref_62","unstructured":"Rey, A., Freitag, M., and Neumann, T. (2023). Seamless Integration of Parquet Files into Data Processing. Datenbanksysteme fur Business, Technologie und Web, Gesellschaft f\u00fcr Informatik e.V."},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Nelli, F. (2015). The Pandas Library\u2014An Introduction. Python Data Analytics, Apress.","DOI":"10.1007\/978-1-4842-0958-5_4"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1109\/MCSE.2011.37","article-title":"The NumPy Array: A Structure for Efficient Numerical Computation","volume":"13","author":"Colbert","year":"2011","journal-title":"Comput. Sci. Eng."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Khan, M.A., Ayub, U., Naqvi, S.A.A., Khakwani, K.Z.R., Sipra, Z.b.R., Raina, A., Zou, S., He, H., Hossein, S.A., and Hasan, B. (2024). Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews. medRxiv.","DOI":"10.1101\/2024.09.20.24314108"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Patel, D., Timsina, P., Raut, G., Freeman, R., Levin, M.A., Nadkarni, G.N., Glicksberg, B.S., and Klang, E. (2024). Exploring Temperature Effects on Large Language Models Across Various Clinical Tasks. medRxiv.","DOI":"10.1101\/2024.07.22.24310824"},{"key":"ref_67","unstructured":"Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., and Vaughan, A. (2024). The Llama 3 Herd of Models. arXiv."},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. (2021). What Makes Good In-Context Examples for GPT-3?. DeeLIO Workshop at ACL, Association for Computational Linguistics.","DOI":"10.18653\/v1\/2022.deelio-1.10"},{"key":"ref_69","first-page":"305","article-title":"Prompt Engineering and Provision of Context in Domain Specific Use of GPT","volume":"379","author":"Ribary","year":"2023","journal-title":"Front. Artif. Intell. Appl."},{"key":"ref_70","unstructured":"Yousuf, R.B., Defelice, N., Sharma, M., Xu, S., and Ramakrishnan, N. (2024). LLM Augmentations to support Analytical Reasoning over Multiple Documents. Proceedings of the 2024 IEEE International Conference on Big Data (BigData), Washington, DC, USA, 15\u201318 December 2024, IEEE."},{"key":"ref_71","unstructured":"Mondorf, P., and Plank, B. (2024). Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models\u2014A Survey. arXiv."},{"key":"ref_72","unstructured":"Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., and Shi, D. (2024). Evaluating large language models in medical applications: A survey. arXiv."},{"key":"ref_73","first-page":"1877","article-title":"Language Models are Few-Shot Learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"101260","DOI":"10.1016\/j.patter.2025.101260","article-title":"Unleashing the potential of prompt engineering for large language models","volume":"6","author":"Chen","year":"2025","journal-title":"Patterns"},{"key":"ref_75","doi-asserted-by":"crossref","first-page":"9","DOI":"10.15446\/dyna.v90n230.111700","article-title":"Prompt Engineering: A methodology for optimizing interactions with AI-Language Models in the field of engineering","volume":"90","year":"2023","journal-title":"Dyna"},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_77","doi-asserted-by":"crossref","unstructured":"Wilkins, E.W., Young, R.J., Houston, D., Kawana, E., Lopez Mora, E., Sunkara, M.S., Riley, Z.A., and Poston, B. (2024). Non-Dominant Hemisphere Excitability Is Unaffected during and after Transcranial Direct Current Stimulation of the Dominant Hemisphere. Brain Sci., 14.","DOI":"10.3390\/brainsci14070694"},{"key":"ref_78","doi-asserted-by":"crossref","unstructured":"Wilkins, E.W., Young, R.J., Davidson, R., Krider, R., Alhwayek, G., Park, J.A., Parikh, A.C., Riley, Z.A., and Poston, B. (2025). The Influence of Transcranial Alternating Current Stimulation on the Excitability of the Unstimulated Contralateral Primary Motor Cortex. Brain Sci., 15.","DOI":"10.3390\/brainsci15050512"},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1016\/0895-4356(90)90158-L","article-title":"High agreement but low kappa: I. The problems of two paradoxes","volume":"43","author":"Feinstein","year":"1990","journal-title":"J. Clin. Epidemiol."},{"key":"ref_80","doi-asserted-by":"crossref","unstructured":"Yadav, T., Tekale, A., Chong, J., and Masum, M. (2026). Understanding Tradeoffs in Clinical Text Extraction: Prompting, Retrieval-Augmented Generation, and Supervised Learning on Electronic Health Records. Algorithms, 19.","DOI":"10.3390\/a19030215"},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"e012545","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"Borah","year":"2017","journal-title":"BMJ Open"},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Wittlinger, S., Meerjansen, J., Wolf, F., Wiest, I.C., Ebert, M.P., Siegel, F., and Belle, S. (2026). Multi-LLM Disagreement as a Scalable Detector of Human Annotation Errors in Structured Data from Clinical Free-Text. medRxiv.","DOI":"10.64898\/2026.05.04.26352392"},{"key":"ref_83","unstructured":"Zhang, B., Liu, Z., Cherry, C., and Firat, O. (2024, January 7\u201311). When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method. Proceedings of the International Conference on Learning Representations, Vienna, Austria."},{"key":"ref_84","doi-asserted-by":"crossref","unstructured":"Beaulieu-Jones, B., and Brenner, S. (2024, January 3\u20137). Leveraging Foundational Models in Computational Biology: Validation, Understanding, and Innovation. Proceedings of the Pacific Symposium on Biocomputing, Kohala Coast, HI, USA.","DOI":"10.1142\/9789819807024_0051"},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Fleurence, R.L., Bian, J., Wang, X., Xu, H., Dawoud, D., Higashi, M., and Chhatwal, J. (2024). Generative AI for Health Technology Assessment: Opportunities, Challenges, and Policy Considerations. arXiv.","DOI":"10.1016\/j.jval.2024.10.3846"},{"key":"ref_86","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1208\/s12248-021-00644-3","article-title":"Machine Learning and Artificial Intelligence in Pharmaceutical Research and Development: A Review","volume":"24","author":"Kolluri","year":"2022","journal-title":"AAPS J."},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"1","DOI":"10.4103\/picr.PICR_312_20","article-title":"Artificial intelligence in managing clinical trial design and conduct: Man and machine still on the learning curve?","volume":"12","author":"Bhatt","year":"2021","journal-title":"Perspect. Clin. Res."},{"key":"ref_88","doi-asserted-by":"crossref","unstructured":"Tao, C., Fan, X., and Yang, Y. (2024). Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation. arXiv.","DOI":"10.1109\/CAIT64506.2024.10962957"},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"100055","DOI":"10.1016\/j.digbus.2023.100055","article-title":"Exploring API-driven business models: Lessons learned from Amadeus\u2019s digital transformation","volume":"3","author":"Heshmatisafa","year":"2023","journal-title":"Digit. Bus."},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"15090","DOI":"10.46298\/jdmdh.15090","article-title":"Out of Context! Managing the Limitations of Context Windows in ChatGPT-4o Text Analyses","volume":"NLP4DH","author":"Mervaala","year":"2025","journal-title":"J. Data Min. Digit. Humanit."},{"key":"ref_91","unstructured":"Gemini Team, Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., and Wang, S. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv."},{"key":"ref_92","doi-asserted-by":"crossref","unstructured":"Shool, S., Adimi, S., Saboori Amleshi, R., Bitaraf, E., Golpira, R., and Tara, M. (2025). A systematic review of large language model (LLM) evaluations in clinical medicine. BMC Med. Inform. Decis. Mak., 25.","DOI":"10.1186\/s12911-025-02954-4"},{"key":"ref_93","doi-asserted-by":"crossref","unstructured":"Sun, Z., Zhang, R., Doi, S.A., Furuya-Kanamori, L., Yu, T., Lin, L., and Xu, C. (2024). How good are large language models for automated data extraction from randomized trials?. medRxiv.","DOI":"10.1101\/2024.02.20.24303083"},{"key":"ref_94","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1038\/s41746-024-01233-2","article-title":"Privacy-preserving large language models for structured medical information retrieval","volume":"7","author":"Wiest","year":"2024","journal-title":"npj Digit. Med."},{"key":"ref_95","doi-asserted-by":"crossref","unstructured":"Wiest, I.C., Wolf, F., Le\u00dfmann, M.E., van Treeck, M., Ferber, D., Zhu, J., Boehme, H., Bressem, K.K., Ulrich, H., and Ebert, M.P. (2024). LLM-AIx: An open source pipeline for Information Extraction from unstructured medical text based on privacy preserving Large Language Models. medRxiv.","DOI":"10.1101\/2024.09.02.24312917"},{"key":"ref_96","doi-asserted-by":"crossref","first-page":"e54580","DOI":"10.2196\/54580","article-title":"An Entity Extraction Pipeline for Medical Text Records Using Large Language Models: Analytical Study","volume":"26","author":"Wang","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"ref_97","unstructured":"Li, D., Kadav, A., Gao, A., Li, R., and Bourgon, R. (2024). Automated Clinical Data Extraction with Knowledge Conditioned LLMs. arXiv."},{"key":"ref_98","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1080\/21642583.2022.2074169","article-title":"Recent advances on cooperative control of heterogeneous multi-agent systems subject to constraints: A survey","volume":"10","author":"Bao","year":"2022","journal-title":"Syst. Sci. Control Eng."},{"key":"ref_99","doi-asserted-by":"crossref","unstructured":"Yue, L., Xing, S., Chen, J., and Fu, T. (2024). ClinicalAgent: Clinical Trial Multi-Agent System with Large Language Model-based Reasoning. Proceedings of the 15th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, Shenzhen, China, 22\u201325 November 2024, Association for Computing Machinery.","DOI":"10.1145\/3698587.3701359"},{"key":"ref_100","doi-asserted-by":"crossref","unstructured":"Zuo, K., Jiang, Y., Mo, F., and Lio, P. (2024). KG4Diagnosis: A Hierarchical Multi-Agent LLM Framework with Knowledge Graph Enhancement for Medical Diagnosis. arXiv.","DOI":"10.1101\/2025.06.03.657521"},{"key":"ref_101","unstructured":"Trirat, P., Jeong, W., and Hwang, S.J. (2024). AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML. arXiv."},{"key":"ref_102","unstructured":"Schroeder, N.L., Jaldi, C.D., and Zhang, S. (2025). Large Language Models with Human-In-The-Loop Validation for Systematic Review Data Extraction. arXiv."},{"key":"ref_103","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1038\/s41746-025-01840-7","article-title":"Accelerating clinical evidence synthesis with large language models","volume":"8","author":"Wang","year":"2025","journal-title":"npj Digit. Med."},{"key":"ref_104","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1017\/rsm.2025.10030","article-title":"Automating the data extraction process for systematic reviews using GPT-4o and o3","volume":"17","author":"Kataoka","year":"2025","journal-title":"Res. Synth. Methods"},{"key":"ref_105","doi-asserted-by":"crossref","unstructured":"Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., and Gong, L.X. (2025). Automation of Systematic Reviews with Large Language Models (otto-SR). medRxiv.","DOI":"10.1101\/2025.06.13.25329541"},{"key":"ref_106","doi-asserted-by":"crossref","first-page":"1014","DOI":"10.1016\/j.adaj.2025.08.011","article-title":"Comparing the performance of ChatGPT, DeepSeek, and Gemini in systematic and umbrella review tasks over time","volume":"156","author":"Emami","year":"2025","journal-title":"J. Am. Dent. Assoc."},{"key":"ref_107","doi-asserted-by":"crossref","first-page":"e70036","DOI":"10.1002\/cesm.70036","article-title":"Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews","volume":"3","author":"Marcussen","year":"2025","journal-title":"Cochrane Evid. Synth. Methods"},{"key":"ref_108","doi-asserted-by":"crossref","first-page":"112221","DOI":"10.1016\/j.jclinepi.2026.112221","article-title":"Large language models show promising performance for some systematic review tasks but call for cautious implementation: A systematic review","volume":"194","author":"Laignelot","year":"2026","journal-title":"J. Clin. Epidemiol."},{"key":"ref_109","doi-asserted-by":"crossref","first-page":"111746","DOI":"10.1016\/j.jclinepi.2025.111746","article-title":"Large language models for conducting systematic reviews: On the rise, but not yet ready for use\u2014A scoping review","volume":"181","author":"Lieberum","year":"2025","journal-title":"J. Clin. Epidemiol."},{"key":"ref_110","doi-asserted-by":"crossref","first-page":"106238","DOI":"10.1016\/j.ebiom.2026.106238","article-title":"Large language models for risk-of-bias assessment in randomised clinical trials\u2014A comparative validation study","volume":"126","author":"Nyrhi","year":"2026","journal-title":"eBioMedicine"},{"key":"ref_111","doi-asserted-by":"crossref","first-page":"105036","DOI":"10.1016\/j.jbi.2026.105036","article-title":"AI-assisted protocol information extraction for improved accuracy and efficiency in clinical trial workflows","volume":"179","author":"Babaeipour","year":"2026","journal-title":"J. Biomed. Inform."},{"key":"ref_112","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1002\/cpt.70096","article-title":"Large Language Models for Clinical Trial Protocol Assessments","volume":"119","author":"Shin","year":"2025","journal-title":"Clin. Pharmacol. Ther."},{"key":"ref_113","doi-asserted-by":"crossref","unstructured":"Tang, T., Li, A., Tan, X., Ji, Q., Si, L., and Bao, L. (2025). Bridging Data Gaps in Oncology: Large Language Models and Collaborative Filtering for Cancer Treatment Recommendations. medRxiv.","DOI":"10.1101\/2025.04.07.25325243"},{"key":"ref_114","doi-asserted-by":"crossref","unstructured":"Achterberg, J., van Dijk, B., Meng, J., Islam, S.U., Epiphaniou, G., Maple, C., Ding, X., Arvanitis, T.N., Brouwer, S., and Haas, M. (2026). OpenExtract: Automated Data Extraction for Systematic Reviews in Health. Studies in Health Technology and Informatics, IOP Publishing.","DOI":"10.3233\/SHTI260316"},{"key":"ref_115","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1007\/s10916-026-02353-w","article-title":"Operationalizing Large Language Models for Clinical Research Data Extraction: Methods, Quality Control, and Governance","volume":"50","author":"Chen","year":"2026","journal-title":"J. Med. Syst."}],"container-title":["Bioengineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5354\/13\/7\/773\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T17:18:29Z","timestamp":1783012709000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5354\/13\/7\/773"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,2]]},"references-count":115,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["bioengineering13070773"],"URL":"https:\/\/doi.org\/10.3390\/bioengineering13070773","relation":{},"ISSN":["2306-5354"],"issn-type":[{"value":"2306-5354","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,2]]}}}