{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T02:05:21Z","timestamp":1783044321868,"version":"3.54.6"},"reference-count":12,"publisher":"American Medical Association (AMA)","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Ophthalmol"],"published-print":{"date-parts":[[2025,6,1]]},"abstract":"<jats:sec id=\"ab-ebr250001-4\"><jats:title>Importance<\/jats:title><jats:p>Recently, it was proved that the large language model Generative Pre-trained Transformer 4 (GPT-4; OpenAI) can fabricate synthetic medical datasets designed to support false scientific evidence.<\/jats:p><\/jats:sec><jats:sec id=\"ab-ebr250001-5\"><jats:title>Objective<\/jats:title><jats:p>To uncover statistical patterns that may suggest fabrication in datasets produced by large language models and to improve these synthetic datasets by attempting to remove detectable marks of nonauthenticity, investigating the limits of generative artificial intelligence.<\/jats:p><\/jats:sec><jats:sec id=\"ab-ebr250001-6\"><jats:title>Design, Setting, and Participants<\/jats:title><jats:p>In this quality improvement study, synthetic datasets were produced for 3 fictional clinical studies designed to compare the outcomes of 2 alternative treatments for specific ocular diseases. Synthetic datasets were produced using the default GPT-4o model and a custom GPT. Data fabrication was conducted in November 2024.<\/jats:p><\/jats:sec><jats:sec id=\"ab-ebr250001-7\"><jats:title>Exposure<\/jats:title><jats:p>Prompts were submitted to GPT-4o to produce 12 \u201cunrefined\u201d datasets, which underwent forensic examination. Based on the outcomes of this analysis, the custom GPT Synthetic Data Creator was built with detailed instructions to generate 12 \u201crefined\u201d datasets designed to evade authenticity checks. Then, forensic analysis was repeated on these enhanced datasets.<\/jats:p><\/jats:sec><jats:sec id=\"ab-ebr250001-8\"><jats:title>Main Outcomes and Measures<\/jats:title><jats:p>Forensic analysis was performed to identify statistical anomalies in demographic data, distribution uniformity, and repetitive patterns of last digits, as well as linear correlations, distribution shape, and outliers of study variables. Datasets were also qualitatively assessed for the presence of unrealistic clinical records.<\/jats:p><\/jats:sec><jats:sec id=\"ab-ebr250001-9\"><jats:title>Results<\/jats:title><jats:p>Forensic analysis identified 103 fabrication marks among 304 tests (33.9%) in unrefined datasets. Notable flaws included mismatch between patient names and gender (n\u2009=\u200912), baseline visits occurring during weekends (n\u2009=\u200912), age calculation errors (n\u2009=\u20099), lack of uniformity (n\u2009=\u20094), and repetitive numerical patterns in last digits (n\u2009=\u20097). Very weak correlations (<jats:italic>r<\/jats:italic>\u2009&amp;amp;lt;\u20090.1) were observed between study variables (n\u2009=\u200912). In addition, variables showed a suspicious distribution shape (n\u2009=\u20096). Compared with unrefined datasets, refined ones showed 29.3% (95% CI, 23.5%-35.1%) fewer signs of fabrication (14 of 304 statistical tests performed [4.6%]). Four refined datasets passed forensic analysis as authentic; however, suspicious distribution shape or other issues were found in others.<\/jats:p><\/jats:sec><jats:sec id=\"ab-ebr250001-10\"><jats:title>Conclusions and Relevance<\/jats:title><jats:p>Sufficiently sophisticated custom GPTs can perform complex statistical tasks and may be abused to fabricate synthetic datasets that can pass forensic analysis as authentic.<\/jats:p><\/jats:sec>","DOI":"10.1001\/jamaophthalmol.2025.0834","type":"journal-article","created":{"date-parts":[[2025,4,24]],"date-time":"2025-04-24T15:31:17Z","timestamp":1745508677000},"page":"517","source":"Crossref","is-referenced-by-count":6,"title":["Exploring Detection Methods for Synthetic Medical Datasets Created With a Large Language Model"],"prefix":"10.1001","volume":"143","author":[{"given":"Andrea","family":"Taloni","sequence":"first","affiliation":[{"name":"Department of Translational Medicine, University of Ferrara, Ferrara, Italy"},{"name":"Department of Ophthalmology, Ospedali Privati Forl\u00ec \u201cVilla Igea,\u201d Forl\u00ec, Italy"},{"name":"Istituto Internazionale per la Ricerca e Formazione in Oftalmologia, Forl\u00ec, Italy"},{"name":"Department of Ophthalmology, \u201cMagna Graecia\u201d University of Catanzaro, Catanzaro, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giulia","family":"Coco","sequence":"additional","affiliation":[{"name":"Department of Clinical Sciences and Translational Medicine, University of Rome Tor Vergata, Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Pellegrini","sequence":"additional","affiliation":[{"name":"Department of Translational Medicine, University of Ferrara, Ferrara, Italy"},{"name":"Department of Ophthalmology, Ospedali Privati Forl\u00ec \u201cVilla Igea,\u201d Forl\u00ec, Italy"},{"name":"Istituto Internazionale per la Ricerca e Formazione in Oftalmologia, Forl\u00ec, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthias","family":"Wjst","sequence":"additional","affiliation":[{"name":"Institut f\u00fcr KI und Informatik in der Medizin, Lehrstuhl f\u00fcr Medizinische Informatik, Klinikum rechts der Isar, M\u00fcnchen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Niccol\u00f2","family":"Salgari","sequence":"additional","affiliation":[{"name":"Department of Translational Medicine, University of Ferrara, Ferrara, Italy"},{"name":"Department of Ophthalmology, Ospedali Privati Forl\u00ec \u201cVilla Igea,\u201d Forl\u00ec, Italy"},{"name":"Istituto Internazionale per la Ricerca e Formazione in Oftalmologia, Forl\u00ec, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giovanna","family":"Carnovale-Scalzo","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, \u201cMagna Graecia\u201d University of Catanzaro, Catanzaro, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vincenzo","family":"Scorcia","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, \u201cMagna Graecia\u201d University of Catanzaro, Catanzaro, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Massimo","family":"Busin","sequence":"additional","affiliation":[{"name":"Department of Translational Medicine, University of Ferrara, Ferrara, Italy"},{"name":"Department of Ophthalmology, Ospedali Privati Forl\u00ec \u201cVilla Igea,\u201d Forl\u00ec, Italy"},{"name":"Istituto Internazionale per la Ricerca e Formazione in Oftalmologia, Forl\u00ec, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giuseppe","family":"Giannaccare","sequence":"additional","affiliation":[{"name":"Eye Clinic, Department of Surgical Sciences, University of Cagliari, Cagliari, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","reference":[{"issue":"1","key":"ebr250001r1","doi-asserted-by":"publisher","first-page":"18562","DOI":"10.1038\/s41598-023-45837-2","article-title":"Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology.","volume":"13","author":"Taloni","year":"2023","journal-title":"Sci Rep"},{"issue":"6","key":"ebr250001r2","doi-asserted-by":"publisher","first-page":"887","DOI":"10.3390\/healthcare11060887","article-title":"ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns.","volume":"11","author":"Sallam","year":"2023","journal-title":"Healthcare (Basel)"},{"issue":"1","key":"ebr250001r3","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1002\/leap.1578","article-title":"Risks of abuse of large language models, like ChatGPT, in scientific publishing: authorship, predatory publishing, and paper mills.","volume":"37","author":"Kendall","year":"2024","journal-title":"Learn Publ"},{"issue":"2","key":"ebr250001r4","doi-asserted-by":"publisher","first-page":"397","DOI":"10.1038\/s41433-023-02678-7","article-title":"Modern threats in academia: evaluating plagiarism and artificial intelligence detection scores of ChatGPT.","volume":"38","author":"Taloni","year":"2024","journal-title":"Eye (Lond)"},{"key":"ebr250001r5","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2023.100546","article-title":"Synthetic data generation: state of the art in health care domain.","volume":"48","author":"Murtaza","year":"2023","journal-title":"Comput Sci Rev"},{"issue":"4","key":"ebr250001r6","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooae114","article-title":"Large language models and synthetic health data: progress and prospects.","volume":"7","author":"Smolyak","year":"2024","journal-title":"JAMIA Open"},{"issue":"12","key":"ebr250001r7","doi-asserted-by":"publisher","first-page":"1174","DOI":"10.1001\/jamaophthalmol.2023.5162","article-title":"Large language model advanced data analysis abuse to create a fake data set in medical research.","volume":"141","author":"Taloni","year":"2023","journal-title":"JAMA Ophthalmol"},{"issue":"7989","key":"ebr250001r8","doi-asserted-by":"publisher","first-page":"895","DOI":"10.1038\/d41586-023-03635-w","article-title":"ChatGPT generates fake data set to support scientific hypothesis.","volume":"623","author":"Naddaf","year":"2023","journal-title":"Nature"},{"issue":"1","key":"ebr250001r9","doi-asserted-by":"publisher","first-page":"370","DOI":"10.1016\/j.fmre.2023.11.011","article-title":"Last digit tendency: lucky numbers and psychological rounding in mobile transactions.","volume":"5","author":"Wang","year":"2025","journal-title":"Fundam Res (Beijing)"},{"issue":"3","key":"ebr250001r10","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1080\/02664760601004940","article-title":"Not the first digit! using Benford\u2019s law to detect fraudulent scientific data.","volume":"34","author":"Diekmann","year":"2007","journal-title":"J Appl Stat"},{"issue":"1","key":"ebr250001r11","doi-asserted-by":"publisher","first-page":"566","DOI":"10.1186\/s13063-019-3671-2","article-title":"Sample size estimation for randomised controlled trials with repeated assessment of patient-reported outcomes: what correlation between baseline and follow-up outcomes should we assume?","volume":"20","author":"Walters","year":"2019","journal-title":"Trials"},{"issue":"16","key":"ebr250001r12","doi-asserted-by":"publisher","first-page":"3477","DOI":"10.1038\/s41433-023-02536-6","article-title":"Impact of topographic localization of corneal ectasia on the outcomes of deep anterior lamellar keratoplasty employing large (9 mm) versus conventional diameter (8 mm) grafts.","volume":"37","author":"Lucisano","year":"2023","journal-title":"Eye (Lond)"}],"container-title":["JAMA Ophthalmology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamaophthalmology\/articlepdf\/2832967\/jamaophthalmology_taloni_2025_br_250001_1749757345.29279.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,20]],"date-time":"2025-06-20T15:00:42Z","timestamp":1750431642000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamaophthalmology\/fullarticle\/2832967"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,1]]},"references-count":12,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,6,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamaophthalmol.2025.0834","relation":{},"ISSN":["2168-6165"],"issn-type":[{"value":"2168-6165","type":"print"}],"subject":[],"published":{"date-parts":[[2025,6,1]]}}}