{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T15:20:16Z","timestamp":1784820016159,"version":"3.55.0"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T00:00:00Z","timestamp":1682467200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T00:00:00Z","timestamp":1682467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Large language models such as ChatGPT can produce increasingly realistic text, with unknown information on the accuracy and integrity of using these models in scientific writing. We gathered fifth research abstracts from five high-impact factor medical journals and asked ChatGPT to generate research abstracts based on their titles and journals. Most generated abstracts were detected using an AI output detector, \u2018GPT-2 Output Detector\u2019, with % \u2018fake\u2019 scores (higher meaning more likely to be generated) of median [interquartile range] of 99.98% \u2018fake\u2019 [12.73%, 99.98%] compared with median 0.02% [IQR 0.02%, 0.09%] for the original abstracts. The AUROC of the AI output detector was 0.94. Generated abstracts scored lower than original abstracts when run through a plagiarism detector website and iThenticate (higher scores meaning more matching text found). When given a mixture of original and general abstracts, blinded human reviewers correctly identified 68% of generated abstracts as being generated by ChatGPT, but incorrectly identified 14% of original abstracts as being generated. Reviewers indicated that it was surprisingly difficult to differentiate between the two, though abstracts they suspected were generated were vaguer and more formulaic. ChatGPT writes believable scientific abstracts, though with completely generated data. Depending on publisher-specific guidelines, AI output detectors may serve as an editorial tool to help maintain scientific standards. The boundaries of ethical and acceptable use of large language models to help scientific writing are still being discussed, and different journals and conferences are adopting varying policies.<\/jats:p>","DOI":"10.1038\/s41746-023-00819-6","type":"journal-article","created":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T07:03:01Z","timestamp":1682492581000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":566,"title":["Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers"],"prefix":"10.1038","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5576-3943","authenticated-orcid":false,"given":"Catherine A.","family":"Gao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6039-1141","authenticated-orcid":false,"given":"Frederick M.","family":"Howard","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3659-4387","authenticated-orcid":false,"given":"Nikolay S.","family":"Markov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5818-3589","authenticated-orcid":false,"given":"Emma C.","family":"Dyer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Siddhi","family":"Ramesh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0195-7456","authenticated-orcid":false,"given":"Yuan","family":"Luo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander T.","family":"Pearson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,4,26]]},"reference":[{"key":"819_CR1","unstructured":"OpenAI. ChatGPT: Optimizing language models for dialogue. OpenAI https:\/\/openai.com\/blog\/chatgpt\/ (2022)."},{"key":"819_CR2","unstructured":"Shankland, S. ChatGPT: Why everyone is obsessed this mind-blowing AI chatbot. CNET https:\/\/www.cnet.com\/tech\/computing\/chatgpt-why-everyone-is-obsessed-this-mind-blowing-ai-chatbot\/ (2022)."},{"key":"819_CR3","unstructured":"Agomuoh, F. ChatGPT: how to use the viral AI chatbot that took the world by storm. Digital Trends https:\/\/www.digitaltrends.com\/computing\/how-to-use-openai-chatgpt-text-generation-chatbot\/ (2022)."},{"key":"819_CR4","unstructured":"Hern, A. AI bot ChatGPT stuns academics with essay-writing skills and usability. The Guardian (2022)."},{"key":"819_CR5","unstructured":"Haque, M. U., Dharmadasa, I., Sworna, Z. T., Rajapakse, R. N. & Ahmad, H. \u201cI think this is the most disruptive technology\u201d: exploring sentiments of ChatGPT early adopters using Twitter data. https:\/\/arxiv.org\/abs\/2212.05856 (2022)."},{"key":"819_CR6","doi-asserted-by":"publisher","unstructured":"Stokel-Walker, C. AI bot ChatGPT writes smart essays-should professors worry? Nature https:\/\/doi.org\/10.1038\/d41586-022-04397-7 (2022).","DOI":"10.1038\/d41586-022-04397-7"},{"key":"819_CR7","unstructured":"Whitford, E. Here\u2019s how Forbes got the ChatGPT AI to write 2 college essays in 20\u2009min Forbes https:\/\/www.forbes.com\/sites\/emmawhitford\/2022\/12\/09\/heres-how-forbes-got-the-chatgpt-ai-to-write-2-college-essays-in-20-minutes\/?sh=2b5a552456ad (2022)."},{"key":"819_CR8","doi-asserted-by":"publisher","unstructured":"Yeadon, W., Inyang, O.-O., Mizouri, A., Peach, A. & Testrow, C. The death of the short-form Physics essay in the coming AI revolution. https:\/\/doi.org\/10.48550\/ARXIV.2212.11661 (2022).","DOI":"10.48550\/ARXIV.2212.11661"},{"key":"819_CR9","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","volume":"2","author":"TH Kung","year":"2023","unstructured":"Kung, T. H. et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit. Health 2, e0000198 (2023).","journal-title":"PLoS Digit. Health"},{"key":"819_CR10","unstructured":"Susnjak, T. ChatGPT: the end of online exam integrity? https:\/\/arxiv.org\/abs\/2212.09292 (2022)."},{"key":"819_CR11","doi-asserted-by":"crossref","unstructured":"Much to discuss in AI ethics. Nat. Mach. Intell. 4, 1055\u20131056 (2022).","DOI":"10.1038\/s42256-022-00598-x"},{"key":"819_CR12","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1126\/science.adg7879","volume":"379","author":"HH Thorp","year":"2023","unstructured":"Thorp, H. H. ChatGPT is fun, but not an author. Science 379, 313 (2023).","journal-title":"Science"},{"key":"819_CR13","doi-asserted-by":"publisher","unstructured":"Flanagin, A., Bibbins-Domingo, K., Berkwits, M. & Christiansen, S. L. Nonhuman \u201cauthors\u201d and implications for the integrity of scientific publication and medical knowledge. JAMA https:\/\/doi.org\/10.1001\/jama.2023.1344 (2023).","DOI":"10.1001\/jama.2023.1344"},{"key":"819_CR14","doi-asserted-by":"crossref","unstructured":"Tools such as ChatGPT threaten transparent science; here are our ground rules for their use. Nature 613, 612 (2023).","DOI":"10.1038\/d41586-023-00191-1"},{"key":"819_CR15","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1038\/s41746-021-00464-x","volume":"4","author":"DM Korngiebel","year":"2021","unstructured":"Korngiebel, D. M. & Mooney, S. D. Considering the possibilities and pitfalls of Generative Pre-trained Transformer 3 (GPT-3) in healthcare delivery. NPJ Digit. Med. 4, 93 (2021).","journal-title":"NPJ Digit. Med."},{"key":"819_CR16","doi-asserted-by":"crossref","unstructured":"Clark, E. et al. All that\u2019s \u2018human\u2019 is not gold: Evaluating human evaluation of generated text. in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol. 1: Long Papers) 7282\u20137296 (Association for Computational Linguistics, 2021).","DOI":"10.18653\/v1\/2021.acl-long.565"},{"key":"819_CR17","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1017\/XPS.2020.37","volume":"9","author":"S Kreps","year":"2022","unstructured":"Kreps, S., Miles McCain, R. & Brundage, M. All the news that\u2019s fit to fabricate: AI-generated text as a tool of media misinformation. J. Exp. Political Sci. 9, 104\u2013117 (2022).","journal-title":"J. Exp. Political Sci."},{"key":"819_CR18","doi-asserted-by":"publisher","first-page":"27","DOI":"10.3389\/fmed.2020.00027","volume":"7","author":"G Briganti","year":"2020","unstructured":"Briganti, G. & Le Moine, O. Artificial intelligence in medicine: today and tomorrow. Front. Med. (Lausanne) 7, 27 (2020).","journal-title":"Front. Med. (Lausanne)"},{"key":"819_CR19","unstructured":"Masa. SciNote Manuscript Writer-using Artificial Intelligence. SciNote https:\/\/www.scinote.net\/blog\/scinote-can-write-draft-scientific-manuscript-using-artificial-intelligence\/ (Masa, 2020)."},{"key":"819_CR20","unstructured":"Writefull. Writefull https:\/\/www.writefull.com\/. Accessed December 2022."},{"key":"819_CR21","unstructured":"Solaiman, I. et al. Release strategies and the social impacts of language models. https:\/\/arxiv.org\/abs\/1908.09203 (2019)."},{"key":"819_CR22","unstructured":"GPT-2 Output Detector. https:\/\/huggingface.co\/openai-detector. Accessed December 2022."},{"key":"819_CR23","unstructured":"Turnitin, I. by Plagiarism detection software. https:\/\/www.ithenticate.com\/. Accessed December 2022."},{"key":"819_CR24","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-24698-1","volume":"12","author":"FM Howard","year":"2021","unstructured":"Howard, F. M. et al. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nat. Commun. 12, 4423 (2021).","journal-title":"Nat. Commun."},{"key":"819_CR25","unstructured":"Banerjee, I. et al. Reading race: AI recognises patient\u2019s racial identity in medical images. https:\/\/arxiv.org\/abs\/2107.10356 (2021)."},{"key":"819_CR26","doi-asserted-by":"publisher","first-page":"513474","DOI":"10.3389\/fpsyg.2020.513474","volume":"11","author":"JM Bishop","year":"2020","unstructured":"Bishop, J. M. Artificial intelligence is stupid and causal reasoning will not fix it. Front. Psychol. 11, 513474 (2020).","journal-title":"Front. Psychol."},{"key":"819_CR27","unstructured":"Tyrrell, J. How easy is it to fool AI content detectors? TechHQ https:\/\/techhq.com\/2023\/02\/how-easy-is-it-to-fool-ai-content-detectors\/ (2023)."},{"key":"819_CR28","doi-asserted-by":"crossref","unstructured":"Hosseini, M., Rasmussen, L. M. & Resnik, D. B. Using AI to write scholarly publications. Account. Res. 1\u20139 https:\/\/www.tandfonline.com\/doi\/full\/10.1080\/08989621.2023.2168535 (2023).","DOI":"10.1080\/08989621.2023.2168535"},{"key":"819_CR29","unstructured":"Arcade, C. Plagiarism checker. Plagiarismdetector.net https:\/\/plagiarismdetector.net\/ (2010)."},{"key":"819_CR30","doi-asserted-by":"publisher","first-page":"3021","DOI":"10.21105\/joss.03021","volume":"6","author":"M Waskom","year":"2021","unstructured":"Waskom, M. seaborn: statistical data visualization. J. Open Source Softw. 6, 3021 (2021).","journal-title":"J. Open Source Softw."},{"key":"819_CR31","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/MCSE.2007.55","volume":"9","author":"JD Hunter","year":"2007","unstructured":"Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90\u201395 (2007).","journal-title":"Comput. Sci. Eng."},{"key":"819_CR32","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"819_CR33","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","volume":"17","author":"P Virtanen","year":"2020","unstructured":"Virtanen, P. et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods 17, 261\u2013272 (2020).","journal-title":"Nat. Methods"},{"key":"819_CR34","doi-asserted-by":"publisher","unstructured":"Charlier, F. et al. trevismd\/statannotations: v0.5. Zenodo https:\/\/doi.org\/10.5281\/ZENODO.7213391 (2022).","DOI":"10.5281\/ZENODO.7213391"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00819-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00819-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00819-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T07:24:31Z","timestamp":1682493871000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00819-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,26]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["819"],"URL":"https:\/\/doi.org\/10.1038\/s41746-023-00819-6","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.12.23.521610","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,26]]},"assertion":[{"value":"1 January 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 March 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 April 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A.T.P. reports no competing interests for this work, and reports personal fees from Prelude Therapeutics Advisory Board, Elevar Advisory Board, AbbVie consulting, Ayala Advisory Board, and Privo Therapeutics, all outside of submitted work. The remaining authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"75"}}