{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T17:57:22Z","timestamp":1791482242212,"version":"4.3.4"},"reference-count":20,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T00:00:00Z","timestamp":1791417600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:p>Large language models (LLMs) are increasingly used for qualitative analysis in substance use research, yet their performance relative to human coders remains underexplored. This study compares ChatGPT-4.0 with human coders in performing qualitative coding tasks using NIH grant abstracts as a test case, focusing on the identification and description of research innovations. Using a sample of NIH HEAL Initiative grant abstracts related to opioid overdose prevention, a total of 125 abstracts were independently coded by ChatGPT and humans to generate innovation descriptions, which were then evaluated by both human raters and ChatGPT for depth\/detail and relevance\/completeness using 5-point Likert scales. Identical instructions were used across all coding and evaluation stages. ChatGPT-generated descriptions were consistently rated higher than human-generated descriptions on both dimensions. Human evaluators rated ChatGPT outputs at an average of 4.47 for both depth\/detail and relevance\/completeness, compared to 3.33 and 3.24 for human outputs, respectively (F(1,176)=133.9, p\u2009&lt;\u20090.001). These findings suggest that LLMs, when carefully prompted, can enhance the efficiency and quality of qualitative research evaluation.<\/jats:p>","DOI":"10.1371\/journal.pone.0343857","type":"journal-article","created":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T17:38:42Z","timestamp":1791481122000},"page":"e0343857","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["AI versus human coding of NIH grant abstracts"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-3185-710X","authenticated-orcid":true,"given":"Sarah A.","family":"Alkhatib","sequence":"first","affiliation":[{"name":"College of Public Health, University of North Texas Health","place":["Fort Worth, Texas, United States of America"],"department":["Department of Population & Community Health"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3772-8508","authenticated-orcid":true,"given":"Nadiya","family":"Jiwa","sequence":"additional","affiliation":[{"name":"College of Public Health, University of North Texas Health","place":["Fort Worth, Texas, United States of America"],"department":["Department of Population & Community Health"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dallin","family":"Judd","sequence":"additional","affiliation":[{"name":"Texas College of Osteopathic Medicine, University of North Texas Health","place":["Fort Worth, Texas, United States of America"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Justin M.","family":"Luningham","sequence":"additional","affiliation":[{"name":"College of Science & Engineering, Texas Christian University","place":["Fort Worth, Texas, United States of America"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ginnie","family":"Sawyer-Morris","sequence":"additional","affiliation":[{"name":"Friends Research Institute","place":["Baltimore, Maryland, United States of America"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Merve","family":"Ulukaya","sequence":"additional","affiliation":[{"name":"George Mason University, Schar School of Policy and Government","place":["Fairfax, Virginia, United States of America"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Todd","family":"Molfenter","sequence":"additional","affiliation":[{"name":"University of Wisconsin Center for Health Enhancement Systems Studies","place":["Madison, Wisconsin, United States of America"],"department":["Department of Industrial & Systems Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Faye S.","family":"Taxman","sequence":"additional","affiliation":[{"name":"George Mason University, Schar School of Policy and Government","place":["Fairfax, Virginia, United States of America"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Scott T.","family":"Walters","sequence":"additional","affiliation":[{"name":"College of Public Health, University of North Texas Health","place":["Fort Worth, Texas, United States of America"],"department":["Department of Population & Community Health"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,10,8]]},"reference":[{"issue":"5","key":"pone.0343857.ref001","doi-asserted-by":"crossref","first-page":"376","DOI":"10.7326\/0003-4819-126-5-199703010-00006","article-title":"Systematic reviews: Synthesis of best evidence for clinical decisions","volume":"126","author":"DJ Cook","year":"1997","journal-title":"Ann Int Med"},{"key":"pone.0343857.ref002","doi-asserted-by":"crossref","unstructured":"Aarons G, Moullin J, Ehrhart M. The role of organizational processes in dissemination and implementation research. 2017. p. 121\u201342.","DOI":"10.1093\/oso\/9780190683214.003.0008"},{"key":"pone.0343857.ref003","doi-asserted-by":"crossref","unstructured":"Firat M. How Chat GPT can transform autodidactic experiences and open education? 2023.","DOI":"10.31219\/osf.io\/9ge8m"},{"issue":"3","key":"pone.0343857.ref004","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1108\/LHTN-01-2023-0009","article-title":"Chatting about ChatGPT: How may AI and GPT impact academia and libraries?","volume":"40","author":"BD Lund","year":"2023","journal-title":"Library Hi Tech News"},{"issue":"1","key":"pone.0343857.ref005","first-page":"27","article-title":"AI-empowered qualitative data analysis: Training future public health researchers","volume":"6","author":"CW Wheldon","year":"2025","journal-title":"Common Health"},{"issue":"7","key":"pone.0343857.ref006","doi-asserted-by":"crossref","first-page":"595","DOI":"10.1177\/10497323231217392","article-title":"Artificial intelligence augmented qualitative analysis: The way of the future?","volume":"34","author":"D Hitch","year":"2023","journal-title":"Qualitative Health Research"},{"key":"pone.0343857.ref007","doi-asserted-by":"crossref","unstructured":"Yan L, Echeverria V, Fernandez-Nieto GM, Jin Y, Swiecki Z, Zhao L. Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations. In: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA; 2024. p. Article 191.","DOI":"10.1145\/3613905.3650732"},{"key":"pone.0343857.ref008","article-title":"Developing and testing an automated qualitative assistant (AQUA) to support qualitative analysis","volume":"9","author":"RP Lennon","year":"2021","journal-title":"Fam Med Community Health"},{"key":"pone.0343857.ref009","volume-title":"Opportunities and challenges for AI-assisted qualitative data analysis: An example from collaborative problem-solving discourse data","author":"LA Siiman","year":"2023"},{"issue":"6","key":"pone.0343857.ref010","doi-asserted-by":"crossref","first-page":"188","DOI":"10.3390\/fi16060188","article-title":"Studying the quality of source code generated by different ai generative engines: An empirical evaluation","volume":"16","author":"D Tosi","year":"2024","journal-title":"Future Internet"},{"key":"pone.0343857.ref011","unstructured":"Ahmad FB, Cisewski JA, Rossen LM, Sutton P. Provisional drug overdose death count. 2025."},{"issue":"1","key":"pone.0343857.ref012","article-title":"Advancing research translation in addiction and pain: A portfolio analysis of the NIH HEAL initiative","volume":"9","author":"G Sawyer-Morris","year":"2025","journal-title":"J Clin Transl Sci"},{"issue":"1","key":"pone.0343857.ref013","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1186\/s13012-022-01245-0","article-title":"The updated consolidated framework for implementation research based on user feedback","volume":"17","author":"LJ Damschroder","year":"2022","journal-title":"Implement Sci"},{"issue":"1","key":"pone.0343857.ref014","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1186\/s13012-018-0736-7","article-title":"Developing measures to assess constructs from the Inner Setting domain of the Consolidated Framework for Implementation Research","volume":"13","author":"ME Fernandez","year":"2018","journal-title":"Implement Sci"},{"key":"pone.0343857.ref015","doi-asserted-by":"crossref","DOI":"10.1177\/16094069231211248","article-title":"Exploring the use of artificial intelligence for qualitative data analysis: The case of ChatGPT","volume":"22","author":"DL Morgan","year":"2023","journal-title":"Int J Qual Methods"},{"key":"pone.0343857.ref016","article-title":"Language models are few-shot learners","author":"TB Brown","year":"2020","journal-title":"CoRR"},{"issue":"10","key":"pone.0343857.ref017","doi-asserted-by":"crossref","DOI":"10.1007\/s10462-024-10888-y","article-title":"Large language models (LLMs): Survey, technical frameworks, and future challenges","volume":"57","author":"P Kumar","year":"2024","journal-title":"Artif Intell Rev"},{"issue":"1","key":"pone.0343857.ref018","doi-asserted-by":"crossref","DOI":"10.1145\/3682069","article-title":"Large language models for conducting advanced text analytics information systems research","volume":"16","author":"B Ampel","year":"2025","journal-title":"ACM Trans Manage Inf Syst"},{"key":"pone.0343857.ref019","article-title":"Comprehensive review of AI hallucinations: impacts and mitigation strategies for financial and business applications","author":"S Joshi","year":"2025","journal-title":"Int J Comp Appl Technol Res"},{"issue":"4","key":"pone.0343857.ref020","doi-asserted-by":"crossref","first-page":"3005","DOI":"10.1007\/s10462-022-10246-w","article-title":"Human-in-the-loop machine learning: A state of the art","volume":"56","author":"E Mosqueira-Rey","year":"2022","journal-title":"Artif Intell Rev"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0343857","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T17:38:48Z","timestamp":1791481128000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0343857"}},"subtitle":[],"editor":[{"given":"Steve","family":"Zimmerman","sequence":"first","affiliation":[{"name":"Public Library of Science","place":["UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,10,8]]},"references-count":20,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,10,8]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0343857","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,8]]}}}