{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T19:24:04Z","timestamp":1787685844445,"version":"build-2784847793"},"reference-count":29,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n While large language model (LLM)\u2013assisted qualitative analysis could improve the efficiency and scalability of feedback-driven curricular refinement in medical education, how best to leverage LLMs for qualitative analysis while ensuring quality outputs remains an open question. Prior work has demonstrated the feasibility of using LLMs for inductive and deductive coding tasks, but more needs to be known about how LLM-assisted thematic coding can best be deployed in a medical education context to maximize its strengths and guard against its weaknesses.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n Our study evaluated LLM performance in inductive code generation and in the deductive application of a human codebook, using a student focus-group transcript, to propose a model for AI collaboration in qualitative analysis.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n The qualitative data for this study consisted of a 1-hour focus group with 4 second-year medical students discussing a required AI-driven clinical-scenario tool (2-Sigma). Three human coders conducted an inductive thematic analysis. Using the same transcript, GPT-4o (version gpt-4o-2024-11-20; OpenAI) generated inductive codes and applied the human codebook deductively. The researchers compared the alignment between the AI inductive codes and the human consensus codebook using 3 categories: agreement, reasonable alternative, and not reasonable. Interrater reliability of AI deductive coding was evaluated using percent agreement and Cohen \u03ba, with textual audits of discrepancies, including \u201cmisses\u201d (failed to apply appropriate codes) and \u201cmisfires\u201d (inappropriately applied codes). Analysis took place between February and July 2025.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n In the inductive condition, GPT-4o generated 137 initial codes, of which 31.4% (n=43) demonstrated agreement with human codes, 26.3% (n=36) represented reasonable alternatives, and 42.3% (n=58) were classified as not reasonable. In the deductive condition, mean percent agreement for AI application of human codes was 96% (SD 4%, range 79%\u2010100%) and the mean \u03ba was 0.71 (SD 0.26, range 0\u20101.00). Of all 2352 coding decisions, there were 57 (2.4%) misfires and 28 (1.2%) misses; common patterns included overinterpretation of tone, failure to recognize continued ideas across excerpts, and difficulty distinguishing hypothetical vs experienced features. Based on our findings, we suggest a roadmap that retains human interpretive control while leveraging AI scalability: humans first develop a contextually grounded codebook through inductive analysis, then use AI both as a creative partner to surface alternative codes and as a tool to apply the validated codebook across the dataset.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n With targeted human oversight, an LLM reliably applied an existing codebook and generated additional inductive codes. These findings support a proposed workflow in which AI serves as an additional perspective within human-driven qualitative analysis, offering a scalable adjunct for qualitative analysis in medical education. Validation across larger and more diverse datasets will help confirm the generalizability of this approach.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/85572","type":"journal-article","created":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T16:45:10Z","timestamp":1783010710000},"page":"e85572-e85572","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Model\u2013Assisted Thematic Coding in Medical Education Research: Comparative Methodological Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-3602-9232","authenticated-orcid":false,"given":"Alexa","family":"DeRegnaucourt","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5746-1404","authenticated-orcid":false,"given":"Katherine Miller","family":"Jennings","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-6456-5258","authenticated-orcid":false,"given":"Andrew","family":"Zahn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7084-7416","authenticated-orcid":false,"given":"Neal","family":"Taliwal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6400-1745","authenticated-orcid":false,"given":"Matthew","family":"Kelleher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2143-1972","authenticated-orcid":false,"given":"Christine Yang","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-3696-5395","authenticated-orcid":false,"given":"Seth","family":"Overla","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8327-8002","authenticated-orcid":false,"given":"Sally A","family":"Santen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4857-6936","authenticated-orcid":false,"given":"Danielle Elliott","family":"Weber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4567-1313","authenticated-orcid":false,"given":"Laurah","family":"Turner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,25]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1145\/3581754.3584136","article-title":"Supporting qualitative analysis with large language models: combining codebook with GPT-3 for deductive coding","author":"Xiao","journal-title":"Companion Proc Int User Interfaces Conf"},{"key":"R2","unstructured":"Chew R Bollenbacher J Wenger M Speer J Kim A . LLM-assisted content analysis: using large language models to support deductive coding. arXiv. Preprint posted online on Jun 23, 2023. [doi: 10.48550\/arXiv.2306.14924]"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/54482","article-title":"Comparing the efficacy and efficiency of human and generative AI: qualitative thematic analyses","volume":"3","author":"Prescott","journal-title":"JMIR AI"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1177\/16094069231201504","article-title":"Exploring the use of AI in qualitative analysis: a comparative study of guaranteed income data","volume":"22","author":"Hamilton","journal-title":"Int J Qual Methods"},{"issue":"4","key":"R5","doi-asserted-by":"publisher","first-page":"997","DOI":"10.1177\/08944393231220483","article-title":"Performing an inductive thematic analysis of semi-structured interviews with a large language model: an exploration and provocation on the limits of the approach","volume":"42","author":"De Paoli","journal-title":"Soc Sci Comput Rev"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.2196\/54974","article-title":"Harnessing ChatGPT for thematic analysis: are we ready?","volume":"26","author":"Lee","journal-title":"J Med Internet Res"},{"issue":"2-3","key":"R7","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1177\/10497323251359445","article-title":"Human analysis vs. artificial intelligence: analyzing of qualitative medical students\u2019 narratives","volume":"36","author":"Misgav","journal-title":"Qual Health Res"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1007\/s40670-025-02587-2","article-title":"Incorporating artificial intelligence in qualitative research: exploring the role of ChatGPT in thematic analysis","volume":"36","author":"Bowden","journal-title":"Med Sci Educ"},{"issue":"12","key":"R9","doi-asserted-by":"publisher","first-page":"1447","DOI":"10.1097\/ACM.0000000000006185","article-title":"Harnessing the generative power of AI to move closer to personalized medical education","volume":"100","author":"Turner","journal-title":"Acad Med"},{"issue":"2","key":"R10","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1191\/1478088706qp063oa","article-title":"Using thematic analysis in psychology","volume":"3","author":"Braun","journal-title":"Qual Res Psychol"},{"issue":"4","key":"R11","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1111\/j.1468-2958.2002.tb00826.x","article-title":"Content analysis in mass communication: assessment and reporting of intercoder reliability","volume":"28","author":"Lombard","journal-title":"Human Comm Res"},{"issue":"3","key":"R12","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: the kappa statistic","volume":"22","author":"McHugh","journal-title":"Biochem Med (Zagreb)"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","journal-title":"Biometrics"},{"key":"R14","unstructured":"Parkington K Teferra BG Rouleau-Tang M et al. Human vs LLM-based thematic analysis for digital mental health research: proof-of-concept comparative study. arXiv. Preprint posted online on May 2, 2025. [doi: 10.48550\/arXiv.2507.08002]"},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Siiman LA Rannastu-Avalos M P\u00f6ys\u00e4-Tarhonen J H\u00e4kkinen P Pedaste M . Opportunities and challenges for AI-assisted qualitative data analysis: an example from collaborative problem-solving discourse data. In: Huang YM Rocha T , editors. Innovative Technologies and Learning 6th International Conference, ICITL 2023, Porto, Portugal, August 28\u201330, 2023, Proceedings. Springer; 87-96. [doi: 10.1007\/978-3-031-40113-8_9]","DOI":"10.1007\/978-3-031-40113-8_9"},{"key":"R16","unstructured":"Mollick E . Co-Intelligence: Living and Working with AI. Portfolio; 2024. URL: https:\/\/www.penguinrandomhouse.com\/books\/741805\/co-intelligence-by-ethan-mollick\/ [Accessed 30-07-2026]"},{"issue":"2","key":"R17","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1177\/14707853251405043","article-title":"Generative AI-augmented thematic analysis","volume":"68","author":"Jayawardene","journal-title":"Int J Mark Res"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1177\/10778004251412871","article-title":"From coding to conversation: a new methodological framework for AI-assisted qualitative analysis","author":"Friese","journal-title":"Qual Inq"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","first-page":"169","DOI":"10.18608\/jla.2025.8575","article-title":"Qualitative coding with GPT-4: where it works better","volume":"12","author":"Liu","journal-title":"J Learn Anal"},{"issue":"10","key":"R20","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1097\/ACM.0000000000006134","article-title":"Artificial intelligence to support qualitative data analysis: promises, approaches, pitfalls","volume":"100","author":"Cook","journal-title":"Acad Med"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1800","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","journal-title":"Adv Neural Inf Process Syst"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1140\/epjds\/s13688-025-00548-8","article-title":"Scaling hermeneutics: a guide to qualitative coding with LLMs for reflexive content analysis","volume":"14","author":"Dunivin","journal-title":"EPJ Data Sci"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-emnlp.121","article-title":"Measuring sycophancy of language models in multi-turn dialogues","author":"Hong","journal-title":"Findings Assoc Comput Linguist"},{"key":"R24","unstructured":"Sycophancy in GPT\u20114o: what happened and what we\u2019re doing about it. OpenAI. Apr 29, 2025. URL: https:\/\/openai.com\/index\/sycophancy-in-gpt-4o\/ [Accessed 02-06-2026]"},{"key":"R25","unstructured":"Sharma M Tong M Korbak T et al. Towards understanding sycophancy in language models. Presented at: The Twelfth International Conference on Learning Representations (ICLR 2024); May 7-11, 2024; Vienna, Austria. URL: https:\/\/openreview.net\/forum?id=tvhaxkMKAn [Accessed 30-07-2026]"},{"key":"R26","doi-asserted-by":"crossref","unstructured":"Wang K Li J Yang S Zhang Z Wang D . When truth is overridden: uncovering the internal origins of sycophancy in large language models. In: Koenig S Jenkins C Taylor ME , editors. Proceedings of the Fortieth AAAI Conference on Artificial Intelligence and Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence and Sixteenth Symposium on Educational Advances in Artificial Intelligence. AAAI Press; 2025:33566-33574. [doi: 10.1609\/aaai.v40i39.40645]","DOI":"10.1609\/aaai.v40i39.40645"},{"issue":"6758","key":"R27","doi-asserted-by":"publisher","DOI":"10.1126\/science.aea3922","article-title":"Why AI chatbots lie to us","volume":"389","author":"Mitchell","journal-title":"Science"},{"key":"R28","unstructured":"Chowdhury N Johnson D Huang V Steinhardt J Schwettmann S . Investigating truthfulness in a pre-release o3 model. Transluce. Apr 16, 2025. URL: https:\/\/transluce.org\/investigating-o3-truthfulness [Accessed 04-01-2026]"},{"issue":"5","key":"R29","doi-asserted-by":"publisher","first-page":"3503","DOI":"10.1007\/s10462-021-10088-y","article-title":"Explainable artificial intelligence: a comprehensive review","volume":"55","author":"Minh","journal-title":"Artif Intell Rev"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T19:12:02Z","timestamp":1787685122000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e85572"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,25]]},"references-count":29,"URL":"https:\/\/doi.org\/10.2196\/85572","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,25]]},"article-number":"v10i4e85572"}}