{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T01:49:20Z","timestamp":1778809760086,"version":"3.51.4"},"reference-count":32,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,5,14]],"date-time":"2023-05-14T00:00:00Z","timestamp":1684022400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"publisher","award":["2113991"],"award-info":[{"award-number":["2113991"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"publisher","award":["2219271"],"award-info":[{"award-number":["2219271"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Fields in the social sciences, such as education research, have started to expand the use of computer-based research methods to supplement traditional research approaches. Natural language processing techniques, such as topic modeling, may support qualitative data analysis by providing early categories that researchers may interpret and refine. This study contributes to this body of research and answers the following research questions: (RQ1) What is the relative coverage of the latent Dirichlet allocation (LDA) topic model and human coding in terms of the breadth of the topics\/themes extracted from the text collection? (RQ2) What is the relative depth or level of detail among identified topics using LDA topic models and human coding approaches? A dataset of student reflections was qualitatively analyzed using LDA topic modeling and human coding approaches, and the results were compared. The findings suggest that topic models can provide reliable coverage and depth of themes present in a textual collection comparable to human coding but require manual interpretation of topics. The breadth and depth of human coding output is heavily dependent on the expertise of coders and the size of the collection; these factors are better handled in the topic modeling approach.<\/jats:p>","DOI":"10.3390\/make5020029","type":"journal-article","created":{"date-parts":[[2023,5,15]],"date-time":"2023-05-15T02:56:57Z","timestamp":1684119417000},"page":"473-490","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Evaluating the Coverage and Depth of Latent Dirichlet Allocation Topic Model in Comparison with Human Coding of Qualitative Data: The Case of Education Research"],"prefix":"10.3390","volume":"5","author":[{"given":"Gaurav","family":"Nanda","sequence":"first","affiliation":[{"name":"School of Engineering Technology Purdue University, West Lafayette, IN 47906, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aparajita","family":"Jaiswal","sequence":"additional","affiliation":[{"name":"Center for Intercultural Learning, Mentorship, Assessment and Research (CILMAR), Purdue University, West Lafayette, IN 47906, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hugo","family":"Castellanos","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Technology, Purdue University, West Lafayette, IN 47906, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuzhe","family":"Zhou","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Technology, Purdue University, West Lafayette, IN 47906, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alex","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Technology, Purdue University, West Lafayette, IN 47906, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6117-7502","authenticated-orcid":false,"given":"Alejandra J.","family":"Magana","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Technology, Purdue University, West Lafayette, IN 47906, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1080\/13645579.2011.625764","article-title":"Using natural language processing technology for qualitative data analysis","volume":"15","author":"Crowston","year":"2012","journal-title":"Int. J. Soc. Res. Methodol."},{"key":"ref_2","unstructured":"Gillies, M., Murthy, D., Brenton, H., and Olaniyan, R. (2022). Theme and Topic: How Qualitative Research and Topic Modeling Can Be Brought Together. arXiv."},{"key":"ref_3","first-page":"145","article-title":"Topic Modeling Using Latent Dirichlet Allocation: A Survey","volume":"54","author":"Chauhan","year":"2021","journal-title":"ACM Comput. Surv."},{"key":"ref_4","first-page":"993","article-title":"Latent Dirichlet Allocation","volume":"3","author":"Blei","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1109\/TLT.2021.3064798","article-title":"Analyzing Large Collections of Open-Ended Feedback from MOOC Learners Using LDA Topic Modeling and Qualitative Analysis","volume":"14","author":"Nanda","year":"2021","journal-title":"IEEE Trans. Learn. Technol."},{"key":"ref_6","unstructured":"Nanda, G., Wei, S., Katz, A., Brinton, C., and Ohland, M. (2022, January 25\u201329). Work-in-Progress: Using Latent Dirichlet Allocation to Uncover Themes in Student Comments from Peer Evaluations of Teamwork. Proceedings of the 2022 ASEE Annual Conference & Exposition, Minneapolis, MN, USA."},{"key":"ref_7","first-page":"525","article-title":"Text Mining of Twitter Data Using a Latent Dirichlet Allocation Topic Model and Sentiment Analysis","volume":"12","author":"Yang","year":"2018","journal-title":"Int. J. Comput. Inf. Eng."},{"key":"ref_8","first-page":"8","article-title":"Legal Document Summarization Using Latent Dirichlet Allocation","volume":"3","author":"Kumar","year":"2012","journal-title":"Int. J. Comput. Sci. Telecommun."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"102098","DOI":"10.1016\/j.wpi.2022.102098","article-title":"Ontology-Based Knowledge Representation and Semantic Topic Modeling for Intelligent Trademark Legal Precedent Research","volume":"68","author":"Li","year":"2022","journal-title":"World Pat. Inf."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"117303","DOI":"10.1016\/j.eswa.2022.117303","article-title":"Preliminary Exploration of Topic Modelling Representations for Electronic Health Records Coding According to the International Classification of Diseases in Spanish","volume":"204","author":"Blanco","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.1007\/s11135-014-0039-2","article-title":"Coding, sorting and sifting of qualitative data analysis: Debates and discussion","volume":"49","author":"Chowdhury","year":"2015","journal-title":"Qual. Quant."},{"key":"ref_12","first-page":"2850","article-title":"Thinking about the coding process in qualitative data analysis","volume":"23","author":"Elliott","year":"2018","journal-title":"Qual. Rep."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Cleland, J. (2015). Researching Medical Education, Wiley Online Library.","DOI":"10.1002\/9781118838983"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1417","DOI":"10.1007\/s11135-011-9455-8","article-title":"The Distinction between Qualitative and Quantitative Research Methods Is Problematic","volume":"46","author":"Allwood","year":"2012","journal-title":"Qual. Quant."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"e87908","DOI":"10.1371\/journal.pone.0087908","article-title":"A computational approach to qualitative analysis in large textual datasets","volume":"9","author":"Evans","year":"2014","journal-title":"PLoS ONE"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1145\/3185515","article-title":"Using Machine Learning to Support Qualitative Coding in Social Science: Shifting the Focus to Ambiguity","volume":"8","author":"Chen","year":"2018","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"ref_17","unstructured":"Crowston, K., Liu, X., and Allen, E.E. (2010, January 22\u201327). Machine Learning and Rule-Based Automated Coding of Qualitative Data. Proceedings of the American Society for Information Science and Technology, Pittsburgh, PA, USA."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1007\/s10956-020-09862-4","article-title":"Combining Machine Learning and Qualitative Methods to Elaborate Students\u2019 Ideas about the Generality of Their Model-Based Explanations","volume":"30","author":"Rosenberg","year":"2021","journal-title":"J. Sci. Educ. Technol."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Marathe, M., and Toyama, K. (2018, January 21\u201326). Semi-automated coding for qualitative research: A user-centered inquiry and initial prototypes. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3173922"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1397","DOI":"10.1002\/asi.23786","article-title":"Comparing Grounded Theory and Topic Modeling: Extreme Divergence or Unlikely Convergence?","volume":"68","author":"Baumer","year":"2017","journal-title":"J. Assoc. Inf. Sci. Technol."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Muller, M., Guha, S., Baumer, E.P., Mimno, D., and Shami, N.S. (2016, January 13\u201316). Machine Learning and Grounded Theory Method: Convergence, Divergence, and Combination. Proceedings of the 19th International Conference on Supporting Group Work, Sanibel Island, FL, USA.","DOI":"10.1145\/2957276.2957280"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1177\/0049124117729703","article-title":"Computational Grounded Theory: A Methodological Framework","volume":"49","author":"Nelson","year":"2020","journal-title":"Sociol. Methods Res."},{"key":"ref_23","unstructured":"Jaiswal, A., Patel, D., Zhu, Y., Lee, J.S., and Magana, A. (2022, January 25\u201329). A Reflection on Action Approach to Teamwork Facilitation. Proceedings of the 2022 ASEE Annual Conference & Exposition, Minneapolis, MN, USA."},{"key":"ref_24","unstructured":"Khandkar, S.H. (2009). Open Coding, University of Calgary."},{"key":"ref_25","first-page":"e2","article-title":"Topic Modeling: A Comprehensive Review","volume":"7","author":"Kherwa","year":"2019","journal-title":"In EAI Endorsed Transactions on Scalable Information Systems"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"15169","DOI":"10.1007\/s11042-018-6894-4","article-title":"Latent Dirichlet Allocation (LDA) and Topic Modeling: Models, Applications, a Survey","volume":"78","author":"Jelodar","year":"2019","journal-title":"Multimed. Tools Appl."},{"key":"ref_27","unstructured":"Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J., and Blei, D. Advances in Neural Information Processing Systems 22: Proceedings of the 23rd Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 7\u201310 December 2009."},{"key":"ref_28","unstructured":"Mimno, D., Wallach, H., Talley, E., Leenders, M., and McCallum, A. (2011, January 27\u201331). Optimizing Semantic Coherence in Topic Models. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"R\u00f6der, M., Both, A., and Hinneburg, A. (2015, January 2\u20136). Exploring the space of topic coherence measures. Proceedings of the 8th ACM International Conference on Web Search and Data Mining, Shanghai, China.","DOI":"10.1145\/2684822.2685324"},{"key":"ref_30","unstructured":"(2023, March 04). Welcome to PyLDAvis\u2019s Documentation PyLDAvis 2.1.2 Documentation. Available online: https:\/\/pyldavis.readthedocs.io\/en\/latest\/."},{"key":"ref_31","unstructured":"(2023, March 04). Topic Modeling. Available online: https:\/\/mimno.github.io\/Mallet\/topics.html."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Rahman, S., and Kandogan, E. (May, January 29). Characterizing practices, limitations, and opportunities related to text information extraction workflows: A hu-man-in-the-loop perspective. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, Orleans, LA, USA.","DOI":"10.1145\/3491102.3502068"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/5\/2\/29\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:34:46Z","timestamp":1760124886000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/5\/2\/29"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,14]]},"references-count":32,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["make5020029"],"URL":"https:\/\/doi.org\/10.3390\/make5020029","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,14]]}}}