{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T10:35:31Z","timestamp":1777113331489,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":37,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,4,27]]},"DOI":"10.1145\/3785022.3785101","type":"proceedings-article","created":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T09:39:01Z","timestamp":1777109941000},"page":"933-939","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Disagreement as Data: Reasoning Trace Analytics in Multi-Agent Systems"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-4716-292X","authenticated-orcid":false,"given":"Elham","family":"Tajik","sequence":"first","affiliation":[{"name":"University of Albany, Albany, New York, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3437-8979","authenticated-orcid":false,"given":"Conrad","family":"Borchers","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, Pennsylvania, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3737-4714","authenticated-orcid":false,"given":"Bahar","family":"Shahrokhian","sequence":"additional","affiliation":[{"name":"Arizona State University, Tempe, Arizona, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3218-2032","authenticated-orcid":false,"given":"Sebastian","family":"Simon","sequence":"additional","affiliation":[{"name":"Le Mans University, Le Mans, Pays de La Loire, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4763-7418","authenticated-orcid":false,"given":"Ali","family":"Keramati","sequence":"additional","affiliation":[{"name":"University of California. Irvine, Irvine, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3470-6369","authenticated-orcid":false,"given":"Sonika","family":"Pal","sequence":"additional","affiliation":[{"name":"Indian Institute of Technology, Mumbai, Mumbai, Maharashtra, India"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9122-6870","authenticated-orcid":false,"given":"Sreecharan","family":"Sankaranarayanan","sequence":"additional","affiliation":[{"name":"Extuitive Inc. (Flagship Pioneering), Cambridge, Massachusetts, USA"}]}],"member":"320","published-online":{"date-parts":[[2026,4,26]]},"reference":[{"key":"e_1_3_3_2_2_2","doi-asserted-by":"crossref","unstructured":"Ryan\u00a0S Baker Stephen Hutt Nigel Bosch Jaclyn Ocumpaugh Gautam Biswas Luc Paquette JM\u00a0Alexandra Andres Nidhi Nasiar and Anabil Munshi. 2024. Detector-driven classroom interviewing: Focusing qualitative researcher time by selecting cases in situ. Educational technology research and development 72 5 (2024) 2841\u20132863.","DOI":"10.1007\/s11423-023-10324-y"},{"key":"e_1_3_3_2_3_2","unstructured":"Aneesha Bakharia Antonette Shibani Lisa-Angelique Lim Trish McCluskey and Simon\u00a0Buckingham Shum. 2025. From Transcripts to Themes: A Trustworthy Workflow for Qualitative Analysis Using Large Language Models. (2025)."},{"key":"e_1_3_3_2_4_2","first-page":"65","volume-title":"Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and\/or summarization","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and\/or summarization. 65\u201372."},{"key":"e_1_3_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-64299-9_10"},{"key":"e_1_3_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2460296.2460316"},{"key":"e_1_3_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3706468.3706484"},{"key":"e_1_3_3_2_8_2","unstructured":"Conrad Borchers Bahar Shahrokhian Francesco Balzan Elham Tajik Sreecharan Sankaranarayanan and Sebastian Simon. 2025. Temperature and Persona Shape LLM Agent Consensus With Minimal Accuracy Gains in Qualitative Coding. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2507.11198 (2025)."},{"key":"e_1_3_3_2_9_2","unstructured":"Conrad Borchers Jiayi Zhang Hendrik Fleischer Sascha Schanze Vincent Aleven Ryan Baker et\u00a0al. 2025. Large Language Models Generalize SRL Prediction to New Languages Within But Not Between Domains. Journal of Educational Data Mining 17 2 (2025) 24\u201354."},{"key":"e_1_3_3_2_10_2","doi-asserted-by":"crossref","unstructured":"Anthony Botelho Sami Baral John\u00a0A Erickson Priyanka Benachamardi and Neil\u00a0T Heffernan. 2023. Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematics. Journal of computer assisted learning 39 3 (2023) 823\u2013840.","DOI":"10.1111\/jcal.12793"},{"key":"e_1_3_3_2_11_2","doi-asserted-by":"crossref","unstructured":"Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3 2 (2006) 77\u2013101.","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_3_2_12_2","doi-asserted-by":"crossref","unstructured":"Virginia Braun and Victoria Clarke. 2021. One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative research in psychology 18 3 (2021) 328\u2013352.","DOI":"10.1080\/14780887.2020.1769238"},{"key":"e_1_3_3_2_13_2","doi-asserted-by":"crossref","unstructured":"Stefano De\u00a0Paoli. 2024. Performing an inductive thematic analysis of semi-structured interviews with a large language model: An exploration and provocation on the limits of the approach. Social Science Computer Review 42 4 (2024) 997\u20131019.","DOI":"10.1177\/08944393231220483"},{"key":"e_1_3_3_2_14_2","first-page":"4171","volume-title":"Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171\u20134186."},{"key":"e_1_3_3_2_15_2","doi-asserted-by":"crossref","unstructured":"Wael\u00a0H Gomaa Aly\u00a0A Fahmy et\u00a0al. 2013. A survey of text similarity approaches. international journal of Computer Applications 68 13 (2013) 13\u201318.","DOI":"10.5120\/11638-7118"},{"key":"e_1_3_3_2_16_2","unstructured":"Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi et\u00a0al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2501.12948 (2025)."},{"key":"e_1_3_3_2_17_2","doi-asserted-by":"crossref","unstructured":"Subbarao Kambhampati Kaya Stechly and Karthik Valmeekam. 2025. (How) Do reasoning models reason? Annals of the New York Academy of Sciences 1547 1 (2025) 33\u201340.","DOI":"10.1111\/nyas.15339"},{"key":"e_1_3_3_2_18_2","unstructured":"Subbarao Kambhampati Kaya Stechly Karthik Valmeekam Lucas Saldyt Siddhant Bhambri Vardhan Palod Atharva Gundawar Soumya\u00a0Rani Samineni Durgesh Kalwar and Upasana Biswas. 2025. Stop Anthropomorphizing Intermediate Tokens as Reasoning\/Thinking Traces! arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2504.09762 (2025)."},{"key":"e_1_3_3_2_19_2","doi-asserted-by":"crossref","unstructured":"Ali Keramati Mohammad\u00a0Reza Keramati and Mohammad\u00a0Hossein Arefian. 2024. Students\u2019 reflection on the effect of collaborative learning on the learning environment and academic achievement in online reflective platforms. Reflective Practice 25 2 (2024) 117\u2013131.","DOI":"10.1080\/14623943.2024.2305868"},{"key":"e_1_3_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/CITSM.2016.7577578"},{"key":"e_1_3_3_2_21_2","first-page":"74","volume-title":"Text summarization branches out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74\u201381."},{"key":"e_1_3_3_2_22_2","doi-asserted-by":"crossref","unstructured":"Xiner Liu Andres\u00a0Felipe Zambrano Ryan\u00a0S Baker Amanda Barany Jaclyn Ocumpaugh Jiayi Zhang Maciej Pankiewicz Nidhi Nasiar and Zhanlan Wei. 2025. Qualitative Coding with GPT-4: Where it Works Better. Journal of Learning Analytics (2025) 1\u201317.","DOI":"10.18608\/jla.2025.8575"},{"key":"e_1_3_3_2_23_2","doi-asserted-by":"crossref","unstructured":"Elisabetta Mazzullo Okan Bulut Tarid Wongvorachan and Bin Tan. 2023. Learning analytics in the era of large language models. Analytics 2 4 (2023) 877\u2013898.","DOI":"10.3390\/analytics2040046"},{"key":"e_1_3_3_2_24_2","unstructured":"Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/1301.3781 (2013)."},{"key":"e_1_3_3_2_25_2","doi-asserted-by":"crossref","unstructured":"Mehrnoush Mohammadi Elham Tajik Roberto Martinez-Maldonado Shazia Sadiq Wojtek Tomaszewski and Hassan Khosravi. 2025. Artificial intelligence in multimodal learning analytics: a systematic literature review. Computers and Education: Artificial Intelligence (2025) 100426.","DOI":"10.1016\/j.caeai.2025.100426"},{"key":"e_1_3_3_2_26_2","first-page":"311","volume-title":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311\u2013318."},{"key":"e_1_3_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3330430.3333622"},{"key":"e_1_3_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3696410.3714595"},{"key":"e_1_3_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3706468.3706564"},{"key":"e_1_3_3_2_30_2","doi-asserted-by":"crossref","unstructured":"Zeeshan Rasheed Muhammad Waseem Aakash Ahmad Kai-Kristian Kemell Wang Xiaofeng Anh\u00a0Nguyen Duc and Pekka Abrahamsson. 2024. Can large language models serve as data analysts? A multi-agent assisted approach for qualitative data analysis. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2402.01386 (2024).","DOI":"10.2139\/ssrn.5149683"},{"key":"e_1_3_3_2_31_2","doi-asserted-by":"publisher","unstructured":"Sreecharan Sankaranarayanan Conrad Borchers Sebastian Simon Elham Tajik Amine\u00a0Hatun Ata\u015f Berkan Celik Francesco Balzan and Bahar Shahrokhian. 2025. Automating Thematic Analysis with Multi-Agent LLM Systems. EdArXiv Preprints (10.35542\/osf.io\/kq8zh_v1) (2025). 10.35542\/osf.io\/kq8zh","DOI":"10.35542\/osf.io\/kq8zh_v1"},{"key":"e_1_3_3_2_32_2","volume-title":"Introduction to information retrieval","author":"Sch\u00fctze Hinrich","year":"2008","unstructured":"Hinrich Sch\u00fctze, Christopher\u00a0D Manning, and Prabhakar Raghavan. 2008. Introduction to information retrieval. Vol.\u00a039. Cambridge University Press Cambridge."},{"key":"e_1_3_3_2_33_2","unstructured":"Bruce Sherin Nicole Kersting and Matthew Berland. 2018. Learning analytics in support of qualitative analysis. International Society of the Learning Sciences Inc.[ISLS]."},{"key":"e_1_3_3_2_34_2","doi-asserted-by":"crossref","unstructured":"Sebastian Simon Sreecharan Sankaranarayanan Elham Tajik Conrad Borchers shahrokhian Bahar Francesco Balzan Sebastian Strau\u00df Sree Viswanathan Amine Ata\u015f Mia \u010carapina Li Liang and Berkan Celik. 2025. Comparing human and LLM-generated inductive thematic analyses: Assessing agreement in coding consistency and interpretative accuracy. Proceedings of 26th International Conference on Artificial Intelligence in Education (AIED) (2025).","DOI":"10.35542\/osf.io\/ez8wc_v1"},{"key":"e_1_3_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3706468.3706530"},{"key":"e_1_3_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3573051.3593378"},{"key":"e_1_3_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613905.3650732"},{"key":"e_1_3_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-47014-1_32"}],"event":{"name":"LAK 2026: LAK26: 16th International Learning Analytics and Knowledge Conference","location":"Bergen Norway","acronym":"LAK 2026"},"container-title":["Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference"],"original-title":[],"deposited":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T09:44:54Z","timestamp":1777110294000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3785022.3785101"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,26]]},"references-count":37,"alternative-id":["10.1145\/3785022.3785101","10.1145\/3785022"],"URL":"https:\/\/doi.org\/10.1145\/3785022.3785101","relation":{},"subject":[],"published":{"date-parts":[[2026,4,26]]},"assertion":[{"value":"2026-04-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}