{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,27]],"date-time":"2026-06-27T19:50:20Z","timestamp":1782589820273,"version":"3.54.5"},"publisher-location":"Cham","reference-count":33,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031643019","type":"print"},{"value":"9783031643026","type":"electronic"}],"license":[{"start":{"date-parts":[[2024,1,1]],"date-time":"2024-01-01T00:00:00Z","timestamp":1704067200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,2]],"date-time":"2024-07-02T00:00:00Z","timestamp":1719878400000},"content-version":"vor","delay-in-days":183,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Conversational tutoring systems (CTSs) offer learning experiences through interactions based on natural language. They are recognized for promoting cognitive engagement and improving learning outcomes, especially in reasoning tasks. Nonetheless, the cost associated with authoring CTS content is a major obstacle to widespread adoption and to research on effective instructional design. In this paper, we discuss and evaluate a novel type of CTS that leverages recent advances in large language models (LLMs) in two ways: First, the system enables AI-assisted content authoring by inducing an easily editable tutoring script automatically from a lesson text. Second, the system automates the script orchestration in a learning-by-teaching format via two LLM-based agents (Ruffle&amp;Riley) acting as a student and a professor. The system allows for free-form conversations that follow the ITS-typical inner and outer loop structure. We evaluate Ruffle&amp;Riley\u2019s ability to support biology lessons in two between-subject online user studies (<jats:inline-formula>\n              <jats:alternatives>\n                <jats:tex-math>$$N = 200$$<\/jats:tex-math>\n                <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mi>N<\/mml:mi>\n                    <mml:mo>=<\/mml:mo>\n                    <mml:mn>200<\/mml:mn>\n                  <\/mml:mrow>\n                <\/mml:math>\n              <\/jats:alternatives>\n            <\/jats:inline-formula>) comparing the system to simpler QA chatbots and reading activity. Analyzing system usage patterns, pre\/post-test scores and user experience surveys, we find that Ruffle&amp;Riley users report high levels of engagement, understanding and perceive the offered support as helpful. Even though Ruffle&amp;Riley users require more time to complete the activity, we did not find significant differences in short-term learning gains over the reading activity. Our system architecture and user study provide various insights for designers of future CTSs. We further open-source our system to support ongoing research on effective instructional design of LLM-based learning technologies.\n<\/jats:p>","DOI":"10.1007\/978-3-031-64302-6_6","type":"book-chapter","created":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T06:04:39Z","timestamp":1719813879000},"page":"75-90","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":38,"title":["Ruffle&amp;Riley: Insights from\u00a0Designing and\u00a0Evaluating a\u00a0Large Language Model-Based Conversational Tutoring System"],"prefix":"10.1007","author":[{"given":"Robin","family":"Schmucker","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meng","family":"Xia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amos","family":"Azaria","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tom","family":"Mitchell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,7,2]]},"reference":[{"key":"6_CR1","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1007\/s40593-015-0088-2","volume":"26","author":"V Aleven","year":"2016","unstructured":"Aleven, V., et al.: Example-tracing tutors: intelligent tutor development for non-programmers. Int. J. Artif. Intell. Educ. 26, 224\u2013269 (2016)","journal-title":"Int. J. Artif. Intell. Educ."},{"key":"6_CR2","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1007\/11774303_39","volume-title":"Intelligent Tutoring Systems","author":"RSJ Baker","year":"2006","unstructured":"Baker, R.S.J., et al.: Adapting to when students game an intelligent tutoring system. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 392\u2013401. Springer, Heidelberg (2006). https:\/\/doi.org\/10.1007\/11774303_39"},{"key":"6_CR3","doi-asserted-by":"publisher","first-page":"823","DOI":"10.1111\/jcal.12793","volume":"39","author":"A Botelho","year":"2023","unstructured":"Botelho, A., Baral, S., Erickson, J.A., Benachamardi, P., Heffernan, N.T.: Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematics. J. Comput. Assist. Learn. 39, 823\u2013840 (2023)","journal-title":"J. Comput. Assist. Learn."},{"key":"6_CR4","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1007\/978-3-030-22341-0_46","volume-title":"Adaptive Instructional Systems","author":"Z Cai","year":"2019","unstructured":"Cai, Z., Hu, X., Graesser, A.C.: Authoring conversational intelligent tutoring systems. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 593\u2013603. Springer, Cham (2019). https:\/\/doi.org\/10.1007\/978-3-030-22341-0_46"},{"issue":"4","key":"6_CR5","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1080\/00461520.2014.965823","volume":"49","author":"MT Chi","year":"2014","unstructured":"Chi, M.T., Wylie, R.: The ICAP framework: linking cognitive engagement to active learning outcomes. Educ. Psychol. 49(4), 219\u2013243 (2014)","journal-title":"Educ. Psychol."},{"key":"6_CR6","unstructured":"Clark, M.A., Douglas, M., Choi, J.: Biology 2e. OpenStax (2018)"},{"key":"6_CR7","doi-asserted-by":"crossref","unstructured":"Demszky, D., Liu, J., Hill, H.C., Jurafsky, D., Piech, C.: Can automated feedback improve teachers\u2019 uptake of student ideas? Evidence from a randomized controlled trial in a large-scale online course. Educ. Eval. Policy Anal. (2023)","DOI":"10.3102\/01623737231169270"},{"key":"6_CR8","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1007\/s40593-017-0157-9","volume":"28","author":"D Dermeval","year":"2018","unstructured":"Dermeval, D., Paiva, R., Bittencourt, I.I., Vassileva, J., Borges, D.: Authoring tools for designing intelligent tutoring systems: a systematic review of the literature. Int. J. Artif. Intell. Educ. 28, 336\u2013384 (2018)","journal-title":"Int. J. Artif. Intell. Educ."},{"key":"6_CR9","doi-asserted-by":"crossref","unstructured":"Duran, D.: Learning-by-teaching. Evidence and implications as a pedagogical mechanism. Innov. Educ. Teach. Int. 54(5), 476\u2013484 (2017)","DOI":"10.1080\/14703297.2016.1156011"},{"key":"6_CR10","doi-asserted-by":"publisher","first-page":"180","DOI":"10.3758\/BF03195563","volume":"36","author":"AC Graesser","year":"2004","unstructured":"Graesser, A.C., et al.: AutoTutor: a tutor with dialogue in natural language. Behav. Res. Meth. Instrum. Comput. 36, 180\u2013192 (2004)","journal-title":"Behav. Res. Meth. Instrum. Comput."},{"issue":"6","key":"6_CR11","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1002\/acp.2350090604","volume":"9","author":"AC Graesser","year":"1995","unstructured":"Graesser, A.C., Person, N.K., Magliano, J.P.: Collaborative dialogue patterns in naturalistic one-to-one tutoring. Appl. Cogn. Psychol. 9(6), 495\u2013522 (1995)","journal-title":"Appl. Cogn. Psychol."},{"key":"6_CR12","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1016\/j.lindif.2016.01.012","volume":"47","author":"X Huang","year":"2016","unstructured":"Huang, X., Craig, S.D., Xie, J., Graesser, A., Hu, X.: Intelligent tutoring systems work as a math gap reducer in 6th grade after-school program. Learn. Individ. Differ. 47, 258\u2013265 (2016)","journal-title":"Learn. Individ. Differ."},{"key":"6_CR13","series-title":"LNCS","doi-asserted-by":"publisher","first-page":"476","DOI":"10.1007\/978-3-031-36272-9_39","volume-title":"AIED 2023","author":"Y Jiao","year":"2023","unstructured":"Jiao, Y., Shridhar, K., Cui, P., Zhou, W., Sachan, M.: Automatic educational question generation with difficulty level controls. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds.) AIED 2023. LNCS, vol. 13916, pp. 476\u2013488. Springer, Cham (2023). https:\/\/doi.org\/10.1007\/978-3-031-36272-9_39"},{"key":"6_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.lindif.2023.102274","volume":"103","author":"E Kasneci","year":"2023","unstructured":"Kasneci, E., et al.: ChatGPT for good? On opportunities and challenges of large language models for education. Learn. Individ. Differ. 103, 102274 (2023)","journal-title":"Learn. Individ. Differ."},{"issue":"6161","key":"6_CR15","doi-asserted-by":"publisher","first-page":"935","DOI":"10.1126\/science.1238056","volume":"342","author":"KR Koedinger","year":"2013","unstructured":"Koedinger, K.R., Booth, J.L., Klahr, D.: Instructional complexity and the science to constrain it. Science 342(6161), 935\u2013937 (2013)","journal-title":"Science"},{"key":"6_CR16","doi-asserted-by":"crossref","unstructured":"Koedinger, K.R., Kim, J., Jia, J.Z., McLaughlin, E.A., Bier, N.L.: Learning is not a spectator sport: doing is better than watching for learning from a MOOC. In: Proceedings of the Second (2015) ACM Conference on Learning@Scale, L@S 2015, pp. 111\u2013120. Association for Computing Machinery, New York (2015)","DOI":"10.1145\/2724660.2724681"},{"issue":"5","key":"6_CR17","doi-asserted-by":"publisher","first-page":"320","DOI":"10.1016\/j.learninstruc.2011.12.002","volume":"22","author":"KJ Kopp","year":"2012","unstructured":"Kopp, K.J., Britt, M.A., Millis, K., Graesser, A.C.: Improving the efficiency of dialogue in tutoring. Learn. Instr. 22(5), 320\u2013330 (2012)","journal-title":"Learn. Instr."},{"issue":"1","key":"6_CR18","doi-asserted-by":"publisher","first-page":"42","DOI":"10.3102\/0034654315581420","volume":"86","author":"JA Kulik","year":"2016","unstructured":"Kulik, J.A., Fletcher, J.D.: Effectiveness of intelligent tutoring systems: a meta-analytic review. Rev. Educ. Res. 86(1), 42\u201378 (2016)","journal-title":"Rev. Educ. Res."},{"key":"6_CR19","doi-asserted-by":"crossref","unstructured":"Labadze, L., Grigolia, M., Machaidze, L.: Role of AI chatbots in education: systematic literature review. Int. J. Educ. Technol. High. Ed. 20(1), 56 (2023)","DOI":"10.1186\/s41239-023-00426-1"},{"key":"6_CR20","doi-asserted-by":"crossref","unstructured":"Lee, Y., Kim, T.S., Kim, S., Yun, Y., Kim, J.: DAPIE: interactive step-by-step explanatory dialogues to answer children\u2019s why and how questions. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1\u201322 (2023)","DOI":"10.1145\/3544548.3581369"},{"issue":"3","key":"6_CR21","doi-asserted-by":"publisher","first-page":"181","DOI":"10.3233\/IRG-2008-18(3)02","volume":"18","author":"K Leelawong","year":"2008","unstructured":"Leelawong, K., Biswas, G.: Designing learning by teaching agents: the Betty\u2019s brain system. Int. J. Artif. Intell. Educ. 18(3), 181\u2013208 (2008)","journal-title":"Int. J. Artif. Intell. Educ."},{"key":"6_CR22","doi-asserted-by":"crossref","unstructured":"Markel, J.M., Opferman, S.G., Landay, J.A., Piech, C.: GPTeach: interactive ta training with GPT-based students. In: Proceedings of the 10th ACM Conference on Learning@Scale, pp. 226\u2013236. ACM, New York (2023)","DOI":"10.1145\/3573051.3593393"},{"key":"6_CR23","unstructured":"Mitrovic, A.: The effect of explaining on learning: a case study with a data normalization tutor. In: AIED, pp. 499\u2013506 (2005)"},{"key":"6_CR24","series-title":"LNCS","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1007\/978-3-031-42682-7_16","volume-title":"EC-TEL 2023","author":"S Moore","year":"2023","unstructured":"Moore, S., Nguyen, H.A., Chen, T., Stamper, J.: Assessing the quality of multiple-choice questions using GPT-4 and rule-based methods. In: Viberg, O., Jivet, I., Mu\u00f1oz-Merino, P., Perifanou, M., Papathoma, T. (eds.) EC-TEL 2023. LNCS, vol. 14200, pp. 229\u2013245. Springer, Cham (2023). https:\/\/doi.org\/10.1007\/978-3-031-42682-7_16"},{"key":"6_CR25","series-title":"LNCS","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1007\/978-3-031-42682-7_19","volume-title":"EC-TEL 2023","author":"HA Nguyen","year":"2023","unstructured":"Nguyen, H.A., Stec, H., Hou, X., Di, S., McLaren, B.M.: Evaluating ChatGPT\u2019s decimal skills and feedback generation in a digital learning game. In: Viberg, O., Jivet, I., Mu\u00f1oz-Merino, P., Perifanou, M., Papathoma, T. (eds.) EC-TEL 2023. LNCS, vol. 14200, pp. 278\u2013293. Springer, Cham (2023). https:\/\/doi.org\/10.1007\/978-3-031-42682-7_19"},{"key":"6_CR26","first-page":"427","volume":"24","author":"BD Nye","year":"2014","unstructured":"Nye, B.D., Graesser, A.C., Hu, X.: Autotutor and family: a review of 17 years of natural language tutoring. Int. J. AIED 24, 427\u2013469 (2014)","journal-title":"Int. J. AIED"},{"key":"6_CR27","doi-asserted-by":"publisher","first-page":"164246","DOI":"10.1109\/ACCESS.2020.3021383","volume":"8","author":"J Paladines","year":"2020","unstructured":"Paladines, J., Ramirez, J.: A systematic literature review of intelligent tutoring systems with dialogue in natural language. IEEE Access 8, 164246\u2013164267 (2020)","journal-title":"IEEE Access"},{"key":"6_CR28","unstructured":"Pardos, Z.A., Bhandari, S.: Learning gain differences between ChatGPT and human tutor generated algebra hints. arXiv preprint arXiv:2302.06871 (2023)"},{"key":"6_CR29","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2022.102898","volume":"167","author":"Z Peng","year":"2022","unstructured":"Peng, Z., Liu, Y., Zhou, H., Xu, Z., Ma, X.: CReBot: exploring interactive question prompts for critical paper reading. Int. J. Hum.-Comput. Stud. 167, 102898 (2022)","journal-title":"Int. J. Hum.-Comput. Stud."},{"issue":"2","key":"6_CR30","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1109\/TLT.2009.23","volume":"2","author":"L Razzaq","year":"2009","unstructured":"Razzaq, L., et al.: The assistment builder: supporting the life cycle of tutoring system content creation. IEEE Trans. Learn. Technol. 2(2), 157\u2013166 (2009)","journal-title":"IEEE Trans. Learn. Technol."},{"key":"6_CR31","unstructured":"Ros\u00e9, C.P.: Interactive conceptual tutoring in Atlas-Andes. In: Artificial Intelligence in Education: AI-Ed in the Wired and Wireless Future, pp. 256\u2013266 (2001)"},{"key":"6_CR32","doi-asserted-by":"crossref","unstructured":"Ruan, S., et al.: BookBuddy: turning digital materials into interactive foreign language lessons through a voice chatbot. In: Proceedings of the Sixth ACM Conference on Learning@ Scale, pp.\u00a01\u20134 (2019)","DOI":"10.1145\/3330430.3333643"},{"key":"6_CR33","doi-asserted-by":"crossref","unstructured":"VanLehn, K.: The behavior of tutoring systems. Int. J. Artif. Intelli. Educ. 16(3), 227\u2013265 (2006)","DOI":"10.3233\/IRG-2006-16(3)02"}],"container-title":["Lecture Notes in Computer Science","Artificial Intelligence in Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-64302-6_6","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,5]],"date-time":"2025-09-05T20:22:56Z","timestamp":1757103776000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-64302-6_6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"ISBN":["9783031643019","9783031643026"],"references-count":33,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-64302-6_6","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024]]},"assertion":[{"value":"2 July 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"The authors have no competing interests to declare.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclosure of Interests"}},{"value":"AIED","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Artificial Intelligence in Education","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Recife","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Brazil","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"8 July 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"12 July 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"25","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"aied2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/aied2024.cesar.school\/home","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}