{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,18]],"date-time":"2025-12-18T19:53:07Z","timestamp":1766087587439,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":50,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,7,7]],"date-time":"2020-07-07T00:00:00Z","timestamp":1594080000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["1660878","1726550","1651909"],"award-info":[{"award-number":["1660878","1726550","1651909"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,7,7]]},"DOI":"10.1145\/3340631.3394848","type":"proceedings-article","created":{"date-parts":[[2020,7,13]],"date-time":"2020-07-13T21:49:55Z","timestamp":1594676995000},"page":"284-292","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Improving Student-System Interaction Through Data-driven Explanations of Hierarchical Reinforcement Learning Induced Pedagogical Policies"],"prefix":"10.1145","author":[{"given":"Guojing","family":"Zhou","sequence":"first","affiliation":[{"name":"North Carolina State University, Raleigh, NC, USA"}]},{"given":"Xi","family":"Yang","sequence":"additional","affiliation":[{"name":"North Carolina State University, Raleigh, NC, USA"}]},{"given":"Hamoon","family":"Azizsoltani","sequence":"additional","affiliation":[{"name":"SAS, Raleigh, NC, USA"}]},{"given":"Tiffany","family":"Barnes","sequence":"additional","affiliation":[{"name":"North Carolina State University, Raleigh, NC, USA"}]},{"given":"Min","family":"Chi","sequence":"additional","affiliation":[{"name":"North Carolina State University, Raleigh, NC, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,7,13]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Cognitive tutors: Lessons learned. The journal of the learning sciences","author":"Anderson John R","year":"1995","unstructured":"John R Anderson , Albert T Corbett , Kenneth R Koedinger , and Ray Pelletier . 1995. Cognitive tutors: Lessons learned. The journal of the learning sciences , Vol. 4 , 2 ( 1995 ), 167--207. John R Anderson, Albert T Corbett, Kenneth R Koedinger, and Ray Pelletier. 1995. Cognitive tutors: Lessons learned. The journal of the learning sciences, Vol. 4, 2 (1995), 167--207."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985741"},{"key":"e_1_3_2_2_3_1","volume-title":"Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems","author":"Barto Andrew G","year":"2003","unstructured":"Andrew G Barto and Sridhar Mahadevan . 2003. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems , Vol. 13 , 1--2 ( 2003 ), 41--77. Andrew G Barto and Sridhar Mahadevan. 2003. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, Vol. 13, 1--2 (2003), 41--77."},{"key":"e_1_3_2_2_4_1","volume-title":"Beverly Park Woolf, and Carole R Beal","author":"Beck Joseph","year":"2000","unstructured":"Joseph Beck , Beverly Park Woolf, and Carole R Beal . 2000 . ADVISOR : A machine learning architecture for intelligent tutor construction. AAAI\/IAAI , Vol. 2000 , 552--557 (2000), 1--2. Joseph Beck, Beverly Park Woolf, and Carole R Beal. 2000. ADVISOR: A machine learning architecture for intelligent tutor construction. AAAI\/IAAI, Vol. 2000, 552--557 (2000), 1--2."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-010-9093-1"},{"key":"e_1_3_2_2_6_1","first-page":"1","article-title":"An evaluation of pedagogical tutorial tactics for a natural language tutoring system: A reinforcement learning approach","volume":"21","author":"Chi Min","year":"2011","unstructured":"Min Chi , Kurt VanLehn , Diane Litman , and Pamela Jordan . 2011 b. An evaluation of pedagogical tutorial tactics for a natural language tutoring system: A reinforcement learning approach . International Journal of Artificial Intelligence in Education , Vol. 21 , 1 -- 2 (2011), 83--113. Min Chi, Kurt VanLehn, Diane Litman, and Pamela Jordan. 2011b. An evaluation of pedagogical tutorial tactics for a natural language tutoring system: A reinforcement learning approach. International Journal of Artificial Intelligence in Education, Vol. 21, 1--2 (2011), 83--113.","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.88.4.715"},{"volume-title":"International Conference on Spatial Cognition. Springer, 319--334","author":"Heriberto","key":"e_1_3_2_2_8_1","unstructured":"Heriberto Cuay\u00e1huitl et al. 2010. Generating adaptive route instructions using hierarchical reinforcement learning . In International Conference on Spatial Cognition. Springer, 319--334 . Heriberto Cuay\u00e1huitl et al. 2010. Generating adaptive route instructions using hierarchical reinforcement learning. In International Conference on Spatial Cognition. Springer, 319--334."},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-6494.1994.tb00797.x"},{"key":"e_1_3_2_2_10_1","volume-title":"Towards Understanding How to Leverage Sense-Making, Induction and Refinement, and Fluency to Improve Robust Learning","author":"Doroudi Shayan","year":"2015","unstructured":"Shayan Doroudi , Kenneth Holstein , Vincent Aleven , and Emma Brunskill . 2015. Towards Understanding How to Leverage Sense-Making, Induction and Refinement, and Fluency to Improve Robust Learning . International Educational Data Mining Society ( 2015 ). Shayan Doroudi, Kenneth Holstein, Vincent Aleven, and Emma Brunskill. 2015. Towards Understanding How to Leverage Sense-Making, Induction and Refinement, and Fluency to Improve Robust Learning. International Educational Data Mining Society (2015)."},{"volume-title":"One-on-one tutoring by humans and computers","author":"Evens Martha","key":"e_1_3_2_2_11_1","unstructured":"Martha Evens and Joel Michael . 2006. One-on-one tutoring by humans and computers . Psychology Press . Martha Evens and Joel Michael. 2006. One-on-one tutoring by humans and computers .Psychology Press."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2018.09.007"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2009.01.007"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0012841"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02298286"},{"key":"e_1_3_2_2_16_1","unstructured":"Kenneth R Koedinger John R Anderson William H Hadley and Mary A Mark. 1997. Intelligent tutoring goes to school in the big city. (1997).  Kenneth R Koedinger John R Anderson William H Hadley and Mary A Mark. 1997. Intelligent tutoring goes to school in the big city. (1997)."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v34i3.2484"},{"key":"e_1_3_2_2_18_1","first-page":"8","article-title":"Choices for children","volume":"75","author":"Kohn Alfie","year":"1993","unstructured":"Alfie Kohn . 1993 . Choices for children . Phi Delta Kappan , Vol. 75 , 1 (1993), 8 -- 20 . Alfie Kohn. 1993. Choices for children. Phi Delta Kappan, Vol. 75, 1 (1993), 8--20.","journal-title":"Phi Delta Kappan"},{"key":"e_1_3_2_2_19_1","unstructured":"Tejas D Kulkarni Karthik Narasimhan Ardavan Saeedi and Josh Tenenbaum. 2016. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems. 3675--3683.  Tejas D Kulkarni Karthik Narasimhan Ardavan Saeedi and Josh Tenenbaum. 2016. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems. 3675--3683."},{"key":"e_1_3_2_2_20_1","volume-title":"Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. Computers as cognitive tools","author":"Lepper Mark R","year":"1993","unstructured":"Mark R Lepper , Maria Woolverton , Donna L Mumme , and J Gurtner . 1993. Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. Computers as cognitive tools , Vol. 1993 ( 1993 ), 75--105. Mark R Lepper, Maria Woolverton, Donna L Mumme, and J Gurtner. 1993. Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. Computers as cognitive tools, Vol. 1993 (1993), 75--105."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33492-4_6"},{"key":"e_1_3_2_2_22_1","volume-title":"Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. International Foundation for Autonomous Agents and Multiagent Systems, 1077--1084","author":"Mandel Travis","year":"2014","unstructured":"Travis Mandel , Yun-En Liu , Sergey Levine , Emma Brunskill , and Zoran Popovic . 2014 . Offline policy evaluation across representations with applications to educational games . In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. International Foundation for Autonomous Agents and Multiagent Systems, 1077--1084 . Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. 2014. Offline policy evaluation across representations with applications to educational games. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. International Foundation for Autonomous Agents and Multiagent Systems, 1077--1084."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21869-9_30"},{"key":"e_1_3_2_2_24_1","volume-title":"Proceedings of the 30th annual conference of the cognitive science society. 2176--2181","author":"McLaren Bruce M","year":"2008","unstructured":"Bruce M McLaren , Sung-Joo Lim , and Kenneth R Koedinger . 2008 . When and how often should worked examples be given to students? New results and a summary of the current state of research . In Proceedings of the 30th annual conference of the cognitive science society. 2176--2181 . Bruce M McLaren, Sung-Joo Lim, and Kenneth R Koedinger. 2008. When and how often should worked examples be given to students? New results and a summary of the current state of research. In Proceedings of the 30th annual conference of the cognitive science society. 2176--2181."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073602"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sbspro.2010.03.641"},{"key":"e_1_3_2_2_28_1","volume-title":"Faster teaching via pomdp planning. Cognitive science","author":"Rafferty Anna N","year":"2016","unstructured":"Anna N Rafferty , Emma Brunskill , Thomas L Griffiths , and Patrick Shafto . 2016. Faster teaching via pomdp planning. Cognitive science , Vol. 40 , 6 ( 2016 ), 1290--1332. Anna N Rafferty, Emma Brunskill, Thomas L Griffiths, and Patrick Shafto. 2016. Faster teaching via pomdp planning. Cognitive science, Vol. 40, 6 (2016), 1290--1332."},{"volume-title":"Advanced lectures on machine learning","author":"Rasmussen Carl Edward","key":"e_1_3_2_2_29_1","unstructured":"Carl Edward Rasmussen . 2004. Gaussian processes in machine learning . In Advanced lectures on machine learning . Springer , 63--71. Carl Edward Rasmussen. 2004. Gaussian processes in machine learning. In Advanced lectures on machine learning. Springer, 63--71."},{"key":"e_1_3_2_2_30_1","volume-title":"Providing a rationale in an autonomy-supportive way as a strategy to motivate others during an uninteresting activity. Motivation and emotion","author":"Reeve Johnmarshall","year":"2002","unstructured":"Johnmarshall Reeve , Hyungshim Jang , Pat Hardre , and Mafumi Omura . 2002. Providing a rationale in an autonomy-supportive way as a strategy to motivate others during an uninteresting activity. Motivation and emotion , Vol. 26 , 3 ( 2002 ), 183--207. Johnmarshall Reeve, Hyungshim Jang, Pat Hardre, and Mafumi Omura. 2002. Providing a rationale in an autonomy-supportive way as a strategy to motivate others during an uninteresting activity. Motivation and emotion, Vol. 26, 3 (2002), 183--207."},{"key":"e_1_3_2_2_31_1","volume-title":"Tenth Artificial Intelligence and Interactive Digital Entertainment Conference .","author":"Rowe Jonathan","year":"2014","unstructured":"Jonathan Rowe , Bradford Mott , and James Lester . 2014 . Optimizing player experience in interactive narrative planning: a modular reinforcement learning approach . In Tenth Artificial Intelligence and Interactive Digital Entertainment Conference . Jonathan Rowe, Bradford Mott, and James Lester. 2014. Optimizing player experience in interactive narrative planning: a modular reinforcement learning approach. In Tenth Artificial Intelligence and Interactive Digital Entertainment Conference ."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-19773-9_42"},{"key":"e_1_3_2_2_33_1","volume-title":"In Proceedings of the 17th International Conference on Machine Learning. Citeseer.","author":"Ryan Malcolm","year":"2000","unstructured":"Malcolm Ryan and Mark Reid . 2000 . Learning to fly: An application of hierarchical reinforcement learning . In In Proceedings of the 17th International Conference on Machine Learning. Citeseer. Malcolm Ryan and Mark Reid. 2000. Learning to fly: An application of hierarchical reinforcement learning. In In Proceedings of the 17th International Conference on Machine Learning. Citeseer."},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11251-009-9107-8"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.63.3.379"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-6494.00070"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.90.4.705"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-017-5650-8"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2008.12.011"},{"key":"e_1_3_2_2_40_1","volume-title":"Aim Low: Correlation-Based Feature Selection for Model-Based Reinforcement Learning","author":"Shen Shitian","year":"2016","unstructured":"Shitian Shen and Min Chi . 2016 a. Aim Low: Correlation-Based Feature Selection for Model-Based Reinforcement Learning . International Educational Data Mining Society ( 2016). Shitian Shen and Min Chi. 2016a. Aim Low: Correlation-Based Feature Selection for Model-Based Reinforcement Learning. International Educational Data Mining Society (2016)."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2930238.2930247"},{"key":"e_1_3_2_2_42_1","first-page":"85","article-title":"Learner control versus program control in interactive videodisc instruction: What are the effects in procedural learning","volume":"19","author":"Shyu Hsin-Yih","year":"1992","unstructured":"Hsin-Yih Shyu and Scott W Brown . 1992 . Learner control versus program control in interactive videodisc instruction: What are the effects in procedural learning . International Journal of Instructional Media , Vol. 19 , 2 (1992), 85 -- 95 . Hsin-Yih Shyu and Scott W Brown. 1992. Learner control versus program control in interactive videodisc instruction: What are the effects in procedural learning. International Journal of Instructional Media, Vol. 19, 2 (1992), 85--95.","journal-title":"International Journal of Instructional Media"},{"key":"e_1_3_2_2_43_1","volume-title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence","author":"Sutton Richard S","year":"1999","unstructured":"Richard S Sutton , Doina Precup , and Satinder Singh . 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence , Vol. 112 , 1--2 ( 1999 ), 181--211. Richard S Sutton, Doina Precup, and Satinder Singh. 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, Vol. 112, 1--2 (1999), 181--211."},{"key":"e_1_3_2_2_44_1","volume-title":"International journal of artificial intelligence in education","author":"Vanlehn Kurt","year":"2006","unstructured":"Kurt Vanlehn . 2006. The behavior of tutoring systems . International journal of artificial intelligence in education , Vol. 16 , 3 ( 2006 ), 227--265. Kurt Vanlehn. 2006. The behavior of tutoring systems. International journal of artificial intelligence in education, Vol. 16, 3 (2006), 227--265."},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00443"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/514672.514675"},{"key":"e_1_3_2_2_47_1","volume-title":"AIED 2019, Chicago, IL, USA, June 25--29, 2019, Proceedings, Part I. Springer, 544--556","author":"Zhou Guojing","year":"2019","unstructured":"Guojing Zhou , Hamoon Azizsoltani , Markel Sanz Ausin , Tiffany Barnes , and Min Chi . 2019 a. Hierarchical reinforcement learning for pedagogical policy induction. In Artificial Intelligence in Education - 20th International Conference , AIED 2019, Chicago, IL, USA, June 25--29, 2019, Proceedings, Part I. Springer, 544--556 . Guojing Zhou, Hamoon Azizsoltani, Markel Sanz Ausin, Tiffany Barnes, and Min Chi. 2019 a. Hierarchical reinforcement learning for pedagogical policy induction. In Artificial Intelligence in Education - 20th International Conference, AIED 2019, Chicago, IL, USA, June 25--29, 2019, Proceedings, Part I. Springer, 544--556."},{"key":"e_1_3_2_2_48_1","volume-title":"Proceedings of the 37th annual conference of the cognitive science society. 2817--2822","author":"Zhou Guojing","year":"2015","unstructured":"Guojing Zhou , Thomas W Price , Collin Lynch , Tiffany Barnes , and Min Chi . 2015 . The Impact of Granularity on Worked Examples and Problem Solving . In Proceedings of the 37th annual conference of the cognitive science society. 2817--2822 . Guojing Zhou, Thomas W Price, Collin Lynch, Tiffany Barnes, and Min Chi. 2015. The Impact of Granularity on Worked Examples and Problem Solving. In Proceedings of the 37th annual conference of the cognitive science society. 2817--2822."},{"key":"e_1_3_2_2_49_1","unstructured":"Guojing Zhou Jianxun Wang Collin Lynch and Min Chi. 2017. Towards Closing the Loop: Bridging Machine-induced Pedagogical Policies to Learning Theories. In EDM .  Guojing Zhou Jianxun Wang Collin Lynch and Min Chi. 2017. Towards Closing the Loop: Bridging Machine-induced Pedagogical Policies to Learning Theories. In EDM ."},{"key":"e_1_3_2_2_50_1","volume-title":"Proceedings of the 41th annual conference of the cognitive science society. 3206--3212","author":"Zhou Guojing","year":"2019","unstructured":"Guojing Zhou , Xi Yang , and Min Chi . 2019 b. Big, Little, or Both? Exploring the Impact of Granularity on Learning for Students with Different Incoming Competence . In Proceedings of the 41th annual conference of the cognitive science society. 3206--3212 . Guojing Zhou, Xi Yang, and Min Chi. 2019 b. Big, Little, or Both? Exploring the Impact of Granularity on Learning for Students with Different Incoming Competence. In Proceedings of the 41th annual conference of the cognitive science society. 3206--3212."}],"event":{"name":"UMAP '20: 28th ACM Conference on User Modeling, Adaptation and Personalization","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGCHI ACM Special Interest Group on Computer-Human Interaction"],"location":"Genoa Italy","acronym":"UMAP '20"},"container-title":["Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3340631.3394848","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3340631.3394848","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:13:29Z","timestamp":1750202009000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3340631.3394848"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,7]]},"references-count":50,"alternative-id":["10.1145\/3340631.3394848","10.1145\/3340631"],"URL":"https:\/\/doi.org\/10.1145\/3340631.3394848","relation":{},"subject":[],"published":{"date-parts":[[2020,7,7]]},"assertion":[{"value":"2020-07-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}