{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T16:48:11Z","timestamp":1780332491986,"version":"3.54.1"},"reference-count":48,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2015,6,17]],"date-time":"2015-06-17T00:00:00Z","timestamp":1434499200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>We present KnoWDiaL, an approach for Learning and using task-relevant Knowledge from human-robot Dialog and access to the Web. KnoWDiaL assumes that there is an autonomous agent that performs tasks, as requested by humans through speech. The agent needs to \u201cunderstand\u201d the request, (i.e., to fully ground the task until it can proceed to plan for and execute it). KnoWDiaL contributes such understanding by using and updating a Knowledge Base, by dialoguing with the user, and by accessing the web. We believe that KnoWDiaL, as we present it, can be applied to general autonomous agents. However, we focus on our work with our autonomous collaborative robot, CoBot, which executes service tasks in a building, moving around and transporting objects between locations. Hence, the knowledge acquired and accessed consists of groundings of language to robot actions, and building locations, persons, and objects. KnoWDiaL handles the interpretation of voice commands, is robust regarding speech recognition errors, and is able to learn commands involving referring expressions in an open domain, (i.e., without requiring a lexicon). We present in detail the multiple components of KnoWDiaL, namely a frame-semantic parser, a probabilistic grounding model, a web-based predicate evaluator, a dialog manager, and the weighted predicate-based Knowledge Base. We illustrate the knowledge access and updates from the dialog and Web access, through detailed and complete examples. We further evaluate the correctness of the predicate instances learned into the Knowledge Base, and show the increase in dialog efficiency as a function of the number of interactions. We have extensively and successfully used KnoWDiaL in CoBot dialoguing and accessing the Web, and extract a few corresponding example sequences from captured videos.<\/jats:p>","DOI":"10.3390\/robotics4020223","type":"journal-article","created":{"date-parts":[[2015,6,17]],"date-time":"2015-06-17T11:19:34Z","timestamp":1434539974000},"page":"223-252","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Learning Task Knowledge from Dialog and Web Access"],"prefix":"10.3390","volume":"4","author":[{"given":"Vittorio","family":"Perera","sequence":"first","affiliation":[{"name":"School of Computer Science, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robin","family":"Soetens","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Eindhoven University of Technology, Den Dolech 2, Eindhoven"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Kollar","sequence":"additional","affiliation":[{"name":"School of Computer Science, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mehdi","family":"Samadi","sequence":"additional","affiliation":[{"name":"School of Computer Science, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yichao","family":"Sun","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Industrial Control Technology, Zhejiang University, 38 Zheda Road, Hangzhou 456555, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniele","family":"Nardi","sequence":"additional","affiliation":[{"name":"Department of Computer, Control, and Management Engineering \"Antonio Ruberti\", \"Sapienza\" University of Rome Via Ariosto 25, Rome 00185, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ren\u00e9","family":"Van de Molengraft","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Eindhoven University of Technology, Den Dolech 2, Eindhoven"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manuela","family":"Veloso","sequence":"additional","affiliation":[{"name":"School of Computer Science, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2015,6,17]]},"reference":[{"key":"ref_1","unstructured":"Samadi, M., Veloso, M., and Blum, M. (2011, January 18). Evaluating correctness of propositions using the web. Proceedings of the Workshop on Learning by Reading and its Applications in Intelligent Question-Answering, Barcelona, Spain."},{"key":"ref_2","unstructured":"Samadi, M., Kollar, T., and Veloso, M.M. (2012, January 22\u201326). Using the web to interactively learn to find objects. Proceedings of the 26th Conference on Artifical Intelligence (AAAI-12), Toronto, ON, Canada."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1679","DOI":"10.1177\/0278364913503892","article-title":"Localization and navigation of the cobots over long-term deployments","volume":"32","author":"Biswas","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_4","unstructured":"Hanheide, M., Gretton, C., Dearden, R., Hawes, N., Wyatt, J., Pronobis, A., Aydemir, A., G\u00f6belbecker, M., and Zender, H. (2011, January 16\u201322). Exploiting probabilistic knowledge under uncertain sensing for efficient robot behaviour. Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, Barcelona, Spain."},{"key":"ref_5","unstructured":"Porzel, R., and Gurevych, I. (2003). Modeling and Using Context, Springer."},{"key":"ref_6","unstructured":"Rosenthal, S., Biswas, J., and Veloso, M. (2010, January 9\u201314). An effective personal mobile robot agent through symbiotic human-robot interaction. Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, Toronto, ON, Canada."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Biswas, J., Coltin, B., and Veloso, M. (2011, January 25\u201330). Corrective Gradient Refinement for Mobile Robot Localization. Proceedings of the IROS\u201911, the IEEE\/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA.","DOI":"10.1109\/IROS.2011.6048263"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Kollar, T., Perera, V., Nardi, D., and Veloso, M. (2013, January 6\u201310). Learning environmental knowledge from task-based human-robot dialog. Proceedings of the International Conference on Robotics and Automation, Karlsruhe, Geramny.","DOI":"10.1109\/ICRA.2013.6631186"},{"key":"ref_9","unstructured":"Jackendoff, R.S. (1983). Semantics and Cognition, MIT Press."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1017\/S0140525X00029927","article-title":"Whence and whither in spatial language and spatial cognition?","volume":"16","author":"Landau","year":"1993","journal-title":"Behav. Brain Sci."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Hamp, B. (2005). From Perception to Meaning: Image Schemas in Cognitive Linguistics, Mouton de Gruyter.","DOI":"10.1515\/9783110197532"},{"key":"ref_12","unstructured":"Katz, B. (1988). Artificial Intelligence at MIT Expanding Frontiers, MIT Press."},{"key":"ref_13","unstructured":"Wallach, H. (2004). Conditional Random Fields: An Introduction, Department of Computer and Information Science, University of Pennsylvania. Rapport Technique MS-CIS-04-21."},{"key":"ref_14","unstructured":"Kudo, T. CRF++: Yet Another CRF Toolkit. Available online: http:\/\/sourceforge.net\/projects\/crfpp\/."},{"key":"ref_15","unstructured":"Bird, S., Klein, E., and Loper, E. (2009). Natural Language Processing with Python, O\u2019Reilly."},{"key":"ref_16","unstructured":"Winograd, T. (1970). Procedures as a Representation for Data in a Computer Program for Understanding Natural Language. [Ph.D. Thesis, Massachusetts Institute of Technology]."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1080\/09540090802445113","article-title":"Object schemas for grounding language in a responsive robot","volume":"20","author":"Hsiao","year":"2008","journal-title":"Connect. Sci."},{"key":"ref_18","unstructured":"MacMahon, M., Stankiewicz, B., and Kuipers, B. (2006, January 16\u201320). Walk the talk: Connecting language, knowledge, and action in route instructions. Proceedings of the 21st National Conference on Artificial Intelligence-AAAI, Boston, MA, USA."},{"key":"ref_19","unstructured":"Task-Based Dialog on CoBot. Available online: https:\/\/www.youtube.com\/watch?v=2LojY9gFK5A."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1109\/TSMCC.2004.826273","article-title":"Spatial language for human-robot dialogs","volume":"34","author":"Skubic","year":"2004","journal-title":"IEEE Trans. Syst. Man Cybern. C"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Dzifcak, J., Scheutz, M., Baral, C., and Schermerhorn, P. (2009, January 12\u201317). What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan.","DOI":"10.1109\/ROBOT.2009.5152776"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Matuszek, C., Fox, D., and Koscher, K. (2010, January 2\u20135). Following directions using statistical machine translation. Proceedings of the 5th ACM\/IEEE International Conference on Human-robot Interaction, New York, NY, USA.","DOI":"10.1109\/HRI.2010.5453189"},{"key":"ref_23","unstructured":"Vogel, A., and Jurafsky, D. (2010, January 11\u201316). Learning to follow navigational directions. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden."},{"key":"ref_24","unstructured":"Dantam, N., and Stilman, M. (2011). Robotics: Science and Systems (RSS), MIT Press."},{"key":"ref_25","unstructured":"Shimizu, N., and Haas, A. (2009, January 11\u201317). Learning to follow navigational route instructions. Proceedings of the 21st International Jont Conference on Artifical Intelligence, Pasadena, CA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1004","DOI":"10.1016\/j.robot.2008.08.012","article-title":"A dialogue approach to learning object descriptions and semantic categories","volume":"56","author":"Holzapfel","year":"2008","journal-title":"Robot. Auton. Syst."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Kaiser, P., Lewis, M., Petrick, R., Asfour, T., and Steedman, M. (June, January 31). Extracting common sense knowledge from text for robot planning. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China.","DOI":"10.1109\/ICRA.2014.6907402"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kollar, T., Tellex, S., Roy, D., and Roy, N. Toward understanding natural language directions. Proceedings of HRI-2010; : Piscataway.","DOI":"10.1109\/HRI.2010.5453186"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee, A., Teller, S., and Roy, N. (2011, January 7\u201311). Understanding natural language commands for robotic navigation and mobile manipulation. Proceedings of the National Conference on Artificial Intelligence (AAAI), San Francisco, CA, USA.","DOI":"10.1609\/aaai.v25i1.7979"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/s11370-008-0016-5","article-title":"Using dialog and human observations to dictate tasks to a learning robot assistant","volume":"1","author":"Rybski","year":"2008","journal-title":"Intell. Serv. Robot."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Chen, X., Stone, P., Sucar, L.-E., and van-der Zant, T. (2012). RoboCup 2012: Robot Soccer World Cup XVI, Springer Verlag.","DOI":"10.1007\/978-3-642-39250-4"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Mu, Y., and Yin, Y. (2010, January 29\u201331). Task-oriented spoken dialogue system for humanoid robot. Proceedings of the IEEE 2010 International Conference on Multimedia Technology (ICMT), Ningbo, China.","DOI":"10.1109\/ICMULT.2010.5631205"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1007\/s12369-011-0123-x","article-title":"Grounding the interaction: Anchoring situated discourse in everyday human-robot interaction","volume":"4","author":"Lemaignan","year":"2012","journal-title":"Int. J. Soc. Robot."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Breuing, A., Waltinger, U., and Wachsmuth, I. (2011, January 22\u201327). Harvesting wikipedia knowledge to identify topics in ongoing natural language dialogs. Proceedings of the 2011 IEEE\/WIC\/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), Lyon, France.","DOI":"10.1109\/WI-IAT.2011.158"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Tenorth, M., Nyga, D., and Beetz, M. (2010, January 3\u20137). Understanding and executing instructions for everyday manipulation tasks from the world wide web. Proceedings of the 2010 IEEE International Conference on Robotics and Automation (ICRA), Anchorage, AK, USA.","DOI":"10.1109\/ROBOT.2010.5509955"},{"key":"ref_36","unstructured":"Gupta, R., and Kochenderfer, M.J. (2004, January 25\u201329). Common sense data acquisition for indoor mobile robots. Proceedings of the National Conference on Artificial Intelligence, San Jose, CA, USA."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"311","DOI":"10.3233\/AIS-2009-0040","article-title":"Grounding commonsense knowledge in intelligent systems","volume":"1","author":"Daoutis","year":"2009","journal-title":"J. Ambient Intell. Smart Environ."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"538","DOI":"10.14778\/1453856.1453916","article-title":"Webtables: Exploring the power of tables on the web","volume":"1","author":"Cafarella","year":"2008","journal-title":"Proc. VLDB Endow."},{"key":"ref_39","unstructured":"Schiff, S. (The New Yorker, 2006). Know it all, The New Yorker."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Carlson, A., Betteridge, J., Wang, R.C., Hruschka, E.R., and Mitchell, T.M. (2010, January 3\u20136). Coupled semi-supervised learning for information extraction. Proceedings of the Third ACM International Conference on Web Search and Data Mining, New York, NY, USA.","DOI":"10.1145\/1718487.1718501"},{"key":"ref_41","unstructured":"Fader, A., Soderland, S., and Etzioni, O. (2011, January 27\u201331). Identifying relations for open information extraction. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Edinburgh, UK."},{"key":"ref_42","unstructured":"Nie, Z., Wu, F., Wen, J.-R., and Ma, W.-Y. (2006, January 3\u20138). Extracting objects from the web. Proceedings of the 22nd International Conference on Data Engineering, Atlanta, GA, USA."},{"key":"ref_43","unstructured":"Matuszek, C., Witbrock, M., Kahlert, R.C., Cabral, J., Schneider, D., Shah, P., and Lenat, D. (2005, January 9\u201313). Searching for common sense: Populating cycTM from the web. Proceedings of the National Conference on Artificial Intelligence, Pittsburgh, PA, USA."},{"key":"ref_44","first-page":"100","article-title":"Toward open knowledge enabling for human-robot interaction","volume":"1","author":"Chen","year":"2012","journal-title":"J. Hum. Robot Interact."},{"key":"ref_45","unstructured":"Chen, X., Ji, J., Sui, Z., and Xie, J. (2013, January 3\u20139). Handling open knowledge for service robots. Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, Beijing, China."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Zhou, K., Zillich, M., Zender, H., and Vincze, M. (2012, January 7\u201312). Web mining driven object locality knowledge acquisition for efficient robot behavior. Proceedings of the International Conference on Intelligent Robots and Systems (IROS), Vilamoura, Portugal.","DOI":"10.1109\/IROS.2012.6385931"},{"key":"ref_47","unstructured":"Samadi, M. (2015). Facts and Reasons: Anytime Web Information Querying to Support Agents and Human Decision Making. [Ph.D. Thesis, Computer Science Department, Carnegie Mellon University]."},{"key":"ref_48","unstructured":"Samadi, M., Veloso, M., and Blum, M. (2013, January 14\u201318). OpenEval: Web information query evaluation. Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence (AAAI-13), Bellevue, WA, USA."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/4\/2\/223\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T20:48:02Z","timestamp":1760215682000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/4\/2\/223"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,6,17]]},"references-count":48,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2015,6]]}},"alternative-id":["robotics4020223"],"URL":"https:\/\/doi.org\/10.3390\/robotics4020223","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,6,17]]}}}