{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T00:35:30Z","timestamp":1783384530181,"version":"3.54.6"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"22","license":[{"start":{"date-parts":[[2023,8,22]],"date-time":"2023-08-22T00:00:00Z","timestamp":1692662400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,8,22]],"date-time":"2023-08-22T00:00:00Z","timestamp":1692662400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["742671"],"award-info":[{"award-number":["742671"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007052","name":"Universit\u00e0 degli Studi di Verona","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100007052","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Appl Intell"],"published-print":{"date-parts":[[2023,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Natural language annotations and manuals can provide useful procedural information and relations for the highly specialized scenario of autonomous robotic task planning. In this paper, we propose and publicly release AUTOMATE, a pipeline for automatic task knowledge extraction from expert-written domain texts. AUTOMATE integrates semantic sentence classification, semantic role labeling, and identification of procedural connectors, in order to extract templates of Linear Temporal Logic (LTL) relations that can be directly implemented in any sufficiently expressive logic programming formalism for autonomous reasoning, assuming some low-level commonsense and domain-independent knowledge is available. This is the first work that bridges natural language descriptions of complex LTL relations and the automation of full robotic tasks. Unlike most recent similar works that assume strict language constraints in substantially simplified domains, we test our pipeline on texts that reflect the expressiveness of natural language used in available textbooks and manuals. In fact, we test AUTOMATE in the surgical robotic scenario, defining realistic language constraints based on a publicly available dataset. In the context of two benchmark training tasks with texts constrained as above, we show that automatically extracted LTL templates, after translation to a suitable logic programming paradigm, achieve comparable planning success in reduced time, with respect to logic programs written by expert programmers.<\/jats:p>","DOI":"10.1007\/s10489-023-04882-0","type":"journal-article","created":{"date-parts":[[2023,8,22]],"date-time":"2023-08-22T11:01:49Z","timestamp":1692702109000},"page":"26351-26363","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Mapping natural language procedures descriptions to linear temporal logic templates: an application in the surgical robotic domain"],"prefix":"10.1007","volume":"53","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8607-8495","authenticated-orcid":false,"given":"Marco","family":"Bombieri","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniele","family":"Meli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diego","family":"Dall\u2019Alba","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Rospocher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paolo","family":"Fiorini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,8,22]]},"reference":[{"key":"4882_CR1","unstructured":"Proposal for a regulation of the European parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain Union legislative acts (2021).https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/PDF\/?uri=CELEX:52021PC0206"},{"issue":"2","key":"4882_CR2","first-page":"1","volume":"13","author":"P Haslum","year":"2019","unstructured":"Haslum P, Lipovetzky N, Magazzeni D, Muise C (2019) An introduction to the planning domain definition language. Synth Lect Artif Intell Mach Learn 13(2):1\u2013187","journal-title":"Synth Lect Artif Intell Mach Learn"},{"key":"4882_CR3","doi-asserted-by":"crossref","unstructured":"Apt KR (1990) Logic programming. Handbook of Theoretical Computer Science, Volume B: Formal Models and Sematics (B), 493\u2013574","DOI":"10.1016\/B978-0-444-88074-1.50015-9"},{"key":"4882_CR4","first-page":"351","volume":"2018","author":"H Park","year":"2018","unstructured":"Park H, Motahari Nezhad HR (2018) Learning procedures from text: Codifying how-to procedures in deep neural networks. Comp Proc Web Conf 2018:351\u2013358","journal-title":"Comp Proc Web Conf"},{"key":"4882_CR5","doi-asserted-by":"crossref","unstructured":"Hsiung E, Mehta H, Chu J, Liu X, Patel R, Tellex S, Konidaris G (2022) Generalizing to new domains by mapping natural language to lifted ltl. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 3624\u20133630. IEEE","DOI":"10.1109\/ICRA46639.2022.9812169"},{"key":"4882_CR6","doi-asserted-by":"crossref","unstructured":"Pnueli A (1977) The temporal logic of programs. In: 18th Annual Symposium on Foundations of Computer Science (1977), pp. 46 \u2013 57. IEEE","DOI":"10.1109\/SFCS.1977.32"},{"issue":"4","key":"4882_CR7","doi-asserted-by":"publisher","first-page":"8638","DOI":"10.1126\/scirobotics.aam8638","volume":"2","author":"G-Z Yang","year":"2017","unstructured":"Yang G-Z, Cambias J, Cleary K, Daimler E, Drake J, Dupont PE, Hata N, Kazanzides P, Martel S, Patel RV et al (2017) Medical robotics-regulatory, ethical, and legal considerations for increasing levels of autonomy. Sci Robot 2(4):8638","journal-title":"Sci Robot"},{"key":"4882_CR8","doi-asserted-by":"crossref","unstructured":"Mujtaba D, Mahapatra N (2019) Recent trends in natural language understanding for procedural knowledge. In: 2019 International Conference on Computational Science and Computational Intelligence (CSCI), pp. 420\u2013424","DOI":"10.1109\/CSCI49370.2019.00082"},{"key":"4882_CR9","unstructured":"Brunello A, Montanari A, Reynolds, M (2019) Synthesis of LTL formulas from natural language texts: State of the art and research directions. In: 26th International Symposium on Temporal Representation and Reasoning, TIME 2019, October 16-19, 2019. LIPIcs, vol. 147, pp. 17\u201311719. Schloss Dagstuhl-Leibniz-Zentrum f\u00fcr Informatik, M\u00e1laga, Spain"},{"key":"4882_CR10","first-page":"1","volume-title":"52nd Hawaii International Conference on System Sciences, HICSS 2019","author":"T Wambsganss","year":"2019","unstructured":"Wambsganss T, Fromm H (2019) Mining user-generated repair instructions from automotive web communities. In: Bui T (ed) 52nd Hawaii International Conference on System Sciences, HICSS 2019. ScholarSpace, Grand Wailea, Maui, Hawaii, USA, pp 1\u201310"},{"key":"4882_CR11","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1007\/978-3-030-49435-3_17","volume-title":"Advanced Information Systems Engineering","author":"C Qian","year":"2020","unstructured":"Qian C, Wen L, Kumar A, Lin L, Lin L, Zong Z, Li S, Wang J (2020) An approach for process model extraction by multi-grained text classification. In: Dustdar S, Yu E, Salinesi C, Rieu D, Pant V (eds) Advanced Information Systems Engineering. Springer, Cham, pp 268\u2013282"},{"key":"4882_CR12","doi-asserted-by":"crossref","unstructured":"Nabizadeh N, Wersing H, Kolossa D (2021) Leveraging inter-step dependencies for information extraction from procedural task instructions . In: Text, Speech, and Dialogue - 24th International Conference, TSD, Proceedings. Lecture Notes in Computer Science , vol. 12848 , pp. 341 \u2013 353. Springer, Olomouc, Czech Republic","DOI":"10.1007\/978-3-030-83527-9_29"},{"issue":"4","key":"4882_CR13","doi-asserted-by":"publisher","first-page":"4748","DOI":"10.1007\/s10489-022-03692-0","volume":"53","author":"Y Jiang","year":"2023","unstructured":"Jiang Y, Zaporojets K, Deleu J, Demeester T, Develder C (2023) Cookdial: a dataset for task-oriented dialogs grounded in procedural documents. Appl Intell 53(4):4748\u20134766","journal-title":"Appl Intell"},{"key":"4882_CR14","doi-asserted-by":"publisher","first-page":"101003","DOI":"10.1016\/j.aei.2019.101003","volume":"43","author":"B Zhong","year":"2020","unstructured":"Zhong B, Xing X, Luo H, Zhou Q, Li H, Rose TM, Fang W (2020) Deep learning-based extraction of construction procedural constraints from construction regulations. Adv Eng Inf 43:101003","journal-title":"Adv Eng Inf"},{"key":"4882_CR15","doi-asserted-by":"publisher","first-page":"104696","DOI":"10.1016\/j.autcon.2022.104696","volume":"147","author":"X Wang","year":"2023","unstructured":"Wang X, El-Gohary N (2023) Deep learning-based relation extraction and knowledge graph-based representation of construction safety requirements. Autom Const 147:104696","journal-title":"Autom Const"},{"key":"4882_CR16","doi-asserted-by":"crossref","unstructured":"Chambers AJ, Stringfellow AM, Luo BB, Underwood SJ, Allard TG, Johnston IA, Brockman S, Shing L, Wollaber AB, VanDam C (2020) Automated business process discovery from unstructured natural-language documents. In: Business Process Management Workshops - BPM 2020 International Workshops. Lecture Notes in Business Information Processing , vol. 397, pp. 232\u2013243. Springer, Seville, Spain","DOI":"10.1007\/978-3-030-66498-5_18"},{"key":"4882_CR17","doi-asserted-by":"crossref","unstructured":"Ning Q, Zhou B, Feng Z, Peng H, Roth D (2018) Cogcomptime: A tool for understanding time in natural language . In: Blanco , E. , Lu , W. (eds.) Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018 , pp. 72 \u2013 77 . Association for Computational Linguistics, Belgium","DOI":"10.18653\/v1\/D18-2013"},{"issue":"8","key":"4882_CR18","doi-asserted-by":"publisher","first-page":"1287","DOI":"10.1007\/s11548-021-02370-9","volume":"16","author":"M Bombieri","year":"2021","unstructured":"Bombieri M, Rospocher M, Dall\u2019Alba D, Fiorini P (2021) Automatic detection of procedural knowledge in robotic-assisted surgical texts. Int J Comput Assist Radiol Surg 16(8):1287\u20131295","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"4882_CR19","doi-asserted-by":"crossref","unstructured":"Fellbau C (1998) Wordnet: An electronic lexical database","DOI":"10.7551\/mitpress\/7287.001.0001"},{"key":"4882_CR20","unstructured":"Bentivogli L, Bocco A, Pianta E (2004) Archiwordnet: integrating wordnet with domain-specific knowledge. In: Proceedings of the 2nd International Global Wordnet Conference, pp. 39\u201347"},{"key":"4882_CR21","doi-asserted-by":"crossref","unstructured":"Kazanzides P, Chen Z, Deguet A, Fischer GS, Taylor RH, DiMaio SP (2014) An open-source research kit for the da vinci surgical system. In: 2014 IEEE International Conference on Robotics and Automation, ICRA 2014, May 31 - June 7, 2014 , pp. 6434 \u2013 6439. IEEE , Hong Kong, China","DOI":"10.1109\/ICRA.2014.6907809"},{"issue":"9","key":"4882_CR22","first-page":"30","volume":"93","author":"NJ Soper","year":"2008","unstructured":"Soper NJ, Fried GM (2008) The fundamentals of laparoscopic surgery: its time has come. Bull Am Coll Surg 93(9):30\u201332","journal-title":"Bull Am Coll Surg"},{"key":"4882_CR23","doi-asserted-by":"crossref","unstructured":"Nagy TD, Haidegger TP (2021) Towards standard approaches for the evaluation of autonomous surgical subtask execution. In: 2021 IEEE 25th International Conference on Intelligent Engineering Systems (INES), pp. 67\u201374. IEEE","DOI":"10.1109\/INES52918.2021.9512901"},{"key":"4882_CR24","doi-asserted-by":"publisher","unstructured":"Meli D, Tagliabue E, Dall\u2019Alba D, Fiorini P (2021) Autonomous tissue retraction with a biomechanically informed logic based framework. In: 2021 International Symposium on Medical Robotics (ISMR). IEEE, Atlanta, GA, pp 1\u20137. https:\/\/doi.org\/10.1109\/ISMR48346.2021.9661573","DOI":"10.1109\/ISMR48346.2021.9661573"},{"key":"4882_CR25","doi-asserted-by":"crossref","unstructured":"Tagliabue E, Meli D, Dall\u2019alba D, Fiorini P (2022) Deliberation in autonomous robotic surgery: a framework for handling anatomical uncertainty. In: Proceedings-IEEE International Conference on Robotics and Automation, pp. 11080\u201311086","DOI":"10.1109\/ICRA46639.2022.9811820"},{"issue":"9","key":"4882_CR26","doi-asserted-by":"publisher","first-page":"1397","DOI":"10.1007\/s11548-018-1824-5","volume":"13","author":"B Gibaud","year":"2018","unstructured":"Gibaud B, Forestier G, Feldmann C, Ferrigno G, Gon\u00e7alves P, Haidegger T, Julliard C, Kati D, Kenngott H, Maier-Hein L et al (2018) Toward a standard ontology of surgical process models. Int J Comput Assist Radiol Surg. 13(9):1397\u20131408","journal-title":"Int J Comput Assist Radiol Surg."},{"issue":"1","key":"4882_CR27","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1109\/MRA.2016.2645444","volume":"24","author":"SR Fiorini","year":"2017","unstructured":"Fiorini SR, Bermejo-Alonso J, Gon\u00e7alves P, De Freitas EP, Alarcos AO, Olszewska JI, Prestes E, Schlenoff C, Ragavan SV, Redfield S et al (2017) A suite of ontologies for robotics and automation [industrial activities]. IEEE Robot Autom Mag 24(1):8\u201311","journal-title":"IEEE Robot Autom Mag"},{"key":"4882_CR28","unstructured":"Shi P, Lin J (2019) Simple BERT models for relation extraction and semantic role labeling. CoRR abs\/1904.05255"},{"key":"4882_CR29","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)"},{"key":"4882_CR30","doi-asserted-by":"crossref","unstructured":"Bird S (2006) NLTK: The Natural Language Toolkit. In: Proceedings of the COLING\/ACL 2006 Interactive Presentation Sessions , pp. 69 \u2013 72 . Association for Computational Linguistics , Sydney, Australia","DOI":"10.3115\/1225403.1225421"},{"issue":"1","key":"4882_CR31","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1162\/0891201053630264","volume":"31","author":"M Palmer","year":"2005","unstructured":"Palmer M, Kingsbury PR, Gildea D (2005) The proposition bank: An annotated corpus of semantic roles. Comput Linguistics 31(1):71\u2013106","journal-title":"Comput Linguistics"},{"key":"4882_CR32","doi-asserted-by":"crossref","unstructured":"Dobrovolskii V (2021) Word-level coreference resolution . In: Moens , M. , Huang , X. , Specia , L. , Yih , S.W. (eds.) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event, 7-11 November, 2021 , pp. 7670 \u2013 7675 . Association for Computational Linguistics , Punta Cana, Dominican Republic","DOI":"10.18653\/v1\/2021.emnlp-main.605"},{"key":"4882_CR33","unstructured":"Dogmus Z, Gezici G, Patoglu V, Erdem E (2012) Developing and maintaining an ontology for rehabilitation robotics. In: KEOD , pp. 389 \u2013 395"},{"key":"4882_CR34","doi-asserted-by":"crossref","unstructured":"Schlenoff C, Prestes E, Madhavan R, Goncalves P, Li H, Balakirsky S, Kramer T, Miguelanez E (2012) An ieee standard ontology for robotics and automation. In: 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems , pp. 1337 \u2013 1342. IEEE","DOI":"10.1109\/IROS.2012.6385518"},{"key":"4882_CR35","doi-asserted-by":"crossref","unstructured":"Guerram, T, Mellal N (2018) A domain independent approach for ontology semantic enrichment. Computer Science & Information Technology, 13\u201319","DOI":"10.5121\/csit.2018.80202"},{"key":"4882_CR36","doi-asserted-by":"publisher","first-page":"9011","DOI":"10.1007\/s10462-022-10389-w","volume":"56","author":"D Meli","year":"2023","unstructured":"Meli D, Nakawala H, Fiorini P (2023) Logic programming for deliberative robotic task planning. Artif Intell Rev 56:9011\u20139049","journal-title":"Artif Intell Rev"},{"key":"4882_CR37","unstructured":"Gebser M, Kaminski R, Kaufmann B, Ostrowski M, Schaub T, Wanko P (2016) Theory solving made easy with clingo 5. In: Technical Communications of the 32nd International Conference on Logic Programming (ICLP 2016). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik"},{"key":"4882_CR38","doi-asserted-by":"crossref","unstructured":"Ginesi M, Meli D, Roberti A, Sansonetto N, Fiorini P (2020) Autonomous task planning and situation awareness in robotic surgery. In: IEEE\/RSJ International Conference on Intelligent Robots and Systems , pp. 3144 \u2013 3150. IEEE","DOI":"10.1109\/IROS45743.2020.9341382"},{"issue":"4","key":"4882_CR39","doi-asserted-by":"publisher","first-page":"569","DOI":"10.1109\/TMRB.2020.3033670","volume":"2","author":"A Roberti","year":"2020","unstructured":"Roberti A, Piccinelli N, Meli D, Muradore R, Fiorini P (2020) Improving rigid 3-d calibration for robotic surgery. IEEE Trans Med Robot Bionics 2(4):569\u2013573","journal-title":"IEEE Trans Med Robot Bionics"},{"issue":"4","key":"4882_CR40","doi-asserted-by":"publisher","first-page":"8205","DOI":"10.1109\/LRA.2021.3104880","volume":"6","author":"D Meli","year":"2021","unstructured":"Meli D, Fiorini P (2021) Unsupervised identification of surgical robotic actions from small non-homogeneous datasets. IEEE Robot Autom Lett 6(4):8205\u20138212","journal-title":"IEEE Robot Autom Lett"},{"key":"4882_CR41","doi-asserted-by":"publisher","first-page":"1739","DOI":"10.1007\/s10994-021-06013-7","volume":"110","author":"D Meli","year":"2021","unstructured":"Meli D, Sridharan M, Fiorini P (2021) Inductive learning of answer set programs for autonomous surgical task planning. Mach Learn 110:1739\u20131763","journal-title":"Mach Learn"},{"key":"4882_CR42","doi-asserted-by":"crossref","unstructured":"Bombieri M, Rospocher M, Ponzetto SP, Fiorini P (2022) The Robotic Surgery Procedural Framebank . In: Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC 2022). European Language Resources Association (ELRA), Marseille, France","DOI":"10.1007\/s10579-023-09668-x"},{"key":"4882_CR43","doi-asserted-by":"publisher","first-page":"106415","DOI":"10.1016\/j.compbiomed.2022.106415","volume":"152","author":"M Bombieri","year":"2023","unstructured":"Bombieri M, Rospocher M, Ponzetto SP, Fiorini P (2023) Machine understanding surgical actions from intervention procedure textbooks. Comput Biol Med 152:106415","journal-title":"Comput Biol Med"}],"container-title":["Applied Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-023-04882-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10489-023-04882-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-023-04882-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,25]],"date-time":"2023-10-25T23:05:00Z","timestamp":1698275100000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10489-023-04882-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,22]]},"references-count":43,"journal-issue":{"issue":"22","published-print":{"date-parts":[[2023,11]]}},"alternative-id":["4882"],"URL":"https:\/\/doi.org\/10.1007\/s10489-023-04882-0","relation":{},"ISSN":["0924-669X","1573-7497"],"issn-type":[{"value":"0924-669X","type":"print"},{"value":"1573-7497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,22]]},"assertion":[{"value":"10 July 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 August 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All the authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}},{"value":"This article does not contain studies with human participants or animals.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Statement of informed consent is not applicable since the manuscript does not contain any patient data.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}}]}}