{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T14:27:20Z","timestamp":1762957640759,"version":"3.41.0"},"reference-count":24,"publisher":"World Scientific Pub Co Pte Ltd","issue":"01","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Semantic Computing"],"published-print":{"date-parts":[[2025,3]]},"abstract":"<jats:p> Designing plans that allow robots to carry out actions such as grasping an object or cutting a fruit is a time-consuming activity requiring specific skills and knowledge. The recent success of Generative Large Language Models (LLMs) has opened new avenues for code generation. In order to evaluate the ability of LLMs to generate code representing manipulation plans, we carry out experiments with different LLMs in the CRAM framework and its PyCRAM implementation. <\/jats:p><jats:p> In our experimental framework, we ask an LLM such as ChatGPT or GPT-4 to generate a plan for a specific target action given the plan for a given reference action as an example. We evaluate the generated designators against a ground truth designator using machine translation and code generation metrics, as well as assessing whether they compile or can be simulated. We find that GPT-4 slightly outperforms ChatGPT, but both models achieve a solid performance above all evaluated metrics. However, only 35% of the generated CRAM designators compile successfully and the generated PyCRAM designators fulfill roughly 42% of the action\u2019s simulation success criteria. In addition, we assess how the chosen reference action influences the code generation quality as well as the compilation success. Unexpectedly, the action similarity negatively correlates with compilation success. With respect to the metrics, we obtain either a positive or negative correlation depending on the model and programming language used. <\/jats:p>","DOI":"10.1142\/s1793351x25410041","type":"journal-article","created":{"date-parts":[[2025,3,7]],"date-time":"2025-03-07T15:09:42Z","timestamp":1741360182000},"page":"79-103","source":"Crossref","is-referenced-by-count":4,"title":["Generation of Robot Manipulation Plans Using Generative Large Language Models"],"prefix":"10.1142","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0434-6781","authenticated-orcid":false,"given":"Jan-Philipp","family":"T\u00f6berg","sequence":"first","affiliation":[{"name":"Center for Cognitive Interaction Technology (CITEC) and Joint Research Center on Cooperative and Cognition-enabled AI (CoAI JRC), Bielefeld University, Bielefeld, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5009-7158","authenticated-orcid":false,"given":"Philip","family":"Frese","sequence":"additional","affiliation":[{"name":"Center for Cognitive Interaction Technology (CITEC) and Joint Research Center on Cooperative and Cognition-enabled AI (CoAI JRC), Bielefeld University, Bielefeld, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4771-441X","authenticated-orcid":false,"given":"Philipp","family":"Cimiano","sequence":"additional","affiliation":[{"name":"Center for Cognitive Interaction Technology (CITEC) and Joint Research Center on Cooperative and Cognition-enabled AI (CoAI JRC), Bielefeld University, Bielefeld, Germany"}]}],"member":"219","published-online":{"date-parts":[[2025,4,24]]},"reference":[{"key":"S1793351X25410041BIB001","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-018-9646-y"},{"key":"S1793351X25410041BIB004","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2010.5650146"},{"key":"S1793351X25410041BIB006","doi-asserted-by":"publisher","DOI":"10.1109\/IRC59093.2023.00039"},{"key":"S1793351X25410041BIB007","first-page":"9118","volume-title":"Proc. 39th Int. Conf. Machine Learning","author":"Huang W.","year":"2022"},{"key":"S1793351X25410041BIB008","doi-asserted-by":"publisher","DOI":"10.3390\/buildings13071772"},{"key":"S1793351X25410041BIB009","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48891.2023.10160591"},{"key":"S1793351X25410041BIB010","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-023-10135-3"},{"volume-title":"Workshop on Language and Robot Learning (LangRob) at the Conference on Robot Learning","year":"2023","author":"B\u00e4rmann L.","key":"S1793351X25410041BIB011"},{"volume-title":"Foundation Models for Decision Making @NeurIPS","year":"2023","author":"Xu M.","key":"S1793351X25410041BIB012"},{"volume-title":"AAAI 2023 Spring Symposium Challenges Requiring the Combination of Machine Learning Knowledge Engineering","year":"2023","author":"Cao Y.","key":"S1793351X25410041BIB013"},{"key":"S1793351X25410041BIB015","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-023-10133-5"},{"key":"S1793351X25410041BIB016","first-page":"1","volume-title":"Proc. 9th Annual Conf. Advances in Cognitive Systems","author":"Wray R. E.","year":"2021"},{"key":"S1793351X25410041BIB017","first-page":"1877","volume-title":"Advances in Neural Information Processing Systems","author":"Brown T.","year":"2020"},{"key":"S1793351X25410041BIB018","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219748"},{"key":"S1793351X25410041BIB019","doi-asserted-by":"publisher","DOI":"10.3115\/981732.981751"},{"key":"S1793351X25410041BIB020","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"S1793351X25410041BIB021","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-022-01965-7"},{"key":"S1793351X25410041BIB022","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"S1793351X25410041BIB023","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin C.-Y.","year":"2004"},{"key":"S1793351X25410041BIB024","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3049"},{"key":"S1793351X25410041BIB025","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111741"},{"key":"S1793351X25410041BIB027","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.859"},{"key":"S1793351X25410041BIB028","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.5317da47"},{"key":"S1793351X25410041BIB029","doi-asserted-by":"publisher","DOI":"10.1145\/3510454.3528648"}],"container-title":["International Journal of Semantic Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S1793351X25410041","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,23]],"date-time":"2025-06-23T03:13:03Z","timestamp":1750648383000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S1793351X25410041"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3]]},"references-count":24,"journal-issue":{"issue":"01","published-print":{"date-parts":[[2025,3]]}},"alternative-id":["10.1142\/S1793351X25410041"],"URL":"https:\/\/doi.org\/10.1142\/s1793351x25410041","relation":{},"ISSN":["1793-351X","1793-7108"],"issn-type":[{"type":"print","value":"1793-351X"},{"type":"electronic","value":"1793-7108"}],"subject":[],"published":{"date-parts":[[2025,3]]}}}