{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T17:45:41Z","timestamp":1785606341801,"version":"3.56.0"},"reference-count":20,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T00:00:00Z","timestamp":1684713600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T00:00:00Z","timestamp":1684713600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100009473","name":"Universidad de M\u00e1laga","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100009473","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Softw Syst Model"],"published-print":{"date-parts":[[2023,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Most experts agree that large language models (LLMs), such as those used by Copilot and ChatGPT, are expected to revolutionize the way in which software is developed. Many papers are currently devoted to analyzing the potential advantages and limitations of these generative AI models for writing code. However, the analysis of the current state of LLMs with respect to software modeling has received little attention. In this paper, we investigate the current capabilities of ChatGPT to perform modeling tasks and to assist modelers, while also trying to identify its main shortcomings. Our findings show that, in contrast to code generation, the performance of the current version of ChatGPT for software modeling is limited, with various syntactic and semantic deficiencies, lack of consistency in responses and scalability issues. We also outline our views on how we perceive the role that LLMs can play in the software modeling discipline in the short term, and how the modeling community can help to improve the current capabilities of ChatGPT and the coming LLMs for software modeling.<\/jats:p>","DOI":"10.1007\/s10270-023-01105-5","type":"journal-article","created":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T06:02:03Z","timestamp":1684735323000},"page":"781-793","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":192,"title":["On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML"],"prefix":"10.1007","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6717-4775","authenticated-orcid":false,"given":"Javier","family":"C\u00e1mara","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1314-9694","authenticated-orcid":false,"given":"Javier","family":"Troya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7779-8810","authenticated-orcid":false,"given":"Lola","family":"Burgue\u00f1o","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8139-9986","authenticated-orcid":false,"given":"Antonio","family":"Vallecillo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,5,22]]},"reference":[{"key":"1105_CR1","unstructured":"Atenea Research Group: Git repository: chatgpt-uml (2023). https:\/\/github.com\/atenearesearchgroup\/chatgpt-uml"},{"key":"1105_CR2","doi-asserted-by":"crossref","unstructured":"Barke, S., James, M.B., Polikarpova, N.: Grounded copilot: How programmers interact with code-generating models. (2022). CoRR arXiv:2206.15000","DOI":"10.1145\/3586030"},{"key":"1105_CR3","doi-asserted-by":"crossref","unstructured":"Borji, A.: A categorical archive of chatgpt failures. (2023). CoRR arXiv:2302.03494","DOI":"10.21203\/rs.3.rs-2895792\/v1"},{"key":"1105_CR4","doi-asserted-by":"publisher","unstructured":"Burgue\u00f1o, L., Claris\u00f3, R., G\u00e9rard, S., Li, S., Cabot, J.: An NLP-based architecture for the autocompletion of partial domain models. In: Proc. of CAiSE\u201921, LNCS, vol. 12751, pp. 91\u2013106. Springer (2021). https:\/\/doi.org\/10.1007\/978-3-030-79382-1_6","DOI":"10.1007\/978-3-030-79382-1_6"},{"key":"1105_CR5","doi-asserted-by":"publisher","unstructured":"Cabot, J., Ravent\u00f3s, R.: Roles as entity types: a conceptual modelling pattern. In: Proc. of ER\u201904, LNCS, vol. 3288, pp. 69\u201382. Springer (2004). https:\/\/doi.org\/10.1007\/978-3-540-30464-7_7","DOI":"10.1007\/978-3-540-30464-7_7"},{"issue":"3","key":"1105_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.5381\/jot.2022.21.3.a4","volume":"21","author":"T Capuano","year":"2022","unstructured":"Capuano, T., Sahraoui, H.A., Fr\u00e9nay, B., Vanderose, B.: Learning from code repositories to recommend model classes. J. Object Technol. 21(3), 1\u201311 (2022). https:\/\/doi.org\/10.5381\/jot.2022.21.3.a4","journal-title":"J. Object Technol."},{"key":"1105_CR7","doi-asserted-by":"crossref","unstructured":"Chaaben, M.B., Burgue\u00f1o, L., Sahraoui, H.: Towards using few-shot prompt learning for automating model completion. In: Proc. of ICSE (NIER)\u201923. IEEE\/ACM (2023)","DOI":"10.1109\/ICSE-NIER58687.2023.00008"},{"key":"1105_CR8","unstructured":"D\u00f6derlein, J., Acher, M., Khelladi, D.E., Combemale, B.: Piloting copilot and codex: Hot temperature, cold prompts, or black magic? (2022). CoRR arXiv:2210.14699"},{"key":"1105_CR9","unstructured":"GitHub: Copilot: Your AI pair programmer (2023). https:\/\/github.com\/features\/copilot\/"},{"issue":"10","key":"1105_CR10","doi-asserted-by":"publisher","first-page":"1737","DOI":"10.14778\/3401960.3401970","volume":"13","author":"H Kim","year":"2020","unstructured":"Kim, H., So, B.H., Han, W.S., Lee, H.: Natural language to SQL: Where are we today? Proc. VLDB Endow. 13(10), 1737\u20131750 (2020). https:\/\/doi.org\/10.14778\/3401960.3401970","journal-title":"Proc. VLDB Endow."},{"key":"1105_CR11","unstructured":"Marcusarchive, G., Davisarchive, E.: GPT-3, Bloviator: OpenAI\u2019s language generator has no idea what it\u2019s talking about (2020). https:\/\/www.technologyreview.com\/2020\/08\/22\/1007539\/gpt3-openai-language-generator-artificial-intelligence-ai-opinion\/"},{"key":"1105_CR12","unstructured":"Meyer, B.: What Do ChatGPT and AI-based Automatic Program Generation Mean for the Future of Software. Commun. ACM 65(12), 5 (2022). https:\/\/cacm.acm.org\/blogs\/blog-cacm\/268103-what-do-chatgpt-and-ai-based-automatic-program-generation-mean-for-the-future-of-software\/fulltext"},{"key":"1105_CR13","unstructured":"Mok, A.: \u2018Prompt engineering\u2019 is one of the hottest jobs in generative AI. Here\u2019s how it works. Business Insider (2023). https:\/\/www.businessinsider.com\/prompt-engineering-ai-chatgpt-jobs-explained-2023-3"},{"key":"1105_CR14","unstructured":"Open AI: ChatGPT (2023). https:\/\/chat.openai.com\/chat"},{"key":"1105_CR15","unstructured":"Pirotte, A., Zim\u00e1nyi, E., Massart, D., Yakusheva, T.: Materialization: A powerful and ubiquitous abstraction pattern. In: Proc. of VLDB\u201994, pp. 630\u2013641. Morgan Kaufmann (1994). http:\/\/www.vldb.org\/conf\/1994\/P630.PDF"},{"key":"1105_CR16","doi-asserted-by":"publisher","unstructured":"Rocco, J.D., Sipio, C.D., Ruscio, D.D., Nguyen, P.T.: A GNN-based recommender system to assist the specification of metamodels and models. In: Proc. of MODELS\u201922, pp. 70\u201381. IEEE (2021). https:\/\/doi.org\/10.1109\/MODELS50736.2021.00016","DOI":"10.1109\/MODELS50736.2021.00016"},{"issue":"3","key":"1105_CR17","doi-asserted-by":"publisher","first-page":"1015","DOI":"10.1007\/s10270-021-00942-6","volume":"21","author":"R Saini","year":"2022","unstructured":"Saini, R., Mussbacher, G., Guo, J.L.C., Kienzle, J.: Automated, interactive, and traceable domain modeling empowered by artificial intelligence. Softw. Syst. Model. 21(3), 1015\u20131045 (2022). https:\/\/doi.org\/10.1007\/s10270-021-00942-6","journal-title":"Softw. Syst. Model."},{"issue":"3","key":"1105_CR18","doi-asserted-by":"publisher","first-page":"856","DOI":"10.1002\/spe.3170","volume":"53","author":"M Savary-Leblanc","year":"2023","unstructured":"Savary-Leblanc, M., Burgue\u00f1o, L., Cabot, J., Pallec, X.L., G\u00e9rard, S.: Software assistants in software engineering: a systematic mapping study. Softw. Pract .Exp. 53(3), 856\u2013892 (2023). https:\/\/doi.org\/10.1002\/spe.3170","journal-title":"Softw. Pract .Exp."},{"key":"1105_CR19","doi-asserted-by":"publisher","unstructured":"Vaithilingam, P., Zhang, T., Glassman, E.L.: Expectation vs. Experience: evaluating the usability of code generation tools powered by large language models. In: Proc. of CHI\u201922, pp. 332:1\u2013332:7. ACM (2022). https:\/\/doi.org\/10.1145\/3491101.3519665","DOI":"10.1145\/3491101.3519665"},{"issue":"3","key":"1105_CR20","doi-asserted-by":"publisher","first-page":"1071","DOI":"10.1007\/s10270-022-00975-5","volume":"21","author":"M Weyssow","year":"2022","unstructured":"Weyssow, M., Sahraoui, H.A., Syriani, E.: Recommending metamodel concepts during modeling activities with pre-trained language models. Softw. Syst. Model. 21(3), 1071\u20131089 (2022). https:\/\/doi.org\/10.1007\/s10270-022-00975-5","journal-title":"Softw. Syst. Model."}],"container-title":["Software and Systems Modeling"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10270-023-01105-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10270-023-01105-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10270-023-01105-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,13]],"date-time":"2023-12-13T14:19:51Z","timestamp":1702477191000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10270-023-01105-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,22]]},"references-count":20,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,6]]}},"alternative-id":["1105"],"URL":"https:\/\/doi.org\/10.1007\/s10270-023-01105-5","relation":{},"ISSN":["1619-1366","1619-1374"],"issn-type":[{"value":"1619-1366","type":"print"},{"value":"1619-1374","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,22]]},"assertion":[{"value":"15 March 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 April 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 April 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 May 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}