{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T04:29:57Z","timestamp":1778214597642,"version":"3.51.4"},"reference-count":24,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2024,11,1]],"date-time":"2024-11-01T00:00:00Z","timestamp":1730419200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Applied Ontology"],"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:p>\n                    \n                    \n                    \n                    In the continuously evolving landscape of knowledge engineering, the symbiosis and teaming of humans and machines emerge as a pivotal new domain. This article explores the multifaceted realms of human and machine collaborative ontology engineering (OE). The goal of the presented work is to explore the potential of Large Language Models (LLMs) to speed up and automate the processes of collaborative OE, experimenting with different levels of LLM involvement. The proposed approach is based on a human-centered approach, that is, the HCOME approach to collaborative OE, and follows a process of exploring the declining involvement of humans and the parallel increase of LLM involvement, concluding at a level of automation where the OE is exclusively performed by LLMs. This experimentation is organized based on a series of human\/LLM collaboration levels (a spectrum of OE), each one aligned to a specific OE methodology, that is, Level-0 HCOME (Human), Level-1 X-HCOME (Human and LLMs), Level-2 SimX-HCOME (LLMs and Human), and Level-3 Sim-HCOME (LLMs). The evaluation of these methodologies (one per level) is performed by measuring the similarity of the generated ontologies against\n                    \n                    \u201creference\u201d ontologies\u00a0(precision, recall, and F1-score of reference-to-LLM-generated ontological mappings). The results presented in this paper demonstrate that while LLMs significantly expedite the OE process, the accuracy and completeness of the resulting ontologies are notably enhanced by maintaining a high level of human involvement. This study is expected to contribute to a deeper understanding of evolving dynamics in LLM-based\/enhanced OE, paving the way for future advancements toward more effective collaborative OE frameworks.\n                  <\/jats:p>","DOI":"10.1177\/15705838241305067","type":"journal-article","created":{"date-parts":[[2025,6,25]],"date-time":"2025-06-25T03:06:30Z","timestamp":1750820790000},"page":"334-367","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":15,"title":["From Human- to LLM-Centered Collaborative Ontology Engineering"],"prefix":"10.1177","volume":"19","author":[{"given":"Dimitrios","family":"Doumanas","sequence":"first","affiliation":[{"name":"Intelligent Systems Laboratory, Department of Cultural Technology and Communication, University of the Aegean, Mytilene, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Georgios","family":"Bouchouras","sequence":"additional","affiliation":[{"name":"Intelligent Systems Laboratory, Department of Cultural Technology and Communication, University of the Aegean, Mytilene, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andreas","family":"Soularidis","sequence":"additional","affiliation":[{"name":"Intelligent Systems Laboratory, Department of Cultural Technology and Communication, University of the Aegean, Mytilene, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Konstantinos","family":"Kotis","sequence":"additional","affiliation":[{"name":"Intelligent Systems Laboratory, Department of Cultural Technology and Communication, University of the Aegean, Mytilene, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"George","family":"Vouros","sequence":"additional","affiliation":[{"name":"Artificial Intelligence Laboratory, Department of Digital Systems, University of Pireaus, Pireaus, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2025,1,31]]},"reference":[{"key":"e_1_3_3_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-47240-4_22"},{"key":"e_1_3_3_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-88873-4_45"},{"key":"e_1_3_3_4_1","doi-asserted-by":"crossref","unstructured":"Caufield J. H. Hegde H. Emonet V. Harris N. L. Joachimiak M. P. Matentzoglu N. Kim H. Moxon S. A. T. Reese J. T. Haendel M. A. Robinson P. N. & Mungall C. J. (2023 April 5). Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning. arXiv.Org. https:\/\/arxiv.org\/abs\/2304.02711v1","DOI":"10.1093\/bioinformatics\/btae104"},{"key":"e_1_3_3_5_1","unstructured":"Funk M. Hosemann S. Jung J. C. Lutz C. (2023a). Towards Ontology Construction With Language Models (arXiv:2309.09898). arXiv. http:\/\/arxiv.org\/abs\/2309.09898"},{"key":"e_1_3_3_6_1","unstructured":"Inga J. Ruess M. Robens J. H. Nelius T. Kille S. Dahlinger P. Thomaschke R. Neumann G. Matthiesen S. Hohmann S. Kiesel A. (2021). Human\u2013Machine Symbiosis: A Multivariate Perspective for Physically Coupled Human\u2013Machine Systems (arXiv:2111.14681). arXiv. http:\/\/arxiv.org\/abs\/2111.14681"},{"key":"e_1_3_3_7_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888920000065"},{"key":"e_1_3_3_8_1","first-page":"345","volume-title":"Learning useful kick-off ontologies from query logs: HCOME revised2010 International Conference on Complex, Intelligent and Software Intensive Systems","author":"Kotis K.","year":"2010","unstructured":"Kotis K., Papasalouros A. (2010). Learning useful kick-off ontologies from query logs: HCOME revised. In 2010 International Conference on Complex, Intelligent and Software Intensive Systems (pp. 345\u2013351)."},{"issue":"12","key":"e_1_3_3_9_1","first-page":"1710","article-title":"Enhancing the collective knowledge for the engineering of ontologies in open and socially constructed learning spaces","volume":"17","author":"Kotis K.","year":"2011","unstructured":"Kotis K., Papasalouros A., Vouros G., Pappas N., Zoumpatianos K. (2011). Enhancing the collective knowledge for the engineering of ontologies in open and socially constructed learning spaces. Journal of Universal Computer Science, 17(12), 1710\u20131742. https:\/\/doi.org\/10.3217\/jucs-017-12-1710","journal-title":"Journal of Universal Computer Science"},{"key":"e_1_3_3_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-005-0227-4"},{"key":"e_1_3_3_11_1","doi-asserted-by":"crossref","unstructured":"Liu T. Xiong Q. Zhang S. (2023). When to use large language model: Upper bound analysis of BM25 algorithms in reading comprehension task. 2023 5th International Conference on Natural Language Processing (ICNLP) Guangzhou China pp. 1\u20134. https:\/\/doi.org\/10.1109\/ICNLP58431.2023.00049","DOI":"10.1109\/ICNLP58431.2023.00049"},{"key":"e_1_3_3_12_1","first-page":"1","volume-title":"Ontology-based modelling and reasoning for forest fire emergencies in resilient societiesProceedings of the 12th Hellenic Conference on Artificial Intelligence","author":"Masa P.","year":"2022","unstructured":"Masa P., Meditskos G., Kintzios S., Vrochidis S., Kompatsiaris I. (2022). Ontology-based modelling and reasoning for forest fire emergencies in resilient societies. Proceedings of the 12th Hellenic Conference on Artificial Intelligence, pp. 1\u20139."},{"key":"e_1_3_3_13_1","unstructured":"Mateiu P. Groza A. (2023). Ontology Engineering With Large Language Models (arXiv:2307.16699). arXiv. http:\/\/arxiv.org\/abs\/2307.16699"},{"key":"e_1_3_3_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.657"},{"key":"e_1_3_3_15_1","doi-asserted-by":"publisher","DOI":"10.3233\/AO-230072"},{"key":"e_1_3_3_16_1","first-page":"46","article-title":"Knowledge base construction from pre-trained language models by prompt learning","author":"Ning X.","year":"2022","unstructured":"Ning X., Celebi R. (2022). Knowledge base construction from pre-trained language models by prompt learning. In Knowledge Base Construction from Pre-trained Language Models 2022. vol. 3274, CEUR Workshop Proceedings (pp 46\u201354). 2022 Semantic Web Challenge on Knowledge Base Construction from Pre-Trained Language Models, Hanghzou, China, 1\/01\/22. https:\/\/ceur-ws.org\/Vol-3274\/paper4.pdf","journal-title":"Knowledge Base Construction from Pre-trained Language Models 2022. vol. 3274, CEUR Workshop Proceedings"},{"key":"e_1_3_3_17_1","unstructured":"Pan J. Z. Razniewski S. Kalo J.-C. Singhania S. Chen J. Dietze S. Jabeen H. Omeliyanenko J. Zhang W. Lissandrini M. Biswas R. de Melo G. Bonifati A. Vakaj E. Dragoni M. Graux D. (2023a). Large Language Models and Knowledge Graphs: Opportunities and Challenges (arXiv:2308.06374). arXiv. http:\/\/arxiv.org\/abs\/2308.06374"},{"key":"e_1_3_3_18_1","unstructured":"Pan S. Luo L. Wang Y. Chen C. Wang J. Wu X. (2023b). Unifying Large Language Models and Knowledge Graphs: A Roadmap (arXiv:2306.08302). arXiv. http:\/\/arxiv.org\/abs\/2306.08302"},{"key":"e_1_3_3_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3586183.3606763"},{"key":"e_1_3_3_20_1","doi-asserted-by":"publisher","DOI":"10.1055\/a-2264-5631"},{"key":"e_1_3_3_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-76298-0_76"},{"key":"e_1_3_3_22_1","doi-asserted-by":"crossref","unstructured":"Wang Z. Mao S. Wu W. Ge T. Wei F. Ji H. (2024). Unleashing the emergent cognitive synergy in large language models: a task-solving agent through multi-persona self-collaboration (arXiv:2307.05300). arXiv. http:\/\/arxiv.org\/abs\/2307.05300","DOI":"10.18653\/v1\/2024.naacl-long.15"},{"key":"e_1_3_3_23_1","doi-asserted-by":"publisher","DOI":"10.3390\/iot2040037"},{"key":"e_1_3_3_24_1","volume-title":"Wear4pdmove: An ontology for knowledge-based personalized health monitoring of PD patientsISWC 2023 Posters and Demos: 22nd International Semantic Web Conference","author":"Zafeiropolos N.","year":"2023","unstructured":"Zafeiropolos N., Bitilis P., Kotis K. (2023). Wear4pdmove: An ontology for knowledge-based personalized health monitoring of PD patients. In ISWC 2023 Posters and Demos: 22nd International Semantic Web Conference, November 6\u201310, 2023, Athens, Greece. chrome-extension: \/\/efaidnbmnnnibpcajpcglclefindmkaj\/https:\/\/hozo.jp\/ISWC2023_PD-Industry-proc\/ISWC2023_paper_433.pdf"},{"key":"e_1_3_3_25_1","unstructured":"Zhang Y. Chen Z. Fang Y. Cheng L. Lu Y. Li F. Zhang W. Chen H. (2023). Knowledgeable Preference Alignment for LLMs in Domain-Specific Question Answering (arXiv:2311.06503). arXiv. http:\/\/arxiv.org\/abs\/2311.06503"}],"container-title":["Applied Ontology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/15705838241305067","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/15705838241305067","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/15705838241305067","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T11:46:30Z","timestamp":1777376790000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/15705838241305067"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11]]},"references-count":24,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,11]]}},"alternative-id":["10.1177\/15705838241305067"],"URL":"https:\/\/doi.org\/10.1177\/15705838241305067","relation":{},"ISSN":["1570-5838","1875-8533"],"issn-type":[{"value":"1570-5838","type":"print"},{"value":"1875-8533","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11]]}}}