{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T15:45:21Z","timestamp":1770997521709,"version":"3.50.1"},"reference-count":35,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Autonomous Systems Initiative (ASI)"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>The introduction and ever-growing size of the transformer deep-learning architecture have had a tremendous impact not only in the field of natural language processing but also in other fields. The transformer-based language models have contributed to a renewed interest in commonsense knowledge due to the abilities of deep learning models. Recent literature has focused on analyzing commonsense embedded within the pre-trained parameters of these models and embedding missing commonsense using knowledge graphs and fine-tuning. We base our current work on the empirically proven language understanding of very large transformer-based language models to expand a limited commonsense knowledge graph, initially generated only on visual data. The few-shot-prompted pre-trained language models can learn the context of an initial knowledge graph with less bias than language models fine-tuned on a large initial corpus. It is also shown that these models can offer new concepts that are added to the vision-based knowledge graph. This two-step approach of vision mining and language model prompts results in the auto-generation of a commonsense knowledge graph well equipped with physical commonsense, which is human commonsense gained by interacting with the physical world. To prompt the language models, we adapted the chain-of-thought method of prompting. To the best of our knowledge, it is a novel contribution to the domain of the generation of commonsense knowledge, which can result in a five-fold cost reduction compared to the state-of-the-art. Another contribution is assigning fuzzy linguistic terms to the generated triples. The process is end to end in the context of knowledge graphs. It means the triples are verbalized to natural language, and after being processed, the results are converted back to triples and added to the commonsense knowledge graph.<\/jats:p>","DOI":"10.3390\/sym14081715","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T22:53:30Z","timestamp":1660776810000},"page":"1715","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Utilizing Language Models to Expand Vision-Based Commonsense Knowledge Graphs"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3945-9439","authenticated-orcid":false,"given":"Navid","family":"Rezaei","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T6G 1H9, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4783-0717","authenticated-orcid":false,"given":"Marek Z.","family":"Reformat","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T6G 1H9, Canada"},{"name":"Information Technology Institute, University of Social Sciences, 90-113 \u0141\u00f3d\u017a, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Hwang, J.D., Bhagavatula, C., Bras, R.L., Da, J., Sakaguchi, K., Bosselut, A., and Choi, Y. (2021, January 2\u20139). COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs. Proceedings of the AAAI, Virtual Conference.","DOI":"10.1609\/aaai.v35i7.16792"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"West, P., Bhagavatula, C., Hessel, J., Hwang, J.D., Jiang, L., Bras, R.L., Lu, X., Welleck, S., and Choi, Y. (2021). Symbolic knowledge distillation: From general language models to commonsense models. arXiv.","DOI":"10.18653\/v1\/2022.naacl-main.341"},{"key":"ref_3","unstructured":"LeCun, Y. (2022, June 27). A Path Towards Autonomous Machine Intelligence Version 0.9.2. Available online: https:\/\/openreview.net\/pdf?id=BZ5a1r-kVsf."},{"key":"ref_4","first-page":"139","article-title":"The Curious Case of Commonsense Intelligence","volume":"151","author":"Choi","year":"2022","journal-title":"J. Am. Acad. Arts Sci."},{"key":"ref_5","unstructured":"Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., and Brunskill, E. (2021). On the opportunities and risks of foundation models. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016, January 1\u20135). SQuAD: 100,000+ Questions for Machine Comprehension of Text. Proceedings of the EMNLP, Austin, TX, USA.","DOI":"10.18653\/v1\/D16-1264"},{"key":"ref_7","first-page":"415","article-title":"Image-Based World-perceiving Knowledge Graph (WpKG) with Imprecision","volume":"1237","author":"Rezaei","year":"2020","journal-title":"Inf. Process. Manag. Uncertain Knowl. Based Syst."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Rezaei, N., Reformat, M.Z., and Yager, R.R. (2022, January 11\u201315). Generating Contextual Weighted Commonsense Knowledge Graphs. Proceedings of the International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, Milan, Italy.","DOI":"10.1007\/978-3-031-08971-8_49"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Tang, K., Niu, Y., Huang, J., Shi, J., and Zhang, H. (2020, January 13\u201319). Unbiased Scene Graph Generation From Biased Training. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00377"},{"key":"ref_10","unstructured":"McCarthy, J. (1990). Formalizing Common Sense, Intellect Books."},{"key":"ref_11","unstructured":"Burges, C.J., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K.Q. Translating Embeddings for Modeling Multi-relational Data. Proceedings of the Advances in Neural Information Processing Systems."},{"key":"ref_12","unstructured":"Bengio, Y., and LeCun, Y. (2015, January 7\u20139). Embedding Entities and Relations for Learning and Inference in Knowledge Bases. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA. Conference Track Proceedings."},{"key":"ref_13","unstructured":"Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_14","unstructured":"Wang, C., Liu, X., and Song, D.X. (2020). Language Models are Open Knowledge Graphs. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Petroni, F., Rockt\u00e4schel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. (2019, January 3\u20137). Language Models as Knowledge Bases?. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1250"},{"key":"ref_16","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2\u20137). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA. Volume 1 (Long and Short Papers)."},{"key":"ref_17","first-page":"1877","article-title":"Language Models are Few-Shot Learners","volume":"Volume 33","author":"Larochelle","year":"2020","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_18","unstructured":"Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., and Lin, X.V. (2022). Opt: Open pre-trained transformer language models. arXiv."},{"key":"ref_19","unstructured":"Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., and Gehrmann, S. (2022). Palm: Scaling language modeling with pathways. arXiv."},{"key":"ref_20","unstructured":"Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022). Chain of thought prompting elicits reasoning in large language models. arXiv."},{"key":"ref_21","unstructured":"Rezaei, N., and Reformat, M.Z. (2022). Super-Prompting: Utilizing Model-Independent Contextual Data to Reduce Data Annotation Required in Visual Commonsense Tasks. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Khot, T., Sabharwal, A., and Clark, P. (2019, January 3\u20137). What\u2019s Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1281"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Fabbri, A.R., Ng, P., Wang, Z., Nallapati, R., and Xiang, B. (2020, January 5\u201310). Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.413"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Jastrz\u0119bski, S., Bahdanau, D., Hosseini, S., Noukhovitch, M., Bengio, Y., and Cheung, J. (2018, January 8\u201314). Commonsense mining as knowledge base completion? A study on the impact of novelty. Proceedings of the Workshop on Generalization in the Age of Deep Learning, Munich, Germany.","DOI":"10.18653\/v1\/W18-1002"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Davison, J., Feldman, J., and Rush, A.M. (2019, January 3\u20137). Commonsense knowledge mining from pretrained models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1109"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Chen, X., Shrivastava, A., and Gupta, A. (2013, January 1\u20138). NEIL: Extracting Visual Knowledge from Web Data. Proceedings of the IEEE International Conference on Computer Vision 2013, Sydney, Australia.","DOI":"10.1109\/ICCV.2013.178"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Speer, R., Chin, J., and Havasi, C. (2017, January 4\u20139). Conceptnet 5.5: An open multilingual graph of general knowledge. Proceedings of the Thirty-first AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.11164"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Xie, S., Girshick, R., Doll\u00e1r, P., Tu, Z., and He, K. (2017, January 21\u201326). Aggregated residual transformations for deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.634"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zellers, R., Yatskar, M., Thomson, S., and Choi, Y. (2018, January 18\u201323). Neural Motifs: Scene Graph Parsing with Global Context. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00611"},{"key":"ref_31","unstructured":"Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2019, January 6\u20139). The Curious Case of Neural Text Degeneration. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1080\/19312450709336664","article-title":"Answering the Call for a Standard Reliability Measure for Coding Data","volume":"1","author":"Hayes","year":"2007","journal-title":"Commun. Methods Meas."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Li, X., Taheri, A., Tu, L., and Gimpel, K. (2016, January 7\u201312). Commonsense Knowledge Base Completion. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany. (Volume 1: Long Papers).","DOI":"10.18653\/v1\/P16-1137"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zhang, H., Khashabi, D., Song, Y., and Roth, D. (2021, January 7\u201315). TransOMCS: From linguistic graphs to commonsense knowledge. Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, Yokohama, Japan.","DOI":"10.24963\/ijcai.2020\/554"},{"key":"ref_35","unstructured":"Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/14\/8\/1715\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:10:58Z","timestamp":1760141458000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/14\/8\/1715"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,17]]},"references-count":35,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["sym14081715"],"URL":"https:\/\/doi.org\/10.3390\/sym14081715","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,17]]}}}