{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T17:33:08Z","timestamp":1754155988926,"version":"3.41.2"},"reference-count":37,"publisher":"Emerald","issue":"3","license":[{"start":{"date-parts":[[2019,9,16]],"date-time":"2019-09-16T00:00:00Z","timestamp":1568592000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["LHT"],"published-print":{"date-parts":[[2019,9,16]]},"abstract":"Purpose<\/jats:title>The purpose of this paper is to formulate apposite criteria for ontology evaluation and test them through assessments of existing ontologies.<\/jats:p><\/jats:sec>Design\/methodology\/approach<\/jats:title>A literature review provided the basis from which to extract the categories relevant to an evaluation of internal ontology components. According to the ontology evaluation categories, a panel of experts provided the evaluation criteria for each category via Delphi survey. Reliability was gauged by applying the criteria to assessments of existing smartphone ontologies.<\/jats:p><\/jats:sec>Findings<\/jats:title>Existing research tends to approach ontology evaluation through comparison with well-engineered ontologies, implementation in target applications and appropriateness\/interconnection appraisals in relation to raw data, but such methodologies fall short of shedding light on the internal workings of ontologies, such as structure, semantic representation and interoperability. This study adopts its evaluation categories from previous research while also collecting concrete evaluation criteria from an expert panel and verifying the reliability of the resulting 53 criteria.<\/jats:p><\/jats:sec>Originality\/value<\/jats:title>This is the first published study to extract ontology evaluation criteria in terms of syntax, semantics and pragmatics. The results can be used as an evaluation index following ontology construction.<\/jats:p><\/jats:sec>","DOI":"10.1108\/lht-01-2019-0012","type":"journal-article","created":{"date-parts":[[2019,7,30]],"date-time":"2019-07-30T08:18:10Z","timestamp":1564474690000},"page":"338-354","source":"Crossref","is-referenced-by-count":0,"title":["Extracting and applying evaluation criteria for ontology quality assessment"],"prefix":"10.1108","volume":"37","author":[{"given":"Seonghun","family":"Kim","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sam G.","family":"Oh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"key2019092008524699300_ref001","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jbi.2018.02.010","article-title":"Assessing the practice of biomedical ontology evaluation: gaps and opportunities","volume":"80","year":"2018","journal-title":"Journal of Biomedical Informatics"},{"key":"key2019092008524699300_ref002","unstructured":"Brank, J., Mladenic, D. and Grobelnik, M. (2006), \u201cGold standard based ontology evaluation using instance assignment\u201d, paper presented at the 4th International Workshop on Evaluation of Ontologies for the Web(EON) at the 15th International World Wide Web Conference, available at: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.71.3681&rep=rep1&type=pdf (accessed February 20, 2018)."},{"key":"key2019092008524699300_ref003","unstructured":"Brewster, C., Alani, H., Dasmahapatra, S. and Wilks, Y. (2004), \u201cData driven ontology evaluation\u201d, paper presented at International Conference on Language Resources and Evaluation, available at: www.cbrewster.com\/papers\/BrewsterLREC.pdf (accessed February 22, 2018)."},{"key":"key2019092008524699300_ref301","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.datak.2004.11.010","article-title":"A semiotic metrics suite for assessing the quality of ontologies","volume":"55","year":"2005","journal-title":"Data & Knowledge Engineering"},{"issue":"3","key":"key2019092008524699300_ref004","first-page":"75","article-title":"Implementation of geo-ontology for geographical concepts: a case of coastal topology","volume":"13","year":"2013","journal-title":"Journal of the Korean Cartographic Association"},{"key":"key2019092008524699300_ref005","doi-asserted-by":"crossref","unstructured":"Degbelo, A. (2017), \u201cA snapshot of ontology evaluation criteria and strategies\u201d, paper presented at Semantics, September 11-14, Amsterdam, available at: www.researchgate.net\/publication\/319979589_A_Snapshot_of_Ontology_Evaluation_Criteria_and_Strategies (accessed October 21, 2018).","DOI":"10.1145\/3132218.3132219"},{"journal-title":"Logical Foundations of Artificial Intelligence","year":"1987","key":"key2019092008524699300_ref302"},{"issue":"2","key":"key2019092008524699300_ref006","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1006\/knac.1993.1008","article-title":"A translation approach to portable ontology specifications","volume":"5","year":"1993","journal-title":"Knowledge Acquisition"},{"year":"2014","key":"key2019092008524699300_ref007","article-title":"DBpedia ontology population coverage enhancement with framenet"},{"key":"key2019092008524699300_ref008","unstructured":"Kim, S. (2015), \u201cDeveloping an ontology and its usability to social data analytics\u201d, PhD dissertation, Sungkyunkwan University."},{"issue":"3","key":"key2019092008524699300_ref009","doi-asserted-by":"crossref","first-page":"119","DOI":"10.3743\/KOSIM.2007.24.3.119","article-title":"An implementation of inference-based web ontology for intelligent image retrieval system","volume":"24","year":"2007","journal-title":"Journal of the Korean Society for Information Management"},{"issue":"1","key":"key2019092008524699300_ref010","doi-asserted-by":"crossref","first-page":"149","DOI":"10.3743\/KOSIM.2008.25.1.149","article-title":"Web ontology modeling based on description logic and SWRL","volume":"25","year":"2008","journal-title":"Journal of the Korean Society for Information Management"},{"issue":"4","key":"key2019092008524699300_ref011","doi-asserted-by":"crossref","first-page":"97","DOI":"10.3743\/KOSIM.2005.22.4.097","article-title":"A study on the naming rules of metadata based on ontology","volume":"22","year":"2005","journal-title":"Journal of the Korean Society for Information Management"},{"issue":"3","key":"key2019092008524699300_ref012","doi-asserted-by":"crossref","first-page":"67","DOI":"10.14404\/JKSARM.2013.13.3.067","article-title":"Design and evaluation of an individual instance-based ontology retrieval system for archival records of the \u2018Saemaul movement\u2019","volume":"13","year":"2013","journal-title":"Journal of Korean Society of Archives and Records Management"},{"volume-title":"Research Methodology","year":"2009","key":"key2019092008524699300_ref013"},{"issue":"3","key":"key2019092008524699300_ref014","doi-asserted-by":"crossref","first-page":"343","DOI":"10.3743\/KOSIM.2007.24.3.343","article-title":"A design and implementation of ontology-based retrieval system for the electronic records of universities","volume":"24","year":"2007","journal-title":"Journal of the Korean Society for Information Management"},{"issue":"3","key":"key2019092008524699300_ref015","doi-asserted-by":"crossref","first-page":"43","DOI":"10.3743\/KOSIM.2007.24.3.043","article-title":"Automatic recommendation of panel pool using a probabilistic ontology and researcher networks","volume":"24","year":"2007","journal-title":"Journal of the Korean Society for Information Management"},{"issue":"15","key":"key2019092008524699300_ref016","first-page":"1","article-title":"Ontometric: a method to choose the appropriate ontology","volume":"2","year":"2004","journal-title":"Journal of Database Management"},{"key":"key2019092008524699300_ref017","unstructured":"Maedche, A. and Staab, S. (2002), \u201cMeasuring similarity between ontologies\u201d, paper presented at 13th International Conference, EKAW, Sig\u00fcenza, October 1-4, available at: www.researchgate.net\/profile\/Steffen_Staab\/publication\/226450264_Measuring_Similarity_between_Ontologies\/links\/0deec52b95ed1a9923000000\/Measuring-Similarity-between-Ontologies.pdf?origin=publication_detail (accessed May 10, 2018)."},{"issue":"2","key":"key2019092008524699300_ref018","doi-asserted-by":"crossref","first-page":"177","DOI":"10.3743\/KOSIM.2011.28.2.177","article-title":"A study on constructing the ontology of LIS journal","volume":"28","year":"2011","journal-title":"Journal of the Korean Society for Information Management"},{"issue":"4","key":"key2019092008524699300_ref019","first-page":"69","article-title":"Measurement criteria for ontology extraction tools","volume":"14","year":"2008","journal-title":"Journal of Intelligent Information Systems"},{"key":"key2019092008524699300_ref020","unstructured":"Porzel, R. and Malaka, R. (2004), \u201cA task-based approach for ontology evaluation\u201d, paper presented at ECAI Workshop on Ontology Learning and Population, Valencia, available at: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.93.6109&rep=rep1&type=pdf (accessed April 21, 2018)."},{"key":"key2019092008524699300_ref021","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1016\/j.compchemeng.2015.12.003","article-title":"Ontology evaluation for reuse in the domain of process systems engineering","volume":"85","year":"2016","journal-title":"Computers & Chemical Engineering"},{"issue":"3","key":"key2019092008524699300_ref022","doi-asserted-by":"crossref","first-page":"137","DOI":"10.5859\/KAIS.2012.21.3.137","article-title":"Using ontologies for semantic text mining","volume":"21","year":"2012","journal-title":"The Journal of Information Systems"},{"key":"key2019092008524699300_ref023","unstructured":"Gomez-Perez, A., Fernandez, M. and de Vicente, A.J. (1996), \u201cTowards a method to conceptualize domain ontologies\u201d, paper presented at ECAI-96 Workshop on Ontological Engineering, Budapest, available at: http:\/\/oa.upm.es\/7228\/1\/Towards_a_Method_.pdf (accessed April 21, 2018)."},{"key":"key2019092008524699300_ref024","unstructured":"Gr\u00fcninger, M. and Fox, M.S. (1994), \u201cThe design and evaluation of ontologies for enterprise engineering\u201d, paper presented at Workshop on Implemented Ontologies, European Conference on Artificial Intelligence, Amsterdam, available at: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.29.7053&rep=rep1&type=pdf (accessed April 21, 2018)."},{"key":"key2019092008524699300_ref025","unstructured":"Gr\u00fcninger, M. and Fox, M.S. (1995a), \u201cMethodology for the design and evaluation of ontologies\u201d, paper presented at IJCAI-95 Workshop on Basic Ontological Issues in Knowledge Sharing, August 19-20, Montreal, available at: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download;jsessionid=43DEE42CC7E6543B1AA300BF26940F40?doi=10.1.1.44.8723&rep=rep1&type=pdf (accessed April 21, 2018)."},{"key":"key2019092008524699300_ref026","doi-asserted-by":"crossref","unstructured":"Gr\u00fcninger, M. and Fox, M.S. (1995b), \u201cThe role of competency questions in enterprise engineering\u201d, in Rolstad\u00e5s, A. (Ed.), Benchmarking \u2013 Theory and Practice, Springer, Boston, MA, pp. 22-31.","DOI":"10.1007\/978-0-387-34847-6_3"},{"issue":"5","key":"key2019092008524699300_ref029","first-page":"765","article-title":"Study on a methodology for developing Shanghanlun ontology","volume":"25","year":"2011","journal-title":"Journal of Physiology & Pathology in Korean Medicine"},{"issue":"3","key":"key2019092008524699300_ref027","first-page":"275","article-title":"Implementation a philosophy ontology based on knowledge of text contents","volume":"11","year":"2006","journal-title":"Journal of KIISE: Computing Practices and Letters"},{"key":"key2019092008524699300_ref028","first-page":"89","article-title":"A case study on building verb-ontology: Korean locomotion verbs","volume":"38","year":"2006","journal-title":"Language and Linguistics"},{"key":"key2019092008524699300_ref032","unstructured":"Noy, N.F. and McGuinness, D.L. (2001), \u201cOntology development 101: a guide to creating your first ontology\u201d, available at: www.corais.org\/sites\/default\/files\/ontology_development_101_aguide_to_creating_your_first_ontology.pdf (accessed April 21, 2018)."},{"key":"key2019092008524699300_ref030","unstructured":"Ra, M., Yoo, D., No, S., Shin, J. and Han, C. (2012), \u201cThe mixed ontology building methodology using database information\u201d, paper presented at the International MultiConference of Engineers and Computer Scientists, available at: www.iaeng.org\/publication\/IMECS2012\/IMECS2012_pp68-73.pdf (accessed April 21, 2018)."},{"issue":"2","key":"key2019092008524699300_ref031","first-page":"161","article-title":"An establishment of entrepreneurship ontology through analysis of intellectual structure in entrepreneurship research","volume":"20","year":"2013","journal-title":"Journal of Information Technology Applications & Management"},{"key":"key2019092008524699300_ref034","unstructured":"Uschold, M. (1996a), \u201cConverting an informal ontology into Ontolingua: some experiences\u201d, paper presented at ECAI-96 Workshop on Ontological Engineering, Budapest, August 13, available at: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.24.647&rep=rep1&type=pdf (accessed April 21, 2018)."},{"key":"key2019092008524699300_ref035","unstructured":"Uschold, M. (1996b), \u201cBuilding ontologies: towards a unified methodology\u201d, paper presented at Expert Systems 96, the 16th Annual Conference of the British Computer Society Specialist Group on Expert Systems, December 16-18, Cambridge, available at: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.466.9461&rep=rep1&type=pdf (accessed April 21, 2018)."},{"key":"key2019092008524699300_ref033","unstructured":"Uschold, M. and King, M. (1995), \u201cTowards a methodology for building ontologies\u201d, paper presented at IJCAI-95 Workshop on Basic Ontological Issues in Knowledge Sharing, Montreal, available at: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.480.1214&rep=rep1&type=pdf (accessed April 21, 2018)."}],"container-title":["Library Hi Tech"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/LHT-01-2019-0012\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/LHT-01-2019-0012\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T22:13:54Z","timestamp":1753395234000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/lht\/article\/37\/3\/338-354\/263084"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,16]]},"references-count":37,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,9,16]]}},"alternative-id":["10.1108\/LHT-01-2019-0012"],"URL":"https:\/\/doi.org\/10.1108\/lht-01-2019-0012","relation":{},"ISSN":["0737-8831"],"issn-type":[{"type":"print","value":"0737-8831"}],"subject":[],"published":{"date-parts":[[2019,9,16]]}}}