{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T15:26:04Z","timestamp":1781105164324,"version":"3.54.1"},"reference-count":39,"publisher":"IGI Global Scientific Publishing","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2013,1,1]]},"abstract":"<p>With the emergence of tools for collaborative ontology engineering, more and more data about the creation process behind collaborative construction of ontologies is becoming available. Today, collaborative ontology engineering tools such as Collaborative Prot\u00e9g\u00e9 offer rich and structured logs of changes, thereby opening up new challenges and opportunities to study and analyze the creation of collaboratively constructed ontologies. While there exists a plethora of visualization tools for ontologies, they have primarily been built to visualize aspects of the final product (the ontology) and not the collaborative processes behind construction (e.g. the changes made by contributors over time). To the best of the authors\u2019 knowledge, there exists no ontology visualization tool today that focuses primarily on visualizing the history behind collaboratively constructed ontologies. Since the ontology engineering processes can influence the quality of the final ontology, they believe that visualizing process data represents an important stepping-stone towards better understanding of managing the collaborative construction of ontologies in the future. In this application paper, the authors present a tool \u2013 PragmatiX \u2013 which taps into structured change logs provided by tools such as Collaborative Prot\u00e9g\u00e9 to visualize various pragmatic aspects of collaborative ontology engineering. The tool is aimed at managers and leaders of collaborative ontology engineering projects to help them in monitoring progress, in exploring issues and problems, and in tracking quality-related issues such as overrides and coordination among contributors. The paper makes the following contributions: (i) They present PragmatiX, a tool for visualizing the creation process behind collaboratively constructed ontologies (ii) the authors illustrate the functionality and generality of the tool by applying it to structured logs of changes of two large collaborative ontology-engineering projects and (iii) they conduct a heuristic evaluation of the tool with domain experts to uncover early design challenges and opportunities for improvement. Finally, the authors hope that this work sparks a new line of research on visualization tools for collaborative ontology engineering projects.<\/p>","DOI":"10.4018\/jswis.2013010103","type":"journal-article","created":{"date-parts":[[2013,6,20]],"date-time":"2013-06-20T12:04:07Z","timestamp":1371729847000},"page":"45-78","source":"Crossref","is-referenced-by-count":11,"title":["PragmatiX"],"prefix":"10.4018","volume":"9","author":[{"given":"Simon","family":"Walk","sequence":"first","affiliation":[{"name":"Knowledge Management Institute, Graz University of Technology, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jan","family":"P\u00f6schko","sequence":"additional","affiliation":[{"name":"Knowledge Management Institute, Graz University of Technology, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Markus","family":"Strohmaier","sequence":"additional","affiliation":[{"name":"Knowledge Management Institute, Graz University of Technology, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keith","family":"Andrews","sequence":"additional","affiliation":[{"name":"Institute for Information Systems and Computer Media, Graz University of Technology, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tania","family":"Tudorache","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natalya F.","family":"Noy","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Csongor","family":"Nyulas","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark A.","family":"Musen","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"2432","reference":[{"key":"jswis.2013010103-0","unstructured":"Alani, H. (2003). TGVizTab: An ontology visualisation extension for prot\u00e9g\u00e9. In Knowledge Capture 03 - Workshop on Visualizing Information in Knowledge Engineering (pp. 2-7). Sanibel Island, FL: ACM."},{"key":"jswis.2013010103-1","unstructured":"Andrews, K. (2011). Human-computer interaction: Lecture notes. Retrieved April 5, 2011, from http:\/\/courses.iicm.tugraz.at\/hci\/hci.pdf"},{"key":"jswis.2013010103-2","doi-asserted-by":"crossref","unstructured":"Auer, S., Dietzold, S., & Riechert, T. (2006, November 5-9). OntoWiki - A tool for social, semantic collaboration. In I. F. Cruz et al. (Eds.), The semantic web - ISWC 2006, 5th International Semantic Web Conference, ISWC 2006, Athens, GA (Vol. 4273, pp. 736-749). Berlin \/ Heidelberg, Germany: Springer.","DOI":"10.1007\/11926078_53"},{"key":"jswis.2013010103-3","unstructured":"Bao, J., & Honavar, V. (2004, October). Collaborative ontology building with wiki@nt - a multi-agent based ontology building environment. In Proceedings of the 3rd International Workshop on Evaluation of Ontology-Based Tools (pp. 1-10)."},{"key":"jswis.2013010103-4","author":"C. M.Barnum","year":"2010","journal-title":"Usability testing essentials: Ready, set\u2026 test!"},{"key":"jswis.2013010103-5","article-title":"Ontosphere: More than a 3D ontology visualization tool","volume":"Vol. 166","author":"A.Bosca","year":"2005","journal-title":"Swap"},{"key":"jswis.2013010103-6","unstructured":"Brank, J., Grobelnik, M., & Mladenic, D. (2005). A survey of ontology evaluation techniques. In Proceedings of the Conference on Data Mining and Data Warehouses (pp. 166-170). Lubljana, Slovenia"},{"key":"jswis.2013010103-7","doi-asserted-by":"publisher","DOI":"10.4018\/jswis.2005040103"},{"key":"jswis.2013010103-8","unstructured":"Eklund, P. W., Roberts, N., & Green, S. P. (2002). Ontorama: Browsing an RDF ontology using a hyperbolic-like browser. In Proceedings of the First International Symposium on Cyberworlds (p. 405-411). IEEE press."},{"key":"jswis.2013010103-9","doi-asserted-by":"crossref","unstructured":"Ellson, J., Gansner, E. R., Koutsofios, E., North, S. C., & Woodhull, G. (2001). Graphviz - Open source graph drawing tools. In Graph drawing (p. 483-484).","DOI":"10.1007\/3-540-45848-4_57"},{"key":"jswis.2013010103-10","unstructured":"Falconer, S. (12.04.2010). OntoGraf. In Prot\u00e9g\u00e9 Wiki. Retrieved October 17, 2012, from http:\/\/protegewiki.stanford.edu\/wiki\/OntoGraf"},{"key":"jswis.2013010103-11","doi-asserted-by":"crossref","unstructured":"Falconer, S. M., Tudorache, T., & Noy, N. F. (2011). An analysis of collaborative patterns in large-scale ontology development projects. In M. A. Musen & Oscar Corcho (Eds.), Proceedings of the Sixth International Conference on Knowledge Capture (pp. 25-32). New York, NY: ACM.","DOI":"10.1145\/1999676.1999682"},{"key":"jswis.2013010103-12","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2003.07.007"},{"key":"jswis.2013010103-13","unstructured":"Hagberg, A. A., Schult, D. A., & Swart, P. J. (2008). Exploring network structure, dynamics, and function using networkx. In G. Varoquaux, T. Vaught, & J. Millman (Eds.), Proceedings of the 7th Python in Science Conference (p. 11-15). Pasadena, CA."},{"key":"jswis.2013010103-14","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkh036"},{"key":"jswis.2013010103-15","unstructured":"Horridge, M. (2012). OWLViz \u2013 A visualization plugin for the Prot\u00e9g\u00e9 OWL plugin. Retrieved June 02, 2012, from http:\/\/www.co-ode.org\/downloads\/owlviz\/OWLVizGuide.pdf"},{"key":"jswis.2013010103-16","unstructured":"Lanzenberger, M., & Sampson, J. (2006). Alviz - A tool for visual ontology alignment. In Proceedings of the Conference on Information Visualization (p. 430-440). Washington, DC: IEEE Computer Society."},{"key":"jswis.2013010103-17","doi-asserted-by":"crossref","unstructured":"Maedche, A., & Staab, S. (2000). Semi-automatic engineering of ontologies from text. In Proceedings of the 12th Internal Conference on Software and Knowledge Engineering. Chicago, IL: KSI.","DOI":"10.1007\/3-540-39967-4_14"},{"key":"jswis.2013010103-18","doi-asserted-by":"crossref","unstructured":"Nielsen, J. (1992). Finding usability problems through heuristic evaluation. In Proceedings of the Conference on Human Factors in Computing Systems (pp. 373-380). Monterey, CA: ACM.","DOI":"10.1145\/142750.142834"},{"key":"jswis.2013010103-19","doi-asserted-by":"crossref","unstructured":"Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating Interdependence (pp. 152-158). New York, NY: ACM.","DOI":"10.1145\/191666.191729"},{"key":"jswis.2013010103-20","unstructured":"Nielsen, J. (1994). Heuristic evaluation. In Proceedings of the Conference on Human Factors in Computing Systems (pp. 152-158). Boston, MA: ACM."},{"key":"jswis.2013010103-21","author":"J.Nielsen","year":"1994","journal-title":"Usability inspection methods"},{"key":"jswis.2013010103-22","doi-asserted-by":"crossref","unstructured":"Noy, N., Chugh, A., Liu, W., & Musen, M. E. (2006). A framework for ontology evolution in collaborative environments. In Proceedings of International the Semantic Web Conference (ISWC 2006) (pp. 544-558).","DOI":"10.1007\/11926078_39"},{"key":"jswis.2013010103-23","unstructured":"Noy, N., & McGuinness, D. (2001). Ontology development 101: A guide to creating your first ontology. Stanford knowledge systems laboratory technical report KSL-01-05 and Stanford medical informatics technical report SMI-2001-0880. Retrieved from http:\/\/www.ksl.stanford.edu\/people\/dlm\/papers\/ontology-tutorial-noy-mcguinness-abstrac"},{"key":"jswis.2013010103-24","article-title":"Collaborative ontology development on the (semantic) web.","author":"N. F.Noy","year":"2008","journal-title":"Nature Biotechnology"},{"key":"jswis.2013010103-25","unstructured":"P\u00f6schko, J., Strohmaier, M., Tudorache, T., Noy, N. F., & Musen, M. A. (2012). Pragmatic analysis of crowd-based knowledge production systems with iCAT Analytics: Visualizing changes to the ICD-11 ontology. In Proceedings of the AAAI Spring Symposium 2012: Wisdom of the Crowd."},{"key":"jswis.2013010103-26","author":"J.Rubin","year":"2008","journal-title":"Handbook of usability testing: How to plan, design, and conduct effective tests"},{"key":"jswis.2013010103-27","author":"G.Schreiber","year":"2000","journal-title":"Knowledge engineering and management: The CommonKADS methodology"},{"key":"jswis.2013010103-28","unstructured":"Singh, G., Prabhakar, T., & Chatterjee, J. (2006, November 9-11). Ontoviz: Visualizing ontologies and thesauri using layout algorithms. In V. C. Patil & S. Ninomiya (Eds.), Afita 2006: The Fifth International Conference of the Asian Federation for Information Technology in Agriculture, Bangalore, India, (pp. 709-719)."},{"key":"jswis.2013010103-29","doi-asserted-by":"publisher","DOI":"10.1145\/637411.637413"},{"key":"jswis.2013010103-30","doi-asserted-by":"publisher","DOI":"10.1145\/506443.506459"},{"key":"jswis.2013010103-31","doi-asserted-by":"crossref","unstructured":"Storey, M.-A., Noy, N. F., Musen, M., Best, C., Fergerson, R., & Ernst, N. (2002b). Jambalaya: An interactive environment for exploring ontologies. In Proceedings of the 7th International Conference on Intelligent User Interfaces (pp. 239-239). New York, NY: ACM.","DOI":"10.1145\/502716.502778"},{"key":"jswis.2013010103-32","doi-asserted-by":"crossref","unstructured":"Sure, Y., Erdmann, M., Angele, J., Staab, S., Studer, R., & Wenke, D. (2002). OntoEdit: Collaborative ontology development for the semantic web. In I. Horrocks & J. A. Hendler (Eds.), Proceedings of the International Semantic Web Conference (Vol. 2342, pp. 221-235). Springer.","DOI":"10.1007\/3-540-48005-6_18"},{"key":"jswis.2013010103-33","doi-asserted-by":"crossref","unstructured":"Tudorache, T., Falconer, S., Noy, N. F., Nyulas, C., \u00dcst\u00fcn, T. B., Storey, M.-A., & Musen, M. A. (2010). Ontology development for the masses: Creating ICD-11 in WebProt\u00e9g\u00e9. In Proceedings of the 17th International Conference on Knowledge Engineering and Management by the Masses (pp. 74-89). Berlin, Heidelberg, Germany: Springer-Verlag.","DOI":"10.1007\/978-3-642-16438-5_6"},{"key":"jswis.2013010103-34","doi-asserted-by":"crossref","unstructured":"Tudorache, T., Falconer, S. M., Nyulas, C., Noy, N. F., & Musen, M. A. (2010). Will semantic web technologies work for the development of ICD-11? In P. F. Patel-Schneider et al. (Eds.), International Semantic Web Conference (Vol. 6497, p. 257-272). Springer.","DOI":"10.1007\/978-3-642-17749-1_17"},{"key":"jswis.2013010103-35","doi-asserted-by":"crossref","unstructured":"Tudorache, T., Noy, N. F., Tu, S., & Musen, M. A. (2008, October). Supporting collaborative ontology development in prot\u00e9g\u00e9. In Proceedings of the 7th International Semantic Web Conference 2008 (Vol. 5318, pp. 17-32). Karlsruhe, Germany: Springer.","DOI":"10.1007\/978-3-540-88564-1_2"},{"key":"jswis.2013010103-36","unstructured":"Tudorache, T., Nyulas, C., Noy, N. F., & Musen, M. A. (2011). WebProt\u00e9g\u00e9: A distributed ontology editor and knowledge acquisition tool for the web. Semantic Web Journal, 11-165."},{"key":"jswis.2013010103-37","unstructured":"Walk, S., & Andrews, K. (2012). Heuristic evaluation report. Retrieved July 6, 2012 from http:\/\/heuristiceval.simonwalk.at\/he.html"},{"key":"jswis.2013010103-38","unstructured":"Walk, S., Strohmaier, M., Tudorache, T., Noy, N. F., Nyulas, C., & Musen, M. A. (2012). Recommending concepts to experts: An exploration of recommender techniques for collaborative ontology engineering platforms in the biomedical domain. In Proceedings of the 3rd International Conference on Biomedical Ontology (ICBO 2012), Graz, Austria."}],"container-title":["International Journal on Semantic Web and Information Systems"],"original-title":[],"language":"ng","link":[{"URL":"https:\/\/www.igi-global.com\/viewtitle.aspx?TitleId=77824","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,1]],"date-time":"2022-06-01T18:55:10Z","timestamp":1654109710000},"score":1,"resource":{"primary":{"URL":"https:\/\/services.igi-global.com\/resolvedoi\/resolve.aspx?doi=10.4018\/jswis.2013010103"}},"subtitle":["An Interactive Tool for Visualizing the Creation Process Behind Collaboratively Engineered Ontologies"],"short-title":[],"issued":{"date-parts":[[2013,1,1]]},"references-count":39,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2013,1]]}},"URL":"https:\/\/doi.org\/10.4018\/jswis.2013010103","relation":{},"ISSN":["1552-6283","1552-6291"],"issn-type":[{"value":"1552-6283","type":"print"},{"value":"1552-6291","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,1,1]]}}}