{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T09:52:33Z","timestamp":1787219553429,"version":"build-2736575974"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,11,3]],"date-time":"2020-11-03T00:00:00Z","timestamp":1604361600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,11,3]],"date-time":"2020-11-03T00:00:00Z","timestamp":1604361600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100008131","name":"Rheinische Friedrich-Wilhelms-Universit\u00e4t Bonn","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008131","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2021,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Systematic assessment of scientific events has become increasingly important for research communities. A range of metrics (e.g., citations, h-index) have been developed by different research communities to make such assessments effectual. However, most of the metrics for assessing the quality of less formal publication venues and events have not yet deeply investigated. It is also rather challenging to develop respective metrics because each research community has its own formal and informal rules of communication and quality standards. In this article, we develop a comprehensive framework of assessment metrics for evaluating scientific events and involved stakeholders. The resulting quality metrics are determined with respect to three general categories\u2014events, persons, and bibliometrics. Our assessment methodology is empirically applied to several series of computer science events, such as conferences and workshops, using publicly available data for determining quality metrics. We show that the metrics\u2019 values coincide with the intuitive agreement of the community on its \u201ctop conferences\u201d. Our results demonstrate that highly-ranked events share similar profiles, including the provision of outstanding reviews, visiting diverse locations, having reputed people involved, and renowned sponsors.<\/jats:p>","DOI":"10.1007\/s11192-020-03758-1","type":"journal-article","created":{"date-parts":[[2020,11,3]],"date-time":"2020-11-03T23:02:22Z","timestamp":1604444542000},"page":"641-682","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["A comprehensive quality assessment framework for scientific events"],"prefix":"10.1007","volume":"126","author":[{"given":"Sahar","family":"Vahdati","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4455-2854","authenticated-orcid":false,"given":"Said","family":"Fathalla","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christoph","family":"Lange","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andreas","family":"Behrend","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aysegul","family":"Say","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zeynep","family":"Say","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"S\u00f6ren","family":"Auer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,11,3]]},"reference":[{"issue":"1","key":"3758_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.websem.2008.02.005","volume":"7","author":"C Bizer","year":"2009","unstructured":"Bizer, C., & Cyganiak, R. (2009). Quality-driven information filtering using the WIQA policy framework. Journal of Web Semantics, 7(1), 1\u201310.","journal-title":"Journal of Web Semantics"},{"issue":"3","key":"3758_CR2","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1007\/s11192-011-0569-5","volume":"91","author":"L Bornmann","year":"2012","unstructured":"Bornmann, L., Wolf, M., & Daniel, H.-D. (2012). Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics, 91(3), 843\u2013856.","journal-title":"Scientometrics"},{"key":"3758_CR3","unstructured":"Bowyer, K. W. (2012). Mentoring Advice on \u201cConferences Versus Journals\u201d for CSE Faculty."},{"key":"3758_CR4","unstructured":"Bryl, V., Birukou, A., Eckert, K., & Kessler, M. (2014). What\u2019s in the proceedings? Combining publisher\u2019s and researcher\u2019s perspectives. In SePublica. CEUR Workshop Proceedings."},{"key":"3758_CR5","unstructured":"Caires, L. (2015). Again, the role of conference papers in computer science and informatics. http:\/\/ctp.di.fct.unl.pt\/~lcaires\/papers\/conferencesCS.v4.pdf."},{"key":"3758_CR6","unstructured":"Caires, L. (2015). Again, the role of conference papers in computer science and informatics."},{"key":"3758_CR7","unstructured":"Capadisli, S., Riedl, R., & Auer, S. (2015). Enabling accessible knowledge. In Ce- DEM, pp. 257\u2013272. http:\/\/csarven.ca\/enabling-accessibleknowledge."},{"key":"3758_CR8","unstructured":"Ciancarini, P., Di Iorio, A., Nuzzolese, A. G., Peroni, S., & Vitali, F. (2013). Characterising citations in scholarly articles: An experiment. In First International Workshop on Artificial Intelligence and Cognition, AIC. CEURWorkshop Proceedings 1100 (pp. 124\u2013129)."},{"issue":"3","key":"3758_CR9","doi-asserted-by":"publisher","first-page":"129","DOI":"10.3138\/jsp.35.3.129","volume":"35","author":"CN Davidson","year":"2004","unstructured":"Davidson, C. N. (2004). The futures of scholarly publishing. Scholarly Publshing, 35(3), 129\u2013142.","journal-title":"Scholarly Publshing"},{"key":"3758_CR10","doi-asserted-by":"crossref","unstructured":"Debattista, J., Lange, C., & Auer, S. (2014). Representing dataset quality metadata using multi-dimensional views. In Proceedings of the 10th International Conference on Semantic Systems (pp. 92\u201399).","DOI":"10.1145\/2660517.2660525"},{"key":"3758_CR11","doi-asserted-by":"crossref","unstructured":"Di Iorio, A., Lange, C., Dimou, A., & Vahdati, S. (2015). Semantic publishing challenge-assessing the quality of scientific output by information extraction and interlinking. In Semantic Web Evaluation Challenges (pp. 65\u201380). Springer.","DOI":"10.1007\/978-3-319-25518-7_6"},{"key":"3758_CR12","unstructured":"Ernst, M. (2006). Choosing a venue: Conference or journal? https:\/\/homes.cs.washington.edu\/~mernst\/advice\/conferences-vs-journals.html."},{"key":"3758_CR13","unstructured":"Fathalla, S., & Lange, C. (2017). EVENTS: A dataset on the history of top-prestigious events in five computer science communities. In Semantics, Analytics, Visualization (pp. 110\u2013120). Springer."},{"key":"3758_CR48","doi-asserted-by":"crossref","unstructured":"Fathalla, S., & Lange, C. (2018). EVENTSKG: A knowledge graph representation for top-prestigious computer science events metadata. In N. Nguyen, E. Pimenidis, Z. Khan, & B. Trawi\u0144ski (Eds.), Computational collective intelligence. ICCCI 2018. Lecture notes in computer science, (Vol. 11055, pp. 53\u201363). Springer, Cham.","DOI":"10.1007\/978-3-319-98443-8_6"},{"key":"3758_CR14","doi-asserted-by":"crossref","unstructured":"Fathalla, S., Vahdati, S., Auer, S., & Lange, C. (2017). Towards a knowledge graph representing research findings by semantifying survey articles. In International Conference on Theory and Practice of Digital Libraries (pp. 315\u2013327). Springer.","DOI":"10.1007\/978-3-319-67008-9_25"},{"key":"3758_CR15","doi-asserted-by":"crossref","unstructured":"Fathalla, S., Vahdati, S., Auer, S., & Lange, C. (2018). Metadata analysis of scholarly events of computer science, physics, engineering, and mathematics. In International conference on theory and practice of digital libraries (pp. 116\u2013128). Springer.","DOI":"10.1007\/978-3-030-00066-0_10"},{"key":"3758_CR16","doi-asserted-by":"crossref","unstructured":"Fathalla, S., Vahdati, S., Lange, C. & Auer, S. (2017). Analysing scholarly communication metadata of computer science events. In International Conference on Theory and Practice of Digital Libraries (pp. 342\u2013354). Springer.","DOI":"10.1007\/978-3-319-67008-9_27"},{"key":"3758_CR49","doi-asserted-by":"crossref","unstructured":"Fathalla, S., Lange, C., & Auer, S. (2019a). EVENTSKG: A 5-star dataset of top-ranked events in eight computer science communities. In P. Hitzler et al. (Eds.), The semantic web. ESWC 2019. Lecture notes in computer science, (Vol. 11503, pp. 427\u2013442). Springer, Cham.","DOI":"10.1007\/978-3-030-21348-0_28"},{"key":"3758_CR17","doi-asserted-by":"crossref","unstructured":"Fathalla, S., Vahdati, S., Lange, C. & Auer, S. (2019b). SEO: A scientific events data model. In International semantic web conference (pp. 79\u201395). Springer.","DOI":"10.1007\/978-3-030-30796-7_6"},{"key":"3758_CR18","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1016\/j.procs.2018.09.015","volume":"137","author":"S Fathalla","year":"2018","unstructured":"Fathalla, S., Vahdati, S., Auer, S., & Lange, C. (2018). SemSur: A core ontology for the semantic representation of research findings. Procedia Computer Science, 137, 151\u2013162.","journal-title":"Procedia Computer Science"},{"issue":"2","key":"3758_CR19","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1007\/s11192-020-03391-y","volume":"123","author":"S Fathalla","year":"2020","unstructured":"Fathalla, S., Vahdati, S., Lange, C., & Auer, S. (2020). Scholarly event characteristics in four fields of science: A metrics-based analysis. Scientometrics, 123(2), 677\u2013705.","journal-title":"Scientometrics"},{"issue":"11","key":"3758_CR20","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1145\/1839676.1839701","volume":"53","author":"J Freyne","year":"2010","unstructured":"Freyne, J., Coyle, L., Smyth, B., & Cunningham, P. (2010). Relative status of journal and conference publications in computer science. Communications of ACM, 53(11), 124\u2013132.","journal-title":"Communications of ACM"},{"key":"3758_CR21","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1007\/s00799-007-0011-z","volume":"8","author":"N Fuhr","year":"2007","unstructured":"Fuhr, N., Tsakonas, G., Aalberg, T., Agosti, M., Hansen, P., Kapidakis, S., et al. (2007). Evaluation of digital libraries. International Journal of Digital Libraries, 8, 21\u201338.","journal-title":"International Journal of Digital Libraries"},{"issue":"2","key":"3758_CR22","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1086\/208547","volume":"17","author":"M Goldberg","year":"1990","unstructured":"Goldberg, M., & Hartwick, J. (1990). The effect of advertiser reputation and extremity of advertising claim on advertising effectiveness. Consumer Research, 17(2), 172\u2013179.","journal-title":"Consumer Research"},{"issue":"4","key":"3758_CR23","doi-asserted-by":"publisher","first-page":"887","DOI":"10.1016\/j.joi.2013.08.005","volume":"7","author":"Z Hu","year":"2013","unstructured":"Hu, Z., Chen, C., & Liu, Z. (2013). Where are citations located in the body of scientific articles? A study of the distributions of citation locations. Journal of Informetrics, 7(4), 887\u2013896.","journal-title":"Journal of Informetrics"},{"issue":"3","key":"3758_CR24","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1145\/1462571.1462582","volume":"37","author":"HV Jagadish","year":"2008","unstructured":"Jagadish, H. V. (2008). The conference reviewing crisis and a proposed solution. SIGMOD Record, 37(3), 40\u201345.","journal-title":"SIGMOD Record"},{"key":"3758_CR25","unstructured":"Juran, J. M. (1974). Juran\u2019s Quality Control Handbook (4th edition). McGraw-Hill, London."},{"issue":"5","key":"3758_CR26","doi-asserted-by":"publisher","first-page":"159","DOI":"10.28945\/493","volume":"8","author":"S-A Knight","year":"2005","unstructured":"Knight, S.-A., & Burn, J. M. (2005). Developing a framework for assessing information quality on the World Wide Web. Informing Science: International Journal of an Emerging Transdiscipline, 8(5), 159\u2013172.","journal-title":"Informing Science: International Journal of an Emerging Transdiscipline"},{"key":"3758_CR27","unstructured":"Kotler, P. (2000) Marketing Management. Pearson Education, London."},{"key":"3758_CR28","unstructured":"Kumar, M. J. (2009). Evaluating Scientists: Citations, Impact Factor, h-Index, Online Page Hits and What Else? In IETE Technical Review 26.3."},{"key":"3758_CR29","unstructured":"Lange, C., & Di Iorio, A. (2014). Semantic publishing challenge. Assessing the quality of scientific output. In SemanticWeb Evaluation Challenges. SemWebEval 2014 at ESWC 2014. Communications in Computer and Information Science (Vol. 475, pp. 61\u201376). Springer, Berlin"},{"issue":"23","key":"3758_CR30","doi-asserted-by":"publisher","first-page":"3699","DOI":"10.1016\/j.cub.2017.10.032","volume":"27","author":"DF Little","year":"2017","unstructured":"Little, D. F., Zhang, Y.-X., & Wright, B. A. (2017). Disruption of perceptual learning by a brief practice break. Current Biology, 27(23), 3699\u20133705.","journal-title":"Current Biology"},{"key":"3758_CR31","doi-asserted-by":"crossref","unstructured":"Luong, H. P., Huynh, T., Gauch, S., Do, L., & Hoang, K. (2012). Publication venue recommendation using author network\u2019s publication history. In Intelligent Information and Database Systems, Proceedings, Part III. ACIIDS\u20144th Asian Conference. Lecture Notes in Computer Science (Vol. 7198, pp. 426\u2013435). Springer, Berlin.","DOI":"10.1007\/978-3-642-28493-9_45"},{"key":"3758_CR32","unstructured":"Martin, J. (1983). Managing the Data Base Environment. Prentice Hall, Prentice."},{"key":"3758_CR33","unstructured":"Morais, C. D. (2015). Fortune 1000 Companies List for 2015. http:\/\/www.geolounge.com\/fortune-1000-companies-list-for-2015\/."},{"key":"3758_CR34","doi-asserted-by":"crossref","unstructured":"Nuzzolese, A. G., Gentile, A. L., Presutti, V., & Gangemi, A. (2016). Semantic web conference ontology\u2014a refactoring solution. In International Semantic Web Conference (pp. 84\u201387). Springer, Berlin.","DOI":"10.1007\/978-3-319-47602-5_18"},{"key":"3758_CR35","doi-asserted-by":"crossref","unstructured":"Osborne, F., & Motta, E. (2014). Understanding research dynamics. In Semantic Web Evaluation Challenges. SemWebEval 2014 at ESWC 2014. Communications in Computer and Information Science (Vol. 475, pp. 101\u2013107). Springer, Berlin.","DOI":"10.1007\/978-3-319-12024-9_13"},{"key":"3758_CR36","doi-asserted-by":"crossref","unstructured":"Peroni, S., Osborne, F., Di Iorio, A., Nuzzolese, A. G., Poggi, F., Vitali, F., & Motta, E. (2017). Research articles in simplified HTML: A Web-first format for HTML based scholarly article. In PeerJ Computer Science (Vol. 3, p. e132).","DOI":"10.7717\/peerj-cs.132"},{"key":"3758_CR37","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/j.websem.2012.08.001","volume":"17","author":"S Peroni","year":"2012","unstructured":"Peroni, S., & Shotton, D. (2012). FaBiO and CiTO: Ontologies for describing bibliographic resources and citations. Web Semantics, 17, 33\u201343.","journal-title":"Web Semantics"},{"key":"3758_CR38","unstructured":"Remler, D. (2017). Are 90% of academic papers really never cited? Reviewing the literature on academic citations. Accessed 20 October 1997."},{"key":"3758_CR39","unstructured":"Semantic Web Evaluation Challenges. (2014). SemWebEval 2014 at ESWC 2014. In Communications in Computer and Information Science (vol. 475). Springer, Berlin."},{"key":"3758_CR40","doi-asserted-by":"publisher","first-page":"280","DOI":"10.1016\/j.neuron.2010.01.001","volume":"65","author":"A Tambini","year":"2010","unstructured":"Tambini, A., Ketz, N., & Davachi, L. (2010). Enhanced brain correlations during rest are related to memory for recent experiences. Neuron, 65, 280\u2013290.","journal-title":"Neuron"},{"issue":"1","key":"3758_CR41","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1145\/2542506","volume":"57","author":"D Terry","year":"2014","unstructured":"Terry, D. (2014). Publish now, judge later. Communications of the ACM, 57(1), 44\u201346.","journal-title":"Communications of the ACM"},{"key":"3758_CR42","unstructured":"Vahdati, S. (2019). Collaborative integration, publishing and analysis of distributed scholarly metadata. Ph.D. thesis. Universit\u00e4ts-und Landesbibliothek Bonn."},{"key":"3758_CR43","doi-asserted-by":"crossref","unstructured":"Vahdati, S., Arndt, N., Auer, S., & Lange, C. (2016). OpenResearch: Collaborative Management of Scholarly Communication Metadata. In 20th International Conference on Knowledge Engineering and Knowledge Management. EKAW (Bologna, Italy). Lecture Notes in Computer Science (Vol. 10024, pp. 778\u2013793).","DOI":"10.1007\/978-3-319-49004-5_50"},{"key":"3758_CR44","doi-asserted-by":"crossref","unstructured":"Weale, A. R., Bailey, M., & Lear, P. A. (2004). The level of non-citation of articles within a journal as a measure of quality: A comparison to the impact factor. In BMC Medical Research Methodology (Vol. 4, p. 14).","DOI":"10.1186\/1471-2288-4-14"},{"key":"3758_CR45","unstructured":"What is peer review? Elsevier (2016). https:\/\/www.elsevier.com\/reviewers\/what-is-peer-review."},{"key":"3758_CR46","doi-asserted-by":"crossref","unstructured":"Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016). The FAIR guiding principles for scientific data management and stewardship. In Scientific data (Vol. 3).","DOI":"10.1038\/sdata.2016.18"},{"key":"3758_CR47","doi-asserted-by":"crossref","unstructured":"Yang, Z. & Davison, B. D. (2012). Venue recommendation: Submitting your paper with style. In 11th International Conference on Machine Learning and Applications, ICMLA (pp. 681\u2013686). IEEE.","DOI":"10.1109\/ICMLA.2012.127"}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03758-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11192-020-03758-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03758-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,21]],"date-time":"2021-01-21T19:19:50Z","timestamp":1611256790000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11192-020-03758-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,3]]},"references-count":49,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1]]}},"alternative-id":["3758"],"URL":"https:\/\/doi.org\/10.1007\/s11192-020-03758-1","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,3]]},"assertion":[{"value":"3 July 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 November 2020","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}