{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,5]],"date-time":"2025-10-05T19:57:14Z","timestamp":1759694234861,"version":"3.41.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"2-3","license":[{"start":{"date-parts":[[2019,3,27]],"date-time":"2019-03-27T00:00:00Z","timestamp":1553644800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Austrian COMET Program","award":["CNS-0435060, CCR-0325197 and EN-CS-0329609."],"award-info":[{"award-number":["CNS-0435060, CCR-0325197 and EN-CS-0329609."]}]},{"name":"WIQ-EI project within EC FP7-PEOPLE program","award":["269180)"],"award-info":[{"award-number":["269180)"]}]},{"name":"H2020 AFEL project","award":["687916)."],"award-info":[{"award-number":["687916)."]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2019,9,30]]},"abstract":"<jats:p>Digital libraries and services enable users to access large amounts of data on demand. Yet, quality assessment of information encountered on the Internet remains an elusive open issue. For example, Wikipedia, one of the most visited platforms on the Web, hosts thousands of user-generated articles and undergoes 12 million edits\/contributions per month. User-generated content is undoubtedly one of the keys to its success but also a hindrance to good quality. Although Wikipedia has established guidelines for the \u201cperfect article,\u201d authors find it difficult to assert whether their contributions comply with them and reviewers cannot cope with the ever-growing amount of articles pending review. Great efforts have been invested in algorithmic methods for automatic classification of Wikipedia articles (as featured or non-featured) and for quality flaw detection. Instead, our contribution is an interactive tool that combines automatic classification methods and human interaction in a toolkit, whereby experts can experiment with new quality metrics and share them with authors that need to identify weaknesses to improve a particular article. A design study shows that experts are able to effectively create complex quality metrics in a visual analytics environment. In turn, a user study evidences that regular users can identify flaws, as well as high-quality content based on the inspection of automatic quality scores.<\/jats:p>","DOI":"10.1145\/3150973","type":"journal-article","created":{"date-parts":[[2019,3,28]],"date-time":"2019-03-28T12:23:24Z","timestamp":1553775804000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Interactive Quality Analytics of User-generated Content"],"prefix":"10.1145","volume":"9","author":[{"given":"Cecilia Di","family":"Sciascio","sequence":"first","affiliation":[{"name":"Know-Center GmbH, Graz, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Strohmaier","sequence":"additional","affiliation":[{"name":"Know-Center GmbH, Graz, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marcelo","family":"Errecalde","sequence":"additional","affiliation":[{"name":"National University of San Luis, San Luis, Argentina"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eduardo","family":"Veas","sequence":"additional","affiliation":[{"name":"Know-Center GmbH, Graz, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,3,27]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1822258.1822293"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242608"},{"key":"e_1_2_1_3_1","volume-title":"Tate","author":"Alexander Janet E.","year":"1999","unstructured":"Janet E. Alexander and Marsha A . Tate . 1999 . Web Wisdom : How to Evaluate and Create Information Quality on the Webb (1st ed.). L. Erlbaum Associates Inc ., Hillsdale, NJ. Janet E. Alexander and Marsha A. Tate. 1999. Web Wisdom: How to Evaluate and Create Information Quality on the Webb (1st ed.). L. Erlbaum Associates Inc., Hillsdale, NJ."},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Maik Anderka. 2013. Analyzing and Predicting Quality Flaws in User-generated Content: The Case of Wikipedia. Dissertation. Bauhaus-Universit\u00e4t Weimar.  Maik Anderka. 2013. Analyzing and Predicting Quality Flaws in User-generated Content: The Case of Wikipedia. Dissertation. Bauhaus-Universit\u00e4t Weimar.","DOI":"10.1145\/2348283.2348413"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063954"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1963192.1963196"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348413"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/1043463109336804"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.2753\/MIS0742-1222270403"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526993.1526995"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1367497.1367673"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526808"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/PACIFICVIS.2010.5429611"},{"key":"e_1_2_1_14_1","volume-title":"Wikipedia: Wikipedians.","author":"Wikipedia","year":"2015","unstructured":"Wikipedia community. 2015 . Wikipedia: Wikipedians. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:Wikipedians. Wikipedia community. 2015. Wikipedia: Wikipedians. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:Wikipedians."},{"key":"e_1_2_1_15_1","unstructured":"Wikipedia community. 2017. Gadget usage statistics. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Special:GadgetUsage.  Wikipedia community. 2017. Gadget usage statistics. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Special:GadgetUsage."},{"key":"e_1_2_1_16_1","unstructured":"Answers Corporation. 2015. Answer Wikipedia. Retrieved from http:\/\/www.answers.com\/.  Answers Corporation. 2015. Answer Wikipedia. Retrieved from http:\/\/www.answers.com\/."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1998076.1998190"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025201"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2005.149"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/DEXA.2014.52"},{"volume-title":"Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201912)","author":"Ferretti E.","key":"e_1_2_1_21_1","unstructured":"E. Ferretti , D. H. Fusilier , R. G. Cabrera , M. Montes-y- G\u00f3mez , M. Errecalde , and P. Rosso . 2012. On the use of PU learning for quality flaw prediction in Wikipedia: Notebook for PAN at CLEF 2012 . In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201912) . E. Ferretti, D. H. Fusilier, R. G. Cabrera, M. Montes-y-G\u00f3mez, M. Errecalde, and P. Rosso. 2012. On the use of PU learning for quality flaw prediction in Wikipedia: Notebook for PAN at CLEF 2012. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201912)."},{"volume-title":"Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201912)","author":"Ferschke O.","key":"e_1_2_1_22_1","unstructured":"O. Ferschke , I. Gurevych , and M. Rittberger . 2012. FlawFinder: A modular system for predicting quality flaws in Wikipedia: Notebook for PAN at CLEF 2012 . In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201912) . O. Ferschke, I. Gurevych, and M. Rittberger.2012. FlawFinder: A modular system for predicting quality flaws in Wikipedia: Notebook for PAN at CLEF 2012. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201912)."},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL\u201913)","volume":"1","author":"Ferschke Oliver","year":"2013","unstructured":"Oliver Ferschke , Iryna Gurevych , and Marc Rittberger . 2013 . The impact of topic bias on quality flaw prediction in Wikipedia . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL\u201913) , Vol. 1 . Association for Computational Linguistics, Stroudsburg, PA, 721--730. Oliver Ferschke, Iryna Gurevych, and Marc Rittberger. 2013. The impact of topic bias on quality flaw prediction in Wikipedia. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL\u201913), Vol. 1. Association for Computational Linguistics, Stroudsburg, PA, 721--730."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-31865-1_25"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.173"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1555400.1555449"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321476"},{"key":"e_1_2_1_28_1","unstructured":"IBM. 2015. IBM History Flow. Retrieved from https:\/\/en.wikipedia.org\/wiki\/IBM_History_Flow_tool.  IBM. 2015. IBM History Flow. Retrieved from https:\/\/en.wikipedia.org\/wiki\/IBM_History_Flow_tool."},{"key":"e_1_2_1_29_1","first-page":"602","article-title":"Network analysis of user generated content quality in Wikipedia. Online Info","volume":"37","author":"Ingawale Myshkin","year":"2013","unstructured":"Myshkin Ingawale , Amitava Dutta , Rahul Roy , and Priya Seetharaman . 2013 . Network analysis of user generated content quality in Wikipedia. Online Info . Rev. 37 , 4 (2013), 602 -- 619 . Myshkin Ingawale, Amitava Dutta, Rahul Roy, and Priya Seetharaman. 2013. Network analysis of user generated content quality in Wikipedia. Online Info. Rev. 37, 4 (2013), 602--619.","journal-title":"Rev."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2184305.2184308"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 5th International Symposium on Online Journalism. 16--17","author":"Lih Andrew","year":"2004","unstructured":"Andrew Lih . 2004 . Wikipedia as participatory journalism: Reliable sources? Metrics for evaluating collaborative media as a news resource . In Proceedings of the 5th International Symposium on Online Journalism. 16--17 . Andrew Lih. 2004. Wikipedia as participatory journalism: Reliable sources? Metrics for evaluating collaborative media as a news resource. In Proceedings of the 5th International Symposium on Online Journalism. 16--17."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/WI.2006.115"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772847"},{"key":"e_1_2_1_34_1","unstructured":"Jeph Paul. 2015. Replay Edits. Retrieved from http:\/\/cosmiclattes.github.io\/wikireplay\/player.html.  Jeph Paul. 2015. Replay Edits. Retrieved from http:\/\/cosmiclattes.github.io\/wikireplay\/player.html."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518929"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1842993.1843004"},{"key":"e_1_2_1_37_1","volume-title":"Lewis","author":"Sauro Jeff","year":"2012","unstructured":"Jeff Sauro and James R . Lewis . 2012 . Quantifying the User Experience : Practical Statistics for User Research. Elsevier . Jeff Sauro and James R. Lewis. 2012. Quantifying the User Experience: Practical Statistics for User Research. Elsevier."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.213"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025233"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1038\/nmeth.2807"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 10th International Conference on Information Quality (ICIQ\u201905)","author":"Stvilia Besiki","year":"2005","unstructured":"Besiki Stvilia , Michael B. Twidale , Linda C. Smith , and Les Gasser . 2005 . Assessing information quality of a community-based encyclopedia . In Proceedings of the 10th International Conference on Information Quality (ICIQ\u201905) . MIT, 442--454. Besiki Stvilia, Michael B. Twidale, Linda C. Smith, and Les Gasser. 2005. Assessing information quality of a community-based encyclopedia. In Proceedings of the 10th International Conference on Information Quality (ICIQ\u201905). MIT, 442--454."},{"key":"e_1_2_1_42_1","volume-title":"Pyrospirit","author":"User","year":"2015","unstructured":"User : Pyrospirit . 2015 . User : Pyrospiri. Retrieved from https:\/\/en.wikipedia.org\/wiki\/User:Pyrospirit\/metadata. User: Pyrospirit. 2015. User: Pyrospiri. Retrieved from https:\/\/en.wikipedia.org\/wiki\/User:Pyrospirit\/metadata."},{"key":"e_1_2_1_43_1","unstructured":"Ivo van Kamp. 2015. Axon. Retrieved from https:\/\/addons.mozilla.org\/en-US\/firefox\/addon\/axon\/.  Ivo van Kamp. 2015. Axon. Retrieved from https:\/\/addons.mozilla.org\/en-US\/firefox\/addon\/axon\/."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2006.80"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985765"},{"key":"e_1_2_1_46_1","unstructured":"Wikipedia Community. 2015. AutoWikiBrowser. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:AutoWikiBrowser.  Wikipedia Community. 2015. AutoWikiBrowser. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:AutoWikiBrowser."},{"key":"e_1_2_1_47_1","unstructured":"Wikipedia Community. 2015. wikiED. Retrieved from https:\/\/en.wikipedia.org\/wiki\/User:Cacycle\/wikEd.  Wikipedia Community. 2015. wikiED. Retrieved from https:\/\/en.wikipedia.org\/wiki\/User:Cacycle\/wikEd."},{"key":"e_1_2_1_48_1","volume-title":"Wikipedia: Featured article criteria.","author":"Community Wikipedia","year":"2015","unstructured":"Wikipedia Community . 2015 . Wikipedia: Featured article criteria. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:Featured_article_criteria. Wikipedia Community. 2015. Wikipedia: Featured article criteria. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:Featured_article_criteria."},{"key":"e_1_2_1_49_1","volume-title":"Wikipedia: WikiProject articles for creation\/helper script.","author":"Community Wikipedia","year":"2017","unstructured":"Wikipedia Community . 2017 . Wikipedia: WikiProject articles for creation\/helper script. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:WikiProject_Articles_for_creation\/Helper_script. Wikipedia Community. 2017. Wikipedia: WikiProject articles for creation\/helper script. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:WikiProject_Articles_for_creation\/Helper_script."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1296951.1296968"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1641309.1641333"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3150973","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3150973","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:13:39Z","timestamp":1750212819000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3150973"}},"subtitle":["An Integrated Toolkit for the Case of Wikipedia"],"short-title":[],"issued":{"date-parts":[[2019,3,27]]},"references-count":51,"journal-issue":{"issue":"2-3","published-print":{"date-parts":[[2019,9,30]]}},"alternative-id":["10.1145\/3150973"],"URL":"https:\/\/doi.org\/10.1145\/3150973","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"type":"print","value":"2160-6455"},{"type":"electronic","value":"2160-6463"}],"subject":[],"published":{"date-parts":[[2019,3,27]]},"assertion":[{"value":"2017-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-05-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-03-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}