{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:16:05Z","timestamp":1750306565267,"version":"3.41.0"},"reference-count":15,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2014,12,23]],"date-time":"2014-12-23T00:00:00Z","timestamp":1419292800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2014,12,23]]},"abstract":"<jats:p>Visual data exploration is a well-known technique in the data mining toolbox for analyzing and understanding large amounts of data. In information retrieval, visualization is still perceived as a front-end or user facing feature while, at the same time, it can be very effective in the back-end for data analysis. Currently, the research community does not sufficiently analyze the reasons underlying the assessments they use for evaluation, which may lead to bias or error when reporting system performance. In this paper we examine the integration of information visualization techniques for visually exploring and analyzing relevance assessments. We argue that visual tools should be integrated as a core component of the infrastructure for relevance assessment and evaluation.<\/jats:p>","DOI":"10.1145\/2701583.2701585","type":"journal-article","created":{"date-parts":[[2015,1,5]],"date-time":"2015-01-05T13:27:09Z","timestamp":1420464429000},"page":"14-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Visualization for Relevance Assessments"],"prefix":"10.1145","volume":"48","author":[{"given":"Omar","family":"Alonso","sequence":"first","affiliation":[{"name":"Microsoft"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2014,12,23]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/2502742"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2637002.2637025"},{"key":"e_1_2_1_3_1","unstructured":"Omar Alonso Catherine Marshall and Marc Najork. \"Crowdsourcing a Subjective Labeling Task: A Human-Centered Framework to Ensure Reliable Results\" MSR-TR-2014-91 http:\/\/research.microsoft.com\/apps\/pubs\/default.aspx?id=219755  Omar Alonso Catherine Marshall and Marc Najork. \"Crowdsourcing a Subjective Labeling Task: A Human-Centered Framework to Ensure Reliable Results\" MSR-TR-2014-91 http:\/\/research.microsoft.com\/apps\/pubs\/default.aspx?id=219755"},{"key":"e_1_2_1_4_1","volume-title":"Proc. of AIRS","author":"Alonso Omar","year":"2013","unstructured":"Omar Alonso , Dennis Fetterly , and Mark Manasse . \" Duplicate News Story Detection Revisited\". In Proc. of AIRS 2013 , 203--214. Omar Alonso, Dennis Fetterly, and Mark Manasse. \"Duplicate News Story Detection Revisited\". In Proc. of AIRS 2013, 203--214."},{"key":"e_1_2_1_5_1","volume-title":"Morgan Kaufmann","author":"Card Stuart","year":"1999","unstructured":"Stuart Card , Jock Mackinlay , and Ben Shneiderman . Readings in Information Visualization. Using Vision to Think . Morgan Kaufmann , 1999 . Stuart Card, Jock Mackinlay, and Ben Shneiderman. Readings in Information Visualization. Using Vision to Think. Morgan Kaufmann, 1999."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00799-011-0067-7"},{"key":"e_1_2_1_7_1","unstructured":"Claudia Hauff. \"Ranking Retrieval Systems without Relevance Judgments\". http:\/\/www.st.ewi.tudelft.nl\/~hauff\/visualization\/trecVis.html  Claudia Hauff. \"Ranking Retrieval Systems without Relevance Judgments\". http:\/\/www.st.ewi.tudelft.nl\/~hauff\/visualization\/trecVis.html"},{"key":"e_1_2_1_8_1","unstructured":"Claudia Hauff. \"The Uniqueness of QREL Contributions\". http:\/\/www.st.ewi.tudelft.nl\/~hauff\/trec_vis\/trecQrelVis.html  Claudia Hauff. \"The Uniqueness of QREL Contributions\". http:\/\/www.st.ewi.tudelft.nl\/~hauff\/trec_vis\/trecQrelVis.html"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5555\/1631268"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242672"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-70956-5_7"},{"key":"e_1_2_1_12_1","unstructured":"Microsoft Pivot. http:\/\/research.microsoft.com\/en-us\/downloads\/dd4a479f-92d6-496f-867d-666c87fbaada\/default.aspx  Microsoft Pivot. http:\/\/research.microsoft.com\/en-us\/downloads\/dd4a479f-92d6-496f-867d-666c87fbaada\/default.aspx"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1108\/00220410710758986"},{"key":"e_1_2_1_14_1","volume-title":"Information Retrieval Experiment. Butterworths","author":"Tague Jean","year":"1981","unstructured":"Jean Tague . \" The Pragmatics of Information Retrieval Experimentation\" in K. Sp\u00e4rck Jones (Ed.), Information Retrieval Experiment. Butterworths , 1981 . Jean Tague. \"The Pragmatics of Information Retrieval Experimentation\" in K. Sp\u00e4rck Jones (Ed.), Information Retrieval Experiment. Butterworths, 1981."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484150"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2701583.2701585","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2701583.2701585","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:13:11Z","timestamp":1750227191000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2701583.2701585"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,12,23]]},"references-count":15,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2014,12,23]]}},"alternative-id":["10.1145\/2701583.2701585"],"URL":"https:\/\/doi.org\/10.1145\/2701583.2701585","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2014,12,23]]},"assertion":[{"value":"2014-12-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}