{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,5]],"date-time":"2025-10-05T19:52:51Z","timestamp":1759693971684,"version":"3.41.0"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2017,1,4]],"date-time":"2017-01-04T00:00:00Z","timestamp":1483488000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["DP130104007"],"award-info":[{"award-number":["DP130104007"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Google Faculty Research Award"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2017,7,31]]},"abstract":"<jats:p>Magnitude estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of magnitude estimation for judging the relevance of documents for information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting over 50,000 magnitude estimation judgments using crowdsourcing. Our analysis shows that magnitude estimation judgments can be reliably collected using crowdsourcing, are competitive in terms of assessor cost, and are, on average, rank-aligned with ordinal judgments made by expert relevance assessors.<\/jats:p>\n          <jats:p>We explore the application of magnitude estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and magnitude estimation relevance shows substantial variation; in particular, the top systems ranked using magnitude estimation and ordinal judgments differ substantially. Analysis of the magnitude estimation scores shows that this effect is due in part to varying perceptions of relevance: different users have different perceptions of the impact of relative differences in document relevance. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold.<\/jats:p>","DOI":"10.1145\/3002172","type":"journal-article","created":{"date-parts":[[2017,1,4]],"date-time":"2017-01-04T17:02:36Z","timestamp":1483549356000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":40,"title":["On Crowdsourcing Relevance Magnitudes for Information Retrieval Evaluation"],"prefix":"10.1145","volume":"35","author":[{"given":"Eddy","family":"Maddalena","sequence":"first","affiliation":[{"name":"University of Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefano","family":"Mizzaro","sequence":"additional","affiliation":[{"name":"University of Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Falk","family":"Scholer","sequence":"additional","affiliation":[{"name":"RMIT University, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2559-8769","authenticated-orcid":false,"given":"Andrew","family":"Turpin","sequence":"additional","affiliation":[{"name":"University of Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,1,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2012.01.004"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2507884"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.2307\/416793"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102363"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835540"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_1_8_1","volume-title":"Overview of the TREC 2004 terabyte track. In The 14th Text REtrieval Conference (TREC\u201905)","author":"Clarke Charles","year":"2005","unstructured":"Charles Clarke , Nick Craswell , and Ian Soboroff . 2005 . Overview of the TREC 2004 terabyte track. In The 14th Text REtrieval Conference (TREC\u201905) . NIST. Charles Clarke, Nick Craswell, and Ian Soboroff. 2005. Overview of the TREC 2004 terabyte track. In The 14th Text REtrieval Conference (TREC\u201905). NIST."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.95"},{"volume-title":"TREC 2013 web track overview. In 22nd Text REtrieval Conference (TREC\u201913)","author":"Collins-Thompson Kevyn","key":"e_1_2_1_10_1","unstructured":"Kevyn Collins-Thompson , Paul Bennett , Fernando Diaz , Charles L. A. Clarke , and Ellen M. Voorhees . 2014 . TREC 2013 web track overview. In 22nd Text REtrieval Conference (TREC\u201913) . Kevyn Collins-Thompson, Paul Bennett, Fernando Diaz, Charles L. A. Clarke, and Ellen M. Voorhees. 2014. TREC 2013 web track overview. In 22nd Text REtrieval Conference (TREC\u201913)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.2307\/3150495"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-58552-4_43"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(88)90042-8"},{"key":"e_1_2_1_14_1","volume-title":"Psychophysics: The Fundamentals","author":"Gescheider George","year":"1997","unstructured":"George Gescheider . 1997 . Psychophysics: The Fundamentals ( 3 rd ed.). Lawrence Erlbaum Associates . George Gescheider. 1997. Psychophysics: The Fundamentals (3rd ed.). Lawrence Erlbaum Associates.","edition":"3"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1080\/19312450709336664"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-28997-2_16"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1177\/154193121005401229"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 3rd Human Computation Workshop. 88--90","author":"Jung H. J.","year":"2011","unstructured":"H. J. Jung and Matthew Lease . 2011 . Improving consensus accuracy via z-score and weighted voting . In Proceedings of the 3rd Human Computation Workshop. 88--90 . H. J. Jung and Matthew Lease. 2011. Improving consensus accuracy via z-score and weighted voting. In Proceedings of the 3rd Human Computation Workshop. 88--90."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646032"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9205-0"},{"volume-title":"MySpace, YouTube, and The Rest of Today\u2019s User-Generated Media are Killing Our Culture and Economy","author":"Keen Andrew","key":"e_1_2_1_22_1","unstructured":"Andrew Keen . 2008. The Cult of the Amateur: How Blogs , MySpace, YouTube, and The Rest of Today\u2019s User-Generated Media are Killing Our Culture and Economy . Random House . Andrew Keen. 2008. The Cult of the Amateur: How Blogs, MySpace, YouTube, and The Rest of Today\u2019s User-Generated Media are Killing Our Culture and Economy. Random House."},{"key":"e_1_2_1_23_1","volume-title":"Computing Krippendorff\u2019s alpha reliability. Departmental Papers (ASC) 1-25-2011","author":"Krippendorff Klaus","year":"2011","unstructured":"Klaus Krippendorff . 2011. Computing Krippendorff\u2019s alpha reliability. Departmental Papers (ASC) 1-25-2011 ( 2011 ). Klaus Krippendorff. 2011. Computing Krippendorff\u2019s alpha reliability. Departmental Papers (ASC) 1-25-2011 (2011)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-16354-3_23"},{"key":"e_1_2_1_25_1","volume-title":"Sensory Processes: The New Psychophysics","author":"Marks Lawrence","year":"1974","unstructured":"Lawrence Marks . 1974 . Sensory Processes: The New Psychophysics . Academic Press . Lawrence Marks. 1974. Sensory Processes: The New Psychophysics. Academic Press."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the ACM WSDM International Conference on Web Search and Data Mining. 23--26","author":"McCreadie Richard","year":"2011","unstructured":"Richard McCreadie , Craig Macdonald , and Iadh Ounis . 2011 . Crowdsourcing blog track top news judgments at TREC . In Proceedings of the ACM WSDM International Conference on Web Search and Data Mining. 23--26 . Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2011. Crowdsourcing blog track top news judgments at TREC. In Proceedings of the ACM WSDM International Conference on Web Search and Data Mining. 23--26."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1177\/154193120304700406"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-4557.1977.tb00942.x"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2006.07.020"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20682"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 6th International Workshop on Evaluating Information Access (EVIA\u201914)","author":"Scholer Falk","year":"2014","unstructured":"Falk Scholer , Eddy Maddalena , Stefano Mizzaro , and Andrew Turpin . 2014 . Magnitudes of relevance: Relevance judgements, magnitude estimation, and crowdsourcing . In Proceedings of the 6th International Workshop on Evaluating Information Access (EVIA\u201914) . Falk Scholer, Eddy Maddalena, Stefano Mizzaro, and Andrew Turpin. 2014. Magnitudes of relevance: Relevance judgements, magnitude estimation, and crowdsourcing. In Proceedings of the 6th International Workshop on Evaluating Information Access (EVIA\u201914)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010057"},{"key":"e_1_2_1_33_1","volume-title":"Handbook of Parametric and Nonparametric Statistical Procedures","author":"Sheskin David","unstructured":"David Sheskin . 2007. Handbook of Parametric and Nonparametric Statistical Procedures ( 4 th ed.). CRC Press . David Sheskin. 2007. Handbook of Parametric and Nonparametric Statistical Procedures (4th ed.). CRC Press.","edition":"4"},{"key":"e_1_2_1_34_1","volume-title":"Overview of the TREC 2013 crowdsourcing track. In Proceedings of the 22nd NIST Text Retrieval Conference (TREC\u201913)","author":"Smucker Mark","year":"2014","unstructured":"Mark Smucker , Gabriella Kazai , and Matthew Lease . 2014 . Overview of the TREC 2013 crowdsourcing track. In Proceedings of the 22nd NIST Text Retrieval Conference (TREC\u201913) . Mark Smucker, Gabriella Kazai, and Matthew Lease. 2014. Overview of the TREC 2013 crowdsourcing track. In Proceedings of the 22nd NIST Text Retrieval Conference (TREC\u201913)."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564433"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1002\/1097-4571(2000)9999:9999%3C::AID-ASI1564%3E3.0.CO;2-L"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.151.3710.530"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(1999)50:3%3C254::AID-ASI8%3E3.0.CO;2-Y"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767760"},{"volume-title":"The 8th Text REtrieval Conference (TREC-8\u201999)","author":"Ellen","key":"e_1_2_1_40_1","unstructured":"Ellen M. Voorhees and Donna K. Harman. 1999. Overview of the eighth text retrieval conference (TREC-8) . In The 8th Text REtrieval Conference (TREC-8\u201999) . 1--24. Ellen M. Voorhees and Donna K. Harman. 1999. Overview of the eighth text retrieval conference (TREC-8). In The 8th Text REtrieval Conference (TREC-8\u201999). 1--24."},{"key":"e_1_2_1_41_1","volume-title":"Harman","author":"Voorhees Ellen M.","year":"2005","unstructured":"Ellen M. Voorhees and Donna K . Harman . 2005 . TREC : Experiment and Evaluation in Information Retrieval. MIT Press . Ellen M. Voorhees and Donna K. Harman. 2005. TREC: Experiment and Evaluation in Information Retrieval. MIT Press."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484156"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609577"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9206-z"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3002172","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3002172","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:49:54Z","timestamp":1750218594000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3002172"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,1,4]]},"references-count":44,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2017,7,31]]}},"alternative-id":["10.1145\/3002172"],"URL":"https:\/\/doi.org\/10.1145\/3002172","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"type":"print","value":"1046-8188"},{"type":"electronic","value":"1558-2868"}],"subject":[],"published":{"date-parts":[[2017,1,4]]},"assertion":[{"value":"2016-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-01-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}