{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T12:34:34Z","timestamp":1776083674238,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":52,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T00:00:00Z","timestamp":1557705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,5,13]]},"DOI":"10.1145\/3308560.3317083","type":"proceedings-article","created":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T12:17:59Z","timestamp":1557749879000},"page":"1100-1105","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":36,"title":["Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions"],"prefix":"10.1145","author":[{"given":"Lora","family":"Aroyo","sequence":"first","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lucas","family":"Dixon","sequence":"additional","affiliation":[{"name":"Jigsaw"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nithum","family":"Thain","sequence":"additional","affiliation":[{"name":"Jigsaw"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Olivia","family":"Redfield","sequence":"additional","affiliation":[{"name":"Jigsaw"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rachel","family":"Rosen","sequence":"additional","affiliation":[{"name":"Jigsaw"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,5,13]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2013.0497"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2012.01.004"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.15346\/hc.v1i1.3"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_3_2_1_5_1","unstructured":"Jeremy Barnes Roman Klinger and Sabine Schulte im Walde. 2017. Assessing State-of-the-Art Sentiment Models on State-of-the-Art Sentiment Datasets. CoRR abs\/1709.04219(2017). arxiv:1709.04219http:\/\/arxiv.org\/abs\/1709.04219  Jeremy Barnes Roman Klinger and Sabine Schulte im Walde. 2017. Assessing State-of-the-Art Sentiment Models on State-of-the-Art Sentiment Datasets. CoRR abs\/1709.04219(2017). arxiv:1709.04219http:\/\/arxiv.org\/abs\/1709.04219"},{"key":"e_1_3_2_1_6_1","first-page":"33","article-title":"It befalls words to fall down: pejoration as a type of semantic change","volume":"47","author":"Borkowska Paulina","year":"2007","unstructured":"Paulina Borkowska and G. Kleparski . 2007 . It befalls words to fall down: pejoration as a type of semantic change . Studia Anglica Resoviensia 47 , 4 (2007), 33 \u2013 50 . Paulina Borkowska and G. Kleparski. 2007. It befalls words to fall down: pejoration as a type of semantic change. Studia Anglica Resoviensia 47, 4 (2007), 33\u201350.","journal-title":"Studia Anglica Resoviensia"},{"key":"e_1_3_2_1_7_1","volume-title":"Proceedings of Internet, Policy and Politics Conference","author":"Burnap Peter","year":"2014","unstructured":"Peter Burnap and Matthew\u00a0Leighton Williams . 2014 . Hate speech, machine classification and statistical modelling of information flows on Twitter: Interpretation and communication for policy decision making .. In Proceedings of Internet, Policy and Politics Conference . Oxford, United Kingdom. Peter Burnap and Matthew\u00a0Leighton Williams. 2014. Hate speech, machine classification and statistical modelling of information flows on Twitter: Interpretation and communication for policy decision making.. In Proceedings of Internet, Policy and Politics Conference. Oxford, United Kingdom."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5032\/jae.1994.04031"},{"key":"e_1_3_2_1_10_1","unstructured":"Herbet\u00a0Aron David. 1963. The method of paired comparisons. (1963).  Herbet\u00a0Aron David. 1963. The method of paired comparisons. (1963)."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2362394.2362400"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159654"},{"key":"e_1_3_2_1_13_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction, Vol.\u00a01. ACM","author":"Adam Glynn Jacob\u00a0Eisenstein Anirudh Srinivasan","year":"2017","unstructured":"Anirudh Srinivasan Adam Glynn Jacob\u00a0Eisenstein Eshwar\u00a0Chandrasekharan, Umashanthi\u00a0Pavalanathan and Eric Gilbert . 2017 . You Can\u2019t Stay Here: The Efficacy of Reddit\u2019s 2015 Ban Examined Through Hate Speech . In Proceedings of the ACM on Human-Computer Interaction, Vol.\u00a01. ACM , New York, NY, USA, 22. Anirudh Srinivasan Adam Glynn Jacob\u00a0Eisenstein Eshwar\u00a0Chandrasekharan, Umashanthi\u00a0Pavalanathan and Eric Gilbert. 2017. You Can\u2019t Stay Here: The Efficacy of Reddit\u2019s 2015 Ban Examined Through Hate Speech. In Proceedings of the ACM on Human-Computer Interaction, Vol.\u00a01. ACM, New York, NY, USA, 22."},{"key":"e_1_3_2_1_14_1","volume-title":"Theory of optimal experiments","author":"Fedorov V.V.","year":"1972","unstructured":"V.V. Fedorov . 1972. Theory of optimal experiments . Elsevier ( 1972 ). V.V. Fedorov. 1972. Theory of optimal experiments. Elsevier (1972)."},{"key":"e_1_3_2_1_15_1","volume-title":"Rasch models: Foundations, recent developments, and applications","author":"Fischer H","unstructured":"Gerhard\u00a0 H Fischer and Ivo\u00a0 W Molenaar . 2012. Rasch models: Foundations, recent developments, and applications . Springer Science & Business Media . Gerhard\u00a0H Fischer and Ivo\u00a0W Molenaar. 2012. Rasch models: Foundations, recent developments, and applications. Springer Science & Business Media."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3200947.3208069"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393521"},{"key":"e_1_3_2_1_18_1","unstructured":"Joy\u00a0Paul Guilford. 1954. Psychometric methods. (1954).  Joy\u00a0Paul Guilford. 1954. Psychometric methods. (1954)."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025781"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Toby Hopp Chris\u00a0J Vargo Lucas Dixon and Nithum Thain. 2018. Correlating Self-Report and Trace Data Measures of Incivility: A Proof of Concept. Social Science Computer Review(2018) 0894439318814241.  Toby Hopp Chris\u00a0J Vargo Lucas Dixon and Nithum Thain. 2018. Correlating Self-Report and Trace Data Measures of Incivility: A Proof of Concept. Social Science Computer Review(2018) 0894439318814241.","DOI":"10.1177\/0894439318814241"},{"key":"e_1_3_2_1_21_1","unstructured":"Hossein Hosseini Sreeram Kannan Baosen Zhang and Radha Poovendran. 2017. Deceiving Google\u2019s Perspective API Built for Detecting Toxic Comments. arXiv preprint arXiv:1702.08138(2017).  Hossein Hosseini Sreeram Kannan Baosen Zhang and Radha Poovendran. 2017. Deceiving Google\u2019s Perspective API Built for Detecting Toxic Comments. arXiv preprint arXiv:1702.08138(2017)."},{"key":"e_1_3_2_1_22_1","volume-title":"Diffusion of lexical change in social media.PLoS ONE 9, 11","author":"Smith Jacob\u00a0Eisenstein Noah","year":"2014","unstructured":"Noah A.\u00a0 Smith Jacob\u00a0Eisenstein , Brendan\u00a0 O\u2019Connor and Eric\u00a0P. Xing . 2014. Diffusion of lexical change in social media.PLoS ONE 9, 11 ( 2014 ). Noah A.\u00a0Smith Jacob\u00a0Eisenstein, Brendan\u00a0O\u2019Connor and Eric\u00a0P. Xing. 2014. Diffusion of lexical change in social media.PLoS ONE 9, 11 (2014)."},{"key":"e_1_3_2_1_23_1","volume-title":"Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing (HCOMP) (01","author":"Jain Jinfeng\u00a0Yi Shaili Jain","year":"2013","unstructured":"Shaili Jain Anil K.\u00a0 Jain Jinfeng\u00a0Yi , Rong\u00a0Jin. 2013 . Inferring Users\u2019 Preferences from Crowdsourced Pairwise Comparisons: A Matrix Completion Approach . Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing (HCOMP) (01 2013), 207\u2013215. Shaili Jain Anil K.\u00a0Jain Jinfeng\u00a0Yi, Rong\u00a0Jin. 2013. Inferring Users\u2019 Preferences from Crowdsourced Pairwise Comparisons: A Matrix Completion Approach. Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing (HCOMP) (01 2013), 207\u2013215."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/288155.288184"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Svetlana Kiritchenko and Saif\u00a0M. Mohammad. 2016. Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best\u2013Worst Scaling. (2016).  Svetlana Kiritchenko and Saif\u00a0M. Mohammad. 2016. Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best\u2013Worst Scaling. (2016).","DOI":"10.18653\/v1\/N16-1095"},{"key":"e_1_3_2_1_27_1","volume-title":"Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence(AAAI\u201913)","author":"Kwok Irene","year":"2013","unstructured":"Irene Kwok and Yuzhou Wang . 2013 . Locate the Hate: Detecting Tweets Against Blacks . In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence(AAAI\u201913) . AAAI Press, 1621\u20131622. http:\/\/dl.acm.org\/citation.cfm?id=2891460.2891697 Irene Kwok and Yuzhou Wang. 2013. Locate the Hate: Detecting Tweets Against Blacks. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence(AAAI\u201913). AAAI Press, 1621\u20131622. http:\/\/dl.acm.org\/citation.cfm?id=2891460.2891697"},{"key":"e_1_3_2_1_28_1","unstructured":"Rensis Likert. 1932. A technique for the measurement of attitudes.Archives of psychology(1932).  Rensis Likert. 1932. A technique for the measurement of attitudes.Archives of psychology(1932)."},{"key":"e_1_3_2_1_29_1","volume-title":"Annotating flames in Usenet newsgroups: a corpus study. For NSF Minority Institution Infrastructure Grant Site Visit to NMSU CS department","author":"Martin J","year":"2002","unstructured":"Melanie\u00a0 J Martin . 2002. Annotating flames in Usenet newsgroups: a corpus study. For NSF Minority Institution Infrastructure Grant Site Visit to NMSU CS department ( 2002 ). Melanie\u00a0J Martin. 2002. Annotating flames in Usenet newsgroups: a corpus study. For NSF Minority Institution Infrastructure Grant Site Visit to NMSU CS department (2002)."},{"key":"e_1_3_2_1_30_1","volume-title":"Is there an optimal number of alternatives for Likert scale items? Study I: Reliability and validity. Educational and psychological measurement 31, 3","author":"Matell S","year":"1971","unstructured":"Michael\u00a0 S Matell and Jacob Jacoby . 1971. Is there an optimal number of alternatives for Likert scale items? Study I: Reliability and validity. Educational and psychological measurement 31, 3 ( 1971 ), 657\u2013674. Michael\u00a0S Matell and Jacob Jacoby. 1971. Is there an optimal number of alternatives for Likert scale items? Study I: Reliability and validity. Educational and psychological measurement 31, 3 (1971), 657\u2013674."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Tyler McDonnell Matthew Lease Tamer Elsayad and Mucahid Kutlu. 2016. Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. In HCOMP. 10.  Tyler McDonnell Matthew Lease Tamer Elsayad and Mucahid Kutlu. 2016. Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. In HCOMP. 10.","DOI":"10.24963\/ijcai.2017\/692"},{"key":"e_1_3_2_1_32_1","volume-title":"Claus and Graham Kalton","author":"Moser Sir","year":"1972","unstructured":"Sir Moser , Claus and Graham Kalton . 1972 . Survey methods in social investigation(2nd american ed ed.). New York : Basic Books . Previous ed.: 1958. Sir Moser, Claus and Graham Kalton. 1972. Survey methods in social investigation(2nd american ed ed.). New York : Basic Books. Previous ed.: 1958."},{"key":"e_1_3_2_1_33_1","volume-title":"From Rankings to Ratings: Rank Scoring via Active Learning. (10","author":"O\u2019Neill Jack","year":"2018","unstructured":"Jack O\u2019Neill , Sarah Delany , and Brian Mac\u00a0Namee . 2018. From Rankings to Ratings: Rank Scoring via Active Learning. (10 2018 ). Jack O\u2019Neill, Sarah Delany, and Brian Mac\u00a0Namee. 2018. From Rankings to Ratings: Rank Scoring via Active Learning. (10 2018)."},{"key":"e_1_3_2_1_34_1","unstructured":"Jack O\u2019Neill Sarah\u00a0Jane Delany and Brian\u00a0Mac Namee. 2017. Rating by Ranking: An Improved Scale for Judgement-Based Labels. In IntRS@RecSys.  Jack O\u2019Neill Sarah\u00a0Jane Delany and Brian\u00a0Mac Namee. 2017. Rating by Ranking: An Improved Scale for Judgement-Based Labels. In IntRS@RecSys."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000011"},{"key":"e_1_3_2_1_36_1","volume-title":"Human Computation Must Be Reproducible. In CrowdSearch: WWW Workshop on Crowdsourcing Web Search. 20\u201325","author":"Paritosh Praveen","year":"2012","unstructured":"Praveen Paritosh . 2012 . Human Computation Must Be Reproducible. In CrowdSearch: WWW Workshop on Crowdsourcing Web Search. 20\u201325 . Praveen Paritosh. 2012. Human Computation Must Be Reproducible. In CrowdSearch: WWW Workshop on Crowdsourcing Web Search. 20\u201325."},{"key":"e_1_3_2_1_37_1","volume-title":"Raykar and Shipeng Yu","author":"C.","year":"2012","unstructured":"Vikas\u00a0 C. Raykar and Shipeng Yu . 2012 . Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks. J. Mach. Learn. Res . 13 (Feb. 2012), 491\u2013518. http:\/\/dl.acm.org\/citation.cfm?id=2188385.2188401 Vikas\u00a0C. Raykar and Shipeng Yu. 2012. Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks. J. Mach. Learn. Res. 13 (Feb. 2012), 491\u2013518. http:\/\/dl.acm.org\/citation.cfm?id=2188385.2188401"},{"key":"e_1_3_2_1_38_1","volume-title":"Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW \u201915)","author":"Benjamin Hillmann Matt Lesicko Rebecca Gold","unstructured":"Rebecca Gold Benjamin Hillmann Matt Lesicko Samuel Naden Jesse Russell Zixiao (Ken)\u00a0 Wang Shilad\u00a0Sen , Margaret E. \u00a0Gieseland Brent Hecht. 2015. Turkers, Scholars, \u201dArafat\u201d and \u201dPeace\u201d: Cultural Communities and Algorithmic Gold Standards . In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW \u201915) . ACM, New York, NY, USA, 826\u2013838. Rebecca Gold Benjamin Hillmann Matt Lesicko Samuel Naden Jesse Russell Zixiao (Ken)\u00a0Wang Shilad\u00a0Sen, Margaret E.\u00a0Gieseland Brent Hecht. 2015. Turkers, Scholars, \u201dArafat\u201d and \u201dPeace\u201d: Cultural Communities and Algorithmic Gold Standards. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW \u201915). ACM, New York, NY, USA, 826\u2013838."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207243.2207252"},{"key":"e_1_3_2_1_40_1","unstructured":"Saif M.\u00a0Mohammad Svetlana\u00a0Kiritchenko. 2017. Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation. (2017).  Saif M.\u00a0Mohammad Svetlana\u00a0Kiritchenko. 2017. Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation. (2017)."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"crossref","unstructured":"L.\u00a0L. Thurstone. 1927. A Law of Comparative Judgment. (1927).  L.\u00a0L. Thurstone. 1927. A Law of Comparative Judgment. (1927).","DOI":"10.1037\/h0070288"},{"key":"e_1_3_2_1_42_1","volume-title":"Judgment under Uncertainty: Heuristics and Biases. Science 185, 4157","author":"Tversky Amos","year":"1974","unstructured":"Amos Tversky and Daniel Kahneman . 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185, 4157 ( 1974 ), 1124\u20131131. arXiv:http:\/\/science.sciencemag.org\/content\/185\/4157\/1124.full.pdf Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185, 4157 (1974), 1124\u20131131. arXiv:http:\/\/science.sciencemag.org\/content\/185\/4157\/1124.full.pdf"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"e_1_3_2_1_44_1","volume-title":"CLEF, Vol.\u00a01","author":"Voorhees M","unstructured":"Ellen\u00a0 M Voorhees . 2001. The philosophy of information retrieval evaluation . In CLEF, Vol.\u00a01 . Springer , 355\u2013370. Ellen\u00a0M Voorhees. 2001. The philosophy of information retrieval evaluation. In CLEF, Vol.\u00a01. Springer, 355\u2013370."},{"key":"e_1_3_2_1_45_1","volume-title":"Proceedings of the Second Workshop on Language in Social Media. Association for Computational Linguistics, 19\u201326","author":"Warner William","year":"2012","unstructured":"William Warner and Julia Hirschberg . 2012 . Detecting hate speech on the world wide web . In Proceedings of the Second Workshop on Language in Social Media. Association for Computational Linguistics, 19\u201326 . William Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In Proceedings of the Second Workshop on Language in Social Media. Association for Computational Linguistics, 19\u201326."},{"key":"e_1_3_2_1_46_1","volume-title":"Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference. 25\u201332","author":"Welinder P.","unstructured":"P. Welinder and P. Perona . 2010. Online crowdsourcing: Rating annotators and obtaining cost-effective labels . In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference. 25\u201332 . P. Welinder and P. Perona. 2010. Online crowdsourcing: Rating annotators and obtaining cost-effective labels. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference. 25\u201332."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052591"},{"key":"e_1_3_2_1_48_1","volume-title":"Rankr: A Mobile System for Crowdsourcing Opinions.","author":"Huberman Yarun\u00a0Luon Bernardo","year":"2012","unstructured":"Bernardo A.\u00a0 Huberman Yarun\u00a0Luon , Christina\u00a0Aperjis. 2012 . Rankr: A Mobile System for Crowdsourcing Opinions. (2012). Bernardo A.\u00a0Huberman Yarun\u00a0Luon, Christina\u00a0Aperjis. 2012. Rankr: A Mobile System for Crowdsourcing Opinions. (2012)."},{"key":"e_1_3_2_1_49_1","first-page":"1","article-title":"Detection of harassment on web 2.0","volume":"2","author":"Yin Dawei","year":"2009","unstructured":"Dawei Yin , Zhenzhen Xue , Liangjie Hong , Brian\u00a0 D Davison , April Kontostathis , and Lynne Edwards . 2009 . Detection of harassment on web 2.0 . Proceedings of the Content Analysis in the WEB 2 (2009), 1 \u2013 7 . Dawei Yin, Zhenzhen Xue, Liangjie Hong, Brian\u00a0D Davison, April Kontostathis, and Lynne Edwards. 2009. Detection of harassment on web 2.0. Proceedings of the Content Analysis in the WEB 2 (2009), 1\u20137.","journal-title":"Proceedings of the Content Analysis in the WEB"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1125"},{"key":"e_1_3_2_1_51_1","first-page":"427","article-title":"On Some Biases Encountered in Modern Audio Quality Listening Tests-A Review","volume":"56","author":"Zielinski Slawomir","year":"2008","unstructured":"Slawomir Zielinski , Francis Rumsey , and S\u00f8ren Bech . 2008 . On Some Biases Encountered in Modern Audio Quality Listening Tests-A Review . J. Audio Eng. Soc 56 , 6 (2008), 427 \u2013 451 . http:\/\/www.aes.org\/e-lib\/browse.cfm?elib=14393 Slawomir Zielinski, Francis Rumsey, and S\u00f8ren Bech. 2008. On Some Biases Encountered in Modern Audio Quality Listening Tests-A Review. J. Audio Eng. Soc 56, 6 (2008), 427\u2013451. http:\/\/www.aes.org\/e-lib\/browse.cfm?elib=14393","journal-title":"J. Audio Eng. Soc"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3185045"}],"event":{"name":"WWW '19: The Web Conference","location":"San Francisco USA","acronym":"WWW '19","sponsor":["IW3C2 International World Wide Web Conference Committee"]},"container-title":["Companion Proceedings of The 2019 World Wide Web Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308560.3317083","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3308560.3317083","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:53:35Z","timestamp":1750204415000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308560.3317083"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,13]]},"references-count":52,"alternative-id":["10.1145\/3308560.3317083","10.1145\/3308560"],"URL":"https:\/\/doi.org\/10.1145\/3308560.3317083","relation":{},"subject":[],"published":{"date-parts":[[2019,5,13]]},"assertion":[{"value":"2019-05-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}