{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T00:50:54Z","timestamp":1774399854960,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":29,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,11,3]],"date-time":"2019-11-03T00:00:00Z","timestamp":1572739200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100010663","name":"H2020 European Research Council","doi-asserted-by":"publisher","award":["732328"],"award-info":[{"award-number":["732328"]}],"id":[{"id":"10.13039\/100010663","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Australian Research Council (ARC) Discovery Project","award":["DP190102141"],"award-info":[{"award-number":["DP190102141"]}]},{"name":"Australia Research Council (ARC) Discovery Project","award":["DP190102141"],"award-info":[{"award-number":["DP190102141"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,11,3]]},"DOI":"10.1145\/3357384.3357988","type":"proceedings-article","created":{"date-parts":[[2019,11,4]],"date-time":"2019-11-04T14:11:35Z","timestamp":1572876695000},"page":"39-48","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["On Transforming Relevance Scales"],"prefix":"10.1145","author":[{"given":"Lei","family":"Han","sequence":"first","affiliation":[{"name":"University of Queensland, Brisbane, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kevin","family":"Roitero","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eddy","family":"Maddalena","sequence":"additional","affiliation":[{"name":"University of Southampton, Southampton, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefano","family":"Mizzaro","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"University of Queensland, Brisbane, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,11,3]]},"reference":[{"key":"e_1_3_2_1_1_1","first-page":"26","article-title":"Benchmarking Twitter Sentiment Analysis Tools","volume":"14","author":"Abbasi Ahmed","year":"2014","unstructured":"Ahmed Abbasi , Ammar Hassan , and Milan Dhar . 2014 . Benchmarking Twitter Sentiment Analysis Tools .. In LREC , Vol. 14. 26 -- 31 . Ahmed Abbasi, Ammar Hassan, and Milan Dhar. 2014. Benchmarking Twitter Sentiment Analysis Tools.. In LREC, Vol. 14. 26--31.","journal-title":"LREC"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572017"},{"key":"e_1_3_2_1_3_1","volume-title":"Fifth AAAI Conference on Human Computation and Crowdsourcing .","author":"Checco Alessandro","year":"2017","unstructured":"Alessandro Checco , Kevin Roitero , Eddy Maddalena , Stefano Mizzaro , and Gianluca Demartini . 2017 . Let's Agree to Disagree: Fixing Agreement Measures for Crowdsourcing . In Fifth AAAI Conference on Human Computation and Crowdsourcing . Alessandro Checco, Kevin Roitero, Eddy Maddalena, Stefano Mizzaro, and Gianluca Demartini. 2017. Let's Agree to Disagree: Fixing Agreement Measures for Crowdsourcing. In Fifth AAAI Conference on Human Computation and Crowdsourcing ."},{"key":"e_1_3_2_1_4_1","unstructured":"Cyril W Cleverdon. 1962. Report on the testing and analysis of an investigation into the comparative efficiency of indexing systems. (1962).  Cyril W Cleverdon. 1962. Report on the testing and analysis of an investigation into the comparative efficiency of indexing systems. (1962)."},{"key":"e_1_3_2_1_5_1","volume-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics","author":"Dawid Alexander Philip","year":"1979","unstructured":"Alexander Philip Dawid and Allan M Skene . 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics ( 1979 ), 20--28. Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics (1979), 20--28."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/11735106_48"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"crossref","unstructured":"Koji Eguchi Keizo Oyama Emi Ishida Noriko Kando and Kazuko Kuriyama. 2002. Overview of the Web Retrieval Task at the Third NTCIR Workshop.. In NTCIR .  Koji Eguchi Keizo Oyama Emi Ishida Noriko Kando and Kazuko Kuriyama. 2002. Overview of the Web Retrieval Task at the Third NTCIR Workshop.. In NTCIR .","DOI":"10.1145\/564376.564451"},{"key":"e_1_3_2_1_8_1","volume-title":"INEX 2003 Workshop Proceedings. 1--11","author":"Fuhr Norbert","year":"2004","unstructured":"Norbert Fuhr , Saadia Malik , and Mounia Lalmas . 2004 . Overview of the INitiative for the Evaluation of XML Retrieval (INEX) 2003 . In INEX 2003 Workshop Proceedings. 1--11 . Norbert Fuhr, Saadia Malik, and Mounia Lalmas. 2004. Overview of the INitiative for the Evaluation of XML Retrieval (INEX) 2003. In INEX 2003 Workshop Proceedings. 1--11."},{"key":"e_1_3_2_1_9_1","volume-title":"The first text retrieval conference (TREC-1)","author":"Harman Donna K","unstructured":"Donna K Harman . 1993. The first text retrieval conference (TREC-1) . Vol. 500 . US Department of Commerce , National Institute of Standards and Technology. Donna K Harman. 1993. The first text retrieval conference (TREC-1). Vol. 500. US Department of Commerce, National Institute of Standards and Technology."},{"key":"e_1_3_2_1_10_1","volume-title":"ACM Transactions on Information Systems (TOIS)","volume":"20","author":"Jaana Kalervo","year":"2002","unstructured":"Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002 . Cumulated gain-based evaluation of IR techniques . ACM Transactions on Information Systems (TOIS) , Vol. 20 , 4 (2002), 422--446. Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) , Vol. 20, 4 (2002), 422--446."},{"key":"e_1_3_2_1_11_1","unstructured":"Kazuaki Kishida Kuang-hua Chen Sukhoon Lee Kazuko Kuriyama Noriko Kando Hsin-Hsi Chen and Sung Hyon Myaeng. 2007. Overview of CLIR Task at the Sixth NTCIR Workshop.. In NTCIR .  Kazuaki Kishida Kuang-hua Chen Sukhoon Lee Kazuko Kuriyama Noriko Kando Hsin-Hsi Chen and Sung Hyon Myaeng. 2007. Overview of CLIR Task at the Sixth NTCIR Workshop.. In NTCIR ."},{"key":"e_1_3_2_1_12_1","volume-title":"Working Notes of the Fourth NTCIR Workshop Meeting, June 2--4. 1--59","author":"Kishida Kazuaki","year":"2004","unstructured":"Kazuaki Kishida , Kuang-hua Chen, Sukhoon Lee , Kazuko Kuriyama , Noriko Kando , Hsin-Hsi Chen , Sung Hyon Myaeng , and Koji Eguchi . 2004 . Overview of CLIR task at the fourth NTCIR workshop . In Working Notes of the Fourth NTCIR Workshop Meeting, June 2--4. 1--59 . Kazuaki Kishida, Kuang-hua Chen, Sukhoon Lee, Kazuko Kuriyama, Noriko Kando, Hsin-Hsi Chen, Sung Hyon Myaeng, and Koji Eguchi. 2004. Overview of CLIR task at the fourth NTCIR workshop. In Working Notes of the Fourth NTCIR Workshop Meeting, June 2--4. 1--59."},{"key":"e_1_3_2_1_13_1","volume-title":"Computing Krippendorff's alpha reliability . Departmental papers (ASC)","author":"Krippendorff Klaus","year":"2007","unstructured":"Klaus Krippendorff . 2007. Computing Krippendorff's alpha reliability . Departmental papers (ASC) ( 2007 ), 43. Klaus Krippendorff. 2007. Computing Krippendorff's alpha reliability . Departmental papers (ASC) (2007), 43."},{"key":"e_1_3_2_1_14_1","volume-title":"Fourth AAAI Conference on Human Computation and Crowdsourcing .","author":"Maddalena Eddy","year":"2016","unstructured":"Eddy Maddalena , Marco Basaldella , Dario De Nart , Dante Degl'Innocenti , Stefano Mizzaro , and Gianluca Demartini . 2016 . Crowdsourcing relevance assessments: The unexpected benefits of limiting the time to judge . In Fourth AAAI Conference on Human Computation and Crowdsourcing . Eddy Maddalena, Marco Basaldella, Dario De Nart, Dante Degl'Innocenti, Stefano Mizzaro, and Gianluca Demartini. 2016. Crowdsourcing relevance assessments: The unexpected benefits of limiting the time to judge. In Fourth AAAI Conference on Human Computation and Crowdsourcing ."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3002172"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3121050.3121060"},{"key":"e_1_3_2_1_17_1","volume-title":"Overview of INEX","author":"Malik Saadia","year":"2004","unstructured":"Saadia Malik , Mounia Lalmas , and Norbert Fuhr . 2005. Overview of INEX 2004 . In Advances in XML Information Retrieval , , Norbert Fuhr, Mounia Lalmas, Saadia Malik, and Zolt\u00e1n Szl\u00e1vik (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg, 1--15. Saadia Malik, Mounia Lalmas, and Norbert Fuhr. 2005. Overview of INEX 2004. In Advances in XML Information Retrieval , , Norbert Fuhr, Mounia Lalmas, Saadia Malik, and Zolt\u00e1n Szl\u00e1vik (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1--15."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2012.07.059"},{"key":"e_1_3_2_1_20_1","volume-title":"Proc. 5th NTCIR Workshop Meeting on Evaluation of Information Access Technologies","author":"Oyama Keizo","year":"2005","unstructured":"Keizo Oyama . 2005 . Overview of the NTCIR-5 WEB navigational retrieval subtask 2 (Navi-2) . In Proc. 5th NTCIR Workshop Meeting on Evaluation of Information Access Technologies , 2005 . Keizo Oyama. 2005. Overview of the NTCIR-5 WEB navigational retrieval subtask 2 (Navi-2). In Proc. 5th NTCIR Workshop Meeting on Evaluation of Information Access Technologies, 2005 ."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210052"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484090"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564433"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767760"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_3_2_1_26_1","volume-title":"Overview of the Eighth Text REtrieval Conference (TREC-8). In The Eighth Text REtrieval Conference (TREC-8). 1--24","author":"Ellen","unstructured":"Ellen M. Voorhees and Donna K. Harman. 1999 . Overview of the Eighth Text REtrieval Conference (TREC-8). In The Eighth Text REtrieval Conference (TREC-8). 1--24 . Ellen M. Voorhees and Donna K. Harman. 1999. Overview of the Eighth Text REtrieval Conference (TREC-8). In The Eighth Text REtrieval Conference (TREC-8). 1--24."},{"key":"e_1_3_2_1_27_1","volume-title":"Proceeding of 55th ACL. 422--426.","author":"Wang William Yang","unstructured":"William Yang Wang . 2017. \"Liar , Liar Pants on Fire\": A New Benchmark Dataset for Fake News Detection . In Proceeding of 55th ACL. 422--426. William Yang Wang. 2017. \"Liar, Liar Pants on Fire\": A New Benchmark Dataset for Fake News Detection. In Proceeding of 55th ACL. 422--426."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/208098.208105"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"}],"event":{"name":"CIKM '19: The 28th ACM International Conference on Information and Knowledge Management","location":"Beijing China","acronym":"CIKM '19","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 28th ACM International Conference on Information and Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3357384.3357988","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3357384.3357988","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:23:05Z","timestamp":1750202585000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3357384.3357988"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,3]]},"references-count":29,"alternative-id":["10.1145\/3357384.3357988","10.1145\/3357384"],"URL":"https:\/\/doi.org\/10.1145\/3357384.3357988","relation":{},"subject":[],"published":{"date-parts":[[2019,11,3]]},"assertion":[{"value":"2019-11-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}