{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:06:19Z","timestamp":1755839179737,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":34,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,8,14]],"date-time":"2021-08-14T00:00:00Z","timestamp":1628899200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100001243","name":"Micron Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100001243","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,14]]},"DOI":"10.1145\/3447548.3467411","type":"proceedings-article","created":{"date-parts":[[2021,8,12]],"date-time":"2021-08-12T06:12:03Z","timestamp":1628748723000},"page":"86-94","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Aggregating Complex Annotations via Merging and Matching"],"prefix":"10.1145","author":[{"given":"Alexander","family":"Braylan","sequence":"first","affiliation":[{"name":"The University of Texas at Austin, Austin, TX, USA"}]},{"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"The University of Texas Austin, Austin, TX, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,8,14]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"[n.d.]. Common Objects in Context (COCO). https:\/\/cocodataset.org\/  [n.d.]. Common Objects in Context (COCO). https:\/\/cocodataset.org\/"},{"key":"e_1_3_2_2_2_1","volume-title":"Computational Linguistics","volume":"34","author":"Artstein Ron","year":"2008","unstructured":"Ron Artstein and Massimo Poesio . 2008 . Inter-coder agreement for computational linguistics . Computational Linguistics , Vol. 34 , 4 (2008). Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, Vol. 34, 4 (2008)."},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1142055.1142067"},{"key":"e_1_3_2_2_4_1","unstructured":"JC de Borda. 1784. M\u00e9moire sur les \u00e9lections au scrutin. Histoire de l'Academie Royale des Sciences (1784).  JC de Borda. 1784. M\u00e9moire sur les \u00e9lections au scrutin. Histoire de l'Academie Royale des Sciences (1784)."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.647"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380250"},{"key":"e_1_3_2_2_7_1","volume-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics","author":"Dawid Alexander Philip","year":"1979","unstructured":"Alexander Philip Dawid and Allan M Skene . 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics ( 1979 ). Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics (1979)."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/10361146.2014.900530"},{"key":"e_1_3_2_2_10_1","article-title":"Object serialization, Java, and C++","volume":"23","author":"Haverlock Kevin","year":"1998","unstructured":"Kevin Haverlock . 1998 . Object serialization, Java, and C++ . Dr. Dobb's Journal: Software Tools for the Professional Programmer , Vol. 23 , 8 (1998). Kevin Haverlock. 1998. Object serialization, Java, and C++. Dr. Dobb's Journal: Software Tools for the Professional Programmer, Vol. 23, 8 (1998).","journal-title":"Dr. Dobb's Journal: Software Tools for the Professional Programmer"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41154-0_1"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2010.79"},{"key":"e_1_3_2_2_13_1","volume-title":"6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track. CEUR Workshop.","author":"Lee D","year":"2018","unstructured":"D Lee , A Das Sarma , and Aditya Parameswaran . 2018 . Aggregating crowdsourced image segmentations . In 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track. CEUR Workshop. D Lee, A Das Sarma, and Aditya Parameswaran. 2018. Aggregating crowdsourced image segmentations. In 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track. CEUR Workshop."},{"key":"e_1_3_2_2_14_1","volume-title":"Crowdsourced Text Sequence Aggregation Based on Hybrid Reliability and Representation. In 43rd international ACM SIGIR conference.","author":"Li Jiyi","year":"2020","unstructured":"Jiyi Li . 2020 . Crowdsourced Text Sequence Aggregation Based on Hybrid Reliability and Representation. In 43rd international ACM SIGIR conference. Jiyi Li. 2020. Crowdsourced Text Sequence Aggregation Based on Hybrid Reliability and Representation. In 43rd international ACM SIGIR conference."},{"key":"e_1_3_2_2_15_1","volume-title":"GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling. arXiv preprint arXiv:2003.00475","author":"Li Jing","year":"2020","unstructured":"Jing Li , Suiyi Ling , Junle Wang , Zhi Li , and Patrick Le Callet . 2020 . GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling. arXiv preprint arXiv:2003.00475 (2020). Jing Li, Suiyi Ling, Junle Wang, Zhi Li, and Patrick Le Callet. 2020. GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling. arXiv preprint arXiv:2003.00475 (2020)."},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735496.2735505"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313459"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_2_2_19_1","volume-title":"Truth Discovery via Average Proximity. arxiv","author":"Meir Reshef","year":"1905","unstructured":"Reshef Meir , Ofra Amir , Gal Cohensius , Omer Ben-Porat , Tsviel Ben-Shabat , and Lirong Xia . 2020. Truth Discovery via Average Proximity. arxiv : 1905 .00629 [cs.AI] Reshef Meir, Ofra Amir, Gal Cohensius, Omer Ben-Porat, Tsviel Ben-Shabat, and Lirong Xia. 2020. Truth Discovery via Average Proximity. arxiv: 1905.00629 [cs.AI]"},{"key":"e_1_3_2_2_20_1","volume-title":"Ani Nenkova, and Matthew Lease.","author":"Nguyen An T","year":"2017","unstructured":"An T Nguyen , Byron C Wallace , Junyi Jessy Li , Ani Nenkova, and Matthew Lease. 2017 . Aggregating and predicting sequence labels from crowd annotations. In Association for Computational Linguistics ., Vol. 2017 . NIH Public Access. An T Nguyen, Byron C Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. 2017. Aggregating and predicting sequence labels from crowd annotations. In Association for Computational Linguistics., Vol. 2017. NIH Public Access."},{"key":"e_1_3_2_2_21_1","volume-title":"36th international ACM SIGIR conference.","author":"Hung Nguyen Quoc Viet","year":"2013","unstructured":"Quoc Viet Hung Nguyen , Thanh Tam Nguyen , Ngoc Tran Lam , and Karl Aberer . 2013 . Batc: a benchmark for aggregation techniques in crowdsourcing . In 36th international ACM SIGIR conference. Quoc Viet Hung Nguyen, Thanh Tam Nguyen, Ngoc Tran Lam, and Karl Aberer. 2013. Batc: a benchmark for aggregation techniques in crowdsourcing. In 36th international ACM SIGIR conference."},{"key":"e_1_3_2_2_22_1","volume-title":"Roma Patel, Yinfei Yang, Iain J Marshall, Ani Nenkova, and Byron C Wallace.","author":"Nye Benjamin","year":"2018","unstructured":"Benjamin Nye , Junyi Jessy Li , Roma Patel, Yinfei Yang, Iain J Marshall, Ani Nenkova, and Byron C Wallace. 2018 . A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Association for Computational Linguistics . NIH Public Access. Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain J Marshall, Ani Nenkova, and Byron C Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Association for Computational Linguistics. NIH Public Access."},{"key":"e_1_3_2_2_23_1","volume-title":"Akash Das Sarma, and Vipul Venkataraman","author":"Parameswaran Aditya","year":"2016","unstructured":"Aditya Parameswaran , Akash Das Sarma, and Vipul Venkataraman . 2016 . Optimizing open-ended crowdsourcing: The next frontier in crowdsourced data management. Bulletin of the Technical Committee on Data Engineering , Vol. 39 (2016). Aditya Parameswaran, Akash Das Sarma, and Vipul Venkataraman. 2016. Optimizing open-ended crowdsourcing: The next frontier in crowdsourced data management. Bulletin of the Technical Committee on Data Engineering, Vol. 39 (2016)."},{"key":"e_1_3_2_2_24_1","article-title":"The benefits of a model of annotation","volume":"2","author":"Passonneau Rebecca J","year":"2014","unstructured":"Rebecca J Passonneau and Bob Carpenter . 2014 . The benefits of a model of annotation . Transactions of the Association for Computational Linguistics , Vol. 2 (2014). Rebecca J Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, Vol. 2 (2014).","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"e_1_3_2_2_25_1","volume-title":"et almbox","author":"Pedregosa Fabian","year":"2011","unstructured":"Fabian Pedregosa , Ga\u00ebl Varoquaux , Alexandre Gramfort , Vincent Michel , Bertrand Thirion , Olivier Grisel , Mathieu Blondel , Peter Prettenhofer , Ron Weiss , Vincent Dubourg , et almbox . 2011 . Scikit-learn : Machine learning in Python. the Journal of machine Learning research, Vol. 12 (2011). Fabian Pedregosa, Ga\u00ebl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et almbox. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research, Vol. 12 (2011)."},{"key":"e_1_3_2_2_26_1","volume-title":"43rd international ACM SIGIR conference.","author":"Roitero Kevin","year":"2020","unstructured":"Kevin Roitero , Michael Soprano , Shaoyang Fan , Damiano Spina , Stefano Mizzaro , and Gianluca Demartini . 2020 . Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background . In 43rd international ACM SIGIR conference. Kevin Roitero, Michael Soprano, Shaoyang Fan, Damiano Spina, Stefano Mizzaro, and Gianluca Demartini. 2020. Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background. In 43rd international ACM SIGIR conference."},{"key":"e_1_3_2_2_27_1","volume-title":"SQUARE: A Benchmark for Research on Computing Crowd Consensus. In 1st AAAI Conference on Human Computation.","author":"Sheshadri Aashish","year":"2013","unstructured":"Aashish Sheshadri and Matthew Lease . 2013 . SQUARE: A Benchmark for Research on Computing Crowd Consensus. In 1st AAAI Conference on Human Computation. Aashish Sheshadri and Matthew Lease. 2013. SQUARE: A Benchmark for Research on Computing Crowd Consensus. In 1st AAAI Conference on Human Computation."},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1101"},{"key":"e_1_3_2_2_29_1","unstructured":"Padhraic Smyth Usama M Fayyad Michael C Burl Pietro Perona and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of venus images. In NeurIPS.  Padhraic Smyth Usama M Fayyad Michael C Burl Pietro Perona and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of venus images. In NeurIPS."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613751"},{"key":"e_1_3_2_2_31_1","unstructured":"James Surowiecki. 2005. The wisdom of crowds. Anchor.  James Surowiecki. 2005. The wisdom of crowds. Anchor."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1963.10500845"},{"key":"e_1_3_2_2_33_1","unstructured":"Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier R Movellan and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NeurIPS.  Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier R Movellan and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NeurIPS."},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"}],"event":{"name":"KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"],"location":"Virtual Event Singapore","acronym":"KDD '21"},"container-title":["Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3447548.3467411","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3447548.3467411","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:36Z","timestamp":1750191516000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3447548.3467411"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,14]]},"references-count":34,"alternative-id":["10.1145\/3447548.3467411","10.1145\/3447548"],"URL":"https:\/\/doi.org\/10.1145\/3447548.3467411","relation":{},"subject":[],"published":{"date-parts":[[2021,8,14]]},"assertion":[{"value":"2021-08-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}