{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:25:10Z","timestamp":1750220710602,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":63,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,4,20]],"date-time":"2020-04-20T00:00:00Z","timestamp":1587340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,4,20]]},"DOI":"10.1145\/3366423.3380250","type":"proceedings-article","created":{"date-parts":[[2020,5,4]],"date-time":"2020-05-04T08:11:44Z","timestamp":1588579904000},"page":"1807-1818","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Modeling and Aggregation of Complex Annotations via Annotation Distances"],"prefix":"10.1145","author":[{"given":"Alexander","family":"Braylan","sequence":"first","affiliation":[{"name":"The University of Texas at Austin"}]},{"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"The University of Texas at Austin"}]}],"member":"320","published-online":{"date-parts":[[2020,4,20]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.07-034-R2"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.384007"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_3_2_1_4_1","volume-title":"Multilevel bayesian models of categorical data annotation. Unpublished manuscript 17, 122","author":"Carpenter Bob","year":"2008","unstructured":"Bob Carpenter . 2008. Multilevel bayesian models of categorical data annotation. Unpublished manuscript 17, 122 ( 2008 ), 45\u201350. Bob Carpenter. 2008. Multilevel bayesian models of categorical data annotation. Unpublished manuscript 17, 122 (2008), 45\u201350."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v076.i01"},{"key":"e_1_3_2_1_6_1","volume-title":"Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing. arXiv preprint arXiv:1810.10733(2018).","author":"Chen Quanze","year":"2018","unstructured":"Quanze Chen , Jonathan Bragg , Lydia\u00a0 B Chilton , and Daniel\u00a0 S Weld . 2018 . Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing. arXiv preprint arXiv:1810.10733(2018). Quanze Chen, Jonathan Bragg, Lydia\u00a0B Chilton, and Daniel\u00a0S Weld. 2018. Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing. arXiv preprint arXiv:1810.10733(2018)."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/2898607.2898793"},{"key":"e_1_3_2_1_8_1","volume-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics","author":"Dawid Alexander\u00a0Philip","year":"1979","unstructured":"Alexander\u00a0Philip Dawid and Allan\u00a0 M Skene . 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics ( 1979 ), 20\u201328. Alexander\u00a0Philip Dawid and Allan\u00a0M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics (1979), 20\u201328."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1977.tb01600.x"},{"key":"e_1_3_2_1_11_1","volume-title":"Proceedings of the IEEE International Conference on Computer Vision. 1313\u20131320","author":"Dutt\u00a0Jain Suyog","year":"2013","unstructured":"Suyog Dutt\u00a0Jain and Kristen Grauman . 2013 . Predicting sufficient annotation strength for interactive foreground segmentation . In Proceedings of the IEEE International Conference on Computer Vision. 1313\u20131320 . Suyog Dutt\u00a0Jain and Kristen Grauman. 2013. Predicting sufficient annotation strength for interactive foreground segmentation. In Proceedings of the IEEE International Conference on Computer Vision. 1313\u20131320."},{"key":"e_1_3_2_1_12_1","volume-title":"Brown Corpus manual: Manual of information to accompany a standard corpus of present-day edited American English for use with digital computers","author":"Francis W\u00a0Nelson","year":"1979","unstructured":"W\u00a0Nelson Francis and Henry Kucera . 1979. Brown Corpus manual: Manual of information to accompany a standard corpus of present-day edited American English for use with digital computers . Brown University, Providence, Rhode Island , USA ( 1979 ). W\u00a0Nelson Francis and Henry Kucera. 1979. Brown Corpus manual: Manual of information to accompany a standard corpus of present-day edited American English for use with digital computers. Brown University, Providence, Rhode Island, USA (1979)."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISIT.2004.1365067"},{"key":"e_1_3_2_1_14_1","volume-title":"6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). 41\u201349","author":"Goyal Tanya","year":"2018","unstructured":"Tanya Goyal , Tyler McDonnell , Mucahid Kutlu , Tamer Elsayed , and Matthew Lease . 2018 . Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations . In 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). 41\u201349 . Tanya Goyal, Tyler McDonnell, Mucahid Kutlu, Tamer Elsayed, and Matthew Lease. 2018. Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations. In 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). 41\u201349."},{"key":"e_1_3_2_1_15_1","volume-title":"Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass","author":"Gray L","year":"2019","unstructured":"Mary\u00a0 L Gray and Siddharth Suri . 2019 . Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass . Eamon Dolan Books . Mary\u00a0L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Eamon Dolan Books."},{"key":"e_1_3_2_1_16_1","unstructured":"David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017).  David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017)."},{"key":"e_1_3_2_1_17_1","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1120\u20131130","author":"Hovy Dirk","year":"2013","unstructured":"Dirk Hovy , Taylor Berg-Kirkpatrick , Ashish Vaswani , and Eduard Hovy . 2013 . Learning whom to trust with MACE . In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1120\u20131130 . Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1120\u20131130."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1261"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41154-0_1"},{"key":"e_1_3_2_1_20_1","unstructured":"Panos Ipeirotis. 2010. A Computer Scientist in a Business School. https:\/\/www.behind-the-enemy-lines.com\/2010\/09\/worker-evaluation-in-crowdsourcing-gold.html September 15.  Panos Ipeirotis. 2010. A Computer Scientist in a Business School. https:\/\/www.behind-the-enemy-lines.com\/2010\/09\/worker-evaluation-in-crowdsourcing-gold.html September 15."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2010.79"},{"key":"e_1_3_2_1_23_1","volume-title":"Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43\u201352","author":"Kittur Aniket","year":"2011","unstructured":"Aniket Kittur , Boris Smus , Susheel Khamkar , and Robert\u00a0 E Kraut . 2011 . Crowdforge: Crowdsourcing complex work . In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43\u201352 . Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert\u00a0E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43\u201352."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.502"},{"key":"e_1_3_2_1_26_1","volume-title":"Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence.","author":"Lin H","year":"2012","unstructured":"Christopher\u00a0 H Lin , Mausam Mausam , and Daniel\u00a0 S Weld . 2012 . Crowdsourcing control: Moving beyond multiple choice . In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence. Christopher\u00a0H Lin, Mausam Mausam, and Daniel\u00a0S Weld. 2012. Crowdsourcing control: Moving beyond multiple choice. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_2_1_27_1","volume-title":"Proceedings of the 29th International Coference on International Conference on Machine Learning. Omnipress, 17\u201324","author":"Liu Chao","year":"2012","unstructured":"Chao Liu and Yi-Min Wang . 2012 . TrueLabel+ confusions: a spectrum of probabilistic models in analyzing multiple ratings . In Proceedings of the 29th International Coference on International Conference on Machine Learning. Omnipress, 17\u201324 . Chao Liu and Yi-Min Wang. 2012. TrueLabel+ confusions: a spectrum of probabilistic models in analyzing multiple ratings. In Proceedings of the 29th International Coference on International Conference on Machine Learning. Omnipress, 17\u201324."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. arXiv preprint cs\/0205028(2002).  Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. arXiv preprint cs\/0205028(2002).","DOI":"10.3115\/1118108.1118117"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1571994"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-5010"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Mitchell Marcus Beatrice Santorini and Mary\u00a0Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. (1993).  Mitchell Marcus Beatrice Santorini and Mary\u00a0Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. (1993).","DOI":"10.21236\/ADA273556"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2464464.2464485"},{"volume-title":"Proceedings of the main conference on human language technology conference of the North American","author":"McClosky David","key":"e_1_3_2_1_33_1","unstructured":"David McClosky , Eugene Charniak , and Mark Johnson . 2006. Effective self-training for parsing . In Proceedings of the main conference on human language technology conference of the North American Chapter of the Association of Computational Linguistics. Association for Computational Linguistics , 152\u2013159. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the main conference on human language technology conference of the North American Chapter of the Association of Computational Linguistics. Association for Computational Linguistics, 152\u2013159."},{"key":"e_1_3_2_1_34_1","first-page":"27","article-title":"Review of the development of multidimensional scaling methods","volume":"41","author":"Mead Al","year":"1992","unstructured":"Al Mead . 1992 . Review of the development of multidimensional scaling methods . Journal of the Royal Statistical Society: Series D (The Statistician) 41 , 1(1992), 27 \u2013 39 . Al Mead. 1992. Review of the development of multidimensional scaling methods. Journal of the Royal Statistical Society: Series D (The Statistician) 41, 1(1992), 27\u201339.","journal-title":"Journal of the Royal Statistical Society: Series D (The Statistician)"},{"key":"e_1_3_2_1_35_1","unstructured":"Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781(2013).  Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781(2013)."},{"key":"e_1_3_2_1_36_1","volume-title":"Third AAAI conference on human computation and crowdsourcing.","author":"Nguyen An\u00a0Thanh","year":"2015","unstructured":"An\u00a0Thanh Nguyen , Byron\u00a0 C Wallace , and Matthew Lease . 2015 . Combining crowd and expert labels using decision theoretic active learning . In Third AAAI conference on human computation and crowdsourcing. An\u00a0Thanh Nguyen, Byron\u00a0C Wallace, and Matthew Lease. 2015. Combining crowd and expert labels using decision theoretic active learning. In Third AAAI conference on human computation and crowdsourcing."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1028"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324906004505"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047198"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1019"},{"key":"e_1_3_2_1_41_1","volume-title":"Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311\u2013318","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . 2002 . BLEU: a method for automatic evaluation of machine translation . In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311\u2013318 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311\u2013318."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00185"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1218"},{"key":"e_1_3_2_1_44_1","volume-title":"EMNLP Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. Association for Computational Linguistics","author":"Paun Silviu","year":"2019","unstructured":"Silviu Paun and Dirk Hovy . 2019 . EMNLP Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. Association for Computational Linguistics , Hong Kong. http:\/\/dali.eecs.qmul.ac.uk\/annonlp Silviu Paun and Dirk Hovy. 2019. EMNLP Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. Association for Computational Linguistics, Hong Kong. http:\/\/dali.eecs.qmul.ac.uk\/annonlp"},{"key":"e_1_3_2_1_45_1","volume-title":"Proceedings of the Seventh Workshop on Statistical Machine Translation. Association for Computational Linguistics, 401\u2013409","author":"Post Matt","year":"2012","unstructured":"Matt Post , Chris Callison-Burch , and Miles Osborne . 2012 . Constructing parallel corpora for six indian languages via crowdsourcing . In Proceedings of the Seventh Workshop on Statistical Machine Translation. Association for Computational Linguistics, 401\u2013409 . Matt Post, Chris Callison-Burch, and Miles Osborne. 2012. Constructing parallel corpora for six indian languages via crowdsourcing. In Proceedings of the Seventh Workshop on Statistical Machine Translation. Association for Computational Linguistics, 401\u2013409."},{"key":"e_1_3_2_1_46_1","unstructured":"Tao Qin Xiubo Geng and Tie-Yan Liu. 2010. A new probabilistic model for rank aggregation. In Advances in neural information processing systems. 1948\u20131956.  Tao Qin Xiubo Geng and Tie-Yan Liu. 2010. A new probabilistic model for rank aggregation. In Advances in neural information processing systems. 1948\u20131956."},{"key":"e_1_3_2_1_47_1","first-page":"1297","article-title":"Learning from crowds","author":"Raykar C","year":"2010","unstructured":"Vikas\u00a0 C Raykar , Shipeng Yu , Linda\u00a0 H Zhao , Gerardo\u00a0Hermosillo Valadez , Charles Florin , Luca Bogoni , and Linda Moy . 2010 . Learning from crowds . Journal of Machine Learning Research 11 , Apr (2010), 1297 \u2013 1322 . Vikas\u00a0C Raykar, Shipeng Yu, Linda\u00a0H Zhao, Gerardo\u00a0Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research 11, Apr (2010), 1297\u20131322.","journal-title":"Journal of Machine Learning Research 11"},{"key":"e_1_3_2_1_48_1","volume-title":"Sequence labeling with multiple annotators. Machine learning 95, 2","author":"Rodrigues Filipe","year":"2014","unstructured":"Filipe Rodrigues , Francisco Pereira , and Bernardete Ribeiro . 2014. Sequence labeling with multiple annotators. Machine learning 95, 2 ( 2014 ), 165\u2013181. Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2014. Sequence labeling with multiple annotators. Machine learning 95, 2 (2014), 165\u2013181."},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047199"},{"key":"e_1_3_2_1_50_1","unstructured":"Satoshi Sekine and Michael Collins. 1997. EvalB: a bracket scoring program. http:\/\/ nlp.cs.nyu.edu\/evalb\/  Satoshi Sekine and Michael Collins. 1997. EvalB: a bracket scoring program. http:\/\/ nlp.cs.nyu.edu\/evalb\/"},{"key":"e_1_3_2_1_51_1","volume-title":"Proceedings of the 1st AAAI Conference on Human Computation (HCOMP). 156\u2013164","author":"Sheshadri Aashish","year":"2013","unstructured":"Aashish Sheshadri and Matthew Lease . 2013 . SQUARE: A Benchmark for Research on Computing Crowd Consensus . In Proceedings of the 1st AAAI Conference on Human Computation (HCOMP). 156\u2013164 . Aashish Sheshadri and Matthew Lease. 2013. SQUARE: A Benchmark for Research on Computing Crowd Consensus. In Proceedings of the 1st AAAI Conference on Human Computation (HCOMP). 156\u2013164."},{"key":"e_1_3_2_1_52_1","unstructured":"Yu\u00a0A Shreider. 1974. What Is Distance? Popular Lectures in Mathematics.(1974).  Yu\u00a0A Shreider. 1974. What Is Distance? Popular Lectures in Mathematics.(1974)."},{"key":"e_1_3_2_1_53_1","unstructured":"Padhraic Smyth Usama\u00a0M Fayyad Michael\u00a0C Burl Pietro Perona and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of venus images. In Advances in neural information processing systems. 1085\u20131092.  Padhraic Smyth Usama\u00a0M Fayyad Michael\u00a0C Burl Pietro Perona and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of venus images. In Advances in neural information processing systems. 1085\u20131092."},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613751"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242604"},{"key":"e_1_3_2_1_56_1","volume-title":"SIGIR 2011 workshop on crowdsourcing for information retrieval (CIR). 1\u20136.","author":"Tang Wei","year":"2011","unstructured":"Wei Tang and Matthew Lease . 2011 . Semi-supervised consensus labeling for crowdsourcing . In SIGIR 2011 workshop on crowdsourcing for information retrieval (CIR). 1\u20136. Wei Tang and Matthew Lease. 2011. Semi-supervised consensus labeling for crowdsourcing. In SIGIR 2011 workshop on crowdsourcing for information retrieval (CIR). 1\u20136."},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339571"},{"key":"e_1_3_2_1_58_1","unstructured":"Jing Wang Panagiotis\u00a0G Ipeirotis and Foster Provost. 2011. Managing crowdsourcing workers. In The 2011 winter conference on business intelligence. Citeseer 10\u201312.  Jing Wang Panagiotis\u00a0G Ipeirotis and Foster Provost. 2011. Managing crowdsourcing workers. In The 2011 winter conference on business intelligence. Citeseer 10\u201312."},{"key":"e_1_3_2_1_59_1","unstructured":"Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier\u00a0R Movellan and Paul\u00a0L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems. 2035\u20132043.  Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier\u00a0R Movellan and Paul\u00a0L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems. 2035\u20132043."},{"key":"e_1_3_2_1_60_1","unstructured":"Christopher\u00a0KI Williams. 2001. On a connection between kernel PCA and metric multidimensional scaling. In Advances in neural information processing systems. 675\u2013681.  Christopher\u00a0KI Williams. 2001. On a connection between kernel PCA and metric multidimensional scaling. In Advances in neural information processing systems. 675\u2013681."},{"key":"e_1_3_2_1_61_1","unstructured":"Yonghui Wu Mike Schuster Zhifeng Chen Quoc\u00a0V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey 2016. Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144(2016).  Yonghui Wu Mike Schuster Zhifeng Chen Quoc\u00a0V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey 2016. Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144(2016)."},{"key":"e_1_3_2_1_62_1","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, 1220\u20131229","author":"Zaidan F","year":"2011","unstructured":"Omar\u00a0 F Zaidan and Chris Callison-Burch . 2011 . Crowdsourcing translation: Professional quality from non-professionals . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, 1220\u20131229 . Omar\u00a0F Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, 1220\u20131229."},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"}],"event":{"name":"WWW '20: The Web Conference 2020","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"],"location":"Taipei Taiwan","acronym":"WWW '20"},"container-title":["Proceedings of The Web Conference 2020"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3366423.3380250","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3366423.3380250","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:33:10Z","timestamp":1750199590000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3366423.3380250"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,20]]},"references-count":63,"alternative-id":["10.1145\/3366423.3380250","10.1145\/3366423"],"URL":"https:\/\/doi.org\/10.1145\/3366423.3380250","relation":{},"subject":[],"published":{"date-parts":[[2020,4,20]]},"assertion":[{"value":"2020-04-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}