{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,31]],"date-time":"2025-12-31T08:36:21Z","timestamp":1767170181960,"version":"build-2238731810"},"reference-count":85,"publisher":"MIT Press","issue":"2","license":[{"start":{"date-parts":[[2022,3,8]],"date-time":"2022-03-08T00:00:00Z","timestamp":1646697600000},"content-version":"vor","delay-in-days":66,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,6,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Annotation studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations; especially in citizen science or crowdsourcing scenarios where domain expertise is not required. To alleviate these issues, this work proposes annotation curricula, a novel approach to implicitly train annotators. The goal is to gradually introduce annotators into the task by ordering instances to be annotated according to a learning curriculum. To do so, this work formalizes annotation curricula for sentence- and paragraph-level annotation tasks, defines an ordering strategy, and identifies well-performing heuristics and interactively trained models on three existing English datasets. Finally, we provide a proof of concept for annotation curricula in a carefully designed user study with 40 voluntary participants who are asked to identify the most fitting misconception for English tweets about the Covid-19 pandemic. The results indicate that using a simple heuristic to order instances can already significantly reduce the total annotation time while preserving a high annotation quality. Annotation curricula thus can be a promising research direction to improve data collection. To facilitate future research\u2014for instance, to adapt annotation curricula to specific tasks and expert annotation scenarios\u2014all code and data from the user study consisting of 2,400 annotations is made available.1<\/jats:p>","DOI":"10.1162\/coli_a_00436","type":"journal-article","created":{"date-parts":[[2022,3,8]],"date-time":"2022-03-08T15:26:26Z","timestamp":1646753186000},"page":"343-373","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":5,"title":["Annotation Curricula to Implicitly Train Non-Expert\n                    Annotators"],"prefix":"10.1162","volume":"48","author":[{"given":"Ji-Ung","family":"Lee","sequence":"first","affiliation":[{"name":"UKP Lab \/ TU Darmstadt. lee@ukp.informatik.tu-darmstadt.de"}]},{"given":"Jan-Christoph","family":"Klie","sequence":"additional","affiliation":[{"name":"UKP Lab \/ TU Darmstadt. klie@ukp.informatik.tu-darmstadt.de"}]},{"given":"Iryna","family":"Gurevych","sequence":"additional","affiliation":[{"name":"UKP Lab \/ TU Darmstadt. gurevych@ukp.informatik.tu-darmstadt.de"}]}],"member":"281","published-online":{"date-parts":[[2022,6,9]]},"reference":[{"key":"2022121415400804500_bib1","first-page":"56","article-title":"Automatic gap-fill question generation\n                        from text books","volume-title":"Proceedings of the Sixth\n                        Workshop on Innovative Use of NLP for Building Educational\n                        Applications","author":"Agarwal","year":"2011"},{"key":"2022121415400804500_bib2","first-page":"1","article-title":"Deep batch active learning by diverse,\n                        uncertain gradient lower bounds","volume-title":"International\n                        Conference on Learning Representations","author":"Ash","year":"2020"},{"key":"2022121415400804500_bib3","volume-title":"The Basics of Item Response Theory","author":"Baker","year":"2001"},{"key":"2022121415400804500_bib4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/2021.acl-long.1","article-title":"Investigating label suggestions for\n                        opinion mining in German Covid-19 social media","volume-title":"Proceedings of the 59th Annual Meeting of the Association for\n                        Computational Linguistics and the 11th International Joint Conference on\n                        Natural Language Processing (Volume 1: Long Papers)","author":"Beck","year":"2021"},{"key":"2022121415400804500_bib5","doi-asserted-by":"publisher","first-page":"390","DOI":"10.3115\/v1\/P14-2064","article-title":"Difficult cases: From data to learning,\n                        and back","volume-title":"Proceedings of the 52nd Annual Meeting\n                        of the Association for Computational Linguistics (Volume 2: Short\n                        Papers)","author":"KlebanovBeata","year":"2014"},{"key":"2022121415400804500_bib6","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1162\/tacl_a_00200","article-title":"Predicting the difficulty of language\n                        proficiency tests","volume":"2","author":"Beinborn","year":"2014","journal-title":"Transactions of the Association\n                        for Computational Linguistics"},{"key":"2022121415400804500_bib7","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1145\/1553374.1553380","article-title":"Curriculum learning","volume-title":"Proceedings of the 26th Annual International Conference on Machine\n                        Learning","author":"Bengio","year":"2009"},{"key":"2022121415400804500_bib8","first-page":"3","article-title":"Teoria statistica delle classi e calcolo\n                        delle probabilita","volume":"8","author":"Bonferroni","year":"1936","journal-title":"Pubblicazioni del R Istituto\n                        Superiore di Scienze Economiche e Commericiali di Firenze"},{"key":"2022121415400804500_bib9","doi-asserted-by":"publisher","first-page":"632","DOI":"10.18653\/v1\/D15-1075","article-title":"A large annotated corpus for learning\n                        natural language inference","volume-title":"Proceedings of the\n                        2015 Conference on Empirical Methods in Natural Language\n                    Processing","author":"Bowman","year":"2015"},{"key":"2022121415400804500_bib10","first-page":"74","article-title":"Understanding the tradeoff between cost and\n                        quality of expert annotations for keyphrase extraction","volume-title":"Proceedings of the 14th Linguistic Annotation Workshop","author":"Chau","year":"2020"},{"key":"2022121415400804500_bib11","volume-title":"Common European Framework of\n                        Reference for Languages: Learning, Teaching, Assessment","author":"","year":"2001"},{"key":"2022121415400804500_bib12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/2020.bea-1.1","article-title":"Linguistic features for\n                        readability assessment.","volume-title":"Proceedings of\n                        the Fifteenth Workshop on Innovative Use of NLP for Building Educational\n                        Applications","author":"Deutsch","year":"2020"},{"key":"2022121415400804500_bib13","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional\n                        transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of\n                        the Association for Computational Linguistics: Human Language Technologies,\n                        Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2022121415400804500_bib14","first-page":"1809","article-title":"Active learning for crowdsourcing using knowledge\n                        transfer","volume-title":"Proceedings of the Twenty-Eighth AAAI\n                        Conference on Artificial Intelligence","author":"Fang","year":"2014"},{"key":"2022121415400804500_bib15","doi-asserted-by":"publisher","first-page":"858","DOI":"10.1109\/ICDM.2012.64","article-title":"Self-taught active learning from\n                        crowds","volume-title":"2012 IEEE 12th International Conference\n                        on Data Mining","author":"Fang","year":"2012"},{"key":"2022121415400804500_bib16","doi-asserted-by":"publisher","first-page":"323","DOI":"10.26615\/978-954-452-056-4_037","article-title":"Entropy as a proxy for gap complexity in\n                        open cloze tests","volume-title":"Proceedings of the\n                        International Conference on Recent Advances in Natural Language Processing\n                        (RANLP 2019)","author":"Felice","year":"2019"},{"key":"2022121415400804500_bib17","first-page":"56","article-title":"Influence of pre-annotation on POS-tagged\n                        corpus development","volume-title":"Proceedings of the Fourth\n                        Linguistic Annotation Workshop","author":"Fort","year":"2010"},{"key":"2022121415400804500_bib18","first-page":"1923","article-title":"Deep Bayesian active learning with image\n                        data","volume-title":"Proceedings of the 34th International\n                        Conference on Machine Learning","author":"Gal","year":"2017"},{"key":"2022121415400804500_bib19","doi-asserted-by":"crossref","first-page":"102","DOI":"10.18653\/v1\/2020.bea-1.10","article-title":"Distractor analysis and selection for\n                        multiple-choice cloze questions for second-language\n                    learners","volume-title":"Proceedings of the Fifteenth Workshop on\n                        Innovative Use of NLP for Building Educational Applications","author":"Gao","year":"2020"},{"issue":"12","key":"2022121415400804500_bib20","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1145\/3458723","article-title":"Datasheets for datasets","volume":"64","author":"Gebru","year":"2021","journal-title":"Communications of ACM"},{"key":"2022121415400804500_bib21","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.18653\/v1\/D19-1107","article-title":"Are we modeling the task or the annotator?\n                        An investigation of annotator bias in natural language understanding\n                        datasets","volume-title":"Proceedings of the 2019 Conference on\n                        Empirical Methods in Natural Language Processing and the 9th International\n                        Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Geva","year":"2019"},{"key":"2022121415400804500_bib22","doi-asserted-by":"publisher","first-page":"8342","DOI":"10.18653\/v1\/2020.acl-main.740","article-title":"Don\u2019t stop pretraining: Adapt language\n                        models to domains and tasks","volume-title":"Proceedings of the\n                        58th Annual Meeting of the Association for Computational\n                        Linguistics","author":"Gururangan","year":"2020"},{"key":"2022121415400804500_bib23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/2020.nlpcovid19-2.11","article-title":"COVIDLies: Detecting COVID-19 misinformation\n                        on social media","volume-title":"Proceedings of the 1st Workshop\n                        on NLP for COVID-19 (Part 2) at EMNLP 2020","author":"Hossain","year":"2020"},{"key":"2022121415400804500_bib24","first-page":"892","article-title":"Active learning by querying informative and representative\n                        examples","volume-title":"Proceedings of the 23rd International\n                        Conference on Neural Information Processing Systems - Volume 1","author":"Huang","year":"2010"},{"key":"2022121415400804500_bib25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.chb.2019.106198","article-title":"Teaching\n                        citizen scientists to categorize glitches using machine learning guided\n                        training","volume":"105","author":"Jackson","year":"2020","journal-title":"Computers in Human Behavior"},{"key":"2022121415400804500_bib26","first-page":"3149","article-title":"LightGBM: A highly efficient gradient boosting decision\n                        tree","volume-title":"Advances in Neural Information Processing\n                        Systems","author":"Ke","year":"2017"},{"key":"2022121415400804500_bib27","volume-title":"The Curriculum: Theory and Practice","author":"Kelly","year":"2009"},{"key":"2022121415400804500_bib28","first-page":"79","article-title":"Aggregation driven progression system for\n                        GWAPs","volume-title":"Workshop on Games and Natural Language\n                        Processing","author":"Kicikoglu","year":"2020"},{"key":"2022121415400804500_bib29","doi-asserted-by":"crossref","unstructured":"Kincaid, J.\n                                Peter, Robert\n                                P. FishburneJr., Richard\n                                L.Rogers, and\n                                Brad S.Chissom.\n                        1975. Derivation of new readability formulas\n                        (automated readability index, fog count and Flesch reading ease formula) for\n                        Navy enlisted personnel. Technical report, Naval\n                        Technical Training Command Millington TN Research Branch. 10.21236\/ADA006655","DOI":"10.21236\/ADA006655"},{"key":"2022121415400804500_bib30","first-page":"7026","article-title":"BatchBALD: Efficient and diverse batch acquisition for deep\n                        Bayesian active learning","volume-title":"Advances in Neural\n                        Information Processing Systems","author":"Kirsch","year":"2019"},{"key":"2022121415400804500_bib31","first-page":"5","article-title":"The INCEpTION platform: Machine-assisted\n                        and knowledge-oriented interactive annotation","volume-title":"Proceedings of the 27th International Conference on Computational\n                        Linguistics: System Demonstrations","author":"Klie","year":"2018"},{"key":"2022121415400804500_bib32","doi-asserted-by":"publisher","first-page":"6982","DOI":"10.18653\/v1\/2020.acl-main.624","article-title":"From zero to hero: Human-in-the-loop\n                        entity linking in low resource domains","volume-title":"Proceedings of the 58th Annual Meeting of the Association for\n                        Computational Linguistics","author":"Klie","year":"2020"},{"key":"2022121415400804500_bib33","volume-title":"Principles and Practice in Second Language\n                        Acquisition","author":"Krashen","year":"1982"},{"issue":"260","key":"2022121415400804500_bib34","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1080\/01621459.1952.10483441","article-title":"Use of ranks in one-criterion variance\n                        analysis","volume":"47","author":"Kruskal","year":"1952","journal-title":"Journal of the American Statistical\n                        Association"},{"key":"2022121415400804500_bib35","first-page":"1189","article-title":"Self-paced learning for latent variable\n                        models","volume-title":"Advances in Neural Information Processing\n                        Systems","author":"Kumar","year":"2010"},{"key":"2022121415400804500_bib36","first-page":"1546","article-title":"Active learning with Amazon Mechanical\n                        Turk","volume-title":"Proceedings of the 2011 Conference on\n                        Empirical Methods in Natural Language Processing","author":"Laws","year":"2011"},{"key":"2022121415400804500_bib37","doi-asserted-by":"crossref","first-page":"4233","DOI":"10.18653\/v1\/2020.acl-main.390","article-title":"Empowering active learning to jointly\n                        optimize system and user demands","volume-title":"Proceedings of\n                        the 58th Annual Meeting of the Association for Computational\n                        Linguistics","author":"Lee","year":"2020"},{"key":"2022121415400804500_bib38","doi-asserted-by":"publisher","first-page":"360","DOI":"10.18653\/v1\/P19-1035","article-title":"Manipulating the difficulty of\n                        c-tests","volume-title":"Proceedings of the 57th Annual Meeting\n                        of the Association for Computational Linguistics","author":"Lee","year":"2019"},{"key":"2022121415400804500_bib39","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-1-4471-2099-5_1","article-title":"A sequential algorithm for training text\n                        classifiers","volume-title":"Proceedings of the 17th Annual\n                        International ACM SIGIR Conference on Research and Development in\n                        Information Retrieval","author":"Lewis","year":"1994"},{"issue":"3","key":"2022121415400804500_bib40","doi-asserted-by":"publisher","first-page":"406","DOI":"10.1136\/amiajnl-2013-001837","article-title":"Evaluating the impact of pre-annotation on\n                        annotation speed and potential bias: Natural language processing gold\n                        standard development for clinical named entity recognition in clinical trial\n                        announcements","volume":"21","author":"Lingren","year":"2014","journal-title":"Journal of the American Medical\n                        Informatics Association"},{"key":"2022121415400804500_bib41","first-page":"3245","article-title":"Textual complexity as a predictor of\n                        difficulty of listening items in language proficiency tests","volume-title":"Proceedings of COLING 2016, the 26th International Conference on\n                        Computational Linguistics: Technical Papers","author":"Loukina","year":"2016"},{"key":"2022121415400804500_bib42","doi-asserted-by":"publisher","first-page":"21","DOI":"10.18653\/v1\/D19-1003","article-title":"Practical obstacles to deploying active\n                        learning","volume-title":"Proceedings of the 2019 Conference on\n                        Empirical Methods in Natural Language Processing and the 9th International\n                        Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Lowell","year":"2019"},{"key":"2022121415400804500_bib43","first-page":"77","article-title":"Progression in a language annotation game\n                        with a purpose","volume-title":"Proceedings of the AAAI\n                        Conference on Human Computation and Crowdsourcing","author":"Madge","year":"2019"},{"key":"2022121415400804500_bib44","first-page":"245","article-title":"Active learning for sense\n                        annotation","volume-title":"Proceedings of the 20th Nordic\n                        Conference of Computational Linguistics (NODALIDA 2015)","author":"Mart\u00ednez Alonso","year":"2015"},{"key":"2022121415400804500_bib45","doi-asserted-by":"publisher","first-page":"151","DOI":"10.18653\/v1\/2020.bea-1.15","article-title":"Should you fine-tune BERT for automated essay\n                        scoring?","volume-title":"Proceedings of the Fifteenth Workshop on\n                        Innovative Use of NLP for Building Educational Applications","author":"Mayfield","year":"2020"},{"key":"2022121415400804500_bib46","first-page":"136","article-title":"Generating diagnostic multiple choice comprehension cloze\n                        questions","volume-title":"Proceedings of the Seventh Workshop on\n                        Building Educational Applications Using NLP","author":"Mostow","year":"2012"},{"key":"2022121415400804500_bib47","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1145\/1015330.1015349","article-title":"Active learning using\n                        pre-clustering","volume-title":"Proceedings of the Twenty-First\n                        International Conference on Machine Learning","author":"Nguyen","year":"2004"},{"key":"2022121415400804500_bib48","doi-asserted-by":"publisher","first-page":"4885","DOI":"10.18653\/v1\/2020.acl-main.441","article-title":"Adversarial NLI: A new benchmark for natural\n                        language understanding","volume-title":"Proceedings of the 58th\n                        Annual Meeting of the Association for Computational Linguistics","author":"Nie","year":"2020"},{"key":"2022121415400804500_bib49","first-page":"1320","article-title":"Twitter as a corpus for sentiment analysis\n                        and opinion mining","volume-title":"Proceedings of the Seventh\n                        International Conference on Language Resources and Evaluation\n                        (LREC\u201910)","author":"Pak","year":"2010"},{"key":"2022121415400804500_bib50","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1145\/3351095.3372843","article-title":"Bias in word embeddings","volume-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and\n                        Transparency","author":"Papakyriakopoulos","year":"2020"},{"key":"2022121415400804500_bib51","doi-asserted-by":"publisher","first-page":"571","DOI":"10.1162\/tacl_a_00040","article-title":"Comparing Bayesian models of\n                        annotation","volume":"6","author":"Paun","year":"2018","journal-title":"Transactions of the Association for\n                        Computational Linguistics"},{"key":"2022121415400804500_bib52","first-page":"2825","article-title":"Scikit-learn: Machine learning in\n                        Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"Journal of Machine Learning\n                        Research"},{"key":"2022121415400804500_bib53","first-page":"2227","article-title":"Deep contextualized word\n                        representations","volume-title":"Proceedings of the 2018\n                        Conference of the North American Chapter of the Association for\n                        Computational Linguistics: Human Language Technologies, Volume 1 (Long\n                        Papers)","author":"Peters","year":"2018"},{"key":"2022121415400804500_bib54","doi-asserted-by":"publisher","first-page":"7","DOI":"10.18653\/v1\/W19-4302","article-title":"To tune or not to tune? Adapting pretrained\n                        representations to diverse tasks","volume-title":"Proceedings of\n                        the 4th Workshop on Representation Learning for NLP\n                    (RepL4NLP-2019)","author":"Peters","year":"2019"},{"key":"2022121415400804500_bib55","doi-asserted-by":"publisher","first-page":"3982","DOI":"10.18653\/v1\/D19-1410","article-title":"Sentence-BERT: Sentence embeddings using\n                        Siamese BERT-networks","volume-title":"Proceedings of the 2019\n                        Conference on Empirical Methods in Natural Language Processing and the 9th\n                        International Joint Conference on Natural Language Processing\n                        (EMNLP-IJCNLP)","author":"Reimers","year":"2019"},{"key":"2022121415400804500_bib56","doi-asserted-by":"publisher","first-page":"2182","DOI":"10.18653\/v1\/2021.acl-long.170","article-title":"Changing the world by changing the\n                        data","volume-title":"Proceedings of the 59th Annual Meeting of\n                        the Association for Computational Linguistics and the 11th International\n                        Joint Conference on Natural Language Processing (Volume 1: Long\n                        Papers)","author":"Rogers","year":"2021"},{"key":"2022121415400804500_bib57","first-page":"441","article-title":"Toward optimal active learning through\n                        sampling estimation of error reduction","volume-title":"Proceedings of the Eighteenth International Conference on Machine\n                        Learning","author":"Roy","year":"2001"},{"issue":"9","key":"2022121415400804500_bib58","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1145\/3474381","article-title":"WinoGrande: An adversarial Winograd Schema Challenge at\n                        scale","volume":"64","author":"Sakaguchi","year":"2021","journal-title":"Communications of ACM"},{"key":"2022121415400804500_bib59","doi-asserted-by":"publisher","first-page":"2699","DOI":"10.18653\/v1\/2020.acl-main.240","article-title":"Masked language model\n                        scoring","volume-title":"Proceedings of the 58th Annual Meeting\n                        of the Association for Computational Linguistics","author":"Salazar","year":"2020"},{"key":"2022121415400804500_bib60","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3411764.3445518","article-title":"\u201cEveryone wants to do the model work,\n                        not the data work\u201d: Data cascades in high-stakes AI","volume-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing\n                        Systems","author":"Sambasivan","year":"2021"},{"key":"2022121415400804500_bib61","article-title":"DistilBERT, a distilled version of BERT: smaller, faster,\n                        cheaper and lighter","volume":"abs\/1910.01108","author":"Sanh","year":"2019","journal-title":"ArXiv"},{"key":"2022121415400804500_bib62","doi-asserted-by":"publisher","first-page":"2761","DOI":"10.18653\/v1\/P19-1265","article-title":"Analysis of automatic annotation\n                        suggestions for hard discourse-level tasks in expert\n                    domains","volume-title":"Proceedings of the 57th Annual Meeting of\n                        the Association for Computational Linguistics","author":"Schulz","year":"2019"},{"key":"2022121415400804500_bib63","doi-asserted-by":"publisher","first-page":"1715","DOI":"10.18653\/v1\/P16-1162","article-title":"Neural machine translation of rare words with\n                        subword units","volume-title":"Proceedings of the 54th Annual\n                        Meeting of the Association for Computational Linguistics (Volume 1: Long\n                        Papers)","author":"Sennrich","year":"2016"},{"issue":"1","key":"2022121415400804500_bib64","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2200\/S00429ED1V01Y201207AIM018","article-title":"Active learning","volume":"6","author":"Settles","year":"2012","journal-title":"Synthesis Lectures on Artificial Intelligence and Machine\n                        Learning"},{"key":"2022121415400804500_bib65","first-page":"1","article-title":"Active learning with real annotation\n                        costs","volume-title":"Proceedings of the NIPS Workshop on\n                        Cost-Sensitive Learning","author":"Settles","year":"2008"},{"key":"2022121415400804500_bib66","doi-asserted-by":"publisher","first-page":"2904","DOI":"10.18653\/v1\/D18-1318","article-title":"Deep Bayesian active learning for natural\n                        language processing: Results of a large-scale empirical\n                        study","volume-title":"Proceedings of the 2018 Conference on\n                        Empirical Methods in Natural Language Processing","author":"Siddhant","year":"2018"},{"key":"2022121415400804500_bib67","doi-asserted-by":"publisher","first-page":"254","DOI":"10.3115\/1613715.1613751","article-title":"Cheap and fast \u2013 but is it good? Evaluating non-expert\n                        annotations for natural language tasks","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural\n                        Language Processing","author":"Snow","year":"2008"},{"key":"2022121415400804500_bib68","doi-asserted-by":"publisher","first-page":"21","DOI":"10.18653\/v1\/N18-5005","article-title":"ArgumenText: Searching for arguments in\n                        heterogeneous sources","volume-title":"Proceedings of the 2018\n                        Conference of the North American Chapter of the Association for\n                        Computational Linguistics: Demonstrations","author":"Stab","year":"2018"},{"issue":"3","key":"2022121415400804500_bib69","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1145\/1077246.1077253","article-title":"GameFlow: A model for evaluating player\n                        enjoyment in games","volume":"3","author":"Sweetser","year":"2005","journal-title":"Computers in\n                        Entertainment"},{"key":"2022121415400804500_bib70","doi-asserted-by":"publisher","DOI":"10.1515\/9781400834440","volume-title":"Numbers Rule: The Vexing Mathematics of\n                        Democracy, from Plato to the Present","author":"Szpiro","year":"2010"},{"key":"2022121415400804500_bib71","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1145\/3379336.3381499","article-title":"The influence of input data complexity on\n                        crowdsourcing quality","volume-title":"Proceedings of the 25th\n                        International Conference on Intelligent User Interfaces Companion","author":"Tauchmann","year":"2020"},{"issue":"4","key":"2022121415400804500_bib72","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1177\/107769905303000401","article-title":"\u201cCloze procedure\u201d: A new tool for measuring\n                        readability","volume":"30","author":"Taylor","year":"1953","journal-title":"Journalism & Mass Communication\n                        Quarterly"},{"key":"2022121415400804500_bib73","doi-asserted-by":"publisher","first-page":"112","DOI":"10.3115\/1698381.1698399","article-title":"Timed annotations: Enhancing MUC7 metadata by the time it\n                        takes to annotate named entities","volume-title":"Proceedings of\n                        the Third Linguistic Annotation Workshop","author":"Tomanek","year":"2009"},{"key":"2022121415400804500_bib74","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.cogpsych.2016.07.003","article-title":"The anchor integration model: A\n                        descriptive model of anchoring effects","volume":"90","author":"Turner","year":"2016","journal-title":"Cognitive\n                        Psychology"},{"issue":"1","key":"2022121415400804500_bib75","doi-asserted-by":"publisher","first-page":"196","DOI":"10.1214\/12-AOS1058","article-title":"On the definition of a\n                        confounder","volume":"41","author":"VanderWeele","year":"2013","journal-title":"Annals of Statistics"},{"key":"2022121415400804500_bib76","volume-title":"Mind in Society: The Development of Higher\n                        Psychological Processes","author":"Vygotsky","year":"1978"},{"key":"2022121415400804500_bib77","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1016\/j.eswa.2017.05.046","article-title":"Active learning through density\n                        clustering","volume":"85","author":"Wang","year":"2017","journal-title":"Expert Systems with\n                        Applications"},{"issue":"3\/4","key":"2022121415400804500_bib78","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1093\/biomet\/38.3-4.330","article-title":"On the comparison of several mean values: An alternative\n                        approach","volume":"38","author":"Welch","year":"1951","journal-title":"Biometrika"},{"key":"2022121415400804500_bib79","doi-asserted-by":"publisher","first-page":"12","DOI":"10.18653\/v1\/W16-0502","article-title":"Text readability assessment for\n                        second language learners","volume-title":"Proceedings of\n                        the 11th Workshop on Innovative Use of NLP for Building Educational\n                        Applications","author":"Xia","year":"2016"},{"key":"2022121415400804500_bib80","doi-asserted-by":"publisher","first-page":"1471","DOI":"10.18653\/v1\/N19-1150","article-title":"Predicting annotation difficulty to\n                        improve task routing and model performance for biomedical information\n                        extraction","volume-title":"Proceedings of the 2019 Conference of\n                        the North American Chapter of the Association for Computational Linguistics:\n                        Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Yang","year":"2019"},{"key":"2022121415400804500_bib81","first-page":"159","article-title":"Difficulty-aware distractor generation for gap-fill\n                        items","volume-title":"Proceedings of the 17th Annual Workshop of\n                        the Australasian Language Technology Association","author":"Yeung","year":"2019"},{"key":"2022121415400804500_bib82","doi-asserted-by":"publisher","first-page":"91","DOI":"10.3115\/v1\/P14-5016","article-title":"Automatic annotation suggestions and\n                        custom annotation layers in WebAnno","volume-title":"Proceedings\n                        of 52nd Annual Meeting of the Association for Computational Linguistics:\n                        System Demonstrations","author":"Yimam","year":"2014"},{"key":"2022121415400804500_bib83","doi-asserted-by":"publisher","first-page":"7935","DOI":"10.18653\/v1\/2020.emnlp-main.637","article-title":"Cold-start active learning through\n                        self-supervised language modeling","volume-title":"Proceedings of\n                        the 2020 Conference on Empirical Methods in Natural Language Processing\n                        (EMNLP)","author":"Yuan","year":"2020"},{"key":"2022121415400804500_bib84","first-page":"703","article-title":"Active learning from weak and strong\n                        labelers","volume-title":"Advances in Neural Information\n                        Processing Systems","author":"Zhang","year":"2015"},{"key":"2022121415400804500_bib85","doi-asserted-by":"publisher","first-page":"1137","DOI":"10.3115\/1599081.1599224","article-title":"Active learning with sampling by uncertainty and density for\n                        word sense disambiguation and text classification","volume-title":"Proceedings of the 22nd International Conference on Computational\n                        Linguistics (Coling 2008)","author":"Zhu","year":"2008"}],"updated-by":[{"DOI":"10.1162\/coli_x_00469","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2022,12,1]],"date-time":"2022-12-01T00:00:00Z","timestamp":1669852800000}}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/48\/2\/343\/2062624\/coli_a_00436.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/48\/2\/343\/2062624\/coli_a_00436.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,14]],"date-time":"2022-12-14T10:40:40Z","timestamp":1671014440000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/48\/2\/343\/109903\/Annotation-Curricula-to-Implicitly-Train-Non"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":85,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,6,9]]},"published-print":{"date-parts":[[2022,6,9]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00436","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}