{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T15:31:25Z","timestamp":1778081485623,"version":"3.51.4"},"reference-count":6,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,3,4]],"date-time":"2015-03-04T00:00:00Z","timestamp":1425427200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Data and Information Quality"],"published-print":{"date-parts":[[2015,3,4]]},"abstract":"<jats:p>Organizations that develop and use technologies around information retrieval, machine learning, recommender systems, and natural language processing depend on labels for engineering and experimentation. These labels, usually gathered via human computation, are used in machine-learned models for prediction and evaluation purposes. In such scenarios, collecting high-quality labels is a very important part of the overall process. We elaborate on these challenges and discuss research directions.<\/jats:p>","DOI":"10.1145\/2724721","type":"journal-article","created":{"date-parts":[[2015,3,9]],"date-time":"2015-03-09T19:03:01Z","timestamp":1425927781000},"page":"1-3","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Challenges with Label Quality for Supervised Learning"],"prefix":"10.1145","volume":"6","author":[{"given":"Omar","family":"Alonso","sequence":"first","affiliation":[{"name":"Microsoft, Mountain View, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2015,3,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2347736.2347755"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i2.2537"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1515693.1516680"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1659225.1659227"},{"key":"e_1_2_1_5_1","volume-title":"Workshop on Active Learning and Experimental Design. 1--18","author":"Settles Burr","year":"2011","unstructured":"Burr Settles . 2011 . From theories to queries: Active learning in practice . In Workshop on Active Learning and Experimental Design. 1--18 . Burr Settles. 2011. From theories to queries: Active learning in practice. In Workshop on Active Learning and Experimental Design. 1--18."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"}],"container-title":["Journal of Data and Information Quality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2724721","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2724721","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:12:12Z","timestamp":1750227132000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2724721"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,3,4]]},"references-count":6,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,3,4]]}},"alternative-id":["10.1145\/2724721"],"URL":"https:\/\/doi.org\/10.1145\/2724721","relation":{},"ISSN":["1936-1955","1936-1963"],"issn-type":[{"value":"1936-1955","type":"print"},{"value":"1936-1963","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,3,4]]},"assertion":[{"value":"2014-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-03-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}