{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,20]],"date-time":"2026-11-20T12:49:40Z","timestamp":1795178980662,"version":"4.10.3"},"reference-count":122,"publisher":"MIT Press","issue":"3","license":[{"start":{"date-parts":[[2021,6,30]],"date-time":"2021-06-30T00:00:00Z","timestamp":1625011200000},"content-version":"vor","delay-in-days":180,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000879","name":"Alfred P. Sloan Foundation","doi-asserted-by":"publisher","award":["2013-10-27"],"award-info":[{"award-number":["2013-10-27"]}],"id":[{"id":"10.13039\/100000879","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000936","name":"Gordon and Betty Moore Foundation","doi-asserted-by":"publisher","award":["GMBF3834"],"award-info":[{"award-number":["GMBF3834"]}],"id":[{"id":"10.13039\/100000936","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,11,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Supervised machine learning, in which models are automatically derived from labeled training data, is only as good as the quality of that data. This study builds on prior work that investigated to what extent \u201cbest practices\u201d around labeling training data were followed in applied ML publications within a single domain (social media platforms). In this paper, we expand by studying publications that apply supervised ML in a far broader spectrum of disciplines, focusing on human-labeled data. We report to what extent a random sample of ML application papers across disciplines give specific details about whether best practices were followed, while acknowledging that a greater range of application fields necessarily produces greater diversity of labeling and annotation methods. Because much of machine learning research and education only focuses on what is done once a \u201cground truth\u201d or \u201cgold standard\u201d of training data is available, it is especially relevant to discuss issues around the equally important aspect of whether such data is reliable in the first place. This determination becomes increasingly complex when applied to a variety of specialized fields, as labeling can range from a task requiring little-to-no background knowledge to one that must be performed by someone with career expertise.<\/jats:p>","DOI":"10.1162\/qss_a_00144","type":"journal-article","created":{"date-parts":[[2021,6,30]],"date-time":"2021-06-30T14:41:58Z","timestamp":1625064118000},"page":"795-827","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":80,"title":["\u201cGarbage in, garbage out\u201d revisited: What do machine learning application papers report about human-labeled training data?"],"prefix":"10.1162","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7215-0532","authenticated-orcid":true,"given":"R. Stuart","family":"Geiger","sequence":"first","affiliation":[{"name":"University of California, San Diego"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2471-772X","authenticated-orcid":true,"given":"Dominique","family":"Cope","sequence":"additional","affiliation":[{"name":"University of California, Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9952-4987","authenticated-orcid":true,"given":"Jamie","family":"Ip","sequence":"additional","affiliation":[{"name":"University of California, Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0362-8947","authenticated-orcid":true,"given":"Marsha","family":"Lotosh","sequence":"additional","affiliation":[{"name":"Webster Pacific"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7029-2008","authenticated-orcid":true,"given":"Aayush","family":"Shah","sequence":"additional","affiliation":[{"name":"University of California, Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1017-8908","authenticated-orcid":true,"given":"Jenny","family":"Weng","sequence":"additional","affiliation":[{"name":"University of California, Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4563-5192","authenticated-orcid":true,"given":"Rebekah","family":"Tang","sequence":"additional","affiliation":[{"name":"University of California, San Diego"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,11,5]]},"reference":[{"key":"2021110817023839000_bib1","doi-asserted-by":"publisher","first-page":"e29319","DOI":"10.7554\/eLife.29319","article-title":"Motivating participation in open science by examining researcher incentives","volume":"6","author":"Ali-Khan","year":"2017","journal-title":"eLife"},{"key":"2021110817023839000_bib2","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1145\/3306618.3314243","article-title":"Uncovering and mitigating algorithmic bias through learned latent structure","volume-title":"Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society, AIES \u201919","author":"Amini","year":"2019"},{"key":"2021110817023839000_bib3","doi-asserted-by":"crossref","DOI":"10.3133\/pp964","volume-title":"A land use and land cover classification system for use with remote sensor data","author":"Anderson","year":"1976"},{"issue":"1","key":"2021110817023839000_bib4","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1162\/qss_a_00019","article-title":"Scopus as a curated, high-quality bibliometric data source for academic research in quantitative science studies","volume":"1","author":"Baas","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"2021110817023839000_bib5","volume-title":"Passages from the life of a philosopher","author":"Babbage","year":"1864"},{"issue":"7","key":"2021110817023839000_bib6","doi-asserted-by":"publisher","first-page":"1049","DOI":"10.1142\/S0218271810017160","article-title":"Data mining and machine learning in astronomy","volume":"19","author":"Ball","year":"2010","journal-title":"International Journal of Modern Physics D"},{"key":"2021110817023839000_bib7","article-title":"Towards traceability in data ecosystems using a bill of materials model","author":"Barclay","year":"2019","journal-title":"arXiv preprint arXiv:1904.04253"},{"key":"2021110817023839000_bib8","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1162\/tacl_a_00041","article-title":"Data statements for NLP: Toward mitigating system bias and enabling better science","volume":"6","author":"Bender","year":"2018","journal-title":"Transactions of the ACL"},{"key":"2021110817023839000_bib9","doi-asserted-by":"publisher","first-page":"320","DOI":"10.1007\/978-3-030-11680-4_30","article-title":"Ethical and socially-aware data labels","volume-title":"Annual International Symposium on Information Management and Big Data","author":"Beretta","year":"2018"},{"key":"2021110817023839000_bib10","volume-title":"Calling bullshit: The art of skepticism in a data-driven world","author":"Bergstrom","year":"2020"},{"key":"2021110817023839000_bib11","doi-asserted-by":"publisher","first-page":"1537","DOI":"10.1109\/WACV48630.2021.00158","article-title":"Large image datasets: A pyrrhic win for computer vision?","volume-title":"Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV)","author":"Birhane","year":"2021"},{"key":"2021110817023839000_bib12","doi-asserted-by":"publisher","first-page":"5454","DOI":"10.18653\/v1\/2020.acl-main.485","article-title":"Language (technology) is power: A critical survey of \u201cbias\u201d in NLP","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Blodgett","year":"2020"},{"issue":"4","key":"2021110817023839000_bib13","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1007\/s10579-013-9215-6","article-title":"GATE Teamware: A web-based, collaborative text annotation framework","volume":"47","author":"Bontcheva","year":"2013","journal-title":"Language Resources and Evaluation"},{"issue":"6","key":"2021110817023839000_bib14","doi-asserted-by":"publisher","first-page":"1059","DOI":"10.1002\/asi.22634","article-title":"The conundrum of sharing research data","volume":"63","author":"Borgman","year":"2012","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"2021110817023839000_bib15","volume-title":"Memory practices in the sciences","author":"Bowker","year":"2005"},{"issue":"3","key":"2021110817023839000_bib16","doi-asserted-by":"publisher","first-page":"927","DOI":"10.1162\/qss_a_00054","article-title":"Numbers or no numbers in science studies","volume":"1","author":"Bowker","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"2021110817023839000_bib17","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/6352.001.0001","volume-title":"Sorting things out: Classification and its consequences","author":"Bowker","year":"1999"},{"issue":"1","key":"2021110817023839000_bib18","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1007\/s13244-016-0534-1","article-title":"Error and discrepancy in radiology: Inevitable or avoidable?","volume":"8","author":"Brady","year":"2016","journal-title":"Insights into Imaging"},{"issue":"2","key":"2021110817023839000_bib19","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1087\/20150211","article-title":"Beyond authorship: Attribution, contribution, collaboration, and credit","volume":"28","author":"Brand","year":"2015","journal-title":"Learned Publishing"},{"key":"2021110817023839000_bib20","first-page":"77","article-title":"Gender shades: Intersectional accuracy disparities in commercial gender classification","volume-title":"ACM Conference on Fairness, Accountability and Transparency","author":"Buolamwini","year":"2018"},{"key":"2021110817023839000_bib21","first-page":"3995","article-title":"Optimized pre-processing for discrimination prevention","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS\u201917","author":"Calmon","year":"2017"},{"issue":"3","key":"2021110817023839000_bib22","doi-asserted-by":"publisher","first-page":"1017","DOI":"10.1162\/qss_a_00055","article-title":"Beyond networks: Aligning qualitative and computational science studies","volume":"1","author":"Cambrosio","year":"2020","journal-title":"Quantitative Science Studies"},{"issue":"9","key":"2021110817023839000_bib23","doi-asserted-by":"publisher","first-page":"1747","DOI":"10.1002\/asi.23358","article-title":"Data journals: A survey","volume":"66","author":"Candela","year":"2015","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"2021110817023839000_bib24","doi-asserted-by":"publisher","first-page":"2334","DOI":"10.1145\/3025453.3026044","article-title":"Revolt: Collaborative crowdsourcing for labeling machine learning datasets","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI \u201917","author":"Chang","year":"2017"},{"key":"2021110817023839000_bib25","doi-asserted-by":"publisher","first-page":"S2","DOI":"10.1186\/1471-2105-12-S15-S2","article-title":"The data paper: A mechanism to incentivize data publishing in biodiversity science","volume":"12","author":"Chavan","year":"2011","journal-title":"BMC Bioinformatics"},{"issue":"5","key":"2021110817023839000_bib26","doi-asserted-by":"publisher","first-page":"418","DOI":"10.1525\/bio.2009.59.5.9","article-title":"Motivating online publication of data","volume":"59","author":"Costello","year":"2009","journal-title":"BioScience"},{"key":"2021110817023839000_bib27","article-title":"Excavating AI: The politics of training sets for machine learning","author":"Crawford","year":"2019"},{"key":"2021110817023839000_bib28","article-title":"Amazon scraps secret AI recruiting tool that showed bias against women","author":"Dastin","year":"2018","journal-title":"Reuters"},{"issue":"12","key":"2021110817023839000_bib29","doi-asserted-by":"publisher","first-page":"2020","DOI":"10.1093\/jamia\/ocaa094","article-title":"Latent bias and the implementation of artificial intelligence in medicine","volume":"27","author":"DeCamp","year":"2020","journal-title":"Journal of the American Medical Informatics Association"},{"key":"2021110817023839000_bib30","first-page":"837","article-title":"The Automatic Content Extraction (ACE) Program: Tasks, data, and evaluation","volume-title":"Proceedings of the 2004 4th International Conference on Language Resources and Evaluation","author":"Doddington","year":"2004"},{"issue":"293","key":"2021110817023839000_bib31","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1080\/01621459.1961.10482090","article-title":"Multiple comparisons among means","volume":"56","author":"Dunn","year":"1961","journal-title":"Journal of the American Statistical Association"},{"key":"2021110817023839000_bib32","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/978-3-319-00026-8_2","article-title":"Open science: One term, five schools of thought","volume-title":"Opening science: The evolving guide on how the internet is changing research, collaboration and scholarly publishing","author":"Fecher","year":"2014"},{"issue":"5","key":"2021110817023839000_bib33","doi-asserted-by":"publisher","first-page":"e96407","DOI":"10.1371\/journal.pone.0096407","article-title":"Blinded by PRISMA: Are systematic reviewers focusing on PRISMA and ignoring other guidelines?","volume":"9","author":"Fleming","year":"2014","journal-title":"PLOS ONE"},{"issue":"2","key":"2021110817023839000_bib34","doi-asserted-by":"publisher","first-page":"e1349","DOI":"10.1002\/widm.1349","article-title":"Surveying the reach and maturity of machine learning and artificial intelligence in astronomy","volume":"10","author":"Fluke","year":"2020","journal-title":"WIREs Data Mining and Knowledge Discovery"},{"key":"2021110817023839000_bib35","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1145\/3287560.3287589","article-title":"A comparative study of fairness-enhancing interventions in machine learning","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* \u201919","author":"Friedler","year":"2019"},{"key":"2021110817023839000_bib36","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The elements of statistical learning: Data mining, inference, and prediction","author":"Friedman","year":"2009","edition":"2nd ed"},{"key":"2021110817023839000_bib37","article-title":"Datasheets for datasets","author":"Gebru","year":"2018","journal-title":"arXiv preprint arXiv:1803.09010"},{"key":"2021110817023839000_bib38","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1145\/3351095.3372862","article-title":"Garbage in, garbage out? Do machine learning application papers in social computing report where human-labeled training data comes from?","volume-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency","author":"Geiger","year":"2020"},{"issue":"3","key":"2021110817023839000_bib39","doi-asserted-by":"publisher","first-page":"e621","DOI":"10.1097\/PR9.0000000000000621","article-title":"Checklist for the preparation and review of pain clinical trial publications: A pain-specific supplement to CONSORT","volume":"4","author":"Gewandter","year":"2019","journal-title":"Pain Reports"},{"key":"2021110817023839000_bib40","doi-asserted-by":"publisher","first-page":"8:1","DOI":"10.1145\/3329486.3329495","article-title":"Automated management of deep learning experiments","volume-title":"Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning, DEEM\u201919","author":"Gharibi","year":"2019"},{"issue":"10","key":"2021110817023839000_bib41","doi-asserted-by":"publisher","first-page":"388","DOI":"10.1002\/2015EA000136","article-title":"Toward the geoscience paper of the future: Best practices for documenting and sharing research from data to software to provenance","volume":"3","author":"Gil","year":"2016","journal-title":"Earth and Space Science"},{"key":"2021110817023839000_bib42","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9302.001.0001","volume-title":"Raw data is an oxymoron","author":"Gitelman","year":"2013"},{"key":"2021110817023839000_bib43","volume-title":"Deep learning","author":"Goodfellow","year":"2016"},{"issue":"4","key":"2021110817023839000_bib44","doi-asserted-by":"publisher","first-page":"e1003542","DOI":"10.1371\/journal.pcbi.1003542","article-title":"Ten simple rules for the care and feeding of scientific data","volume":"10","author":"Goodman","year":"2014","journal-title":"PLOS Computational Biology"},{"issue":"3","key":"2021110817023839000_bib45","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1525\/aa.1994.96.3.02a00100","article-title":"Professional vision","volume":"96","author":"Goodwin","year":"1994","journal-title":"American Anthropologist"},{"key":"2021110817023839000_bib46","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1145\/3351095.3372826","article-title":"Towards a critical race methodology in algorithmic fairness","volume-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* \u201920","author":"Hanna","year":"2020"},{"key":"2021110817023839000_bib47","doi-asserted-by":"publisher","first-page":"3323","DOI":"10.5555\/3157382.3157469","article-title":"Equality of opportunity in supervised learning","volume-title":"Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS\u2019 16","author":"Hardt","year":"2016"},{"key":"2021110817023839000_bib48","article-title":"Increasing trust in AI services through supplier\u2019s declarations of conformity","author":"Hind","year":"2018","journal-title":"arXiv preprint arXiv:1808.07261"},{"key":"2021110817023839000_bib49","article-title":"The dataset nutrition label: A framework to drive higher data quality standards","author":"Holland","year":"2018","journal-title":"arXiv preprint arXiv:1805.03677"},{"issue":"1","key":"2021110817023839000_bib50","first-page":"13","article-title":"Towards a \u2018science\u2019 of corpus annotation: A new methodological challenge for corpus linguistics","volume":"22","author":"Hovy","year":"2010","journal-title":"International Journal of Translation"},{"issue":"3","key":"2021110817023839000_bib51","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/MCSE.2007.55","article-title":"Matplotlib: A 2D graphics environment","volume":"9","author":"Hunter","year":"2007","journal-title":"Computing in Science & Engineering"},{"key":"2021110817023839000_bib52","article-title":"Measurement and fairness","author":"Jacobs","year":"2019","journal-title":"arXiv:1912.05511 [cs]"},{"key":"2021110817023839000_bib53","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-7138-7","volume-title":"An introduction to statistical learning","author":"James","year":"2013"},{"key":"2021110817023839000_bib54","article-title":"SciPy: Open source scientific tools for Python","author":"Jones","year":"2001"},{"issue":"3","key":"2021110817023839000_bib55","doi-asserted-by":"publisher","first-page":"930","DOI":"10.1162\/qss_a_00056","article-title":"Against method: Exploding the boundary between qualitative and quantitative studies of science","volume":"1","author":"Kang","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"2021110817023839000_bib56","doi-asserted-by":"publisher","first-page":"8706","DOI":"10.18653\/v1\/2020.acl-main.769","article-title":"End-to-end bias mitigation by modelling biases in corpora","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Karimi Mahabadi","year":"2020"},{"key":"2021110817023839000_bib57","volume-title":"The practice of reproducible research: Case studies and lessons from the data-intensive sciences","author":"Kitzes","year":"2018"},{"key":"2021110817023839000_bib58","doi-asserted-by":"publisher","first-page":"87","DOI":"10.3233\/978-1-61499-649-1-87","article-title":"Jupyter Notebooks: A publishing format for reproducible computational workfows","volume-title":"Positioning and power in academic publishing: Players, agents and agendas","author":"Kluyver","year":"2016"},{"issue":"1","key":"2021110817023839000_bib59","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1177\/001316447003000105","article-title":"Estimating the reliability, systematic error and random error of interval data","volume":"30","author":"Krippendorff","year":"1970","journal-title":"Educational and Psychological Measurement"},{"key":"2021110817023839000_bib60","doi-asserted-by":"publisher","first-page":"2117","DOI":"10.1145\/2882903.2899409","article-title":"ActiveClean: An interactive data cleaning framework for modern machine learning","volume-title":"Proceedings of the 2016 International Conference on Management of Data, SIGMOD \u201916","author":"Krishnan","year":"2016"},{"issue":"1","key":"2021110817023839000_bib61","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1162\/qss_a_00097","article-title":"Investigating the division of scientific labor using the Contributor Roles Taxonomy (CRediT)","volume":"2","author":"Larivi\u00e8re","year":"2020","journal-title":"Quantitative Science Studies"},{"issue":"1","key":"2021110817023839000_bib62","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.gsf.2015.07.003","article-title":"Machine learning in geosciences and remote sensing","volume":"7","author":"Lary","year":"2016","journal-title":"Geoscience Frontiers"},{"key":"2021110817023839000_bib63","volume-title":"Science in action: How to follow scientists and engineers through society","author":"Latour","year":"1987"},{"issue":"3","key":"2021110817023839000_bib64","doi-asserted-by":"publisher","first-page":"918","DOI":"10.1162\/qss_e_00061","article-title":"Bridging the divide between qualitative and quantitative science studies","volume":"1","author":"Leydesdorff","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"2021110817023839000_bib65","article-title":"ACE (Automatic Content Extraction) English annotation guidelines for entities version 6.6","author":"Linguistic Data Consortium","year":"2008"},{"key":"2021110817023839000_bib66","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1016\/j.isprsjprs.2017.06.001","article-title":"A review of supervised object-based land-cover image classification","volume":"130","author":"Ma","year":"2017","journal-title":"ISPRS Journal of Photogrammetry and Remote Sensing"},{"issue":"CSCW","key":"2021110817023839000_bib67","doi-asserted-by":"publisher","first-page":"72:1","DOI":"10.1145\/3359174","article-title":"Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice","volume":"3","author":"McDonald","year":"2019","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"2021110817023839000_bib68","doi-asserted-by":"publisher","first-page":"51","DOI":"10.25080\/Majora-92bf1922-00a","article-title":"Data Structures for Statistical Computing in Python","volume-title":"Proceedings of the 9th Python in Science Conference","author":"McKinney","year":"2010"},{"key":"2021110817023839000_bib69","article-title":"Teaching integrity in empirical economics: The pedagogy of reproducible science in undergraduate education","volume-title":"Undergraduate research and the academic librarian: Case studies and best practices","author":"Medeiros","year":"2017"},{"key":"2021110817023839000_bib70","article-title":"A survey on bias and fairness in machine learning","author":"Mehrabi","year":"2019"},{"key":"2021110817023839000_bib71","first-page":"66","article-title":"Work with new electronic \u2018brains\u2019 opens field for army math experts","volume":"10","author":"Mellin","year":"1957","journal-title":"The Hammond Times"},{"key":"2021110817023839000_bib72","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1145\/3287560.3287596","article-title":"Model cards for model reporting","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency","author":"Mitchell","year":"2019"},{"key":"2021110817023839000_bib73","volume-title":"Machine learning","author":"Mitchell","year":"1997"},{"issue":"7","key":"2021110817023839000_bib74","doi-asserted-by":"publisher","first-page":"e1000097","DOI":"10.1371\/journal.pmed.1000097","article-title":"Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement","volume":"6","author":"Moher","year":"2009","journal-title":"PLOS Medicine"},{"issue":"1","key":"2021110817023839000_bib75","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1186\/s13063-018-2733-1","article-title":"Reporting randomised trials of social and psychological interventions: The CONSORT-SPI 2018 Extension","volume":"19","author":"Montgomery","year":"2018","journal-title":"Trials"},{"issue":"5","key":"2021110817023839000_bib76","doi-asserted-by":"publisher","first-page":"e0155036","DOI":"10.1371\/journal.pone.0155036","article-title":"Multilingual Twitter sentiment classification: The role of human annotators","volume":"11","author":"Mozeti\u010d","year":"2016","journal-title":"PLOS ONE"},{"key":"2021110817023839000_bib77","article-title":"doccano: Text annotation tool for human","author":"Nakayama","year":"2018"},{"issue":"6464","key":"2021110817023839000_bib78","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"issue":"5","key":"2021110817023839000_bib79","doi-asserted-by":"publisher","first-page":"2703","DOI":"10.1007\/s11135-013-9919-0","article-title":"On the choice of measures of reliability and validity in the content-analysis of texts","volume":"48","author":"Oleinik","year":"2014","journal-title":"Quality & Quantity"},{"issue":"1","key":"2021110817023839000_bib80","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1186\/s13643-017-0663-8","article-title":"Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement and extensions: A scoping review","volume":"6","author":"Page","year":"2017","journal-title":"Systematic Reviews"},{"issue":"6","key":"2021110817023839000_bib81","doi-asserted-by":"publisher","first-page":"663","DOI":"10.1016\/j.ajodo.2015.03.014","article-title":"The CONSORT Statement: Application within and adaptations for orthodontic trials","volume":"147","author":"Pandis","year":"2015","journal-title":"American Journal of Orthodontics and Dentofacial Orthopedics"},{"issue":"3","key":"2021110817023839000_bib82","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1109\/MCSE.2007.53","article-title":"IPython: A system for interactive scientific computing","volume":"9","author":"P\u00e9rez","year":"2007","journal-title":"Computing in Science and Engineering"},{"issue":"2","key":"2021110817023839000_bib83","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1016\/j.cmpb.2014.11.005","article-title":"Marky: A tool supporting annotation consistency in multi-user and iterative document annotation projects","volume":"118","author":"P\u00e9rez-P\u00e9rez","year":"2015","journal-title":"Computer Methods and Programs in Biomedicine"},{"key":"2021110817023839000_bib84","article-title":"SimpleDorff\u2014Calculate Krippendorff\u2019s Alpha on a DataFrame","author":"Perry","year":"2020"},{"issue":"2","key":"2021110817023839000_bib85","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1016\/j.joi.2009.01.003","article-title":"Sentiment analysis: A combined approach","volume":"3","author":"Prabowo","year":"2009","journal-title":"Journal of Informetrics"},{"key":"2021110817023839000_bib86","doi-asserted-by":"publisher","first-page":"113","DOI":"10.25080\/Majora-4af1f417-011","article-title":"Binder 2.0\u2014Reproducible, interactive, sharable environments for science at scale","volume-title":"Proceedings of the 17th Python in Science Conference","author":"Project Jupyter","year":"2018"},{"issue":"4","key":"2021110817023839000_bib87","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1080\/00031305.2016.1141708","article-title":"How robust are multirater interrater reliability indices to changes in frequency distribution?","volume":"70","author":"Quarfoot","year":"2016","journal-title":"The American Statistician"},{"key":"2021110817023839000_bib88","article-title":"ABOUT ML: Annotation and benchmarking on understanding and transparency of machine learning lifecycles","author":"Raji","year":"2019","journal-title":"arXiv:1912.06166 [cs, stat]"},{"key":"2021110817023839000_bib89","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1016\/j.knosys.2015.06.015","article-title":"A survey on opinion mining and sentiment analysis: Tasks, approaches and applications","volume":"89","author":"Ravi","year":"2015","journal-title":"Knowledge-Based Systems"},{"issue":"Feb","key":"2021110817023839000_bib90","doi-asserted-by":"publisher","first-page":"491","DOI":"10.5555\/2188385.2188401","article-title":"Eliminating spammers and ranking annotators for crowdsourced labeling tasks","volume":"13","author":"Raykar","year":"2012","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"2021110817023839000_bib91","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1001\/jama.284.1.89","article-title":"The contributions of authors","volume":"284","author":"Rennie","year":"2000","journal-title":"JAMA"},{"key":"2021110817023839000_bib92","doi-asserted-by":"publisher","DOI":"10.4324\/9780203551691","volume-title":"Analyzing media messages: Using quantitative content analysis in research","author":"Riff","year":"2013"},{"issue":"1","key":"2021110817023839000_bib93","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1145\/3422648.3422657","article-title":"Database repair meets algorithmic fairness","volume":"49","author":"Salimi","year":"2020","journal-title":"ACM SIGMOD Record"},{"issue":"2","key":"2021110817023839000_bib94","doi-asserted-by":"publisher","first-page":"123","DOI":"10.2218\/ijdc.v7i2.235","article-title":"DMP Online and DMPTool: Different strategies towards a shared goal","volume":"7","author":"Sallans","year":"2012","journal-title":"International Journal of Digital Curation"},{"key":"2021110817023839000_bib95","doi-asserted-by":"publisher","first-page":"1668","DOI":"10.18653\/v1\/P19-1163","article-title":"The risk of racial bias in hate speech detection","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Sap","year":"2019"},{"key":"2021110817023839000_bib96","article-title":"Automatically tracking metadata and provenance of machine learning experiments","volume-title":"Machine Learning Systems Workshop at NIPS","author":"Schelter","year":"2017"},{"issue":"12","key":"2021110817023839000_bib97","doi-asserted-by":"publisher","first-page":"1781","DOI":"10.14778\/3229863.3229867","article-title":"Automating large-scale data quality verification","volume":"11","author":"Schelter","year":"2018","journal-title":"Proceedings of the VLDB Endowment"},{"issue":"1","key":"2021110817023839000_bib98","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1097\/00001888-200601000-00010","article-title":"Academic research record-keeping: Best practices for individuals, group leaders, and institutions","volume":"81","author":"Schreier","year":"2006","journal-title":"Academic Medicine: Journal of the Association of American Medical Colleges"},{"issue":"3","key":"2021110817023839000_bib99","doi-asserted-by":"publisher","first-page":"e1000251","DOI":"10.1371\/journal.pmed.1000251","article-title":"CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials","volume":"7","author":"Schulz","year":"2010","journal-title":"PLOS Medicine"},{"key":"2021110817023839000_bib100","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1145\/3287560.3287598","article-title":"Fairness and abstraction in sociotechnical systems","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* \u201919","author":"Selbst","year":"2019"},{"issue":"1","key":"2021110817023839000_bib101","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1038\/nm0102-68","article-title":"Diffuse large B-cell lymphoma outcome prediction by gene-expression profiling and supervised machine learning","volume":"8","author":"Shipp","year":"2002","journal-title":"Nature Medicine"},{"issue":"3","key":"2021110817023839000_bib102","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1145\/3180492","article-title":"Responsible research with crowds: Pay crowdworkers at least minimum wage","volume":"61","author":"Silberman","year":"2018","journal-title":"Communications of the ACM"},{"key":"2021110817023839000_bib103","doi-asserted-by":"publisher","first-page":"1049","DOI":"10.1145\/2567948.2579215","article-title":"Zooniverse: Observing the world\u2019s largest citizen science platform","volume-title":"Proceedings of the 23rd International Conference on World Wide Web, WWW \u201914 Companion","author":"Simpson","year":"2014"},{"key":"2021110817023839000_bib104","doi-asserted-by":"publisher","first-page":"6562","DOI":"10.1109\/ACCESS.2018.2887201","article-title":"Decision provenance: Harnessing data flow for accountable systems","volume":"7","author":"Singh","year":"2019","journal-title":"IEEE Access"},{"issue":"5","key":"2021110817023839000_bib105","doi-asserted-by":"publisher","first-page":"991","DOI":"10.1006\/ijhc.1999.0252","article-title":"Does automation bias decision-making?","volume":"51","author":"Skitka","year":"1999","journal-title":"International Journal of Human-Computer Studies"},{"key":"2021110817023839000_bib106","article-title":"Why isn\u2019t science better? Look at career incentives","author":"Smaldino","year":"2016","journal-title":"The Conversation"},{"key":"2021110817023839000_bib107","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1186\/1756-0500-2-113","article-title":"Data publication: Towards a database of everything","volume":"2","author":"Smith","year":"2009","journal-title":"BMC Research Notes"},{"key":"2021110817023839000_bib108","article-title":"Measuring crowd truth: Disagreement metrics combined with worker behavior filters","volume-title":"CrowdSem 2013 Workshop","author":"Sober\u00f3n","year":"2013"},{"issue":"12","key":"2021110817023839000_bib109","doi-asserted-by":"publisher","first-page":"2544","DOI":"10.1002\/asi.21416","article-title":"Sentiment strength detection in short informal text","volume":"61","author":"Thelwall","year":"2010","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"4","key":"2021110817023839000_bib110","doi-asserted-by":"publisher","first-page":"358","DOI":"10.1037\/h0076640","article-title":"Interrater reliability and agreement of subjective judgments","volume":"22","author":"Tinsley","year":"1975","journal-title":"Journal of Counseling Psychology"},{"issue":"6","key":"2021110817023839000_bib111","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1093\/intqhc\/mzm042","article-title":"Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups","volume":"19","author":"Tong","year":"2007","journal-title":"International Journal for Quality in Health Care"},{"issue":"2","key":"2021110817023839000_bib112","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/MCSE.2011.37","article-title":"The NumPy array: A structure for efficient numerical computation","volume":"13","author":"van der Walt","year":"2011","journal-title":"Computing in Science Engineering"},{"key":"2021110817023839000_bib113","article-title":"Python Library Reference","author":"van Rossum","year":"1995"},{"issue":"11","key":"2021110817023839000_bib114","doi-asserted-by":"publisher","first-page":"e1002689","DOI":"10.1371\/journal.pmed.1002689","article-title":"Machine learning in medicine: Addressing ethical challenges","volume":"15","author":"Vayena","year":"2018","journal-title":"PLOS Medicine"},{"key":"2021110817023839000_bib115","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.592845","article-title":"Seaborn: Statistical data visualization using Matplotlib","author":"Waskom","year":"2018"},{"issue":"1\/2","key":"2021110817023839000_bib116","doi-asserted-by":"publisher","first-page":"28","DOI":"10.2307\/2332510","article-title":"The generalization of Student\u2019s problem when several different population variances are involved","volume":"34","author":"Welch","year":"1947","journal-title":"Biometrika"},{"issue":"6","key":"2021110817023839000_bib117","doi-asserted-by":"publisher","first-page":"e1005510","DOI":"10.1371\/journal.pcbi.1005510","article-title":"Good enough practices in scientific computing","volume":"13","author":"Wilson","year":"2017","journal-title":"PLOS Computational Biology"},{"key":"2021110817023839000_bib118","article-title":"Automated inference on criminality using face images","author":"Wu","year":"2016","journal-title":"arXiv:1611.04135 [cs]"},{"issue":"4","key":"2021110817023839000_bib119","doi-asserted-by":"publisher","first-page":"416","DOI":"10.1038\/nm843","article-title":"Predicting hepatitis B virus\u2013positive metastatic hepatocellular carcinomas using gene expression profiling and supervised machine learning","volume":"9","author":"Ye","year":"2003","journal-title":"Nature Medicine"},{"key":"2021110817023839000_bib120","first-page":"962","article-title":"Fairness constraints: Mechanisms for fair classification","volume-title":"Artificial Intelligence and Statistics","author":"Zafar","year":"2017"},{"key":"2021110817023839000_bib121","article-title":"We\u2019re incentivizing bad science","author":"Zimring","year":"2019","journal-title":"Scientific American"},{"issue":"3","key":"2021110817023839000_bib122","doi-asserted-by":"publisher","first-page":"945","DOI":"10.1162\/qss_a_00065","article-title":"Is \u201cthe time ripe\u201d for quantitative research on misconduct in science?","volume":"1","author":"Zuckerman","year":"2020","journal-title":"Quantitative Science Studies"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/3\/795\/1970760\/qss_a_00144.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/3\/795\/1970760\/qss_a_00144.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,8]],"date-time":"2021-11-08T12:03:18Z","timestamp":1636372998000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/2\/3\/795\/102771\/Garbage-in-garbage-out-revisited-What-do-machine"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":122,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,11,5]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00144","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00144\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00144\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00144\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00144\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00144\/v1\/decision1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}