{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T08:36:24Z","timestamp":1778056584071,"version":"3.51.4"},"reference-count":109,"publisher":"MIS Quarterly","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,9,1]]},"abstract":"<jats:p>Organizational decision-makers need to evaluate AI tools in light of increasing claims that such tools outperform human experts. Yet, measuring the quality of knowledge work is challenging, raising the question of how to evaluate AI performance in such contexts. We investigate this question through a field study of a major U.S. hospital, observing how managers evaluated five different machine-learning (ML) based AI tools. Each tool reported high performance according to standard AI accuracy measures, which were based on ground truth labels provided by qualified experts. Trying these tools out in practice, however, revealed that none of them met expectations. Searching for explanations, managers began confronting the high uncertainty of experts\u2019 know-what knowledge captured in ground truth labels used to train and validate ML models. In practice, experts address this uncertainty by drawing on rich know-how practices, which were not incorporated into these ML-based tools. Discovering the disconnect between AI\u2019s know-what and experts\u2019 know-how enabled managers to better understand the risks and benefits of each tool. This study shows dangers of treating ground truth labels used in ML models objectively when the underlying knowledge is uncertain. We outline implications of our study for developing, training, and evaluating AI for knowledge work.<\/jats:p>","DOI":"10.25300\/misq\/2021\/16564","type":"journal-article","created":{"date-parts":[[2021,9,15]],"date-time":"2021-09-15T16:20:58Z","timestamp":1631722858000},"page":"1501-1526","source":"Crossref","is-referenced-by-count":272,"title":["Is AI Ground Truth Really True? The Dangers of Training and Evaluating AI Tools Based on Experts\u2019 Know-What"],"prefix":"10.25300","volume":"45","author":[{"given":"Sarah","family":"Lebovitz","sequence":"first","affiliation":[{"name":"University of Virginia, McIntire School of Commerce, Charlottesville, VA 22904 U.S.A."}]},{"given":"Natalia","family":"Levina","sequence":"additional","affiliation":[{"name":"New York University, Stern School of Business, New York, NY 10003 U.S.A."}]},{"given":"Hila","family":"Lifshitz-Assaf","sequence":"additional","affiliation":[{"name":"New York University, Stern School of Business, New York, NY 10003 U.S.A."}]}],"member":"10933","published-online":{"date-parts":[[2021,9,1]]},"reference":[{"key":"2025082212251235700_b1-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.7208\/chicago\/9780226189666.001.0001","volume-title":"The System of Professions: An Essay on the Division of Expert Labor","author":"Abbott","year":"1988"},{"key":"2025082212251235700_b2-17_16564_si_lebovitzlevina","unstructured":"Angwin, J., Larson, J., Mattu, S., and Kirchner, L.\u20082016. \u201cMachine Bias,\u201d ProPublica (https:\/\/www.propublica.org\/article\/machinebias-risk-assessments-in-criminal-sentencing)."},{"issue":"4","key":"2025082212251235700_b3-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"661","DOI":"10.5465\/amr.2016.0334","article-title":"To Question or Accept? How Status Differences Influence Responses to New Epistemic Technologies in Knowledge Work","volume":"43","author":"Anthony","year":"2018","journal-title":"Academy of Management Review"},{"issue":"6","key":"2025082212251235700_b4-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"954","DOI":"10.1038\/s41591-019-0447-x","article-title":"End-to-End Lung Cancer Screening with Three-Dimensional Deep Learning on Low-Dose Chest Computed Tomography","volume":"25","author":"Ardila","year":"2019","journal-title":"Nature Medicine"},{"issue":"3","key":"2025082212251235700_b5-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1257\/jep.29.3.3","article-title":"Why Are There Still So Many Jobs? The History and Future of Workplace Automation","volume":"29","author":"Autor","year":"2015","journal-title":"The Journal of Economic Perspectives"},{"key":"2025082212251235700_b6-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI","volume":"58","author":"Barredo Arrieta","year":"2020","journal-title":"Information Fusion"},{"issue":"2","key":"2025082212251235700_b7-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1007\/s12599-021-00683-2","article-title":"Expl(AI)n It to Me\u2014Explainable AI and Information Systems Research","volume":"63","author":"Bauer","year":"2021","journal-title":"Business & Information Systems Engineering"},{"issue":"3","key":"2025082212251235700_b8-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"720","DOI":"10.1086\/379527","article-title":"Object Lessons: Workplace Artifacts as Representations of Occupational Jurisdiction","volume":"109","author":"Bechky","year":"2003","journal-title":"American Journal of Sociology"},{"key":"2025082212251235700_b9-17_16564_si_lebovitzlevina","volume-title":"Blood, Powder, and Residue: How Crime Labs Translate Evidence into Proof","author":"Bechky","year":"2021"},{"issue":"1","key":"2025082212251235700_b10-17_16564_si_lebovitzlevina","volume":"6","author":"Bechmann","year":"2019","journal-title":"Unsupervised by Any Other Name: Hidden Layers of Knowledge Production in Artificial Intelligence on Social Media"},{"key":"2025082212251235700_b11-17_16564_si_lebovitzlevina","volume-title":"The Social Construction of Reality: A Treatise in the Sociology of Knowledge","author":"Berger","year":"1966"},{"issue":"5","key":"2025082212251235700_b12-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"S2","DOI":"10.1016\/j.amjmed.2008.01.001","article-title":"Overconfidence as a Cause of Diagnostic Error in Medicine","volume":"121","author":"Berner","year":"2008","journal-title":"The American Journal of Medicine"},{"key":"2025082212251235700_b13-17_16564_si_lebovitzlevina","volume-title":"Sorting Things Out: Classification and Its Consequences","author":"Bowker","year":"2000"},{"issue":"1","key":"2025082212251235700_b14-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1287\/orsc.2.1.40","article-title":"Organizational Learning and Communities-of-Practice: Toward a Unified View of Working, Learning, and Innovation","volume":"2","author":"Brown","year":"1991","journal-title":"Organization Science"},{"issue":"2","key":"2025082212251235700_b15-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1287\/orsc.12.2.198.10116","article-title":"Knowledge and Organization: A Social-Practice Perspective","volume":"12","author":"Brown","year":"2001","journal-title":"Organization Science"},{"issue":"6","key":"2025082212251235700_b16-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"1668","DOI":"10.1148\/rg.2015150023","article-title":"Understanding and Confronting Our Mistakes: The Epidemiology of Error in Radiology and Strategies for Error Reduction","volume":"35","author":"Bruno","year":"2015","journal-title":"RadioGraphics"},{"key":"2025082212251235700_b17-17_16564_si_lebovitzlevina","first-page":"463","article-title":"RADEX\u2013 Towards a Computer-Based Radiology Consultant","volume-title":"Pattern Recognition in Practice","author":"Chandrasekaran","year":"1980"},{"key":"2025082212251235700_b18-17_16564_si_lebovitzlevina","volume-title":"Constructing Grounded Theory","author":"Charmaz","year":"2014"},{"key":"2025082212251235700_b19-17_16564_si_lebovitzlevina","unstructured":"Christin, A.\n          \u20082014. \u201cWhen it Comes to Chasing Clicks, Journalists Say One Thing but Feel Pressure to Do Another,\u201d Nieman Lab. (https:\/\/www.niemanlab.org\/2014\/08\/when-it-comes-to-chasingclicks-journalists-say-one-thing-but-feel-pressure-to-do-another\/, accessed October 18, 2018)."},{"issue":"5","key":"2025082212251235700_b20-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"897","DOI":"10.1007\/s11186-020-09411-3","article-title":"The Ethnographer and the Algorithm: Beyond the Black Box","volume":"49","author":"Christin","year":"2020","journal-title":"Theory and Society"},{"issue":"4","key":"2025082212251235700_b21-17_16564_si_lebovitzlevina","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2019180096","article-title":"Improving Accuracy and Efficiency with Concurrent Use of Artificial Intelligence for Digital Breast Tomosynthesis","volume":"1","author":"Conant","year":"2019","journal-title":"Radiology Artificial Intelligence"},{"issue":"2","key":"2025082212251235700_b22-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1111\/padm.12115","article-title":"Broken \u2018Two-Way Windows\u2019? An Exploration of Professional Hybrids","volume":"93","author":"Croft","year":"2015","journal-title":"Public Administration"},{"issue":"Suppl 1","key":"2025082212251235700_b23-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"S63","DOI":"10.4103\/2230-8210.145076","article-title":"Hand X-Ray in Pediatric Endocrinology: Skeletal Age Assessment and Beyond","volume":"18","author":"De Sanctis","year":"2014","journal-title":"Indian Journal of Endocrinology and Metabolism"},{"key":"2025082212251235700_b24-17_16564_si_lebovitzlevina","first-page":"248","article-title":"ImageNet: A Large-Scale Hierarchical Image Database","author":"Deng","year":"2009"},{"key":"2025082212251235700_b25-17_16564_si_lebovitzlevina","article-title":"When to Trust Robots with Decisions, and When Not To","author":"Dhar","year":"2016","journal-title":"Harvard Business Review"},{"issue":"5","key":"2025082212251235700_b26-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"1467","DOI":"10.1287\/orsc.1110.0700","article-title":"Digital Science and Knowledge Boundaries in Complex Innovation","volume":"22","author":"Dougherty","year":"2012","journal-title":"Organization Science"},{"key":"2025082212251235700_b27-17_16564_si_lebovitzlevina","volume-title":"Mind Over Machine","author":"Dreyfus","year":"2000"},{"issue":"6","key":"2025082212251235700_b28-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1038\/sj.bjc.6604954","article-title":"Inter-Observer Variability in Mammography Screening and Effect of Type and Number of Readers on Screening Outcome","volume":"100","author":"Duijm","year":"2009","journal-title":"British Journal of Cancer"},{"issue":"3","key":"2025082212251235700_b29-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1017\/S0003975609000150","article-title":"A Sociology of Quantification","volume":"49","author":"Espeland","year":"2008","journal-title":"European Journal of Sociology\/Archives Europ\u00e9ennes de Sociologie"},{"issue":"1","key":"2025082212251235700_b30-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.infoandorg.2018.02.005","article-title":"Working and Organizing in the Age of the Learning Algorithm","volume":"28","author":"Faraj","year":"2018","journal-title":"Information and Organization"},{"issue":"6","key":"2025082212251235700_b31-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1177\/0018726704044952","article-title":"The Culture of Objectivity: Quantification, Uncertainty, and the Evaluation of Risk at NASA","volume":"57","author":"Feldman","year":"2004","journal-title":"Human Relations"},{"key":"2025082212251235700_b32-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1257\/pandp.20181021","article-title":"A Method to Link Advances in Artificial Intelligence to Occupational Abilities","volume":"108","author":"Felten","year":"2018","journal-title":"AEA Papers and Proceedings"},{"key":"2025082212251235700_b33-17_16564_si_lebovitzlevina","unstructured":"Fern\u00e1ndez-Lor\u00eda, C., Provost, F., and Han, X.\u20082020. \u201cExplaining Data-Driven Decisions Made by AI Systems: The Counterfactual Approach,\u201d ArXiv:2001.07417 (https:\/\/arxiv.org\/abs\/2001.07417)."},{"issue":"3","key":"2025082212251235700_b34-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1177\/0306312793023003002","article-title":"Engineering Knowledge: The Construction of Knowledge in Artificial Intelligence","volume":"23","author":"Forsythe","year":"1993","journal-title":"Social Studies of Science"},{"key":"2025082212251235700_b35-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1016\/j.techfore.2016.08.019","article-title":"The Future of Employment: How Susceptible Are Jobs to Computerisation?","volume":"114","author":"Frey","year":"2017","journal-title":"Technological Forecasting and Social Change"},{"key":"2025082212251235700_b36-17_16564_si_lebovitzlevina","first-page":"81","article-title":"On the Distinction between Know-How, KnowWhat, and Know-Why","volume":"14","author":"Garud","year":"1997","journal-title":"Advances in Strategic Management"},{"key":"2025082212251235700_b37-17_16564_si_lebovitzlevina","unstructured":"Gershgorn, D.\n          \u20082017. \u201cThe Data That Transformed AI Research\u2014 and Possibly the World\u201d, Quartz. (https:\/\/qz.com\/1034972\/thedata-that-changed-the-direction-of-ai-research-and-possibly-theworld\/)."},{"key":"2025082212251235700_b38-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9302.001.0001","volume-title":"Raw Data\u201d Is an Oxymoron","author":"Gitelman","year":"2013"},{"issue":"1","key":"2025082212251235700_b39-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.1016\/j.infoandorg.2021.100344","article-title":"Crisis as Opportunity, Disruption and Exposure: Exploring Emergent Responses to Crisis through Digital Technology","volume":"31","author":"Gkeredakis","year":"2021","journal-title":"Information and Organization"},{"key":"2025082212251235700_b40-17_16564_si_lebovitzlevina","volume-title":"Discovering Grounded Theory","author":"Glaser","year":"1967"},{"key":"2025082212251235700_b41-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.4135\/9781412983709","volume-title":"Composing Qualitative Research","author":"Golden-Biddle","year":"2007"},{"key":"2025082212251235700_b42-17_16564_si_lebovitzlevina","unstructured":"Grady, D.\n          \u20082019. \u201cA.I. Took a Test to Detect Lung Cancer. It Got an A,\u201d The New York Times, Health, May\u200820 (https:\/\/www.nytimes.com\/2019\/05\/20\/health\/cancer-artificialintelligence-ct-scans.html)."},{"issue":"5","key":"2025082212251235700_b43-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.1145\/3236009","article-title":"A Survey of Methods for Explaining Black Box Models","volume":"51","author":"Guidotti","year":"2018","journal-title":"ACM Computing Surveys"},{"key":"2025082212251235700_b44-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/1881.001.0001","volume-title":"Cognition in the Wild","author":"Hutchins","year":"1995"},{"key":"2025082212251235700_b45-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.4159\/9780674039681","volume-title":"Epistemic Cultures. How the Sciences Make Knowledge","author":"Knorr Cetina","year":"1999"},{"key":"2025082212251235700_b46-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-49733-4_1","article-title":"What If the Screens Went Black? The Coming of Software Agents","volume-title":"Beyond Interpretivism? New Encounters with Technology and Organizations","author":"Knorr Cetina","year":"2016"},{"issue":"3","key":"2025082212251235700_b47-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1287\/orsc.3.3.383","article-title":"Knowledge of the Firm, Combinative Capabilities, and the Replication of Technology","volume":"3","author":"Kogut","year":"1992","journal-title":"Organization Science"},{"key":"2025082212251235700_b48-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1023\/A:1017181826899","article-title":"Glossary of Terms","volume":"30","author":"Kohavi","year":"1998","journal-title":"Machine Learning"},{"key":"2025082212251235700_b49-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1007\/978-3-319-33762-3_7","article-title":"Ground Truth Data, Content, Metrics, and Analysis","volume-title":"Computer Vision Metrics: Survey, Taxonomy, and Analysis","author":"Krig","year":"2016"},{"issue":"3","key":"2025082212251235700_b50-17_16564_si_lebovitzlevina","first-page":"e190058","article-title":"Will Artificial Intelligence Replace Radiologists?","volume":"1","author":"Langlotz","year":"2019","journal-title":"Radiology: Artificial Intelligence"},{"key":"2025082212251235700_b51-17_16564_si_lebovitzlevina","volume-title":"Science in Action: How to Follow Scientists and Engineers through Society","author":"Latour","year":"1987"},{"key":"2025082212251235700_b52-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511609268","volume-title":"Cognition in Practice: Mind, Mathematics and Culture in Everyday Life","author":"Lave","year":"1988"},{"issue":"2","key":"2025082212251235700_b53-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1148\/radiol.2392042127","article-title":"BI-RADS Lexicon for US and Mammography: Interobserver Variability and Positive Predictive Value","volume":"239","author":"Lazarus","year":"2006","journal-title":"Radiology"},{"key":"2025082212251235700_b54-17_16564_si_lebovitzlevina","article-title":"Diagnostic Doubt and Artificial Intelligence: An Inductive Field Study of Radiology Work","author":"Lebovitz","year":"2019"},{"issue":"1","key":"2025082212251235700_b55-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1148\/radiol.2016161174","article-title":"National Performance Benchmarks for Modern Screening Digital Mammography: Update from Breast Cancer Surveillance Consortium","volume":"283","author":"Lehman","year":"2017","journal-title":"Radiology"},{"key":"2025082212251235700_b56-17_16564_si_lebovitzlevina","volume-title":"Wellsprings of Knowledge: Building and Sustaining the Sources of Innovation","author":"Leonard-Barton","year":"1995"},{"issue":"3","key":"2025082212251235700_b57-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"684","DOI":"10.5465\/amj.2017.0712","article-title":"Minimal and Adaptive Coordination: How Hackathons\u2019 Projects Accelerate Innovation Without Killing It","volume":"64","author":"Lifshitz-Assaf","year":"2021","journal-title":"Academy of Management Journal"},{"issue":"2","key":"2025082212251235700_b58-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1111\/padm.12119","article-title":"Hybrid Manager-Professionals\u2019 Identity Work: The Maintenance and Hybridization of Medical Professionalism in Managerial Contexts","volume":"93","author":"McGivern","year":"2015","journal-title":"Public Administration"},{"issue":"3","key":"2025082212251235700_b59-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"701","DOI":"10.1086\/679105","article-title":"Decisions about Knowledge in Medical Practice: The Effect of Temporal Features of a Task","volume":"120","author":"Menchik","year":"2014","journal-title":"American Journal of Sociology"},{"issue":"5","key":"2025082212251235700_b60-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"595","DOI":"10.1177\/1350507618797216","article-title":"Integrating Knowledge in the Face of Epistemic Uncertainty: Dialogically Drawing Distinctions","volume":"49","author":"Mengis","year":"2018","journal-title":"Management Learning"},{"key":"2025082212251235700_b61-17_16564_si_lebovitzlevina","first-page":"735","article-title":"Challenges of Ground Truth Evaluation of Multi-Target Tracking","author":"Milan","year":"2013"},{"key":"2025082212251235700_b62-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1016\/B978-0-08-051055-2.50016-X","article-title":"LEAP: A Learning Apprentice for VLSI Design","volume-title":"Machine Learning","author":"Mitchell","year":"1990"},{"issue":"1","key":"2025082212251235700_b63-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1109\/TASL.2011.2109382","article-title":"Acoustic Modeling Using Deep Belief Networks","volume":"20","author":"Mohamed","year":"2012","journal-title":"IEEE Transactions on Audio, Speech, and Language Processing"},{"issue":"2","key":"2025082212251235700_b64-17_16564_si_lebovitzlevina","first-page":"e200029","article-title":"Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers","volume":"2","author":"Mongan","year":"2020","journal-title":"Radiology: Artificial Intelligence"},{"key":"2025082212251235700_b65-17_16564_si_lebovitzlevina","unstructured":"Moran, G.\n          \u20082018. \u201cThis Artificial Intelligence Won\u2019t Take Your Job, it Will Help You Do it Better,\u201d Fast Company. (https:\/\/www.fastcompany.com\/90253977\/this-artificialintelligence-wont-take-your-job-it-will-help-you-do-it-better)."},{"key":"2025082212251235700_b66-17_16564_si_lebovitzlevina","unstructured":"Mukherjee, S.\n          \u20082017. \u201cA.I. Versus M.D.,\u201d The New Yorker, Annals of Medicine (https:\/\/www.newyorker.com\/magazine\/2017\/04\/03\/ai-versus-md)."},{"issue":"12","key":"2025082212251235700_b67-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"1391","DOI":"10.1177\/0170840609349875","article-title":"Zooming In and Out: Studying Practices by Switching Theoretical Lenses and Trailing Connections","volume":"30","author":"Nicolini","year":"2009","journal-title":"Organization Studies"},{"key":"2025082212251235700_b68-17_16564_si_lebovitzlevina","volume-title":"Practice Theory, Work, and Organization: An Introduction","author":"Nicolini","year":"2012"},{"issue":"3","key":"2025082212251235700_b69-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1287\/orsc.1080.0412","article-title":"Tacit Knowledge and Knowledge Conversion: Controversy and Advancement in Organizational Knowledge Creation Theory","volume":"20","author":"Nonaka","year":"2009","journal-title":"Organization Science"},{"key":"2025082212251235700_b70-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1017\/CBO9780511488672.003","article-title":"The Pharmaceutical Sector","volume-title":"The Business of Healthcare Innovation","author":"Northrup","year":"2005"},{"issue":"3","key":"2025082212251235700_b71-17_16564_si_lebovitzlevina","article-title":"The Rebirth of CAD: How Is Modern AI Different from the CAD We Know?","volume":"1","author":"Oakden-Rayner","year":"2019","journal-title":"Radiology: Artificial Intelligence"},{"issue":"3","key":"2025082212251235700_b72-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1287\/orsc.3.3.398","article-title":"The Duality of Technology: Rethinking the Concept of Technology in Organizations","volume":"3","author":"Orlikowski","year":"1992","journal-title":"Organization Science"},{"issue":"3","key":"2025082212251235700_b73-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1287\/orsc.13.3.249.2776","article-title":"Knowing in Practice: Enacting a Collective Capability in Distributed Organizing","volume":"13","author":"Orlikowski","year":"2002","journal-title":"Organization Science"},{"issue":"3","key":"2025082212251235700_b74-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"868","DOI":"10.1287\/orsc.2013.0877","article-title":"What Happens When Evaluation Goes Online? Exploring Apparatuses of Valuation in the Travel Sector","volume":"25","author":"Orlikowski","year":"2014","journal-title":"Organization Science"},{"issue":"1","key":"2025082212251235700_b75-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1287\/orsc.2020.1377","article-title":"Make Way for the Algorithms: Symbolic Actions and Change in a Regime of Knowing","volume":"32","author":"Pachidi","year":"2021","journal-title":"Organization Science"},{"key":"2025082212251235700_b76-17_16564_si_lebovitzlevina","unstructured":"Parloff, R.\n          \u20082016. \u201cFrom 2016: Why Deep Learning Is Suddenly Changing Your Life,\u201d Fortune, September\u200828 (https:\/\/fortune.com\/longform\/ai-artificial-intelligence-deepmachine-learning\/)."},{"key":"2025082212251235700_b77-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","DOI":"10.4159\/harvard.9780674736061","volume-title":"The Black Box Society: The Secret Algorithms That Control Money and Information","author":"Pasquale","year":"2015","edition":"Reprint"},{"issue":"7","key":"2025082212251235700_b78-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"605","DOI":"10.1016\/0361-3682(93)90045-8","article-title":"Getting Comfortable with the Numbers: Auditing and the Micro-Production of Macro-Order","volume":"18","author":"Pentland","year":"1993","journal-title":"Accounting, Organizations and Society"},{"issue":"1","key":"2025082212251235700_b79-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/0959-8022(95)90011-X","article-title":"Information Systems and Organizational Learning: The Social Epistemology of Organizational Knowledge Systems","volume":"5","author":"Pentland","year":"1995","journal-title":"Accounting, Management and Information Technologies"},{"key":"2025082212251235700_b80-17_16564_si_lebovitzlevina","first-page":"17","article-title":"The Social Construction of Facts and Artifacts: Or How the Sociology of Science and the Sociology of Technology Might Benefit Each Other","volume-title":"The Social Construction of Technological Systems: New Directions in the Sociology and History of Technology","author":"Pinch","year":"1987"},{"key":"2025082212251235700_b81-17_16564_si_lebovitzlevina","volume-title":"Personal Knowledge: Towards a Post-Critical Philosophy","author":"Polanyi","year":"1958"},{"key":"2025082212251235700_b82-17_16564_si_lebovitzlevina","volume-title":"The Tacit Dimension","author":"Polanyi","year":"1966"},{"issue":"3","key":"2025082212251235700_b83-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1023\/A:1007601015854","article-title":"Robust Classification for Imprecise Environments","volume":"42","author":"Provost","year":"2001","journal-title":"Machine Learning"},{"key":"2025082212251235700_b84-17_16564_si_lebovitzlevina","volume-title":"Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking","author":"Provost","year":"2013"},{"key":"2025082212251235700_b85-17_16564_si_lebovitzlevina","article-title":"Sizing the Prize: What\u2019s the Real Value of AI for Your Business and How Can You Capitalise","volume-title":"Report","author":"Rao","year":"2017"},{"issue":"7787","key":"2025082212251235700_b86-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"S54","DOI":"10.1038\/d41586-019-03847-z","article-title":"Rise of Robot Radiologists","volume":"576","author":"Reardon","year":"2019","journal-title":"Nature"},{"issue":"11","key":"2025082212251235700_b87-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"1476","DOI":"10.1016\/j.jacr.2017.07.007","article-title":"Artificial Intelligence: Threat or Boon to Radiologists?","volume":"14","author":"Recht","year":"2017","journal-title":"Journal of the American College of Radiology"},{"issue":"4","key":"2025082212251235700_b88-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"787","DOI":"10.5465\/amr.2018.0291","article-title":"To Shape or Adapt: Knowledge Problems, Epistemologies, and Strategic Postures under Knightian Uncertainty","volume":"45","author":"Rindova","year":"2020","journal-title":"Academy of Management Review"},{"issue":"3","key":"2025082212251235700_b89-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","article-title":"Common Pitfalls and Recommendations for Using Machine Learning to Detect and Prognosticate for COVID-19 Using Chest Radiographs and CT Scans","volume":"3","author":"Roberts","year":"2021","journal-title":"Nature Machine Intelligence"},{"key":"2025082212251235700_b90-17_16564_si_lebovitzlevina","volume-title":"The Concept of Mind","author":"Ryle","year":"1949"},{"key":"2025082212251235700_b91-17_16564_si_lebovitzlevina","volume-title":"The Reflective Practitioner: How Professionals Think in Action","author":"Sch\u00f6n","year":"1983"},{"issue":"1","key":"2025082212251235700_b92-17_16564_si_lebovitzlevina","first-page":"161","article-title":"AI and the Economy","volume":"19","author":"Seamans","year":"2019","journal-title":"Innovation Policy and the Economy"},{"key":"2025082212251235700_b93-17_16564_si_lebovitzlevina","first-page":"614","article-title":"Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers","author":"Sheng","year":"2008"},{"key":"2025082212251235700_b94-17_16564_si_lebovitzlevina","article-title":"The AI Index 2018 Annual Report","volume-title":"AI Index Steering Committee, Human-Centered AI Initiative, Stanford University","author":"Shoham","year":"2018"},{"issue":"1","key":"2025082212251235700_b95-17_16564_si_lebovitzlevina","first-page":"57","article-title":"Making Management Decisions: The Role of Intuition and Emotion","volume":"1","author":"Simon","year":"1987","journal-title":"The Academy of Management Executive (1987-1989)"},{"key":"2025082212251235700_b96-17_16564_si_lebovitzlevina","first-page":"1085","article-title":"Inferring Ground Truth from Subjective Labelling of Venus Images","author":"Smyth","year":"1994","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025082212251235700_b97-17_16564_si_lebovitzlevina","volume-title":"The Ethnographic Interview","author":"Spradley","year":"1979"},{"key":"2025082212251235700_b98-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/B978-1-55860-092-8.50006-X","article-title":"The Structure of Ill-Structured Solutions: Boundary Objects and Heterogeneous Distributed Problem Solving","volume-title":"Distributed Artificial Intelligence","author":"Star","year":"1989"},{"key":"2025082212251235700_b99-17_16564_si_lebovitzlevina","volume-title":"Ecologies of Knowledge: Work and Politics in Science and Technology","author":"Star","year":"1995"},{"key":"2025082212251235700_b100-17_16564_si_lebovitzlevina","volume-title":"Plans and Situated Actions","author":"Suchman","year":"1987"},{"issue":"S2","key":"2025082212251235700_b101-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1002\/smj.4250171105","article-title":"Exploring Internal Stickiness: Impediments to the Transfer of Best Practice within the Firm","volume":"17","author":"Szulanski","year":"1996","journal-title":"Strategic Management Journal"},{"key":"2025082212251235700_b102-17_16564_si_lebovitzlevina","unstructured":"The Economist\n          . 2018. \u201cAI, Radiology and the Future of Work,\u201d The Economist, June\u20088 (https:\/\/www.economist.com\/leaders\/2018\/06\/07\/ai-radiology-and-the-future-of-work)."},{"key":"2025082212251235700_b103-17_16564_si_lebovitzlevina","volume-title":"The Gold Standard: The Challenge of Evidence-Based Medicine","author":"Timmermans","year":"2003"},{"key":"2025082212251235700_b104-17_16564_si_lebovitzlevina","first-page":"17689","article-title":"Managing Data-Driven Development: An Ethnography of Developing Machine Learning for Recruitment","volume-title":"Academy of Management Proceedings","author":"Van Den Broek","year":"2020"},{"key":"2025082212251235700_b105-17_16564_si_lebovitzlevina","volume-title":"Qualitative Studies of Organizations","author":"Van Maanen","year":"1998"},{"key":"2025082212251235700_b106-17_16564_si_lebovitzlevina","volume-title":"The Sources of Innovation","author":"von Hippel","year":"1988"},{"key":"2025082212251235700_b107-17_16564_si_lebovitzlevina","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1007\/978-3-030-04091-8_8","article-title":"Hotspots and Blind Spots","volume-title":"Living with Monsters? Social Implications of Algorithmic Phenomena, Hybrid Agency, and the Performativity of Technology","author":"Waardenburg","year":"2018"},{"key":"2025082212251235700_b108-17_16564_si_lebovitzlevina","unstructured":"Walch, K.\n          \u20082019. \u201cThe Growth of AI Adoption in Law Enforcement,\u201d Forbes, July\u200826 (https:\/\/www.forbes.com\/sites\/cognitiveworld\/2019\/07\/26\/the-growth-of-ai-adoption-in-lawenforcement\/)."},{"key":"2025082212251235700_b109-17_16564_si_lebovitzlevina","unstructured":"Weissmann, J.\n          \u20082018. \u201cAmazon Created a Hiring Tool Using A.I. It Immediately Started Discriminating Against Women,\u201d Slate, October\u200810 (https:\/\/slate.com\/business\/2018\/10\/amazon-artificialintelligence-hiring-discrimination-women.html; accessed July 28, 2019)."}],"container-title":["MIS Quarterly"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/45\/3\/1501\/7168\/17_16564_si_lebovitzlevina.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/45\/3\/1501\/7168\/17_16564_si_lebovitzlevina.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T16:25:59Z","timestamp":1755879959000},"score":1,"resource":{"primary":{"URL":"https:\/\/misq.umn.edu\/misq\/article\/45\/3\/1501\/1887\/Is-AI-Ground-Truth-Really-True-The-Dangers-of"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,1]]},"references-count":109,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,9,1]]},"published-print":{"date-parts":[[2021,9,1]]}},"URL":"https:\/\/doi.org\/10.25300\/misq\/2021\/16564","relation":{},"ISSN":["0276-7783","2162-9730"],"issn-type":[{"value":"0276-7783","type":"print"},{"value":"2162-9730","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,1]]}}}