{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T16:05:55Z","timestamp":1784390755559,"version":"3.55.0"},"reference-count":74,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2023,5,29]],"date-time":"2023-05-29T00:00:00Z","timestamp":1685318400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Information Technology"],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:p> This paper presents a new perspective on the problem of bias in artificial intelligence (AI)-driven decision-making by examining the fundamental difference between AI and human rationality in making sense of data. Current research has focused primarily on software engineers\u2019 bounded rationality and bias in the data fed to algorithms but has neglected the crucial role of algorithmic rationality in producing bias. Using a Weberian distinction between formal and substantive rationality, we inquire why AI-based algorithms lack the ability to display common sense in data interpretation, leading to flawed decisions. We first conduct a rigorous text analysis to uncover and exemplify contextual nuances within the sampled data. We then combine unsupervised and supervised learning, revealing that algorithmic decision-making characterizes and judges data categories mechanically as it operates through the formal rationality of mathematical optimization procedures. Next, using an AI tool, we demonstrate how formal rationality embedded in AI-based algorithms limits its capacity to perform adequately in complex contexts, thus leading to bias and poor decisions. Finally, we delineate the boundary conditions and limitations of leveraging formal rationality to automatize algorithmic decision-making. Our study provides a deeper understanding of the rationality-based causes of AI\u2019s role in bias and poor decisions, even when data is generated in a largely bias-free context. <\/jats:p>","DOI":"10.1177\/02683962231176842","type":"journal-article","created":{"date-parts":[[2023,5,29]],"date-time":"2023-05-29T22:55:50Z","timestamp":1685400950000},"page":"19-40","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":93,"title":["The formal rationality of artificial intelligence-based algorithms and the problem of bias"],"prefix":"10.1177","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7201-4901","authenticated-orcid":false,"given":"Rohit","family":"Nishant","sequence":"first","affiliation":[{"name":"FSA Ulaval, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8197-4322","authenticated-orcid":false,"given":"Dirk","family":"Schneckenberg","sequence":"additional","affiliation":[{"name":"Rennes School of Business, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3826-9403","authenticated-orcid":false,"given":"MN","family":"Ravishankar","sequence":"additional","affiliation":[{"name":"Queen\u2019s University Belfast, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2023,5,29]]},"reference":[{"key":"bibr1-02683962231176842","volume":"50","author":"\u00c5gerfalk P","year":"2021","journal-title":"Communications of the Association for Information Systems"},{"key":"bibr2-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2018.0789"},{"key":"bibr3-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2021.102387"},{"key":"bibr4-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbusres.2022.01.083"},{"key":"bibr5-02683962231176842","doi-asserted-by":"publisher","DOI":"10.17705\/1jais.00664"},{"key":"bibr6-02683962231176842","volume-title":"Machine Learning (ML) for Natural Language Processing (NLP)","author":"Barba P","year":"2020"},{"key":"bibr73-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1007\/s10677-011-9317-4"},{"key":"bibr7-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2016.1216147"},{"key":"bibr8-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1111\/jpim.12656"},{"key":"bibr9-02683962231176842","volume-title":"When Algorithms Give Real Students Imaginary Grades","author":"Broussard","year":"2020"},{"key":"bibr10-02683962231176842","first-page":"803","volume-title":"International Conference on Machine Learning","author":"Brunet ME","year":"2019"},{"key":"bibr11-02683962231176842","volume-title":"The hundred-page machine learning book","volume":"1","author":"Burkov A","year":"2019"},{"key":"bibr12-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1177\/1461444817706684"},{"key":"bibr13-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1109\/TBIOM.2020.3027269"},{"key":"bibr15-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1145\/3391403.3399545"},{"key":"bibr16-02683962231176842","unstructured":"Dastin (2018) Amazon scraps secret AI recruiting tool that showed bias against women. Retrieved from https:\/\/www.reuters.com\/article\/us-amazon-com-jobs-automation-insight\/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G"},{"key":"bibr17-02683962231176842","unstructured":"DeBois (2020) How machine learning is influencing diversity and inclusion. Retrieved from https:\/\/www.informationweek.com\/strategic-cio\/team-building-and-staffing\/how-machine-learning-is-influencing-diversity-and-inclusion\/a\/d-id\/1338459"},{"key":"bibr18-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaa094"},{"key":"bibr19-02683962231176842","first-page":"451473","volume-title":"A Companion to Linguistic Anthropology","author":"Duranti A","year":"2004"},{"key":"bibr20-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-019-0177-4"},{"key":"bibr21-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-015-8334-3_1"},{"key":"bibr22-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0164374"},{"key":"bibr23-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2020.0990"},{"key":"bibr24-02683962231176842","volume-title":"Learning word vectors for 157 languages","author":"Grave E","year":"2018"},{"key":"bibr25-02683962231176842","doi-asserted-by":"publisher","DOI":"10.5465\/amr.2019.0178"},{"key":"bibr26-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1177\/0008125619864925"},{"key":"bibr27-02683962231176842","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1141"},{"key":"bibr28-02683962231176842","volume":"4","author":"Hao K","year":"2019","journal-title":"MIT Technology Review"},{"key":"bibr29-02683962231176842","unstructured":"Heaven (2020) Google\u2019s medical AI was super accurate in a lab. Real life was a different story. Retrieved from https:\/\/www.technologyreview.com\/2020\/04\/27\/1000658\/google-medical-ai-accurate-lab-real-life-clinic-covid-diabetes-retina-disease\/"},{"key":"bibr30-02683962231176842","unstructured":"Hern (2018) Google\u2019s solution to accidental algorithmic racism: ban gorillas. Retrieved from https:\/\/www.theguardian.com\/technology\/2018\/jan\/12\/google-racism-ban-gorilla-black-people"},{"key":"bibr31-02683962231176842","volume-title":"KH Coder 3 reference manual","author":"Higuchi K","year":"2016"},{"key":"bibr32-02683962231176842","volume-title":"Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus","author":"Horton J","year":"2022"},{"key":"bibr33-02683962231176842","unstructured":"Joshi P (2019) A step-by-step NLP guide to learn ELMo for extracting features from text. Retrieved from https:\/\/www.analyticsvidhya.com\/blog\/2019\/03\/learn-to-use-elmo-to-extract-features-from-text\/"},{"key":"bibr34-02683962231176842","unstructured":"Jiang L, Hwang JD, Bhagavatula C, et al. (2021a) Delphi: towards machine ethics and norms. https:\/\/arxiv.org\/abs\/2110.07574"},{"key":"bibr75-02683962231176842","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1383"},{"key":"bibr35-02683962231176842","unstructured":"Jiang L, Hwang JD, Bhagavatula C, et al. (2021b) Delphi: towards machine ethics and norms. Retrieved from https:\/\/medium.com\/ai2-blog\/towards-machine-ethics-and-norms-d64f2bdde6a3"},{"key":"bibr36-02683962231176842","unstructured":"Kantarci (2021) Bias in AI: what it is, types & examples, how and tools to fix it. Retrieved from https:\/\/research.aimultiple.com\/ai-bias\/#amazon%e2%80%99s-bias-recruiting-tool"},{"key":"bibr37-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84628-754-1"},{"key":"bibr74-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1086\/227128"},{"key":"bibr38-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1177\/0011000084123005"},{"key":"bibr39-02683962231176842","first-page":"270","volume":"1","author":"Kok JN","year":"2009","journal-title":"Artificial intelligence"},{"key":"bibr40-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1080\/0960085X.2021.1927212"},{"key":"bibr41-02683962231176842","doi-asserted-by":"publisher","DOI":"10.4159\/9780674259935"},{"key":"bibr42-02683962231176842","volume-title":"Algorithmic Bias Detection and Mitigation: Best Practices and Policies to Reduce Consumer Harms","author":"Lee NT","year":"2019"},{"key":"bibr43-02683962231176842","author":"Li M","year":"2019","journal-title":"Harvard Business Review"},{"key":"bibr44-02683962231176842","doi-asserted-by":"publisher","DOI":"10.5465\/amr.2018.0181"},{"key":"bibr45-02683962231176842","unstructured":"Maas A, Daly RE, Pham PT, et al. (2011) Learning word vectors for sentiment analysis. In: Paper presented at the Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies,Portland Oregon June 19 - 24, 2011."},{"key":"bibr46-02683962231176842","first-page":"2016","volume":"9","author":"Mann G","year":"2016","journal-title":"Harvard Business Review"},{"key":"bibr47-02683962231176842","author":"Marcus G","year":"2020","journal-title":"Technology Review"},{"key":"bibr49-02683962231176842","volume":"25","author":"Manyika J","year":"2019","journal-title":"Harvard Business Review"},{"key":"bibr50-02683962231176842","unstructured":"Mikolov T, Chen K, Corrado G, et al. (2013) Efficient estimation of word representations in vector space. https:\/\/arxiv.org\/abs\/1301.3781"},{"key":"bibr51-02683962231176842","unstructured":"Mitchell TM (1997) Machine learning. Retrieved from http:\/\/www.cs.cmu.edu\/\u223ctom\/mlbook.html"},{"key":"bibr52-02683962231176842","volume":"295","author":"Nayak P","year":"2019","journal-title":"The Keyword"},{"key":"bibr53-02683962231176842","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"bibr54-02683962231176842","first-page":"37","volume-title":"Illuminating social life: Classical and Contemporary Theory Revisited","author":"Ritzer G","year":"1998"},{"key":"bibr55-02683962231176842","unstructured":"Redmore S (2019) Natural language processing in 5 minutes. Retrieved from https:\/\/customerthink.com\/natural-language-processing-in-5-minutes\/"},{"key":"bibr56-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2022.102505"},{"key":"bibr57-02683962231176842","doi-asserted-by":"crossref","unstructured":"Solomon R.1993.Ethics: A short introduction. Dubuque, IA: Brown & Benchmark.","DOI":"10.5840\/ruffinoup19932"},{"key":"bibr76-02683962231176842","doi-asserted-by":"publisher","DOI":"10.5465\/amr.2014.0252"},{"key":"bibr58-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2021.102441"},{"key":"bibr59-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1287\/orsc.2017.1195"},{"key":"bibr60-02683962231176842","unstructured":"Stern (2021) Social-media algorithms rule how we see the world. Good luck trying to stop them. Retrieved from https:\/\/www.wsj.com\/articles\/social-media-algorithms-rule-how-we-see-the-world-good-luck-trying-to-stop-them-11610884800\/"},{"key":"bibr61-02683962231176842","volume-title":"Business data science: Combining machine learning and economics to optimize, automate, and accelerate business decisions","author":"Taddy M","year":"2019"},{"key":"bibr62-02683962231176842","doi-asserted-by":"crossref","unstructured":"Tafjord O, Mishra BD, Clark P (2020) Proofwriter: generating implications, proofs, and abductive statements over natural language. https:\/\/arxiv.org\/abs\/2012.13048","DOI":"10.18653\/v1\/2021.findings-acl.317"},{"key":"bibr63-02683962231176842","unstructured":"Talat Z, Blix H, Valvoda J, et al. (2021) A word on machine ethics: a response to Jiang et al.(2021) https:\/\/arxiv.org\/abs\/2111.04158"},{"key":"bibr64-02683962231176842","unstructured":"Taulli (2021) IBM Watson: why is healthcare ai so tough? Retrieved from https:\/\/www.forbes.com\/sites\/tomtaulli\/2021\/02\/27\/ibm-watson-why-is-healthcare-ai-so-tough\/?sh=4e799bd85375"},{"key":"bibr65-02683962231176842","unstructured":"TensorFlow (2021) An end-to-end open source machine learning platform. Retrieved from https:\/\/www.tensorflow.org\/"},{"key":"bibr66-02683962231176842","doi-asserted-by":"publisher","DOI":"10.4135\/9781526430212.n12"},{"key":"bibr67-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1093\/mind\/LIX.236.433"},{"key":"bibr68-02683962231176842","unstructured":"Vincent (2016) Twitter taught microsoft\u2019s AI chatbot to be a racist asshole in less than a day. Retrieved from https:\/\/www.theverge.com\/2016\/3\/24\/11297050\/tay-microsoft-chatbot-racist"},{"key":"bibr69-02683962231176842","volume-title":"Economy and Society: An Outline of Interpretive Sociology","volume":"1","author":"Weber M","year":"1978"},{"key":"bibr70-02683962231176842","unstructured":"Weiss (2021) IBM reportedly looking to sell its unprofitable watson health. Retrieved from https:\/\/www.enterpriseai.news\/2021\/02\/25\/ibm-reportedly-looking-to-sell-its-unprofitable-watson-health-business\/"},{"key":"bibr71-02683962231176842","unstructured":"Wiggers K (2023) Anthropic\u2019s claude improves on ChatGPT but still suffers from limitations. Retrieved from https:\/\/techcrunch.com\/2023\/01\/09\/anthropics-claude-improves-on-chatgpt-but-still-suffers-from-limitations\/"},{"key":"bibr72-02683962231176842","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23125"}],"container-title":["Journal of Information Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02683962231176842","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/02683962231176842","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02683962231176842","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,1]],"date-time":"2025-03-01T06:56:26Z","timestamp":1740812186000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/02683962231176842"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,29]]},"references-count":74,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["10.1177\/02683962231176842"],"URL":"https:\/\/doi.org\/10.1177\/02683962231176842","relation":{},"ISSN":["0268-3962","1466-4437"],"issn-type":[{"value":"0268-3962","type":"print"},{"value":"1466-4437","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,29]]}}}