{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T19:18:41Z","timestamp":1784143121740,"version":"3.55.0"},"reference-count":52,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,5,5]],"date-time":"2022-05-05T00:00:00Z","timestamp":1651708800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"COVIDA project"},{"name":"Strategic Alliance TU\/E, WUR, UU en UMC Utrecht"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Fairness and bias are crucial concepts in artificial intelligence, yet they are relatively ignored in machine learning applications in clinical psychiatry. We computed fairness metrics and present bias mitigation strategies using a model trained on clinical mental health data. We collected structured data related to the admission, diagnosis, and treatment of patients in the psychiatry department of the University Medical Center Utrecht. We trained a machine learning model to predict future administrations of benzodiazepines on the basis of past data. We found that gender plays an unexpected role in the predictions\u2014this constitutes bias. Using the AI Fairness 360 package, we implemented reweighing and discrimination-aware regularization as bias mitigation strategies, and we explored their implications for model performance. This is the first application of bias exploration and mitigation in a machine learning model trained on real clinical psychiatry data.<\/jats:p>","DOI":"10.3390\/info13050237","type":"journal-article","created":{"date-parts":[[2022,5,5]],"date-time":"2022-05-05T13:10:26Z","timestamp":1651756226000},"page":"237","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Bias Discovery in Machine Learning Models for Mental Health"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7231-2773","authenticated-orcid":false,"given":"Pablo","family":"Mosteiro","sequence":"first","affiliation":[{"name":"Department of Information and Computing Sciences, Utrecht University, 3584 CS Utrecht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jesse","family":"Kuiper","sequence":"additional","affiliation":[{"name":"Department of Information and Computing Sciences, Utrecht University, 3584 CS Utrecht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Judith","family":"Masthoff","sequence":"additional","affiliation":[{"name":"Department of Information and Computing Sciences, Utrecht University, 3584 CS Utrecht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Floortje","family":"Scheepers","sequence":"additional","affiliation":[{"name":"Afdeling Psychiatrie, University Medical Center Utrecht, 3584 CX Utrecht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9237-221X","authenticated-orcid":false,"given":"Marco","family":"Spruit","sequence":"additional","affiliation":[{"name":"Department of Information and Computing Sciences, Utrecht University, 3584 CS Utrecht, The Netherlands"},{"name":"Department of Public Health and Primary Care, Leiden University Medical Center, 2333 ZA Leiden, The Netherlands"},{"name":"Leiden Institute of Advanced Computer Science, Leiden University, 2311 EZ Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"BII.S4706","DOI":"10.4137\/BII.S4706","article-title":"Suicide Note Classification Using Natural Language Processing: A Content Analysis","volume":"3","author":"Pestian","year":"2010","journal-title":"Biomed. Inform. Insights"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e196709","DOI":"10.1001\/jamanetworkopen.2019.6709","article-title":"Machine Learning Approach to Inpatient Violence Risk Assessment Using Routinely Collected Clinical Notes in Electronic Health Records","volume":"2","author":"Menger","year":"2019","journal-title":"JAMA Netw. Open"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/j.jbi.2018.08.007","article-title":"Risk prediction using natural language processing of electronic mental health records in an inpatient forensic psychiatry setting","volume":"86","author":"Le","year":"2018","journal-title":"J. Biomed. Inform."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1016\/j.psychres.2018.07.004","article-title":"A data science approach to predicting patient aggressive events in a psychiatric hospital","volume":"268","author":"Suchting","year":"2018","journal-title":"Psychiatry Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1016\/j.jad.2020.03.081","article-title":"Predicting future suicidal behaviour in young adults, with different machine learning techniques: A population-based longitudinal study","volume":"271","author":"Wijnen","year":"2020","journal-title":"J. Affect. Disord."},{"key":"ref_6","unstructured":"Kalidas, V. (2020). Siamese Fine-Tuning of BERT for Classification of Small and Imbalanced Datasets, Applied to Prediction of Involuntary Admissions in Mental Healthcare. [Master\u2019s Thesis, Eindhoven University of Technology]."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1136\/jech.2003.008466","article-title":"Bias","volume":"58","author":"Llorca","year":"2004","journal-title":"J. Epidemiol. Community Health"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1002\/ijc.30443","article-title":"Use of benzodiazepine and risk of cancer: A meta-analysis of observational studies","volume":"140","author":"Kim","year":"2017","journal-title":"Int. J. Cancer"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1016\/j.psychres.2012.02.008","article-title":"High dose benzodiazepine dependence: Description of 29 patients treated with flumazenil infusion and stabilised with clonazepam","volume":"198","author":"Quaglio","year":"2012","journal-title":"Psychiatry Res."},{"key":"ref_10","unstructured":"Federatie Medisch Specialisten (2021, November 18). Angststoornissen. Available online: https:\/\/richtlijnendatabase.nl\/richtlijn\/angststoornissen\/gegeneraliseerde_angststoornis_gas\/farmacotherapie_bij_gas\/benzodiazepine_gegeneraliseerde_angststoornis.html."},{"key":"ref_11","first-page":"A4900","article-title":"Kiezen voor de juiste benzodiazepine","volume":"156","author":"Vinkers","year":"2012","journal-title":"Ned. Tijdschr. Geneeskd."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1080\/02813430310001734","article-title":"Factors associated with high prescribing of benzodiazepines and minor opiates","volume":"21","author":"Bjorner","year":"2003","journal-title":"Scand. J. Prim. Health Care"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"595","DOI":"10.1016\/j.genhosppsych.2015.06.004","article-title":"Demographic and clinical factors associated with benzodiazepine prescription at discharge from psychiatric inpatient treatment","volume":"37","author":"Peters","year":"2015","journal-title":"Gen. Hosp. Psychiatry"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.drugalcdep.2018.02.011","article-title":"Examining racial\/ethnic differences in patterns of benzodiazepine prescription and misuse","volume":"187","author":"Cook","year":"2018","journal-title":"Drug Alcohol Depend."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1001\/jamapsychiatry.2014.1763","article-title":"Benzodiazepine Use in the United States","volume":"72","author":"Olfson","year":"2015","journal-title":"JAMA Psychiatry"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1433","DOI":"10.1007\/s00127-020-01989-4","article-title":"The influence of prescriber and patient gender on the prescription of benzodiazepines: Evidence for stereotypes and biases?","volume":"56","author":"McIntyre","year":"2021","journal-title":"Soc. Psychiatry Psychiatr. Epidemiol."},{"key":"ref_17","first-page":"1","article-title":"The influence of prescriber and patient gender on the prescription of benzodiazepines: Results from the Florida Medicaid Dataset","volume":"26","author":"Lui","year":"2021","journal-title":"CNS Spectrums"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.psychres.2017.09.059","article-title":"Prescribing practices in Southeastern Europe\u2014Focus on benzodiazepine prescription at discharge from nine university psychiatric hospitals","volume":"258","author":"Maric","year":"2017","journal-title":"Psychiatry Res."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4:1","DOI":"10.1147\/JRD.2019.2942287","article-title":"AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias","volume":"63","author":"Bellamy","year":"2019","journal-title":"IBM J. Res. Dev."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Baer, T. (2019). Understand, Manage, and Prevent Algorithmic Bias, Apress.","DOI":"10.1007\/978-1-4842-4885-0"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1002\/sim.4780130518","article-title":"Selection bias in observational and experimental studies","volume":"13","author":"Ellenberg","year":"1994","journal-title":"Stat. Med."},{"key":"ref_22","first-page":"671","article-title":"Big Data\u2019s Disparate Impact","volume":"104","author":"Barocas","year":"2016","journal-title":"Calif. Law Rev."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1089\/big.2016.0048","article-title":"Conscientious Classification: A Data Scientist\u2019s Guide to Discrimination-Aware Classification","volume":"5","author":"LaGatta","year":"2017","journal-title":"Big Data"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1006\/juec.1993.1014","article-title":"A Model of Redlining","volume":"33","author":"Lang","year":"1993","journal-title":"J. Urban Econ."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1145\/3376898","article-title":"A Snapshot of the Frontiers of Fairness in Machine Learning","volume":"63","author":"Chouldechova","year":"2020","journal-title":"Commun. ACM"},{"key":"ref_26","unstructured":"Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2021, November 18). Fairness through Awareness. Available online: https:\/\/arxiv.org\/abs\/1104.3913."},{"key":"ref_27","unstructured":"Zemel, R., Wu, Y., Swersky, K., Pitassi, T., and Dwork, C. (2013, January 17\u201319). Learning Fair Representations. Proceedings of the 30th International Conference on Machine Learning, PMLR, Atlanta, GA, USA."},{"key":"ref_28","unstructured":"Joseph, M., Kearns, M., Morgenstern, J., and Roth, A. (2021, November 18). Fairness in Learning: Classic and Contextual Bandits. Available online: https:\/\/arxiv.org\/abs\/1605.07139."},{"key":"ref_29","unstructured":"Friedler, S.A., Scheidegger, C., and Venkatasubramanian, S. (2021, November 18). On the (Im)Possibility of Fairness. Available online: https:\/\/arxiv.org\/abs\/1609.07236."},{"key":"ref_30","unstructured":"Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., and Ghani, R. (2021, November 18). Aequitas: A Bias and Fairness Audit Toolkit. Available online: https:\/\/arxiv.org\/abs\/1811.05577."},{"key":"ref_31","unstructured":"Feldman, M., Friedler, S., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. (2021, November 18). Certifying and Removing Disparate Impact. Available online: https:\/\/arxiv.org\/abs\/1412.3756."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10115-011-0463-8","article-title":"Data preprocessing techniques for classification without discrimination","volume":"33","author":"Kamiran","year":"2012","journal-title":"Knowl. Inf. Syst."},{"key":"ref_33","unstructured":"Flach, P.A., De Bie, T., and Cristianini, N. (2012). Fairness-Aware Classifier with Prejudice Remover Regularizer. Machine Learning and Knowledge Discovery in Databases, Springer."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3392866","article-title":"How We\u2019ve Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis","volume":"4","author":"Scheuerman","year":"2020","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Bartoli, A., and Fusiello, A. (2020, January 23\u201328). Investigating Bias and Fairness in Facial Expression Recognition. Proceedings of the Computer Vision\u2014ECCV 2020 Workshops, Glasgow, UK.","DOI":"10.1007\/978-3-030-67070-2"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Yucer, S., Akcay, S., Al-Moubayed, N., and Breckon, T.P. (2020, January 14\u201319). Exploring Racial Bias Within Face Recognition via Per-Subject Adversarially-Enabled Data Augmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Seattle, DC, USA.","DOI":"10.1109\/CVPRW50498.2020.00017"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Liu, H., Dacon, J., Fan, W., Liu, H., Liu, Z., and Tang, J. (2021, November 18). Does Gender Matter? Towards Fairness in Dialogue Systems. Available online: https:\/\/arxiv.org\/abs\/1910.10486.","DOI":"10.18653\/v1\/2020.coling-main.390"},{"key":"ref_38","unstructured":"Kizilcec, R.F., and Lee, H. (2021, November 18). Algorithmic Fairness in Education. Available online: https:\/\/arxiv.org\/abs\/2007.05443."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Genevi\u00e8ve, L.D., Martani, A., Shaw, D., Elger, B.S., and Wangmo, T. (2020). Structural racism in precision medicine: Leaving no one behind. BMC Med. Ethics, 21.","DOI":"10.1186\/s12910-020-0457-8"},{"key":"ref_40","unstructured":"Tripathi, S., Fritz, B.A., Abdelhack, M., Avidan, M.S., Chen, Y., and King, C.R. (2021, November 18). (Un)Fairness in Post-Operative Complication Prediction Models. Available online: https:\/\/arxiv.org\/abs\/2011.02036."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Singh, H., Mhasawade, V., and Chunara, R. (2021). Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database. medRxiv.","DOI":"10.1101\/2021.07.14.21260493"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Amir, S., van de Meent, J.W., and Wallace, B.C. (2021, November 18). On the Impact of Random Seeds on the Fairness of Clinical Classifiers. Available online: https:\/\/arxiv.org\/abs\/2104.06338.","DOI":"10.18653\/v1\/2021.naacl-main.299"},{"key":"ref_43","first-page":"e58","article-title":"Gender differences in prescribing of zolpidem in the Veterans Health Administration","volume":"25","author":"Jasuja","year":"2019","journal-title":"Am. J. Manag. Care"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1080\/07448481.2010.483714","article-title":"A Meta-analysis of Gender Differences in Attitudes Toward Seeking Professional Psychological Help","volume":"59","author":"Nam","year":"2010","journal-title":"J. Am. Coll. Health"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/0165-0327(96)00028-6","article-title":"Racial influence on diagnosis in psychotic mania","volume":"39","author":"Strakowski","year":"1996","journal-title":"J. Affect. Disord."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"e921","DOI":"10.1038\/tp.2015.182","article-title":"Predicting early psychiatric readmission with natural language processing of narrative discharge summaries","volume":"6","author":"Rumshisky","year":"2016","journal-title":"Transl. Psychiatry"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1038\/s41537-021-00154-3","article-title":"Natural language processing methods are sensitive to sub-clinical linguistic differences in schizophrenia spectrum disorders","volume":"7","author":"Tang","year":"2021","journal-title":"NPJ Schizophr."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1016\/j.ins.2021.12.049","article-title":"Explaining smartphone-based acoustic data in bipolar disorder: Semi-supervised fuzzy clustering and relative linguistic summaries","volume":"588","author":"Casalino","year":"2022","journal-title":"Inf. Sci."},{"key":"ref_49","unstructured":"Nederlands Huisartsen Genootschap (2022, March 22). Omrekentabel Benzodiazepine naar Diazepam 2 mg Tabletten. Available online: https:\/\/www.nhg.org\/sites\/default\/files\/content\/nhg_org\/images\/thema\/omrekentabel_benzodiaz._naar_diazepam_2_mg_tab.pdf."},{"key":"ref_50","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"103621","DOI":"10.1016\/j.jbi.2020.103621","article-title":"An empirical characterization of fair machine learning for clinical risk prediction","volume":"113","author":"Pfohl","year":"2021","journal-title":"J. Biomed. Inform."},{"key":"ref_52","unstructured":"Kuiper, J. (2021). Machine-Learning Based Bias Discovery in Medical Data. [Master\u2019s Thesis, Utrecht University]."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/5\/237\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:06:35Z","timestamp":1760137595000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/5\/237"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,5]]},"references-count":52,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["info13050237"],"URL":"https:\/\/doi.org\/10.3390\/info13050237","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,5]]}}}