{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T18:41:07Z","timestamp":1774550467312,"version":"3.50.1"},"posted":{"date-parts":[[2020,11,24]]},"group-title":"Otolaryngology","reference-count":69,"publisher":"openRxiv","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2024,3,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>Introduction<\/jats:title>\n                  <jats:p>Detecting voice disorders from voice recordings could allow for frequent, remote, and low-cost screening before costly clinical visits and a more invasive laryngoscopy examination. Our goals were to detect unilateral vocal fold paralysis (UVFP) from voice recordings using machine learning, to identify which acoustic variables were important for prediction to increase trust, and to determine model performance relative to clinician performance.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Patients with confirmed UVFP through endoscopic examination (N=77) and controls with normal voices matched for age and sex (N=77) were included. Voice samples were elicited by reading the Rainbow Passage and sustaining phonation of the vowel \u201ca\u201d. Four machine learning models of differing complexity were used. SHapley Additive exPlanations (SHAP) was used to identify important features.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The highest median bootstrapped ROC AUC score was 0.87 and beat clinician\u2019s performance (range: 0.74 \u2013 0.81) based on the recordings. Recording durations were different between UVFP recordings and controls due to how that data was originally processed when storing, which we can show can classify both groups. And counterintuitively, many UVFP recordings had higher intensity than controls, when UVFP patients tend to have weaker voices, revealing a dataset-specific bias which we mitigate in an additional analysis.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>We demonstrate that recording biases in audio duration and intensity created dataset-specific differences between patients and controls, which models used to improve classification. Furthermore, clinician\u2019s ratings provide further evidence that patients were over-projecting their voices and being recorded at a higher amplitude signal than controls. Interestingly, after matching audio duration and removing variables associated with intensity in order to mitigate the biases, the models were able to achieve a similar high performance. We provide a set of recommendations to avoid bias when building and evaluating machine learning models for screening in laryngology.<\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2020.11.23.20235945","type":"posted-content","created":{"date-parts":[[2020,11,24]],"date-time":"2020-11-24T21:51:38Z","timestamp":1606254698000},"source":"Crossref","is-referenced-by-count":2,"title":["Identifying bias in models that detect vocal fold paralysis from audio recordings using explainable machine learning and clinician ratings"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8866-8667","authenticated-orcid":false,"given":"Daniel M.","family":"Low","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vishwanatha","family":"Rao","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5373-9181","authenticated-orcid":false,"given":"Gregory","family":"Randolph","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0206-5441","authenticated-orcid":false,"given":"Phillip C.","family":"Song","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5312-6729","authenticated-orcid":false,"given":"Satrajit S.","family":"Ghosh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"54368","reference":[{"key":"2024032204550600000_2020.11.23.20235945v8.1","doi-asserted-by":"crossref","unstructured":"Wroge TJ , \u00d6zkanca Y , Demiroglu C , Si D. Parkinson\u2019s disease diagnosis using machine learning and voice. 2018 IEEE signal [Internet]. 2018.","DOI":"10.1109\/SPMB.2018.8615607"},{"issue":"1","key":"2024032204550600000_2020.11.23.20235945v8.2","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1002\/lio2.354","article-title":"Automated assessment of psychiatric disorders using speech: A systematic review","volume":"5","year":"2020","journal-title":"Laryngoscope Investig Otolaryngol"},{"key":"2024032204550600000_2020.11.23.20235945v8.3","unstructured":"Quatieri TF . Discrete-Time Speech Signal Processing: Principles and Practice. Pearson Education; 2008. 816 p."},{"key":"2024032204550600000_2020.11.23.20235945v8.4","unstructured":"Molnar C . Interpretable Machine Learning. Lulu.com; 2019. 319 p."},{"key":"2024032204550600000_2020.11.23.20235945v8.5","doi-asserted-by":"publisher","DOI":"10.1177\/0194599817751030"},{"key":"2024032204550600000_2020.11.23.20235945v8.6","doi-asserted-by":"publisher","DOI":"10.1159\/000316320"},{"key":"2024032204550600000_2020.11.23.20235945v8.7","doi-asserted-by":"publisher","DOI":"10.1177\/0194599814531733"},{"key":"2024032204550600000_2020.11.23.20235945v8.8","doi-asserted-by":"crossref","unstructured":"Sritharan N , Chase M , Kamani D . The vagus nerve, recurrent laryngeal nerve, and external branch of the superior laryngeal nerve have unique latencies allowing for intraoperative documentation of \u2026. The [Internet]. 2015.","DOI":"10.1002\/lary.24781"},{"key":"2024032204550600000_2020.11.23.20235945v8.9","doi-asserted-by":"publisher","DOI":"10.1016\/j.surg.2005.08.009"},{"key":"2024032204550600000_2020.11.23.20235945v8.10","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvoice.2010.03.006"},{"key":"2024032204550600000_2020.11.23.20235945v8.11","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvoice.2009.12.010"},{"key":"2024032204550600000_2020.11.23.20235945v8.12","doi-asserted-by":"crossref","unstructured":"Little M , Costello D , Harries M . Objective dysphonia quantification in vocal fold paralysis: comparing nonlinear with classical measures. Nature Precedings. 2009 Apr 21;1\u20131.","DOI":"10.1038\/npre.2009.3106.1"},{"key":"2024032204550600000_2020.11.23.20235945v8.13","doi-asserted-by":"publisher","DOI":"10.1097\/01.MLG.0000199743.99527.9F"},{"key":"2024032204550600000_2020.11.23.20235945v8.14","doi-asserted-by":"publisher","DOI":"10.1007\/s00405-002-0542-2"},{"key":"2024032204550600000_2020.11.23.20235945v8.15","doi-asserted-by":"publisher","DOI":"10.1177\/0194599814521381"},{"key":"2024032204550600000_2020.11.23.20235945v8.16","doi-asserted-by":"publisher","DOI":"10.1111\/j.1742-1241.2008.01875.x"},{"key":"2024032204550600000_2020.11.23.20235945v8.17","doi-asserted-by":"publisher","DOI":"10.1177\/000348940211100803"},{"key":"2024032204550600000_2020.11.23.20235945v8.18","doi-asserted-by":"publisher","DOI":"10.3109\/14015439.2012.696138"},{"key":"2024032204550600000_2020.11.23.20235945v8.19","doi-asserted-by":"publisher","DOI":"10.1177\/000348940511401009"},{"issue":"1","key":"2024032204550600000_2020.11.23.20235945v8.20","first-page":"s41746","article-title":"Digital medicine and the curse of dimensionality","volume":"4","year":"2021","journal-title":"NPJ Digital Medicine"},{"issue":"5","key":"2024032204550600000_2020.11.23.20235945v8.21","doi-asserted-by":"crossref","first-page":"1282","DOI":"10.1002\/mds.28604","article-title":"Reproducibility of Voice Analysis with Machine Learning","volume":"36","year":"2021","journal-title":"Mov Disord"},{"issue":"4","key":"2024032204550600000_2020.11.23.20235945v8.22","first-page":"270","article-title":"Novel approach to acoustical voice analysis using artificial neural networks","volume":"1","year":"2000","journal-title":"J Assoc Res Otolaryngol"},{"key":"2024032204550600000_2020.11.23.20235945v8.23","doi-asserted-by":"publisher","DOI":"10.1109\/TBME.2003.820386"},{"key":"2024032204550600000_2020.11.23.20235945v8.24","doi-asserted-by":"publisher","DOI":"10.1159\/000219950"},{"key":"2024032204550600000_2020.11.23.20235945v8.25","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2010.01.004"},{"issue":"3","key":"2024032204550600000_2020.11.23.20235945v8.26","first-page":"382","article-title":"Delfino da Silva J, da Silva Evangelista D, da N\u00f3brega E Ugulino AC, Oliveira Costa Silva P","volume":"31","year":"2017","journal-title":"Accuracy of Acoustic Analysis Measurements in the Evaluation of Patients With Different Laryngeal Diagnoses. J Voice"},{"issue":"3","key":"2024032204550600000_2020.11.23.20235945v8.27","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1002\/lio2.259","article-title":"Decoding phonation with artificial intelligence (DeP AI): Proof of concept","volume":"4","year":"2019","journal-title":"Laryngoscope Investig Otolaryngol"},{"key":"2024032204550600000_2020.11.23.20235945v8.28","doi-asserted-by":"crossref","unstructured":"Dibazar AA , Narayanan S , Berger TW . Feature analysis for automatic detection of pathological speech. In: Proceedings of the Second Joint 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society] [Engineering in Medicine and Biology. 2002. p. 182\u20133 vol.1.","DOI":"10.1109\/IEMBS.2002.1134447"},{"key":"2024032204550600000_2020.11.23.20235945v8.29","doi-asserted-by":"crossref","unstructured":"Seedat N , Aharonson V , Hamzany Y. Automated and interpretable m-health discrimination of vocal cord pathology enabled by machine learning. In: 2020 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE). 2020 . p. 1\u20136.","DOI":"10.1109\/CSDE50874.2020.9411529"},{"issue":"4","key":"2024032204550600000_2020.11.23.20235945v8.30","first-page":"1","article-title":"Deep Learning Approach for Voice Pathology Detection and Classification","volume":"16","year":"2021","journal-title":"IJHISI"},{"issue":"6","key":"2024032204550600000_2020.11.23.20235945v8.31","doi-asserted-by":"crossref","first-page":"e25247","DOI":"10.2196\/25247","article-title":"Deep Learning Application for Vocal Fold Disease Prediction Through Voice Recognition: Preliminary Development Study","volume":"23","year":"2021","journal-title":"J Med Internet Res"},{"key":"2024032204550600000_2020.11.23.20235945v8.32","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1613\/jair.1.13200","article-title":"Explainable Deep Learning: A Field Guide for the Uninitiated","volume":"73","year":"2022","journal-title":". jair"},{"key":"2024032204550600000_2020.11.23.20235945v8.33","unstructured":"Fairbanks G . Voice and Articulation Drillbook. Harper; 1960. 196 p."},{"issue":"2","key":"2024032204550600000_2020.11.23.20235945v8.34","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1109\/TAFFC.2015.2457417","article-title":"The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing","volume":"7","year":"2016","journal-title":"IEEE Transactions on Affective Computing"},{"key":"2024032204550600000_2020.11.23.20235945v8.35","unstructured":"audEERING GmbH. openSMILE (Version 2.3) [Internet]. 2017. Available from: https:\/\/github.com\/naxingyu\/opensmile\/blob\/3a0968e7b36c1b730a4ffd2977031091ee9abf 7f\/config\/gemaps\/eGeMAPSv01a.conf"},{"key":"2024032204550600000_2020.11.23.20235945v8.36","doi-asserted-by":"publisher","DOI":"10.5281\/ZENODO.4170850"},{"issue":"3","key":"2024032204550600000_2020.11.23.20235945v8.37","first-page":"31","article-title":"The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery","volume":"16","year":"2018","journal-title":"Queueing Syst"},{"key":"2024032204550600000_2020.11.23.20235945v8.38","unstructured":"Raschka S . Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning [Internet]. arXiv [cs.LG]. 2018. Available from: http:\/\/arxiv.org\/abs\/1811.12808"},{"key":"2024032204550600000_2020.11.23.20235945v8.39","doi-asserted-by":"crossref","unstructured":"Ojala M , Garriga GC . Permutation Tests for Studying Classifier Performance. In: 2009 Ninth IEEE International Conference on Data Mining. IEEE; 2009. p. 1833\u201363.","DOI":"10.1109\/ICDM.2009.108"},{"key":"2024032204550600000_2020.11.23.20235945v8.40","unstructured":"Lundberg S , Lee SI . A Unified Approach to Interpreting Model Predictions [Internet]. arXiv [cs.AI]. 2017. Available from: http:\/\/arxiv.org\/abs\/1705.07874"},{"issue":"1","key":"2024032204550600000_2020.11.23.20235945v8.41","first-page":"10237","article-title":"Underspecification presents challenges for credibility in modern machine learning","volume":"23","year":"2022","journal-title":"J Mach Learn Res"},{"key":"2024032204550600000_2020.11.23.20235945v8.42","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbt051"},{"key":"2024032204550600000_2020.11.23.20235945v8.43","doi-asserted-by":"crossref","unstructured":"Sz\u00e9kely GJ , Rizzo ML , Bakirov NK . Measuring and testing dependence by correlation of distances. 2007.","DOI":"10.1214\/009053607000000505"},{"key":"2024032204550600000_2020.11.23.20235945v8.44","doi-asserted-by":"publisher","DOI":"10.1044\/jshr.3902.311"},{"key":"2024032204550600000_2020.11.23.20235945v8.45","doi-asserted-by":"publisher","DOI":"10.1044\/2020_AJSLP-20-00001"},{"key":"2024032204550600000_2020.11.23.20235945v8.46","doi-asserted-by":"crossref","unstructured":"G. Degottex , J. Kane , T. Drugman , T. Raitio and S. Scherer . COVAREP\u2014A collaborative voice analysis repository for speech technologies. Proc IEEE Int Conf Acoust Speech Signal Process [Internet]. 2014 [cited 2023 Oct 21].","DOI":"10.1109\/ICASSP.2014.6853739"},{"key":"2024032204550600000_2020.11.23.20235945v8.47","doi-asserted-by":"publisher","DOI":"10.20982\/tqmp.08.1.p023"},{"key":"2024032204550600000_2020.11.23.20235945v8.48","first-page":"1","article-title":"Package \u201cirr.\u201d","volume":"22","year":"2012","journal-title":"Various coefficients of interrater reliability and agreement"},{"key":"2024032204550600000_2020.11.23.20235945v8.49","doi-asserted-by":"publisher","DOI":"10.1037\/\/1040-3590.6.4.284"},{"key":"2024032204550600000_2020.11.23.20235945v8.50","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1708274114"},{"issue":"6","key":"2024032204550600000_2020.11.23.20235945v8.51","first-page":"1","article-title":"A Survey on Bias and Fairness in Machine Learning","volume":"54","year":"2021","journal-title":"ACM Comput Surv"},{"issue":"1","key":"2024032204550600000_2020.11.23.20235945v8.52","first-page":"6","article-title":"The power of outliers (and why researchers should ALWAYS check for them). Practical Assessment","volume":"9","year":"2019","journal-title":"Research, and Evaluation"},{"key":"2024032204550600000_2020.11.23.20235945v8.53","unstructured":"Kapoor S , Cantrell E , Peng K , Pham TH , Bail CA , Gundersen OE , et al. REFORMS: Reporting Standards for Machine Learning Based Science [Internet]. arXiv [cs.LG]. 2023. Available from: http:\/\/arxiv.org\/abs\/2308.07832"},{"key":"2024032204550600000_2020.11.23.20235945v8.54","doi-asserted-by":"publisher","DOI":"10.1080\/01973533.2016.1277529"},{"key":"2024032204550600000_2020.11.23.20235945v8.55","doi-asserted-by":"crossref","unstructured":"Zhou Y , Ribeiro MT , Shah J . ExSum: From Local Explanations to Model Understanding [Internet]. arXiv [cs.CL]. 2022. Available from: http:\/\/arxiv.org\/abs\/2205.00130","DOI":"10.18653\/v1\/2022.naacl-main.392"},{"key":"2024032204550600000_2020.11.23.20235945v8.56","doi-asserted-by":"publisher","DOI":"10.1145\/3631326"},{"key":"2024032204550600000_2020.11.23.20235945v8.57","doi-asserted-by":"crossref","unstructured":"Dock\u00e8s J , Varoquaux G , Poline JB . Preventing dataset shift from breaking machine-learning biomarkers. Gigascience [Internet]. 2021 Sep 28;10(9).","DOI":"10.1093\/gigascience\/giab055"},{"key":"2024032204550600000_2020.11.23.20235945v8.58","doi-asserted-by":"crossref","unstructured":"Ramig LA , Scherer RC , Titze IR , Ringel SP . Acoustic analysis of voices of patients with neurologic disease: rationale and preliminary data. Ann Otol Rhinol Laryngol. 1988 Mar-Apr;97(2 Pt 1):164\u201372.","DOI":"10.1177\/000348948809700214"},{"key":"2024032204550600000_2020.11.23.20235945v8.59","doi-asserted-by":"publisher","DOI":"10.1159\/000052685"},{"key":"2024032204550600000_2020.11.23.20235945v8.60","doi-asserted-by":"publisher","DOI":"10.1016\/j.conb.2019.04.002"},{"key":"2024032204550600000_2020.11.23.20235945v8.61","doi-asserted-by":"publisher","DOI":"10.1016\/S0892-1997(01)00037-6"},{"issue":"4","key":"2024032204550600000_2020.11.23.20235945v8.62","first-page":"268","article-title":"Acoustic analysis in patients with unilateral arytenoid dislocation and unilateral vocal fold paralysis","volume":"30","year":"2016","journal-title":"Lin Chung Er Bi Yan Hou Tou Jing Wai Ke Za Zhi"},{"issue":"7","key":"2024032204550600000_2020.11.23.20235945v8.63","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1001\/jamaoto.2018.0412","article-title":"The Voice and the Larynx in Older Adults: What\u2019s Normal, and Who Decides?","volume":"144","year":"2018","journal-title":"JAMA Otolaryngol Head Neck Surg"},{"key":"2024032204550600000_2020.11.23.20235945v8.64","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvoice.2009.12.009"},{"key":"2024032204550600000_2020.11.23.20235945v8.65","doi-asserted-by":"publisher","DOI":"10.1044\/1058-0360(2010\/09-0012)"},{"key":"2024032204550600000_2020.11.23.20235945v8.66","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvoice.2005.08.007"},{"key":"2024032204550600000_2020.11.23.20235945v8.67","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvoice.2006.05.001"},{"issue":"5","key":"2024032204550600000_2020.11.23.20235945v8.68","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead","volume":"1","year":"2019","journal-title":"Nat Mach Intell"},{"key":"2024032204550600000_2020.11.23.20235945v8.69","doi-asserted-by":"crossref","unstructured":"Williamson JR , Quatieri TF , Helfer BS , Ciccarelli G , Mehta DD . Vocal and Facial Biomarkers of Depression based on Motor Incoordination and Timing. In: Proceedings of the 4th International Workshop on Audio\/Visual Emotion Challenge. New York, NY, USA: Association for Computing Machinery; 2014. p. 65\u201372. (AVEC \u201914).","DOI":"10.1145\/2661806.2661809"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2020.11.23.20235945","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T13:59:32Z","timestamp":1768485572000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2020.11.23.20235945"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,24]]},"references-count":69,"URL":"https:\/\/doi.org\/10.1101\/2020.11.23.20235945","relation":{},"subject":[],"published":{"date-parts":[[2020,11,24]]},"subtype":"preprint"}}