{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T16:21:43Z","timestamp":1778084503142,"version":"3.51.4"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,10,26]],"date-time":"2021-10-26T00:00:00Z","timestamp":1635206400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,10,26]],"date-time":"2021-10-26T00:00:00Z","timestamp":1635206400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100008967","name":"Philipps-Universit\u00e4t Marburg","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100008967","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Machine learning and artificial intelligence have entered biomedical decision-making for diagnostics, prognostics, or therapy recommendations. However, these methods need to be interpreted with care because of the severe consequences for patients. In contrast to human decision-making, computational models typically make a decision also with low confidence. Machine learning with abstention better reflects human decision-making by introducing a reject option for samples with low confidence. The abstention intervals are typically symmetric intervals around the decision boundary. In the current study, we use asymmetric abstention intervals, which we demonstrate to be better suited for biomedical data that is typically highly imbalanced. We evaluate symmetric and asymmetric abstention on three real-world biomedical datasets and show that both approaches can significantly improve classification performance. However, asymmetric abstention rejects as many or fewer samples compared to symmetric abstention and thus, should be used in imbalanced data.<\/jats:p>","DOI":"10.1186\/s12911-021-01655-y","type":"journal-article","created":{"date-parts":[[2021,10,26]],"date-time":"2021-10-26T12:02:54Z","timestamp":1635249774000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Machine learning with asymmetric abstention for biomedical decision-making"],"prefix":"10.1186","volume":"21","author":[{"given":"Mariem","family":"Gandouz","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hajo","family":"Holzmann","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dominik","family":"Heider","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,10,26]]},"reference":[{"key":"1655_CR1","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1016\/j.xphs.2020.09.055","volume":"110","author":"J Alvarsson","year":"2021","unstructured":"Alvarsson J, McShane SA, Norinder U, Spjuth O. Predicting with confidence: using conformal prediction in drug discovery. J Pharm Sci. 2021;110:42\u20139. https:\/\/doi.org\/10.1016\/j.xphs.2020.09.055.","journal-title":"J Pharm Sci"},{"key":"1655_CR2","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3109\/14767050009053454","volume":"5","author":"D Ayres de Campos","year":"2000","unstructured":"Ayres de Campos D, Bernardes J, Garrido A, de s\u00e1 JM, Pereira-leite L. Sisporto 2.0: a program for automated analysis of cardiotocograms. J Matern Fetal Med. 2000;5:311\u20138. https:\/\/doi.org\/10.3109\/14767050009053454.","journal-title":"J Matern Fetal Med"},{"key":"1655_CR3","first-page":"1823","volume":"9","author":"PL Bartlett","year":"2008","unstructured":"Bartlett PL, Wegkamp MH. Classification with a reject option using a hinge loss. J Mach Learn Res (JMLR). 2008;9:1823\u201340.","journal-title":"J Mach Learn Res (JMLR)"},{"issue":"1","key":"1655_CR4","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1016\/j.canlet.2016.05.033","volume":"382","author":"J-E Bibault","year":"2016","unstructured":"Bibault J-E, Giraud P, Burgun A. Big data and machine learning in radiation oncology: state of the art and future prospects. Cancer Lett. 2016;382(1):110\u20137. https:\/\/doi.org\/10.1016\/j.canlet.2016.05.033.","journal-title":"Cancer Lett"},{"key":"1655_CR5","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1016\/j.ijar.2020.01.010","volume":"119","author":"A Campagner","year":"2020","unstructured":"Campagner A, Cabitza F, Ciucci D. The three-way-in and three-way-out framework to treat and exploit ambiguity in data. Int J Approx Reason. 2020;119:292\u2013312. https:\/\/doi.org\/10.1016\/j.ijar.2020.01.010.","journal-title":"Int J Approx Reason"},{"key":"1655_CR6","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1186\/s12859-018-2090-9","volume":"19","author":"P Chen","year":"2018","unstructured":"Chen P, Pan C. Diabetes classification model based on boosting algorithms. BMC Bioinform. 2018;19:109. https:\/\/doi.org\/10.1186\/s12859-018-2090-9.","journal-title":"BMC Bioinform"},{"key":"1655_CR7","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1186\/s12864-019-6413-7","volume":"21","author":"D Chicco","year":"2020","unstructured":"Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over f1 score and accuracy in binary classification evaluation. BMC Genomics. 2020;21:6.","journal-title":"BMC Genomics"},{"key":"1655_CR8","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1109\/TEC.1957.5222035","volume":"EC\u20136","author":"C Chow","year":"1957","unstructured":"Chow C. An optimum character recognition system using decision functions. IRE Trans Electron Comput. 1957;EC\u20136:247\u201354.","journal-title":"IRE Trans Electron Comput."},{"key":"1655_CR9","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1109\/tit.1970.1054406","volume":"16","author":"C Chow","year":"1970","unstructured":"Chow C. On optimum recognition error and reject tradeoff. IEEE Trans Inf Theory. 1970;16:41\u20136. https:\/\/doi.org\/10.1109\/tit.1970.1054406.","journal-title":"IEEE Trans Inf Theory"},{"key":"1655_CR10","doi-asserted-by":"publisher","first-page":"1559","DOI":"10.1038\/s41591-018-0177-5","volume":"24","author":"N Coudray","year":"2018","unstructured":"Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Feny\u00f6 D, Moreira AL, Razavian N, Tsirigos A. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med. 2018;24:1559\u201367. https:\/\/doi.org\/10.1038\/s41591-018-0177-5.","journal-title":"Nat Med"},{"key":"1655_CR11","unstructured":"Dua D, Graff C. UCI machine learning repository (2017). http:\/\/archive.ics.uci.edu\/ml."},{"key":"1655_CR12","doi-asserted-by":"publisher","first-page":"2099","DOI":"10.1016\/s0031-3203(00)00059-5","volume":"33","author":"G Fumera","year":"2000","unstructured":"Fumera G, Roli F, Giacinto G. Reject option with multiple thresholds. Pattern Recogn. 2000;33:2099\u2013101. https:\/\/doi.org\/10.1016\/s0031-3203(00)00059-5.","journal-title":"Pattern Recogn"},{"issue":"7","key":"1655_CR13","first-page":"102803","volume":"24","author":"A-C Hauschild","year":"2021","unstructured":"Hauschild A-C, Eick L, Wienbeck J, Heider D. Fostering reproducibility, reusability, and technology transfer in health informatics. Science. 2021;24(7):102803\u20131.","journal-title":"Science"},{"key":"1655_CR14","doi-asserted-by":"publisher","DOI":"10.1186\/1756-0381-7-14","author":"D Heider","year":"2014","unstructured":"Heider D, Dybowski JN, Wilms C, Hoffmann D. A simple structure-based model for the prediction of HIV-1 co-receptor tropism. BioData Min. 2014. https:\/\/doi.org\/10.1186\/1756-0381-7-14.","journal-title":"BioData Min"},{"issue":"4","key":"1655_CR15","doi-asserted-by":"publisher","first-page":"709","DOI":"10.1002\/cjs.5550340410","volume":"34","author":"R Herbei","year":"2006","unstructured":"Herbei R, Wegkamp MH. Classification with reject option. Can J Stat. 2006;34(4):709\u201321. https:\/\/doi.org\/10.1002\/cjs.5550340410.","journal-title":"Can J Stat"},{"issue":"12","key":"1655_CR16","doi-asserted-by":"publisher","first-page":"2011","DOI":"10.1093\/jamia\/ocaa088","volume":"27","author":"T Hernandez-Boussard","year":"2020","unstructured":"Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum information for medical AI reporting): developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc. 2020;27(12):2011\u20135.","journal-title":"J Am Med Inform Assoc"},{"key":"1655_CR17","doi-asserted-by":"publisher","DOI":"10.1002\/0471722146","volume-title":"Applied logistic regression","author":"DW Hosmer","year":"2000","unstructured":"Hosmer DW, Lemeshow S. Applied logistic regression. New York: Wiley; 2000."},{"issue":"2","key":"1655_CR18","doi-asserted-by":"publisher","first-page":"325","DOI":"10.3390\/diagnostics8020032","volume":"8","author":"JA Landsheer","year":"2018","unstructured":"Landsheer JA. The clinical relevance of methods for handling inconclusive medical test results: quantification of uncertainty in medical decision-making and screening. Diagnostics. 2018;8(2):325. https:\/\/doi.org\/10.3390\/diagnostics8020032.","journal-title":"Diagnostics"},{"key":"1655_CR19","doi-asserted-by":"publisher","first-page":"790","DOI":"10.1038\/nrmicro1477","volume":"4","author":"T Lengauer","year":"2006","unstructured":"Lengauer T, Sing T. Bioinformatics-assisted anti-HIV therapy. Nat Rev Microb. 2006;4:790\u20137. https:\/\/doi.org\/10.1038\/nrmicro1477.","journal-title":"Nat Rev Microb"},{"issue":"6","key":"1655_CR20","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1038\/nrg3920","volume":"16","author":"MW Libbrecht","year":"2015","unstructured":"Libbrecht MW, Noble WS. Machine learning applications in genetics and genomics. Nat Rev Genet. 2015;16(6):321\u201332.","journal-title":"Nat Rev Genet"},{"key":"1655_CR21","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1016\/j.media.2016.06.037","volume":"33","author":"A Madabhushi","year":"2016","unstructured":"Madabhushi A, Lee G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med Image Anal. 2016;33:170\u20135. https:\/\/doi.org\/10.1016\/j.media.2016.06.037.","journal-title":"Med Image Anal"},{"issue":"4","key":"1655_CR22","doi-asserted-by":"publisher","first-page":"1435","DOI":"10.1007\/s10618-021-00751-x","volume":"35","author":"T Mortier","year":"2021","unstructured":"Mortier T, Wydmuch M, Dembczynski K, H\u00fcllermeier E, Waegeman W. Efficient set-valued prediction in multi-class classification. Data Min Knowl Discov. 2021;35(4):1435\u201369. https:\/\/doi.org\/10.1007\/s10618-021-00751-x.","journal-title":"Data Min Knowl Discov"},{"key":"1655_CR23","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1186\/s13040-016-0114-4","volume":"9","author":"U Neumann","year":"2016","unstructured":"Neumann U, Riemenschneider M, Sowa J-P, Baars T, K\u00e4lsch J, Canbay A, Heider D. Compensation of feature selection biases accompanied with improved predictive performance for binary classification by using a novel ensemble feature selection approach. BioData Min. 2016;9:36. https:\/\/doi.org\/10.1186\/s13040-016-0114-4.","journal-title":"BioData Min"},{"key":"1655_CR24","doi-asserted-by":"publisher","unstructured":"Nguyen V-L, Destercke S, Masson M-H, H\u00fclermeier E. Reliable multi-class classification based on pairwise epistemic and aleatoric uncertainty. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence (IJCAI-18); 2018. p. 5089\u201395. https:\/\/doi.org\/10.24963\/ijcai.2018\/706","DOI":"10.24963\/ijcai.2018\/706"},{"key":"1655_CR25","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1089\/sysm.2018.0002","volume":"1","author":"M Riemenschneider","year":"2018","unstructured":"Riemenschneider M, Wienbeck J, Scherag A, Heider D. Data science for molecular diagnostics applications: from academia to clinic to industry. Syst Med. 2018;1:13\u20137. https:\/\/doi.org\/10.1089\/sysm.2018.0002.","journal-title":"Syst Med"},{"key":"1655_CR26","doi-asserted-by":"publisher","first-page":"2458","DOI":"10.1093\/bioinformatics\/bty984","volume":"35","author":"J Schwarz","year":"2019","unstructured":"Schwarz J, Heider D. Guess: projecting machine learning scores to well-calibrated probability estimates for clinical decision making. Bioinformatics. 2019;35:2458\u201365.","journal-title":"Bioinformatics"},{"key":"1655_CR27","doi-asserted-by":"publisher","first-page":"101706","DOI":"10.1016\/j.artmed.2019.101706","volume":"100","author":"S Sp\u00e4nig","year":"2019","unstructured":"Sp\u00e4nig S, Emberger-Klein A, Sowa J-P, Canbay A, Menrad K, Heider D. The virtual doctor: an interactive clinical-decision-support system based on deep learning for non-invasive prediction of diabetes. Artif Intell Med. 2019;100:101706. https:\/\/doi.org\/10.1016\/j.artmed.2019.101706.","journal-title":"Artif Intell Med"},{"key":"1655_CR28","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/s13040-019-0196-x","volume":"12","author":"S Sp\u00e4nig","year":"2019","unstructured":"Sp\u00e4nig S, Heider D. Encodings and models for antimicrobial peptide classification for multi-resistant pathogens. BioData Min. 2019;12:7. https:\/\/doi.org\/10.1186\/s13040-019-0196-x.","journal-title":"BioData Min"},{"issue":"1","key":"1655_CR29","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1093\/bioinformatics\/btr597","volume":"28","author":"DJ Stekhoven","year":"2012","unstructured":"Stekhoven DJ, B\u00fchlmann P. Missforest\u2014nonparametric missing value imputation for mixed-type data. Bioinformatics. 2012;28(1):112\u20138. https:\/\/doi.org\/10.1093\/bioinformatics\/btr597.","journal-title":"Bioinformatics"},{"key":"1655_CR30","doi-asserted-by":"publisher","unstructured":"Tortorella F. An optimal reject rule for binary classifiers. In: Joint IAPR international workshops on statistical techniques in pattern recognition (SPR) and structural and syntactic pattern recognition (SSPR); 2000. p. 611\u201320. https:\/\/doi.org\/10.1007\/3-540-44522-6_63.","DOI":"10.1007\/3-540-44522-6_63"},{"issue":"2","key":"1655_CR31","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1007\/s10044-004-0209-2","volume":"7","author":"F Tortorella","year":"2004","unstructured":"Tortorella F. Reducing the classification cost of support vector classifiers through an roc-based reject rule. Pattern Anal Appl. 2004;7(2):128\u201343. https:\/\/doi.org\/10.1007\/s10044-004-0209-2.","journal-title":"Pattern Anal Appl"},{"issue":"1","key":"1655_CR32","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1214\/aos\/1079120131","volume":"32","author":"AB Tsybakov","year":"2004","unstructured":"Tsybakov AB. Optimal aggregation of classifiers in statistical learning. Ann Stat. 2004;32(1):135\u201366. https:\/\/doi.org\/10.1214\/aos\/1079120131.","journal-title":"Ann Stat"},{"issue":"23","key":"1655_CR33","doi-asserted-by":"publisher","first-page":"9193","DOI":"10.1073\/pnas.87.23.9193","volume":"87","author":"WH Wolberg","year":"1990","unstructured":"Wolberg WH, Mangasarian OL. Multisurface method of pattern separation for medical diagnosis applied to breast cytology. Proc Natl Acad Sci U S A. 1990;87(23):9193\u20136. https:\/\/doi.org\/10.1073\/pnas.87.23.9193.","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"4","key":"1655_CR34","doi-asserted-by":"publisher","first-page":"582","DOI":"10.1038\/s41591-021-01312-x","volume":"27","author":"E Wu","year":"2021","unstructured":"Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021;27(4):582\u20134. https:\/\/doi.org\/10.1038\/s41591-021-01312-x.","journal-title":"Nat Med."},{"key":"1655_CR35","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1007\/s10549-016-4035-1","volume":"161","author":"A Yala","year":"2017","unstructured":"Yala A, Barzilay R, Salama L, Griffin M, Sollender G, Bardia A, Lehman C, Buckley JM, Coopey SB, Polubriaginof F, Garber JE, Smith BL, Gadd MA, Specht MC, Gudewicz TM, Guidi AJ, Taghian A, Hughes KS. Using machine learning to parse breast pathology reports. Breast Cancer Res Treat. 2017;161:203\u201311. https:\/\/doi.org\/10.1007\/s10549-016-4035-1.","journal-title":"Breast Cancer Res Treat"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01655-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-021-01655-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01655-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,26]],"date-time":"2021-10-26T12:05:04Z","timestamp":1635249904000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-021-01655-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,26]]},"references-count":35,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["1655"],"URL":"https:\/\/doi.org\/10.1186\/s12911-021-01655-y","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,26]]},"assertion":[{"value":"11 August 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 October 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 October 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"294"}}