{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T17:00:51Z","timestamp":1790787651214,"version":"4.1.0"},"reference-count":38,"publisher":"SAGE Publications","issue":"9","license":[{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["J Int Med Res"],"published-print":{"date-parts":[[2026,9]]},"abstract":"\n Objective<\/jats:title>\n Machine learning is increasingly integrated into safety-critical domains such as medical applications. In this context, regulatory frameworks require the assessment and minimization of risks associated with incorrect model predictions. However, classical evaluation methods often focus on quantifying error frequencies, which do not reflect the heterogeneous impact of different types of errors. To address this deficiency, we elaborate an approach for assessing machine learning\u2013based multiclass classification models that conforms to regulatory requirements in the field of medical devices. Additionally, we aim to demonstrate its applicability in a concrete application scenario as an exemplary reference.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n Herein, we provide weighted balanced accuracy\n \n \n W<\/mml:mi>\n B<\/mml:mi>\n \n A<\/mml:mi>\n n<\/mml:mi>\n <\/mml:msub>\n <\/mml:math>\n <\/jats:inline-formula>\n as a risk-based approach for assessing machine learning models for multiclass classification.\n \n \n W<\/mml:mi>\n B<\/mml:mi>\n \n A<\/mml:mi>\n n<\/mml:mi>\n <\/mml:msub>\n <\/mml:math>\n <\/jats:inline-formula>\n is systematically based on regulatory requirements for medical devices. It incorporates core properties such as the representation of risks as a multiplicative combination of likelihood and severity and the relationship between development and real-world scenarios. We demonstrate the application of\n \n \n W<\/mml:mi>\n B<\/mml:mi>\n \n A<\/mml:mi>\n n<\/mml:mi>\n <\/mml:msub>\n <\/mml:math>\n <\/jats:inline-formula>\n using X-ray based detection of lung diseases as a reference scenario.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Our analysis shows that\n \n \n W<\/mml:mi>\n B<\/mml:mi>\n \n A<\/mml:mi>\n n<\/mml:mi>\n <\/mml:msub>\n <\/mml:math>\n <\/jats:inline-formula>\n achieves a risk-based assessment in the lung disease scenario as a reference.\n \n \n W<\/mml:mi>\n B<\/mml:mi>\n \n A<\/mml:mi>\n n<\/mml:mi>\n <\/mml:msub>\n <\/mml:math>\n <\/jats:inline-formula>\n systematically operationalizes the definition of risk from corresponding regulatory requirements and allows the assessment to be adapted according to risk levels that have to be elaborated in the context of the risk-management process.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusion<\/jats:title>\n Although this study uses only a single use case to demonstrate the practical applicability of our approach, it shows the impact of a risk-based approach when assessing the performance of machine learning multiclass classification models in medical applications. Without such an approach, regulatory requirements cannot be implemented in a fully comprehensive way. Consequently, the clinical impact cannot be adequately assessed when applying the model in real-world scenarios.<\/jats:p>\n <\/jats:sec>","DOI":"10.1177\/03000605261486667","type":"journal-article","created":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T14:13:25Z","timestamp":1790604805000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Normalized risk-based evaluation of machine learning\u2013based classification models: A multiclass approach with applications in medical devices"],"prefix":"10.1177","volume":"54","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-3779-2761","authenticated-orcid":false,"given":"Julius","family":"Wiggerthale","sequence":"first","affiliation":[{"name":"Furtwangen University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7025-942X","authenticated-orcid":false,"given":"Martin","family":"Haimerl","sequence":"additional","affiliation":[{"name":"Furtwangen University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9831-2181","authenticated-orcid":false,"given":"Christoph","family":"Reich","sequence":"additional","affiliation":[{"name":"Furtwangen University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,9,28]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics14193830"},{"key":"e_1_3_3_3_2","first-page":"1","volume-title":"Face recognition and identification using deep learning2023 Third international conference on advances in electrical, computing, communication and sustainable technologies (ICAECT)","author":"Vishwanatha CR","unstructured":"Vishwanatha CR, Asha V, Saju B, et al. Face recognition and identification using deep learning. In: 2023 Third international conference on advances in electrical, computing, communication and sustainable technologies (ICAECT), 2023, pp. 1\u20135."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-19-1223-8"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11596-021-2474-3"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-818438-7.00002-2"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature21056"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(20)30292-2"},{"key":"e_1_3_3_9_2","unstructured":"European Parliament and Council. Regulation (EU) 2017\/745 of the European parliament and of the council of 5 April 2017 on medical devices amending directive 2001\/83\/EC regulation (EC) no 178\/2002 and regulation (EC) no 1223\/2009 and repealing council directives 90\/385\/EEC and 93\/42\/EEC (MDR) 2017."},{"key":"e_1_3_3_10_2","unstructured":"European Parliament and Council. Regulation (EU) 2024\/1689 of the European parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain legislative acts (AI Act) 2024."},{"key":"e_1_3_3_11_2","unstructured":"International Organization for Standardization. ISO 14971:2019 Medical devices-application of risk management to medical devices 2019."},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","DOI":"10.1186\/s12911-025-02909-9","article-title":"Risk-based evaluation of machine learning-classification methods used for medical devices","volume":"25","author":"Haimerl M","year":"2025","unstructured":"Haimerl M, Reich C. Risk-based evaluation of machine learning-classification methods used for medical devices. BMC Med Inform Decis Mak 2025; 25: 126.","journal-title":"BMC Med Inform Decis Mak"},{"key":"e_1_3_3_13_2","unstructured":"International Organization for Standardization. ISO 24971:2020-6 Medical devices-guidance on the application of ISO 14971 2020."},{"key":"e_1_3_3_14_2","unstructured":"BSI\/AAMI. BS\/AAMI 34971:2023 Application of ISO 14971 to machine learning in artificial intelligence. Guide. Standard British Standards Institution\/Association for the Advancement of Medical Instrumentation May 2023."},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-023-10652-8"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aci.2018.08.003"},{"key":"e_1_3_3_17_2","volume-title":"Deep learning","author":"Godfellow I","year":"2016","unstructured":"Godfellow I, Bengio Y, Courville A. Deep learning. The MIT Press, 2016."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2018.11.101"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-022-06768-8"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015369"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-019-0192-5"},{"key":"e_1_3_3_22_2","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez A Garc\u00eda S Galar M et al. Learning from imbalanced data sets. Springer 2018.","DOI":"10.1007\/978-3-319-98074-4"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","unstructured":"Legnaro E Guastavino S Marchetti F. Multiclass threshold-based classification and model evaluation. arXiv:2511.21794 Epub ahead of Print 26 November 2025. DOI: 10.48550\/arXiv.2511.21794.","DOI":"10.48550\/arXiv.2511.21794"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2020.101692"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-75768-7_31"},{"key":"e_1_3_3_26_2","article-title":"Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters","author":"Li M","unstructured":"Li M, Gao Q, Yu T. Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters. BMC Cancer 2023; 23: 799.","journal-title":"BMC Cancer"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6579\/abc960"},{"key":"e_1_3_3_28_2","unstructured":"Kermany D Zhang K Goldbaum M. Labeled Optical Coherence Tomography (OCT) and chest X-ray images for classification 2018."},{"key":"e_1_3_3_29_2","unstructured":"Kiran S Jabeen I. Dataset of tuberculosis chest X-rays images 2024."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3031384"},{"key":"e_1_3_3_31_2","article-title":"Two public chest X-ray datasets for computer-aided screening of pulmonary diseases","volume":"4","author":"Jaeger S","year":"2014","unstructured":"Jaeger S, Candemir S, Antani S, et al. Two public chest X-ray datasets for computer-aided screening of pulmonary diseases. Quant Imaging Med Surg 2014; 4.","journal-title":"Quant Imaging Med Surg"},{"key":"e_1_3_3_32_2","unstructured":"Rehan M. Chest X-ray dataset https:\/\/www.kaggle.com\/datasets\/muhammadrehan00\/chest-xray-dataset (2025 accessed 23 October 2025)."},{"key":"e_1_3_3_33_2","first-page":"800","article-title":"The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies","volume":"18","author":"von Elm E","year":"2007","unstructured":"von Elm E, Altman DG, Egger M, et al. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet 2007; 18: 800\u2013804.","journal-title":"Lancet"},{"key":"e_1_3_3_34_2","unstructured":"National Institute of Allergy Infectious Diseases. TB Portals 2026-02-25 (2025)."},{"key":"e_1_3_3_35_2","unstructured":"Simonyan K Zisserman A. Very deep convolutional networks for large-scale image recognition. CoRR; abs\/1409.1556 2014."},{"key":"e_1_3_3_36_2","first-page":"770","article-title":"Deep residual learning for image recognition","volume":"27","author":"He K","unstructured":"He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern, Las Vegas, NV, USA, 27\u201330 June 2016, pp.770\u2013778.","journal-title":"In: 2016 IEEE Conference on Computer Vision and Pattern"},{"key":"e_1_3_3_37_2","first-page":"5987","volume-title":"Aggregated residual transformations for deep neural networks2017 IEEE conference on computer vision and pattern recognition (CVPR)","author":"Xie S","unstructured":"Xie S, Girshick R, Doll\u00e1r P, et al. Aggregated residual transformations for deep neural networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), 2017, pp.5987\u20135995."},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-023-09664-3"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1183\/09031936.00017813"}],"container-title":["Journal of International Medical Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/03000605261486667","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/03000605261486667","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/03000605261486667","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T16:16:31Z","timestamp":1790784991000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/03000605261486667"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9]]},"references-count":38,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2026,9]]}},"alternative-id":["10.1177\/03000605261486667"],"URL":"https:\/\/doi.org\/10.1177\/03000605261486667","relation":{},"ISSN":["0300-0605","1473-2300"],"issn-type":[{"value":"0300-0605","type":"print"},{"value":"1473-2300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9]]},"article-number":"03000605261486667"}}