{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T05:47:55Z","timestamp":1781070475984,"version":"3.54.1"},"reference-count":13,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,4,15]],"date-time":"2024-04-15T00:00:00Z","timestamp":1713139200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,15]],"date-time":"2024-04-15T00:00:00Z","timestamp":1713139200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Cheminform"],"abstract":"<jats:sec>\n                <jats:title>Abstract<\/jats:title>\n                <jats:p>Multiple metrics are used when assessing and validating the performance of quantitative structure\u2013activity relationship (QSAR) models. In the case of binary classification, balanced accuracy is a metric to assess the global performance of such models. In contrast to accuracy, balanced accuracy does not depend on the respective prevalence of the two categories in the test set that is used to validate a QSAR classifier. As such, balanced accuracy is used to overcome the effect of imbalanced test sets on the model\u2019s perceived accuracy. Matthews' correlation coefficient (MCC), an alternative global performance metric, is also known to mitigate the imbalance of the test set. However, in contrast to the balanced accuracy, MCC remains dependent on the respective prevalence of the predicted categories. For simplicity, the rest of this work is based on the positive prevalence. The MCC value may be underestimated at high or extremely low positive prevalence. It contributes to more challenging comparisons between experiments using test sets with different positive prevalences and may lead to incorrect interpretations. The concept of balanced metrics beyond balanced accuracy is, to the best of our knowledge, not yet described in the cheminformatic literature. Therefore, after describing the relevant literature, this manuscript will first formally define a confusion matrix, sensitivity and specificity and then present, with synthetic data, the danger of comparing performance metrics under nonconstant prevalence. Second, it will demonstrate that balanced accuracy is the performance metric accuracy calibrated to a test set with a positive prevalence of 50% (i.e., balanced test set). This concept of balanced accuracy will then be extended to the MCC after showing its dependency on the positive prevalence. Applying the same concept to any other performance metric and widening it to the concept of calibrated metrics will then be briefly discussed. We will show that, like balanced accuracy, any balanced performance metric may be expressed as a function of the well-known values of sensitivity and specificity. Finally, a tale of two MCCs will exemplify the use of this concept of balanced MCC versus MCC with four use cases using synthetic data.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Scientific contribution<\/jats:title>\n                <jats:p>This work provides a formal, unified framework for understanding prevalence dependence in model validation metrics, deriving balanced metric expressions beyond balanced accuracy, and demonstrating their practical utility for common use cases. In contrast to prior literature, it introduces the derived confusion matrix to express metrics as functions of sensitivity, specificity and prevalence without needing additional coefficients. The manuscript extends the concept of balanced metrics to Matthews' correlation coefficient and other widely used performance indicators, enabling robust comparisons under prevalence shifts.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s13321-024-00837-w","type":"journal-article","created":{"date-parts":[[2024,4,15]],"date-time":"2024-04-15T10:02:09Z","timestamp":1713175329000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Mind your prevalence!"],"prefix":"10.1186","volume":"16","author":[{"given":"S\u00e9bastien J. J.","family":"Guesn\u00e9","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thierry","family":"Hanser","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"St\u00e9phane","family":"Werner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel","family":"Boobier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shaylyn","family":"Scott","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,4,15]]},"reference":[{"key":"837_CR1","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1093\/mutage\/gey034","volume":"34","author":"A Cayley","year":"2019","unstructured":"Cayley A, Fowkes A, Williams RV (2019) Important considerations for the validation of QSAR models for in vitro mutagenicity. Mutagenesis 34:25\u201332. https:\/\/doi.org\/10.1093\/mutage\/gey034","journal-title":"Mutagenesis"},{"key":"837_CR2","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1016\/j.yrtph.2015.12.006","volume":"76","author":"C Barber","year":"2016","unstructured":"Barber C, Cayley A, Hanser T et al (2016) Evaluation of a statistics-based Ames mutagenicity QSAR model and interpretation of the results obtained. Regul Toxicol Pharmacol 76:7\u201320. https:\/\/doi.org\/10.1016\/j.yrtph.2015.12.006","journal-title":"Regul Toxicol Pharmacol"},{"key":"837_CR3","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1016\/j.patcog.2019.02.023","volume":"91","author":"A Luque","year":"2019","unstructured":"Luque A, Carrasco A, Mart\u00edn A, de las Heras A (2019) The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recognit 91:216\u2013231. https:\/\/doi.org\/10.1016\/j.patcog.2019.02.023","journal-title":"Pattern Recognit"},{"key":"837_CR4","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/978-3-030-44584-3_36","volume-title":"Advances in intelligent data analysis XVIII","author":"W Siblini","year":"2020","unstructured":"Siblini W, Fr\u00e9ry J, He-Guelton L et al (2020) Master your metrics with calibration. In: Berthold MR, Feelders A, Krempl G (eds) Advances in intelligent data analysis XVIII. Springer International Publishing, Cham, pp 457\u2013469"},{"key":"837_CR5","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1007\/978-3-030-50423-6_6","volume-title":"Computational science\u2014ICCS 2020","author":"J Brabec","year":"2020","unstructured":"Brabec J, Kom\u00e1rek T, Franc V, Machlica L (2020) On model evaluation under non-constant class imbalance. In: Krzhizhanovskaya VV, Z\u00e1vodszky G, Lees MH et al (eds) Computational science\u2014ICCS 2020. Springer International Publishing, Cham, pp 74\u201387"},{"key":"837_CR6","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1038\/bjc.1979.10","volume":"39","author":"JA Cooper II","year":"1979","unstructured":"Cooper JA II, Saracci R, Cole P (1979) Describing the validity of carcinogen screening tests. Br J Cancer 39:87\u201389. https:\/\/doi.org\/10.1038\/bjc.1979.10","journal-title":"Br J Cancer"},{"key":"837_CR7","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1002\/jmri.22466","volume":"33","author":"TF Heston","year":"2011","unstructured":"Heston TF (2011) Standardizing predictive values in diagnostic imaging research. J Magn Reson Imaging 33:505. https:\/\/doi.org\/10.1002\/jmri.22466","journal-title":"J Magn Reson Imaging"},{"key":"837_CR8","doi-asserted-by":"crossref","unstructured":"Jeni LA, Cohn JF, De La Torre F (2013) Facing imbalanced data\u2014recommendations for the use of performance metrics. In: 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE, pp 245\u2013251","DOI":"10.1109\/ACII.2013.47"},{"key":"837_CR9","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1136\/bmj.309.6947.102","volume":"309","author":"DG Altman","year":"1994","unstructured":"Altman DG, Bland JM (1994) Statistics notes: diagnostic tests 2: predictive values. BMJ 309:102\u2013102. https:\/\/doi.org\/10.1136\/bmj.309.6947.102","journal-title":"BMJ"},{"key":"837_CR10","unstructured":"OECD (2017) Guidance Document on the Reporting of Defined Approaches and Individual Information Sources to be Used within Integrated Approaches to Testing and Assessment (IATA) for Skin Sensitisation. OECD"},{"key":"837_CR11","doi-asserted-by":"publisher","DOI":"10.1016\/j.yrtph.2022.105247","volume":"135","author":"A Trejo-Martin","year":"2022","unstructured":"Trejo-Martin A, Bercu JP, Thresher A et al (2022) Use of the bacterial reverse mutation assay to predict carcinogenicity of N-nitrosamines. Regul Toxicol Pharmacol 135:105247. https:\/\/doi.org\/10.1016\/j.yrtph.2022.105247","journal-title":"Regul Toxicol Pharmacol"},{"key":"837_CR12","doi-asserted-by":"publisher","DOI":"10.1016\/j.yrtph.2019.104488","volume":"109","author":"C Landry","year":"2019","unstructured":"Landry C, Kim MT, Kruhlak NL et al (2019) Transitioning to composite bacterial mutagenicity models in ICH M7 (Q)SAR analyses. Regul Toxicol Pharmacol 109:104488. https:\/\/doi.org\/10.1016\/j.yrtph.2019.104488","journal-title":"Regul Toxicol Pharmacol"},{"key":"837_CR13","doi-asserted-by":"publisher","unstructured":"Youden WJ. Index for rating diagnostic tests. Cancer 3:32\u201335. https:\/\/doi.org\/10.1002\/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3","DOI":"10.1002\/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3"}],"container-title":["Journal of Cheminformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-024-00837-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13321-024-00837-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-024-00837-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,15]],"date-time":"2024-04-15T10:17:13Z","timestamp":1713176233000},"score":1,"resource":{"primary":{"URL":"https:\/\/jcheminf.biomedcentral.com\/articles\/10.1186\/s13321-024-00837-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,15]]},"references-count":13,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["837"],"URL":"https:\/\/doi.org\/10.1186\/s13321-024-00837-w","relation":{},"ISSN":["1758-2946"],"issn-type":[{"value":"1758-2946","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,15]]},"assertion":[{"value":"19 December 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 March 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 April 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing financial interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"43"}}