{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:11:10Z","timestamp":1760058670196,"version":"build-2065373602"},"reference-count":24,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2025,4,20]],"date-time":"2025-04-20T00:00:00Z","timestamp":1745107200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The growing interest in machine learning in a critical domain like healthcare emphasizes the need for reliable predictions, as decisions based on these outputs can have significant consequences. This study benchmarks methods for assessing pointwise reliability, focusing on data-driven techniques based on the density principle and the local fit principle. These methods evaluate the reliability of individual predictions by analyzing their similarity to training data and evaluating the performance of the model in local regions. Aiming to establish a standardized comparison, the study introduces a benchmark framework that combines error rate evaluations across reliability intervals with t-distributed Stochastic Neighbor Embedding visualizations to further validate the results. The results demonstrate that methods combining density and local fit principles generally outperform those relying on a single principle, achieving lower error rates for high-reliability predictions. Furthermore, the study identifies challenges such as the adjustment of method parameters and clustering limitations and provides insight into their impact on reliability assessments.<\/jats:p>","DOI":"10.3390\/info16040327","type":"journal-article","created":{"date-parts":[[2025,4,20]],"date-time":"2025-04-20T20:24:16Z","timestamp":1745180656000},"page":"327","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Benchmarking Methods for Pointwise Reliability"],"prefix":"10.3390","volume":"16","author":[{"given":"Cl\u00e1udio","family":"Correia","sequence":"first","affiliation":[{"name":"Coimbra Institute of Engineering, Polytechnic University of Coimbra, Rua da Miseric\u00f3rdia, Lagar dos Corti\u00e7os, S. Martinho do Bispo, 3045-093 Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1021-1173","authenticated-orcid":false,"given":"Sim\u00e3o","family":"Paredes","sequence":"additional","affiliation":[{"name":"Coimbra Institute of Engineering, Polytechnic University of Coimbra, Rua da Miseric\u00f3rdia, Lagar dos Corti\u00e7os, S. Martinho do Bispo, 3045-093 Coimbra, Portugal"},{"name":"CISUC, Center for Informatics and Systems of University of Coimbra, University of Coimbra, P\u00f3lo II, 3030-290 Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3497-7368","authenticated-orcid":false,"given":"Teresa","family":"Rocha","sequence":"additional","affiliation":[{"name":"Coimbra Institute of Engineering, Polytechnic University of Coimbra, Rua da Miseric\u00f3rdia, Lagar dos Corti\u00e7os, S. Martinho do Bispo, 3045-093 Coimbra, Portugal"},{"name":"CISUC, Center for Informatics and Systems of University of Coimbra, University of Coimbra, P\u00f3lo II, 3030-290 Coimbra, Portugal"}]},{"given":"Jorge","family":"Henriques","sequence":"additional","affiliation":[{"name":"CISUC, Center for Informatics and Systems of University of Coimbra, University of Coimbra, P\u00f3lo II, 3030-290 Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9660-2011","authenticated-orcid":false,"given":"Jorge","family":"Bernardino","sequence":"additional","affiliation":[{"name":"Coimbra Institute of Engineering, Polytechnic University of Coimbra, Rua da Miseric\u00f3rdia, Lagar dos Corti\u00e7os, S. Martinho do Bispo, 3045-093 Coimbra, Portugal"},{"name":"CISUC, Center for Informatics and Systems of University of Coimbra, University of Coimbra, P\u00f3lo II, 3030-290 Coimbra, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2025,4,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1109\/RBME.2020.3013489","article-title":"Secure and Robust Machine Learning for Healthcare: A Survey","volume":"14","author":"Qayyum","year":"2021","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"ref_2","first-page":"58","article-title":"Significance of Machine Learning in Healthcare: Features, Pillars and Applications","volume":"3","author":"Javaid","year":"2022","journal-title":"Int. J. Intell. Netw."},{"key":"ref_3","unstructured":"Schulam, P., and Saria, S. (2019, January 16\u201318). Can You Trust This Prediction? Auditing Pointwise Reliability after Learning. Proceedings of the AISTATS 2019\u201422nd International Conference on Artificial Intelligence and Statistics, Okinawa, Japan."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Nicora, G., Rios, M., Abu-Hanna, A., and Bellazzi, R. (2022). Evaluating Pointwise Reliability of Machine Learning Prediction. J. Biomed. Inform., 127.","DOI":"10.1016\/j.jbi.2022.103996"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Jarm, T., \u0160merc, R., and Mahni\u010d-Kalamiza, S. (2024). Pointwise Reliability of Machine Learning Models: Application to Cardiovascular Risk Assessment. 9th European Medical and Biological Engineering Conference, Portoro\u017e, Slovenia, 9\u201313 June 2024, Springer Nature.","DOI":"10.1007\/978-3-031-61628-0"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"108223","DOI":"10.1016\/j.ress.2021.108223","article-title":"Machine Learning-Based Methods in Structural Reliability Analysis: A Review","volume":"219","author":"Enayatollahi","year":"2022","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e2950","DOI":"10.1002\/stc.2950","article-title":"Reliability of Probabilistic Numerical Data for Training Machine Learning Algorithms to Detect Damage in Bridges","volume":"29","author":"Bud","year":"2022","journal-title":"Struct. Control Health Monit."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"van der Waa, J., van Diggelen, J., Neerincx, M., and Raaijmakers, S. (2018). ICM: An Intuitive Model Independent and Accurate Certainty Measure for Machine Learning. 10th International Conference on Agents and Artificial Intelligence, Funchal, Portugal, 16\u201318 January 2018, SCITEPRESS\u2014Science and Technology Publications.","DOI":"10.5220\/0006542603140321"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"102493","DOI":"10.1016\/j.ijhcs.2020.102493","article-title":"Interpretable Confidence Measures for Decision Support Systems","volume":"144","author":"Schoonderwoerd","year":"2020","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1109\/TSSC.1970.300339","article-title":"The Nearest Neighbor Classification Rule with a Reject Option","volume":"6","author":"Hellman","year":"1970","journal-title":"IEEE Trans. Syst. Sci. Cybern."},{"key":"ref_11","unstructured":"Ester, M., Kriegel, H.P., Sander, J., and Xu, X. (1996, January 2\u20134). A Density-Based Algorithm for Discovering Clusters A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. Proceedings of the Proceedings\u20142nd International Conference on Knowledge Discovery and Data Mining, KDD 1996, Portland, OR, USA."},{"key":"ref_12","first-page":"925","article-title":"A Reliable Machine Learning Approach Applied to Single-Cell Classification in Acute Myeloid Leukemia","volume":"2020","author":"Nicora","year":"2021","journal-title":"AMIA Annu. Symp. Proc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1038\/s41524-019-0248-2","article-title":"Reliable and Explainable Machine-Learning Methods for Accelerated Material Discovery","volume":"5","author":"Kailkhura","year":"2019","journal-title":"NPJ Comput. Mater."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Bahrami, S., and Tuncel, E. (2020, January 21\u201326). An Efficient Running Quantile Estimation Technique alongside Correntropy for Outlier Rejection in Online Regression. Proceedings of the 2020 IEEE International Symposium on Information Theory (ISIT), Los Angeles, CA, USA.","DOI":"10.1109\/ISIT44484.2020.9174111"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Briesemeister, S., Rahnenf\u00fchrer, J., and Kohlbacher, O. (2012). No Longer Confidential: Estimating the Confidence of Individual Regression Predictions. PLoS ONE, 7.","DOI":"10.1371\/journal.pone.0048723"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1287\/ijoc.2020.1019","article-title":"Improving Reliability Estimation for Individual Numeric Predictions: A Machine Learning Approach","volume":"34","author":"Adomavicius","year":"2022","journal-title":"INFORMS J. Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1038\/s41746-019-0209-7","article-title":"Identifying Unreliable Predictions in Clinical Risk Models","volume":"3","author":"Myers","year":"2020","journal-title":"NPJ Digit. Med."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"102113","DOI":"10.1016\/j.artmed.2021.102113","article-title":"A New Approach for Interpretability and Reliability in Clinical Risk Prediction: Acute Coronary Syndrome Scenario","volume":"117","author":"Valente","year":"2021","journal-title":"Artif. Intell. Med."},{"key":"ref_19","unstructured":"Correia, C. (2025, April 17). Pointwise-Benchmark. Available online: https:\/\/github.com\/Oak10\/pointwise-benchmark\/releases\/tag\/1.1.1."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1109\/ICJECE.2024.3429273","article-title":"Supervised T-SNE for Metric Learning With Stochastic and Geodesic Distances","volume":"47","author":"Levada","year":"2024","journal-title":"IEEE Can. J. Electr. Comput. Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1038\/s41524-023-01012-9","article-title":"A Critical Examination of Robustness and Generalizability of Machine Learning Prediction of Materials Properties","volume":"9","author":"Li","year":"2023","journal-title":"NPJ Comput. Mater."},{"key":"ref_22","first-page":"2020","article-title":"EHR Dataset for Patient Treatment Classification","volume":"1","author":"Sadikin","year":"2020","journal-title":"Mendeley Data"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"267","DOI":"10.3233\/IFS-1994-2306","article-title":"Fuzzy Model Identification Based on Cluster Estimation","volume":"2","author":"Chiu","year":"1994","journal-title":"J. Intell. Fuzzy Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"13159","DOI":"10.1007\/s10586-017-1321-6","article-title":"Stock Market Prediction Using Subtractive Clustering for a Neuro Fuzzy Hybrid Approach","volume":"22","author":"Chandar","year":"2019","journal-title":"Clust. Comput."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/4\/327\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:18:13Z","timestamp":1760030293000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/4\/327"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,20]]},"references-count":24,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["info16040327"],"URL":"https:\/\/doi.org\/10.3390\/info16040327","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2025,4,20]]}}}