{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:46:03Z","timestamp":1760132763562,"version":"build-2065373602"},"reference-count":42,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T00:00:00Z","timestamp":1697068800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"NRL NISE Program Element Jerome and Isabella Karles Fellowship under Work Unit N20N"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Though an accurate measurement of entropy, or more generally uncertainty, is critical to the success of human\u2013machine teams, the evaluation of the accuracy of such metrics as a probability of machine correctness is often aggregated and not assessed as an iterative control process. The entropy of the decisions made by human\u2013machine teams may not be accurately measured under cold start or at times of data drift unless disagreements between the human and machine are immediately fed back to the classifier iteratively. In this study, we present a stochastic framework by which an uncertainty model may be evaluated iteratively as a probability of machine correctness. We target a novel problem, referred to as the threshold selection problem, which involves a user subjectively selecting the point at which a signal transitions to a low state. This problem is designed to be simple and replicable for human\u2013machine experimentation while exhibiting properties of more complex applications. Finally, we explore the potential of incorporating feedback of machine correctness into a baseline na\u00efve Bayes uncertainty model with a novel reinforcement learning approach. The approach refines a baseline uncertainty model by incorporating machine correctness at every iteration. Experiments are conducted over a large number of realizations to properly evaluate uncertainty at each iteration of the human\u2013machine team. Results show that our novel approach, called closed-loop uncertainty, outperforms the baseline in every case, yielding about 45% improvement on average.<\/jats:p>","DOI":"10.3390\/e25101443","type":"journal-article","created":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T07:28:44Z","timestamp":1697095724000},"page":"1443","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Closed-Loop Uncertainty: The Evaluation and Calibration of Uncertainty for Human\u2013Machine Teams under Data Drift"],"prefix":"10.3390","volume":"25","author":[{"given":"Zachary","family":"Bishof","sequence":"first","affiliation":[{"name":"U.S. Naval Research Laboratory, 1005 Balch Boulevard, Stennis Space Center, St. Louis, MS 39529, USA"}]},{"given":"Jaelle","family":"Scheuerman","sequence":"additional","affiliation":[{"name":"U.S. Naval Research Laboratory, 1005 Balch Boulevard, Stennis Space Center, St. Louis, MS 39529, USA"}]},{"given":"Chris J.","family":"Michael","sequence":"additional","affiliation":[{"name":"U.S. Naval Research Laboratory, 1005 Balch Boulevard, Stennis Space Center, St. Louis, MS 39529, USA"}]}],"member":"1968","published-online":{"date-parts":[[2023,10,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1016\/j.inffus.2021.05.008","article-title":"A review of uncertainty quantification in deep learning: Techniques, applications and challenges","volume":"76","author":"Abdar","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_2","unstructured":"Jiang, H., Kim, B., Guan, M., and Gupta, M. (2018, January 3\u20138). To trust or not to trust a classifier. Proceedings of the Advances in Neural Information Processing Systems 31 (NeurIPS 2018), Montr\u00e9al, Canada."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1016\/j.compchemeng.2003.09.017","article-title":"Optimization under uncertainty: state-of-the-art and opportunities","volume":"28","author":"Sahinidis","year":"2004","journal-title":"Comput. Chem. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Bhatt, U., Antor\u00e1n, J., Zhang, Y., Liao, Q.V., Sattigeri, P., Fogliato, R., Melan\u00e7on, G., Krishnan, R., Stanley, J., and Tickoo, O. (2021, January 19\u201321). Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty. Proceedings of the 2021 AAAI\/ACM Conference on AI, Ethics, and Society, Virtual.","DOI":"10.1145\/3461702.3462571"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"493","DOI":"10.3390\/e10040493","article-title":"Entropy and uncertainty","volume":"10","author":"Robinson","year":"2008","journal-title":"Entropy"},{"key":"ref_6","unstructured":"Aggarwal, C.C., Kong, X., Gu, Q., Han, J., and Philip, S.Y. (2014). Data Classification, Chapman and Hall\/CRC."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","article-title":"Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods","volume":"110","author":"Waegeman","year":"2021","journal-title":"Mach. Learn."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3005","DOI":"10.1007\/s10462-022-10246-w","article-title":"Human-in-the-loop machine learning: A state of the art","volume":"56","year":"2023","journal-title":"Artif. Intell. Rev."},{"key":"ref_9","unstructured":"Monarch, R.M. (2021). Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-Centered AI, Simon and Schuster."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Michael, C.J., Dennis, S.M., Maryan, C., Irving, S., and Palmsten, M.L. (2019, January 5\u20138). A general framework for human-machine digitization of geographic regions from remotely sensed imagery. Proceedings of the 27th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, Chicago, IL, USA.","DOI":"10.1145\/3347146.3359370"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1007\/BF00116895","article-title":"Incremental learning from noisy data","volume":"1","author":"Schlimmer","year":"1986","journal-title":"Mach. Learn."},{"key":"ref_12","unstructured":"Michael, C.J., Acklin, D., and Scheuerman, J. (2020). On interactive machine learning and the potential of cognitive feedback. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Pronk, V., Gutta, S.V., and Verhaegh, W.F. (2005, January 24\u201329). Incorporating confidence in a naive Bayesian classifier. Proceedings of the International Conference on User Modeling, Edinburgh, UK.","DOI":"10.1007\/11527886_41"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1016\/S0951-8320(96)00077-4","article-title":"Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management","volume":"54","author":"Hora","year":"1996","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.neucom.2019.01.103","article-title":"Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks","volume":"338","author":"Wang","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1007\/s40747-019-00124-4","article-title":"Data-driven decision support under concept drift in streamed big data","volume":"6","author":"Lu","year":"2020","journal-title":"Complex Intell. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kl\u00e4s, M., and Vollmer, A.M. (2018, January 18). Uncertainty in machine learning applications: A practice-driven classification of uncertainty. Proceedings of the International Conference on Computer Safety, Reliability, and Security, V\u00e4ster\u00e5s, Sweden.","DOI":"10.1007\/978-3-319-99229-7_36"},{"key":"ref_18","unstructured":"Kaplan, L., Cerutti, F., Sensoy, M., Preece, A.D., and Sullivan, P. (2018). Uncertainty aware AI ML: Why and how. arXiv."},{"key":"ref_19","unstructured":"Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017, January 6\u201311). On calibration of modern neural networks. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, NSW, Australia."},{"key":"ref_20","first-page":"445","article-title":"The case against accuracy estimation for comparing induction algorithms","volume":"98","author":"Provost","year":"1998","journal-title":"ICML"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Abell\u00e1n, J., and Castellano, J.G. (2017). Improving the Naive Bayes classifier via a quick variable selection method using maximum of entropy. Entropy, 19.","DOI":"10.3390\/e19060247"},{"key":"ref_22","unstructured":"Eysenbach, B., and Levine, S. (2021). Maximum entropy rl (provably) solves some robust rl problems. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"337","DOI":"10.3233\/IDA-140645","article-title":"Detecting concept drift: an information entropy based method using an adaptive sliding window","volume":"18","author":"Du","year":"2014","journal-title":"Intell. Data Anal."},{"key":"ref_24","unstructured":"Clements, W.R., Van Delft, B., Robaglia, B.M., Slaoui, R.B., and Toth, S. (2020, January 12\u201318). Estimating risk and uncertainty in deep reinforcement learning. Proceedings of the Uncertainty and Robustness in Deep Learning Workshop at International Conference on Machine Learning, Vienna, Austria."},{"key":"ref_25","unstructured":"Nikolov, N., Kirschner, J., Berkenkamp, F., and Krause, A. (2018). Information-directed exploration for deep reinforcement learning. arXiv."},{"key":"ref_26","unstructured":"Bellemare, M.G., Dabney, W., and Munos, R. (2017, January 6\u201311). A distributional perspective on reinforcement learning. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, NSW, Australia."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1007\/s12650-018-0531-1","article-title":"Recent research advances on interactive machine learning","volume":"22","author":"Jiang","year":"2019","journal-title":"J. Vis."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1251","DOI":"10.1109\/TPAMI.2006.156","article-title":"Confidence-based active learning","volume":"28","author":"Li","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","unstructured":"Teso, S., and Vergari, A. (2022). Efficient and Reliable Probabilistic Interactive Learning with Structured Outputs. arXiv."},{"key":"ref_30","first-page":"3","article-title":"The optimality of naive Bayes","volume":"1","author":"Zhang","year":"2004","journal-title":"Aa"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/s10994-005-4258-6","article-title":"Not so naive Bayes: aggregating one-dependence estimators","volume":"58","author":"Webb","year":"2005","journal-title":"Mach. Learn."},{"key":"ref_32","unstructured":"Xie, Z., Hsu, W., Liu, Z., and Lee, M.L. (2002, January 6\u20138). Snnb: A selective neighborhood based naive Bayes for lazy learning. Proceedings of the Advances in Knowledge Discovery and Data Mining: 6th Pacific-Asia Conference, PAKDD 2002 (Proceedings 6), Taipei, Taiwan."},{"key":"ref_33","unstructured":"John, G.H., and Langley, P. (2013). Estimating continuous distributions in Bayesian classifiers. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_36","unstructured":"Majka, M. (2023, October 05). Naivebayes: High Performance Implementation of the Naive Bayes Algorithm in R; R package version 0.9.7; 2019. Available online: https:\/\/CRAN.R-project.org\/package=naivebayes."},{"key":"ref_37","unstructured":"Pr\u00f6llochs, N., and Feuerriegel, S. (2018). Reinforcement learning in R. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Fails, J.A., and Olsen, D.R. (2003, January 12\u201315). Interactive machine learning. Proceedings of the 8th International Conference on Intelligent User Interfaces, Miami, FL, USA.","DOI":"10.1145\/604045.604056"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1038\/nmeth.2281","article-title":"JAABA: interactive machine learning for automatic annotation of animal behavior","volume":"10","author":"Kabra","year":"2013","journal-title":"Nat. Methods"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Krawczyk, B., and Wozniak, M. (2015, January 9\u201312). Weighted naive bayes classifier with forgetting for drifting data streams. Proceedings of the 2015 IEEE International Conference on Systems, Man, and Cybernetics, Hong Kong, China.","DOI":"10.1109\/SMC.2015.375"},{"key":"ref_41","unstructured":"Hasselt, H.V. (2012). Reinforcement Learning, Springer."},{"key":"ref_42","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/10\/1443\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:05:34Z","timestamp":1760130334000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/10\/1443"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,12]]},"references-count":42,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,10]]}},"alternative-id":["e25101443"],"URL":"https:\/\/doi.org\/10.3390\/e25101443","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2023,10,12]]}}}