{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:41:11Z","timestamp":1760143271551,"version":"build-2065373602"},"reference-count":51,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T00:00:00Z","timestamp":1706659200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Active learning (AL) is a paradigm focused on purposefully selecting training data to enhance a model\u2019s performance by minimizing the need for annotated samples. Typically, strategies assume that the training pool shares the same distribution as the test set, which is not always valid in privacy-sensitive applications where annotating user data is challenging. In this study, we operate within an individual setting and leverage an active learning criterion which selects data points for labeling based on minimizing the min-max regret on a small unlabeled test set sample. Our key contribution lies in the development of an efficient algorithm, addressing the challenging computational complexity associated with approximating this criterion for neural networks. Notably, our results show that, especially in the presence of out-of-distribution data, the proposed algorithm substantially reduces the required training set size by up to 15.4%, 11%, and 35.1% for CIFAR10, EMNIST, and MNIST datasets, respectively.<\/jats:p>","DOI":"10.3390\/e26020129","type":"journal-article","created":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T08:44:06Z","timestamp":1706690646000},"page":"129","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Deep Individual Active Learning: Safeguarding against Out-of-Distribution Challenges in Neural Networks"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7391-213X","authenticated-orcid":false,"given":"Shachar","family":"Shayovitz","sequence":"first","affiliation":[{"name":"School of Electrical Engineering, Tel Aviv University, Tel Aviv 6997801, Israel"}]},{"given":"Koby","family":"Bibas","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering, Tel Aviv University, Tel Aviv 6997801, Israel"}]},{"given":"Meir","family":"Feder","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering, Tel Aviv University, Tel Aviv 6997801, Israel"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,31]]},"reference":[{"key":"ref_1","first-page":"1","article-title":"A survey of deep active learning","volume":"54","author":"Ren","year":"2021","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2591","DOI":"10.1109\/TCSVT.2016.2589879","article-title":"Cost-effective active learning for deep image classification","volume":"27","author":"Wang","year":"2016","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_3","unstructured":"Houlsby, N., Husz\u00e1r, F., Ghahramani, Z., and Lengyel, M. (2011). Bayesian active learning for classification and preference learning. arXiv."},{"key":"ref_4","unstructured":"Gal, Y., Islam, R., and Ghahramani, Z. (2017, January 6\u201311). Deep bayesian active learning with image data. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_5","unstructured":"Sener, O., and Savarese, S. (May, January 30). Active Learning for Convolutional Neural Networks: A Core-Set Approach. Proceedings of the International Conferenc on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"720","DOI":"10.1109\/JSAIT.2021.3073842","article-title":"Universal Active Learning via Conditional Mutual Information Minimization","volume":"2","author":"Shayovitz","year":"2021","journal-title":"IEEE J. Sel. Areas Inf. Theory"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"7036","DOI":"10.1109\/TIT.2011.2154375","article-title":"Information-based complexity, feedback and dynamics in convex programming","volume":"57","author":"Raginsky","year":"2011","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"590","DOI":"10.1162\/neco.1992.4.4.590","article-title":"Information-based objective functions for active data selection","volume":"4","author":"MacKay","year":"1992","journal-title":"Neural Comput."},{"key":"ref_9","unstructured":"Fedorov, V.V. (2013). Theory of Optimal Experiments, Elsevier."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Shayovitz, S., and Feder, M. (2019, January 13\u201316). Minimax Active Learning Via Minimal Model Capacity. Proceedings of the 29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019, Pittsburgh, PA, USA.","DOI":"10.1109\/MLSP.2019.8918907"},{"key":"ref_11","unstructured":"Smith, F.B., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T. (2023, January 25\u201327). Prediction-Oriented Bayesian Active Learning. Proceedings of the International Conference on Artificial Intelligence and Statistics, Valencia, Spain."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Alabduljabbar, A., Abusnaina, A., Meteriz-Yildiran, \u00dc., and Mohaisen, D. (2021, January 15). TLDR: Deep Learning-Based Automated Privacy Policy Annotation with Key Policy Highlights. Proceedings of the 20th Workshop on Workshop on Privacy in the Electronic Society, Seoul, Republic of Korea.","DOI":"10.1145\/3463676.3485608"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2124","DOI":"10.1109\/18.720534","article-title":"Universal prediction","volume":"44","author":"Merhav","year":"1998","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Fogel, Y., and Feder, M. (2018, January 17\u201322). Universal batch learning with log-loss. Proceedings of the International Symposium on Information Theory, Vail, CO, USA.","DOI":"10.1109\/ISIT.2018.8437543"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Bibas, K., Fogel, Y., and Feder, M. (2019, January 7\u201312). A New Look at an Old Problem: A Universal Learning Approach to Linear Regression. Proceedings of the 2019 IEEE International Symposium on Information Theory (ISIT), Paris, France.","DOI":"10.1109\/ISIT.2019.8849398"},{"key":"ref_16","unstructured":"Zhou, A., and Levine, S. (2021, January 18\u201324). Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Yoo, D., and Kweon, I.S. (2019, January 15\u201320). Learning loss for active learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00018"},{"key":"ref_18","unstructured":"Sinha, S., Ebrahimi, S., and Darrell, T. (November, January 27). Variational adversarial active learning. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Karamcheti, S., Krishna, R., Fei-Fei, L., and Manning, C.D. (2021). Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. arXiv.","DOI":"10.18653\/v1\/2021.acl-long.564"},{"key":"ref_20","first-page":"18685","article-title":"Similar: Submodular information measures based active learning in realistic scenarios","volume":"34","author":"Kothawade","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Du, P., Zhao, S., Chen, H., Chai, S., Chen, H., and Li, C. (2021, January 11\u201317). Contrastive coding for active learning under class distribution mismatch. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00880"},{"key":"ref_22","unstructured":"Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 26\u201328). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Teney, D., Abbasnedjad, E., and van den Hengel, A. (2020, January 23\u201328). Learning what makes a difference from counterfactual examples and gradient supervision. Proceedings of the Computer Vision\u2014ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part X 16.","DOI":"10.1007\/978-3-030-58607-2_34"},{"key":"ref_24","unstructured":"Kaushik, D., Hovy, E., and Lipton, Z.C. (2019). Learning the difference that makes a difference with counterfactually-augmented data. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., and Kiela, D. (2019). Adversarial NLI: A new benchmark for natural language understanding. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.441"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Shayovitz, S., and Feder, M. (2022, January 27\u201330). Active Learning for Individual Data via Minimal Stochastic Complexity. Proceedings of the 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, NY, USA.","DOI":"10.1109\/Allerton49937.2022.9929357"},{"key":"ref_27","unstructured":"Krizhevsky, A., Nair, V., and Hinton, G. (2024, January 29). The CIFAR-10 Dataset. Available online: http:\/\/www.cs.toronto.edu\/kriz\/cifar.html."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. (2017, January 14\u201319). EMNIST: Extending MNIST to handwritten letters. Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA.","DOI":"10.1109\/IJCNN.2017.7966217"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1109\/MSP.2012.2211477","article-title":"The mnist database of handwritten digit images for machine learning research [best of the web]","volume":"29","author":"Deng","year":"2012","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_30","unstructured":"Bibas, K., and Feder, M. (2021). Distribution Free Uncertainty for the Minimum Norm Solution of Over-parameterized Linear Regression. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Rosas, F.E., Mediano, P.A., and Gastpar, M. (2020, January 11\u201315). Learning, compression, and leakage: Minimising classification error via meta-universal compression principles. Proceedings of the 2020 IEEE Information Theory Workshop (ITW), Virtual.","DOI":"10.1109\/ITW46852.2021.9457579"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Rissanen, J., and Roos, T. (2007, January 18\u201323). Conditional NML universal models. Proceedings of the 2007 Information Theory and Applications Workshop, San Diego, CA, USA.","DOI":"10.1109\/ITA.2007.4357600"},{"key":"ref_33","first-page":"256","article-title":"On sequentially normalized maximum likelihood models","volume":"27","author":"Roos","year":"2008","journal-title":"Compare"},{"key":"ref_34","unstructured":"Bibas, K., and Feder, M. (2021). The Predictive Normalized Maximum Likelihood for Over-parameterized Linear Regression with Norm Constraint: Regret and Double Descent. arXiv."},{"key":"ref_35","unstructured":"Rosas, F.E., Mediano, P.A., and Gastpar, M. (2020). Learning, compression, and leakage: Minimizing classification error via meta-universal compression principles. arXiv."},{"key":"ref_36","unstructured":"Fu, J., and Levine, S. (2021, January 3\u20137). Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation. Proceedings of the International Conference on Learning Representations, Virtual."},{"key":"ref_37","unstructured":"Bibas, K., Fogel, Y., and Feder, M. (2019). Deep pnml: Predictive normalized maximum likelihood for deep neural networks. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1109\/JSAIT.2020.2991518","article-title":"Maximin active learning in overparameterized model classes","volume":"1","author":"Karzand","year":"2020","journal-title":"IEEE J. Sel. Areas Inf. Theory"},{"key":"ref_39","unstructured":"Kirsch, A., van Amersfoort, J., and Gal, Y. (2019, January 8\u201314). BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning. Proceedings of the Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Vancouver, BC, Canada."},{"key":"ref_40","unstructured":"Maddox, W.J., Izmailov, P., Garipov, T., Vetrov, D.P., and Wilson, A.G. (2019). A simple baseline for bayesian uncertainty in deep learning. Adv. Neural Inf. Process. Syst., 32."},{"key":"ref_41","unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 19\u201324). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the International Conference Machine Learning, New York, NY, USA."},{"key":"ref_42","first-page":"20089","article-title":"Laplace redux-effortless bayesian deep learning","volume":"34","author":"Daxberger","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_43","unstructured":"Wilson, A.G., Hu, Z., Salakhutdinov, R.R., and Xing, E.P. (2016). Stochastic variational deep kernel learning. Adv. Neural Inf. Process. Syst., 29."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2008","DOI":"10.1109\/TPAMI.2018.2889774","article-title":"Advances in variational inference","volume":"41","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1080\/01621459.2017.1285773","article-title":"Variational inference: A review for statisticians","volume":"112","author":"Blei","year":"2017","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_46","first-page":"1303","article-title":"Stochastic variational inference","volume":"14","author":"Hoffman","year":"2013","journal-title":"J. Mach. Learn. Res."},{"key":"ref_47","unstructured":"Simon, H.U. (2015, January 3\u20136). An almost optimal PAC algorithm. Proceedings of the Conference on Learning Theory, Paris, France."},{"key":"ref_48","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv."},{"key":"ref_49","unstructured":"Sermanet, P., Chintala, S., and LeCun, Y. (2012, January 11\u201315). Convolutional neural networks applied to house numbers digit classification. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan."},{"key":"ref_50","unstructured":"Huang, K.H. (2021). DeepAL: Deep Active Learning in Python. arXiv."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/26\/2\/129\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:52:33Z","timestamp":1760104353000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/26\/2\/129"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,31]]},"references-count":51,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,2]]}},"alternative-id":["e26020129"],"URL":"https:\/\/doi.org\/10.3390\/e26020129","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2024,1,31]]}}}