{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,8]],"date-time":"2026-02-08T08:13:33Z","timestamp":1770538413682,"version":"3.49.0"},"reference-count":55,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2020,6,8]],"date-time":"2020-06-08T00:00:00Z","timestamp":1591574400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003407","name":"Ministero dell\u2019Istruzione, dell\u2019Universit\u00e0 e della Ricerca","doi-asserted-by":"publisher","award":["Departments of Excellence (Law 445 232\/2016)"],"award-info":[{"award-number":["Departments of Excellence (Law 445 232\/2016)"]}],"id":[{"id":"10.13039\/501100003407","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>In knowledge-based systems, besides obtaining good output prediction accuracy, it is crucial to understand the subset of input variables that have most influence on the output, with the goal of gaining deeper insight into the underlying process. These requirements call for logistic model estimation techniques that provide a sparse solution, i.e., where coefficients associated with non-important variables are set to zero. In this work we compare the performance of two methods: the first one is based on the well known Least Absolute Shrinkage and Selection Operator (LASSO) which involves regularization with an     \u2113 1     norm; the second one is the Relevance Vector Machine (RVM) which is based on a Bayesian implementation of the linear logistic model. The two methods are extensively compared in this paper, on real and simulated datasets. Results show that, in general, the two approaches are comparable in terms of prediction performance. RVM outperforms the LASSO both in term of structure recovery (estimation of the correct non-zero model coefficients) and prediction accuracy when the dimensionality of the data tends to increase. However, LASSO shows comparable performance to RVM when the dimensionality of the data is much higher than number of samples that is     p &gt; &gt; n    .<\/jats:p>","DOI":"10.3390\/a13060137","type":"journal-article","created":{"date-parts":[[2020,6,9]],"date-time":"2020-06-09T04:19:39Z","timestamp":1591676379000},"page":"137","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Sparse Logistic Regression: Comparison of Regularization and Bayesian Implementations"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9875-6329","authenticated-orcid":false,"given":"Mattia","family":"Zanon","sequence":"first","affiliation":[{"name":"Department of Electronic Engineering, National University of Ireland, Maynooth (NUIM), W23 F2K8 Maynooth, Co. Kildare, Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5839-5250","authenticated-orcid":false,"given":"Giuliano","family":"Zambonin","sequence":"additional","affiliation":[{"name":"Electrolux Italy S.P.A., 33080 Porcia, PN, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5739-9639","authenticated-orcid":false,"given":"Gian Antonio","family":"Susto","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Padova, 35131 Padova, PD, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3016-6197","authenticated-orcid":false,"given":"Se\u00e1n","family":"McLoone","sequence":"additional","affiliation":[{"name":"School of Electronics, Electrical Engineering and Computer Science, Queen\u2019s University Belfast, 16A Malone Road, Belfast BT9 5BN, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,6,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"714","DOI":"10.1093\/bioinformatics\/btp041","article-title":"Genome-wide association analysis by lasso penalized logistic regression","volume":"25","author":"Wu","year":"2009","journal-title":"Bioinformatics"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1109\/TASE.2015.2473659","article-title":"Home Automation Oriented Gesture Classification From Inertial Measurements","volume":"12","author":"Cenedese","year":"2015","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Pampuri, S., Schirru, A., Susto, G.A., De Luca, C., Beghi, A., and De Nicolao, G. (2012, January 20\u201324). Multistep virtual metrology approaches for semiconductor manufacturing processes. Proceedings of the 2012 IEEE International Conference on Automation Science and Engineering (CASE), Seoul, Korea.","DOI":"10.1109\/CoASE.2012.6386484"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Susto, G.A., Beghi, A., and De Luca, C. (2011, January 5\u20139). A virtual metrology system for predicting cvd thickness with equipment variables and qualitative clustering. Proceedings of the ETFA2011, Toulouse, France.","DOI":"10.1109\/ETFA.2011.6059209"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.phycom.2011.07.005","article-title":"Group sparse Lasso for cognitive network sensing robust to model uncertainties and outliers","volume":"5","author":"Bazerque","year":"2012","journal-title":"Phys. Commun."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Hu, X., Tang, L., Tang, J., and Liu, H. (2013, January 4\u20138). Exploiting social relations for sentiment analysis in microblogging. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (2013), Rome, Italy.","DOI":"10.1145\/2433396.2433465"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Lampos, V., and Cristianini, N. (2010, January 14\u201316). Tracking the flu pandemic by monitoring the social web. Proceedings of the 2010 2nd International Workshop on Cognitive Information Processing (CIP), Elba Island, Italy.","DOI":"10.1109\/CIP.2010.5604088"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2507","DOI":"10.1093\/bioinformatics\/btm344","article-title":"A review of feature selection techniques in bioinformatics","volume":"23","author":"Saeys","year":"2007","journal-title":"Bioinformatics"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.artmed.2004.01.007","article-title":"Filter versus wrapper gene selection approaches in DNA microarray domains","volume":"31","author":"Inza","year":"2004","journal-title":"Artif. Intell. Med."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"131","DOI":"10.3233\/IDA-1997-1302","article-title":"Feature selection for classification","volume":"1","author":"Dash","year":"1997","journal-title":"Intell. Data Anal."},{"key":"ref_11","first-page":"1157","article-title":"An introduction to variable and feature selection","volume":"3","author":"Guyon","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J. R. Stat. Soc. Ser. B (Methodol.)"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1137\/S1064827596304010","article-title":"Atomic decomposition by basis pursuit","volume":"20","author":"Chen","year":"1998","journal-title":"Siam J. Sci. Comput."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1111\/j.1467-9868.2005.00503.x","article-title":"Regularization and variable selection via the elastic net","volume":"67","author":"Zou","year":"2005","journal-title":"J. R. Stat. Soc. Ser. (Stat. Methodol."},{"key":"ref_15","first-page":"211","article-title":"Sparse Bayesian Learning and the Relevance Vector Machine","volume":"1","author":"Tipping","year":"2001","journal-title":"J. Mach. Learn. Res."},{"key":"ref_16","unstructured":"Shalev-Shwartz, S., and Zhang, T. (2014, January 22\u201324). Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization. Proceedings of the International Conference on Machine Learning, Bejing, China."},{"key":"ref_17","first-page":"1999","article-title":"An improved glmnet for l1-regularized logistic regression","volume":"13","author":"Yuan","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_18","first-page":"3183","article-title":"A comparison of optimization methods and software for large-scale l1-regularized linear classification","volume":"11","author":"Yuan","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_19","unstructured":"Schmidt, M., Fung, G., and Rosales, R. (2007). Fast optimization methods for l1 regularization: A comparative study and two new approaches. Machine Learning: ECML 2007, Springer."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1947","DOI":"10.3182\/20140824-6-ZA-1003.01885","article-title":"Root cause analysis by a combined sparse classification and Monte 558 Carlo approach","volume":"47","author":"Zanon","year":"2014","journal-title":"IFAC Proc. Vol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1109\/TNN.2003.809398","article-title":"The generalized LASSO","volume":"15","author":"Roth","year":"2004","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1080\/10618600.2000.10474883","article-title":"On the lasso and its dual","volume":"9","author":"Osborne","year":"2000","journal-title":"J. Comput. Graph. Stat."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1016\/j.neunet.2009.07.001","article-title":"Sparse kernel learning with LASSO and Bayesian inference algorithm","volume":"23","author":"Gao","year":"2010","journal-title":"Neural Netw."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.engappai.2017.06.002","article-title":"Predicting seismic events in coal mines based on underground sensor measurements","volume":"64","author":"Janusz","year":"2017","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_25","first-page":"477","article-title":"Boosting algorithms: Regularization, prediction and model fitting","volume":"22","author":"Hothorn","year":"2007","journal-title":"Stat. Sci."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Schapire, R.E. (2003). The boosting approach to machine learning: An overview. Nonlinear Estimation and Classification, Springer.","DOI":"10.1007\/978-0-387-21579-2_9"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1006\/jcss.1997.1504","article-title":"A decision-theoretic generalization of on-line learning and an application to boosting","volume":"55","author":"Freund","year":"1997","journal-title":"J. Comput. Syst. Sci."},{"key":"ref_28","first-page":"148","article-title":"Experiments with a new boosting algorithm","volume":"96","author":"Freund","year":"1996","journal-title":"ICML Citeseer"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1418","DOI":"10.1198\/016214506000000735","article-title":"The adaptive lasso and its oracle properties","volume":"101","author":"Zou","year":"2006","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_30","first-page":"1603","article-title":"Adaptive Lasso for sparse high-dimensional regression models","volume":"18","author":"Huang","year":"2008","journal-title":"Stat. Sin."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1023\/A:1013912006537","article-title":"Logistic regression, AdaBoost and Bregman distances","volume":"48","author":"Collins","year":"2002","journal-title":"Mach. Learn."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/S1570-579X(01)80004-5","article-title":"Joint and separate convexity of the Bregman distance","volume":"Volume 8","author":"Bauschke","year":"2001","journal-title":"Studies in Computational Mathematics"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Friedman, J., Hastie, T., and Tibshirani, R. (2001). The Elements of Statistical Learning, Springer. Number 10.","DOI":"10.1007\/978-0-387-21606-5"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1198\/016214503000125","article-title":"Boosting with the L 2 loss: Regression and classification","volume":"98","author":"Yu","year":"2003","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","article-title":"Greedy function approximation: A gradient boosting machine","volume":"29","author":"Friedman","year":"2001","journal-title":"Ann. Stat."},{"key":"ref_36","unstructured":"Wu, Y., Wipf, D., and Yun, J.M. (2020, May 10). Understanding and Evaluating Sparse Linear Discriminant Analysis. Available online: http:\/\/proceedings.mlr.press\/v38\/wu15.pdf."},{"key":"ref_37","unstructured":"Welling, M. (2020, May 10). Fisher Linear Discriminant Analysis. Available online: https:\/\/www.ics.uci.edu\/~welling\/teaching\/273ASpring09\/Fisher-LDA.pdf."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"8962","DOI":"10.1021\/acs.iecr.8b01110","article-title":"Fault Detection and Classification for Nonlinear Chemical Processes using Lasso and Gaussian Process","volume":"57","author":"Du","year":"2018","journal-title":"Ind. Eng. Chem. Res."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"8685","DOI":"10.1021\/ie101146d","article-title":"Nonlinear soft sensor development based on relevance vector machine","volume":"49","author":"Ge","year":"2010","journal-title":"Ind. Eng. Chem. Res."},{"key":"ref_40","unstructured":"Urhan, A., and Alakent, B. (2019). An Exploratory Analysis of Biased Learners in Soft-Sensing Frames. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v033.i01","article-title":"Regularization paths for generalized linear models via coordinate descent","volume":"33","author":"Friedman","year":"2010","journal-title":"J. Stat. Softw."},{"key":"ref_42","unstructured":"Bertsekas, D.P., and Tsitsiklis, J.N. (1989). Parallel and Distributed Computation: Numerical Methods, Prentice Hall."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"720","DOI":"10.1162\/neco.1992.4.5.720","article-title":"The evidence framework applied to classification networks","volume":"4","author":"MacKay","year":"1992","journal-title":"Neural Comput."},{"key":"ref_44","unstructured":"Tipping, M.E., and Faul, A.C. (2003, January 3\u20136). Fast marginal likelihood maximisation for sparse Bayesian models. Proceedings of the AISTATS, Key West, FL, USA."},{"key":"ref_45","unstructured":"Fletcher, T. (2010). Relevance Vector Machines Explained, University College London."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"448","DOI":"10.1162\/neco.1992.4.3.448","article-title":"A practical Bayesian framework for backpropagation networks","volume":"4","author":"MacKay","year":"1992","journal-title":"Neural Comput."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Neal, R.M. (1996). Bayesian Learning for Neural Networks, Springer-Verlag New York, Inc.","DOI":"10.1007\/978-1-4612-0745-0"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1080\/01621459.1984.10478083","article-title":"Cross-validation of regression models","volume":"79","author":"Picard","year":"1984","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Kuhn, M., and Johnson, K. (2013). Applied Predictive Modeling, Springer.","DOI":"10.1007\/978-1-4614-6849-3"},{"key":"ref_50","first-page":"1137","article-title":"A study of cross-validation and bootstrap for accuracy estimation and model selection","volume":"2","author":"Kohavi","year":"1995","journal-title":"IJCAI Montr. Can."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Fisher, R.A. (1992). Statistical methods for research workers. Breakthroughs in Statistics, Springer.","DOI":"10.1007\/978-1-4612-4380-9_6"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"362","DOI":"10.1002\/asmb.1948","article-title":"A virtual metrology system based on least angle regression and statistical clustering","volume":"29","author":"Susto","year":"2013","journal-title":"Appl. Stoch. Model. Bus. Ind."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Schirru, A., Susto, G.A., Pampuri, S., and McLoone, S. (2012, January 10\u201313). Learning from time series: Supervised aggregative feature extraction. Proceedings of the 2012 IEEE 51st IEEE Conference on Decision and Control (CDC), Maui, HI, USA.","DOI":"10.1109\/CDC.2012.6427042"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"1436","DOI":"10.1214\/009053606000000281","article-title":"High-dimensional graphs and variable selection with the lasso","volume":"34","author":"Meinshausen","year":"2006","journal-title":"Ann. Stat."},{"key":"ref_55","first-page":"2541","article-title":"On model selection consistency of Lasso","volume":"7","author":"Zhao","year":"2006","journal-title":"J. Mach. Learn. Res."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/13\/6\/137\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:36:44Z","timestamp":1760175404000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/13\/6\/137"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,8]]},"references-count":55,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2020,6]]}},"alternative-id":["a13060137"],"URL":"https:\/\/doi.org\/10.3390\/a13060137","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,8]]}}}