{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,28]],"date-time":"2026-07-28T03:55:04Z","timestamp":1785210904856,"version":"3.55.0"},"reference-count":81,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,6,19]],"date-time":"2024-06-19T00:00:00Z","timestamp":1718755200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>Integration between constrained optimization and deep networks has garnered significant interest from both research and industrial laboratories. Optimization techniques can be employed to optimize the choice of network structure based not only on loss and accuracy but also on physical constraints. Additionally, constraints can be imposed during training to enhance the performance of networks in specific contexts. This study surveys the literature on the integration of constrained optimization with deep networks. Specifically, we examine the integration of hyper-parameter tuning with physical constraints, such as the number of FLOPS (FLoating point Operations Per Second), a measure of computational capacity, latency, and other factors. This study also considers the use of context-specific knowledge constraints to improve network performance. We discuss the integration of constraints in neural architecture search (NAS), considering the problem as both a multi-objective optimization (MOO) challenge and through the imposition of penalties in the loss function. Furthermore, we explore various approaches that integrate logic with deep neural networks (DNNs). In particular, we examine logic-neural integration through constrained optimization applied during the training of NNs and the use of semantic loss, which employs the probabilistic output of the networks to enforce constraints on the output.<\/jats:p>","DOI":"10.3389\/frai.2024.1414707","type":"journal-article","created":{"date-parts":[[2024,6,19]],"date-time":"2024-06-19T05:20:14Z","timestamp":1718774414000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Integration between constrained optimization and deep networks: a survey"],"prefix":"10.3389","volume":"7","author":[{"given":"Alice","family":"Bizzarri","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michele","family":"Fraccaroli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Evelina","family":"Lamma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fabrizio","family":"Riguzzi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,6,19]]},"reference":[{"key":"B1","first-page":"20","article-title":"Case-based refinement of knowledge-based neural networks","volume-title":"The International Conference\" Intelligent Systems: A Semiotic Perspective","author":"Agre","year":"1996"},{"key":"B2","first-page":"289","article-title":"Knowledge compilation","volume-title":"Machine Learning: An Artificial Intelligence Approach","author":"Anderson","year":"1983"},{"key":"B3","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1023\/A:1008328630915","article-title":"The connectionist inductive learning and logic programming system","volume":"11","author":"Avila Garcez","year":"1999","journal-title":"Appl. Intellig"},{"key":"B4","doi-asserted-by":"publisher","first-page":"2481","DOI":"10.1109\/TPAMI.2016.2644615","article-title":"Segnet: a deep convolutional encoder-decoder architecture for image segmentation","volume":"39","author":"Badrinarayanan","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2021\/592","article-title":"A comprehensive survey on hardware-aware neural architecture search","author":"Benmeziane","year":"2021","journal-title":"arXiv"},{"key":"B6","first-page":"115","article-title":"Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures","volume-title":"International Conference on Machine Learning","author":"Bergstra","year":"2013"},{"key":"B7","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1057\/palgrave.jors.2600425","article-title":"Nonlinear programming","volume":"48","author":"Bertsekas","year":"1997","journal-title":"J. Operat. Res. Soc"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-4012","article-title":"On the properties of neural machine translation: encoder-decoder approaches","author":"Cho","year":"2014","journal-title":"arXiv"},{"key":"B9","doi-asserted-by":"publisher","first-page":"3965","DOI":"10.48550\/arXiv.2106.04803","article-title":"Coatnet: Marrying convolution and attention for all data sizes","volume":"34","author":"Dai","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B10","first-page":"2462","article-title":"Problog: A probabilistic prolog and its application in link discovery","volume-title":"IJCAI 2007, the 20th International Joint Conference on Artificial Intelligence","author":"De Raedt","year":"2007"},{"key":"B11","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1561\/2000000039","article-title":"Deep learning: methods and applications","volume":"7","author":"Deng","year":"2014","journal-title":"Foundat. Trends Signal Proc."},{"key":"B12","first-page":"517","article-title":"DPP-Net: Device-aware progressive search for pareto-optimal neural architectures","author":"Dong","year":"2018","journal-title":"The European Conference on Computer Vision (ECCV)"},{"key":"B13","article-title":"Efficient multi-objective neural architecture search via lamarckian evolution","author":"Elsken","year":"2018"},{"key":"B14","doi-asserted-by":"publisher","first-page":"1997","DOI":"10.1007\/978-3-030-05318-5_3","article-title":"Neural architecture search: a survey","volume":"20","author":"Elsken","year":"2019","journal-title":"J. Mach. Learn. Res"},{"key":"B15","doi-asserted-by":"publisher","first-page":"4977","DOI":"10.48550\/arXiv.1905.12107","article-title":"Sparse: Sparse architecture search for CNNs on resource-constrained microcontrollers","volume":"32","author":"Fedorov","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B16","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-030-05318-5_1","article-title":"Hyperparameter optimization","volume-title":"Automated Machine Learning","author":"Feurer","year":"2019"},{"key":"B17","article-title":"Efficient and robust automated machine learning","volume-title":"Advances in Neural Information Processing Systems 28 (NIPS 2015)","author":"Feurer","year":"2015"},{"key":"B18","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1007\/s10994-021-06097-1","article-title":"Symbolic DNN-tuner","volume":"111","author":"Fraccaroli","year":"2021","journal-title":"Mach. Learn"},{"key":"B19","doi-asserted-by":"publisher","first-page":"100957","DOI":"10.1016\/j.softx.2021.100957","article-title":"Symbolic dnn-tuner: a python and problog-based system for optimizing deep neural networks hyperparameters","volume":"17","author":"Fraccaroli","year":"2022","journal-title":"SoftwareX"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1905.06088","article-title":"Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning","author":"Garcez","year":"2019"},{"key":"B21","first-page":"1638","article-title":"Squeezenext: Hardware-aware neural network design","volume-title":"The IEEE Conference on Computer Vision and Pattern Recognition Workshops","author":"Gholami","year":"2018"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1907.00909","article-title":"An open source AutoML benchmark","author":"Gijsbers","year":"2019","journal-title":"arXiv"},{"key":"B23","first-page":"1331","article-title":"Protonn: Compressed and accurate knn for resource-scarce devices","volume-title":"International Conference on Machine Learning","author":"Gupta","year":"2017"},{"key":"B24","first-page":"2961","article-title":"Mask R-CNN","volume-title":"The IEEE International Conference on Computer Vision","author":"He","year":"2017"},{"key":"B25","first-page":"5353","article-title":"Convolutional neural networks at constrained time cost","volume-title":"The IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2015"},{"key":"B26","first-page":"770","article-title":"Deep residual learning for image recognition","volume-title":"The IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"B27","doi-asserted-by":"publisher","first-page":"106622","DOI":"10.1016\/j.knosys.2020.106622","article-title":"Automl: A survey of the state-of-the-art","volume":"212","author":"He","year":"2021","journal-title":"Knowl.-Based Syst"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1704.04861","article-title":"Mobilenets: efficient convolutional neural networks for mobile vision applications","author":"Howard","year":"2017","journal-title":"arXiv"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1806.10332","article-title":"Monas: multi-objective neural architecture search using reinforcement learning","author":"Hsu","year":"2018","journal-title":"arXiv"},{"key":"B30","first-page":"2752","article-title":"Condensenet: an efficient densenet using learned group convolutions","volume-title":"The IEEE Conference on Computer Vision and Pattern Recognition","author":"Huang","year":"2018"},{"key":"B31","first-page":"4700","article-title":"Densely connected convolutional networks","volume-title":"The IEEE Conference on Computer Vision and Pattern Recognition","author":"Huang","year":"2017"},{"key":"B32","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1007\/978-1-4612-4380-9_35","article-title":"Robust estimation of a location parameter","volume-title":"Breakthroughs in Statistics","author":"Huber","year":"1992"},{"key":"B33","volume-title":"Multiple Objective Decision Making\u2013Methods and Applications: a State-of-the-Art Survey, Volume 164","author":"Hwang","year":"2012"},{"key":"B34","first-page":"1946","article-title":"Auto-Keras: An efficient neural architecture search system","author":"Jin","year":"2019","journal-title":"25th ACM SIGKDD International Conference on Knowledge Discovery"},{"key":"B35","first-page":"10848","article-title":"Training OOD detectors in their natural habitats","volume-title":"International Conference on Machine Learning","author":"Katz-Samuels","year":"2022"},{"key":"B36","first-page":"4","article-title":"Robo: A flexible and robust bayesian optimization framework in python","volume-title":"NIPS 2017 Bayesian Optimization Workshop","author":"Klein","year":"2017"},{"key":"B37","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1145\/3065386","article-title":"Imagenet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"B38","first-page":"1935","article-title":"Resource-efficient machine learning in 2 kb ram for the internet of things","volume-title":"International Conference on Machine Learning","author":"Kumar","year":"2017"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/679","article-title":"Graph neural networks meet neural-symbolic computing: a survey and perspective","author":"Lamb","year":"2020","journal-title":"arXiv"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1912.06059","article-title":"Grid search, random search, genetic algorithm: a big comparison for nas","author":"Liashchynskyi","year":"2019","journal-title":"arXiv"},{"key":"B41","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1145\/3437984.3458836","article-title":"\u03bcNAS: constrained neural architecture search for microcontrollers","volume-title":"The 1st Workshop on Machine Learning and Systems","author":"Liberis","year":"2021"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1910.05110","article-title":"Neural networks on microcontrollers: saving memory at inference via operator reordering","author":"Liberis","year":"2019","journal-title":"arXiv"},{"key":"B43","doi-asserted-by":"publisher","first-page":"11711","DOI":"10.48550\/arXiv.2007.10319","article-title":"Mcunet: Tiny deep learning on iot devices","volume":"33","author":"Lin","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B44","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2106.04554","article-title":"A survey of transformers","author":"Lin","year":"2021","journal-title":"arXiv"},{"key":"B45","first-page":"2980","article-title":"Focal loss for dense object detection","volume-title":"The IEEE International Conference on Computer Vision","author":"Lin","year":"2017"},{"key":"B46","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis","volume":"42","author":"Litjens","year":"2017","journal-title":"Med. Image Anal"},{"key":"B47","first-page":"19","article-title":"Progressive neural architecture search","volume-title":"The European Conference on Computer Vision (ECCV)","author":"Liu","year":"2018"},{"key":"B48","article-title":"Hierarchical representations for efficient architecture search","author":"Liu","year":"2017"},{"key":"B49","article-title":"Darts: differentiable architecture search","author":"Liu","year":"2018","journal-title":"arXiv"},{"key":"B50","volume-title":"Nsga-net: A Multi-Objective Genetic Algorithm for Neural Architecture Search","author":"Lu","year":"2018"},{"key":"B51","first-page":"116","article-title":"Shufflenet v2: practical guidelines for efficient cnn architecture design","volume-title":"The European Conference on Computer Vision (ECCV)","author":"Ma","year":"2018"},{"key":"B52","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1214\/aoms\/1177730491","article-title":"On a test of whether one of two random variables is stochastically larger than the other","volume":"18","author":"Mann","year":"1947","journal-title":"Ann. Mathem. Statist"},{"key":"B53","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1007\/BF02478259","article-title":"A logical calculus of the ideas immanent in nervous activity","volume":"5","author":"McCulloch","year":"1943","journal-title":"Bull. Math. Biophys"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1707.05589","article-title":"On the state of the art of evaluation in neural language models","author":"Melis","year":"2017","journal-title":"arXiv"},{"key":"B55","volume-title":"Nonlinear Multiobjective Optimization, Volume 12","author":"Miettinen","year":"1999"},{"key":"B56","first-page":"766","article-title":"A flexible framework for multi-objective bayesian optimization using random scalarizations","volume-title":"Uncertainty in Artificial Intelligence","author":"Paria","year":"2020"},{"key":"B57","first-page":"779","article-title":"You only look once: Unified, real-time object detection","volume-title":"The IEEE Conference on Computer Vision and Pattern Recognition","author":"Redmon","year":"2016"},{"key":"B58","doi-asserted-by":"crossref","DOI":"10.1201\/9781003338192","volume-title":"Foundations of Probabilistic Logic Programming: Languages, Semantics, Inference and Learning","author":"Riguzzi","year":"2022"},{"key":"B59","doi-asserted-by":"publisher","first-page":"4609","DOI":"10.1016\/j.eswa.2011.08.130","article-title":"Hybrid expert systems: a survey of current approaches and applications","volume":"39","author":"Sahin","year":"2012","journal-title":"Expert Syst. Appl"},{"key":"B60","first-page":"4510","article-title":"Mobilenetv2: Inverted residuals and linear bottlenecks","volume-title":"The IEEE Conference on Computer Vision and Pattern Recognition","author":"Sandler","year":"2018"},{"key":"B61","first-page":"34","article-title":"Constrained optimization to train neural networks on critical and under-represented classes","volume-title":"Advances in Neural Information Processing Systems","author":"Sangalli","year":"2021"},{"key":"B62","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1409.1556","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan","year":"2014","journal-title":"arXiv"},{"key":"B63","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/0004-3702(90)90007-M","article-title":"Tensor product variable binding and the representation of symbolic structures in connectionist systems","volume":"46","author":"Smolensky","year":"1990","journal-title":"Artif. Intell"},{"key":"B64","first-page":"25","article-title":"Practical bayesian optimization of machine learning algorithms","volume-title":"Advances in Neural Information Processing Systems","author":"Snoek","year":"2012"},{"key":"B65","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.09586","article-title":"Understanding LSTM-a tutorial into long short-term memory recurrent neural networks","author":"Staudemeyer","year":"2019","journal-title":"arXiv"},{"key":"B66","article-title":"Multi-task bayesian optimization","author":"Swersky","year":"2013","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B67","first-page":"1","article-title":"Going deeper with convolutions","volume-title":"The IEEE Conference on Computer Vision and Pattern Recognition","author":"Szegedy","year":"2015"},{"key":"B68","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3439730","article-title":"Automated design of deep neural networks: a survey and unified taxonomy","volume":"54","author":"Talbi","year":"2021","journal-title":"ACM Comp. Surv. (CSUR)"},{"key":"B69","first-page":"2820","article-title":"MNASNET: Platform-aware neural architecture search for mobile","volume-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Tan","year":"2019"},{"key":"B70","first-page":"6105","article-title":"Efficientnet: Rethinking model scaling for convolutional neural networks","volume-title":"International Conference on Machine Learning","author":"Tan","year":"2019"},{"key":"B71","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1452","article-title":"BERT rediscovers the classical NLP pipeline","author":"Tenney","year":"2019","journal-title":"arXiv"},{"key":"B72","doi-asserted-by":"crossref","first-page":"847","DOI":"10.1145\/2487575.2487629","article-title":"Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms","volume-title":"19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Thornton","year":"2013"},{"key":"B73","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4020-6710-5_3","volume-title":"Computing Machinery and Intelligence","author":"Turing","year":"2009"},{"key":"B74","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1007\/978-3-030-05318-5_2","article-title":"Meta-learning","volume-title":"Automated Machine Learning","author":"Vanschoren","year":"2019"},{"key":"B75","first-page":"3492","article-title":"Learning time\/memory-efficient deep architectures with budgeted super networks","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition","author":"Veniat","year":"2018"},{"key":"B76","first-page":"1","article-title":"Evolving deep convolutional neural networks by variable-length particle swarm optimization for image classification","volume-title":"2018 IEEE Congress on Evolutionary Computation (CEC)","author":"Wang","year":"2018"},{"key":"B77","first-page":"5502","article-title":"A semantic loss function for deep learning with symbolic knowledge","author":"Xu","year":"2018"},{"key":"B78","first-page":"6848","article-title":"Shufflenet: an extremely efficient convolutional neural network for mobile devices","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition","author":"Zhang","year":"2018"},{"key":"B79","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1711.07128","article-title":"Hello edge: keyword spotting on microcontrollers","author":"Zhang","year":"2017","journal-title":"arXiv"},{"key":"B80","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1806.07912","article-title":"Resource-efficient neural architect","author":"Zhou","year":"2018"},{"key":"B81","first-page":"8697","article-title":"Learning transferable architectures for scalable image recognition","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition","author":"Zoph","year":"2018"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2024.1414707\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,19]],"date-time":"2024-06-19T05:20:48Z","timestamp":1718774448000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2024.1414707\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,19]]},"references-count":81,"alternative-id":["10.3389\/frai.2024.1414707"],"URL":"https:\/\/doi.org\/10.3389\/frai.2024.1414707","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,19]]},"article-number":"1414707"}}