{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,18]],"date-time":"2025-12-18T14:17:42Z","timestamp":1766067462828,"version":"build-2065373602"},"reference-count":55,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2021,11,28]],"date-time":"2021-11-28T00:00:00Z","timestamp":1638057600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000780","name":"European Commission","doi-asserted-by":"publisher","award":["European Regional Development Fund - Industrial Intensive Innovation Programme"],"award-info":[{"award-number":["European Regional Development Fund - Industrial Intensive Innovation Programme"]}],"id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Automated deep neural architecture generation has gained increasing attention. However, exiting studies either optimize important design choices, without taking advantage of modern strategies such as residual\/dense connections, or they optimize residual\/dense networks but reduce search space by eliminating fine-grained network setting choices. To address the aforementioned weaknesses, we propose a novel particle swarm optimization (PSO)-based deep architecture generation algorithm, to devise deep networks with residual connections, whilst performing a thorough search which optimizes important design choices. A PSO variant is proposed which incorporates a new encoding scheme and a new search mechanism guided by non-uniformly randomly selected neighboring and global promising solutions for the search of optimal architectures. Specifically, the proposed encoding scheme is able to describe convolutional neural network architecture configurations with residual connections. Evaluated using benchmark datasets, the proposed model outperforms existing state-of-the-art methods for architecture generation. Owing to the guidance of diverse non-uniformly selected neighboring promising solutions in combination with the swarm leader at fine-grained and global levels, the proposed model produces a rich assortment of residual architectures with great diversity. Our devised networks show better capabilities in tackling vanishing gradients with up to 4.34% improvement of mean accuracy in comparison with those of existing studies.<\/jats:p>","DOI":"10.3390\/s21237936","type":"journal-article","created":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T01:45:02Z","timestamp":1638323102000},"page":"7936","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Evolving Deep Architecture Generation with Residual Connections for Image Classification Using Particle Swarm Optimization"],"prefix":"10.3390","volume":"21","author":[{"given":"Tom","family":"Lawrence","sequence":"first","affiliation":[{"name":"Department of Computer and Information Sciences, Faculty of Engineering and Environment, Northumbria University, Newcastle upon Tyne NE1 8ST, UK"}]},{"given":"Li","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Royal Holloway, University of London, Egham TW20 0EX, UK"}]},{"given":"Kay","family":"Rogage","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Sciences, Faculty of Engineering and Environment, Northumbria University, Newcastle upon Tyne NE1 8ST, UK"}]},{"given":"Chee Peng","family":"Lim","sequence":"additional","affiliation":[{"name":"Institute for Intelligent Systems Research and Innovation, Deakin University, Waurn Ponds, VIC 3216, Australia"}]}],"member":"1968","published-online":{"date-parts":[[2021,11,28]]},"reference":[{"key":"ref_1","unstructured":"Luo, W., Li, Y., Urtasun, R., and Zemel, R. (2016, January 5\u201310). Understanding the effective receptive field in deep convolutional neural networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_2","first-page":"1","article-title":"Wide Residual Networks","volume":"87","author":"Zagoruyko","year":"2016","journal-title":"British Machine Vision Conference (BMVC)"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014). Microsoft coco: Common objects in context. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_8","unstructured":"Krizhevsky, A., and Hinton, G. (2021, November 15). Learning Multiple Layers of Features from Tiny Images. Available online: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.222.9220&rep=rep1&type=pdf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"10767","DOI":"10.1109\/ACCESS.2019.2891838","article-title":"Effective Combination of DenseNet and BiLSTM for Keyword Spotting","volume":"7","author":"Zeng","year":"2019","journal-title":"IEEE Access"},{"key":"ref_10","first-page":"233","article-title":"Medical image retrieval using Resnet-18","volume":"Volume 10954","author":"Chen","year":"2019","journal-title":"Medical Imaging 2019: Imaging Informatics for Healthcare, Research, and Applications"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40537-019-0197-0","article-title":"A survey on image data augmentation for deep learning","volume":"6","author":"Shorten","year":"2019","journal-title":"J. Big Data"},{"key":"ref_12","unstructured":"Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. (2016). Pruning convolutional neural networks for resource efficient inference. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Rezende, E., Ruppert, G., Carvalho, T., Ramos, F., and De Geus, P. (2017, January 18\u201321). Malicious software classification using transfer learning of resnet-50 deep neural network. Proceedings of the 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), Cancun, Mexico.","DOI":"10.1109\/ICMLA.2017.00-19"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"107537","DOI":"10.1016\/j.asoc.2021.107537","article-title":"Evolving convolutional neural networks by symbiotic organisms search algorithm for image classification","volume":"109","author":"Miao","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.swevo.2019.05.010","article-title":"Particle swarm optimization of deep neural networks architectures for image classification","volume":"49","author":"Junior","year":"2019","journal-title":"Swarm Evol. Comput."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Xie, L., and Yuille, A. (2017, January 22\u201329). Genetic CNN. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.154"},{"key":"ref_17","unstructured":"Kennedy, J., and Eberhart, R. (December, January 27). Particle swarm optimization. Proceedings of the ICNN\u201995\u2014International Conference on Neural Networks, Perth, Australia."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Caruana, R., Lawrence, S., and Giles, C.L. (2001). Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Proceedings of the 13th International Conference on Neural Information Processing Systems, MIT Press.","DOI":"10.1109\/IJCNN.2000.857823"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Fielding, B., and Zhang, L. (2020). Evolving Deep DenseBlock Architecture Ensembles for Image Classification. Electronics, 9.","DOI":"10.3390\/electronics9111880"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wang, B., Xue, B., and Zhang, M. (2020, January 19\u201324). Particle Swarm optimisation for Evolving Deep Neural Networks for Image Classification by Evolving and Stacking Transferable Blocks. Proceedings of the 2020 IEEE Congress on Evolutionary Computation (CEC), Glasgow, UK.","DOI":"10.1109\/CEC48606.2020.9185541"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wang, B., Sun, Y., Xue, B., and Zhang, M. (2018, January 8\u201313). Evolving deep convolutional neural networks by variable-length particle swarm optimization for image classification. Proceedings of the IEEE Congress on Evolutionary Computation (CEC), Rio de Janeiro, Brazil.","DOI":"10.1109\/CEC.2018.8477735"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wang, B., Sun, Y., Xue, B., and Zhang, M. (2019, January 13\u201317). Evolving Deep Neural Networks by Multi-Objective Particle Swarm Optimization for Image Classification. Proceedings of the Genetic and Evolutionary Computation Conference, GECCO\u201919, Prague, Czech Republic.","DOI":"10.1145\/3321707.3321735"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"107976","DOI":"10.1016\/j.asoc.2021.107976","article-title":"Quantum fractional order darwinian particle swarm optimization for hyperspectral multi-level image thresholding","volume":"2021","author":"Dutta","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Szwarcman, D., Civitarese, D., and Vellasco, M. (2019, January 14\u201319). Quantum-Inspired Neural Architecture Search. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.","DOI":"10.1109\/IJCNN.2019.8852453"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"106918","DOI":"10.1016\/j.knosys.2021.106918","article-title":"Intelligent human action recognition using an ensemble model of evolving deep networks with swarm-based optimization","volume":"220","author":"Zhang","year":"2021","journal-title":"Knowl.-Based Syst."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Liu, X., Zhang, C., Cai, Z., Yang, J., Zhou, Z., and Gong, X. (2021). Continuous Particle Swarm Optimization-Based Deep Learning Architecture Search for Hyperspectral Image Classification. Remote Sens., 13.","DOI":"10.3390\/rs13061082"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1142\/9789813270237_0009","article-title":"Optimization of recurrent neural networks using evolutionary group-based particle swarm optimization for hexapod robot gait generation","volume":"84","author":"Juang","year":"2018","journal-title":"Hybrid Metaheuristics Res. Appl."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"105725","DOI":"10.1016\/j.asoc.2019.105725","article-title":"Intelligent skin cancer diagnosis using improved particle swarm optimization and deep learning models","volume":"84","author":"Tan","year":"2019","journal-title":"Appl. Soft Comput."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"106328","DOI":"10.1016\/j.asoc.2020.106328","article-title":"Intelligent optic disc segmentation using improved particle swarm optimization and evolving ensemble models","volume":"92","author":"Zhang","year":"2020","journal-title":"Appl. Soft Comput."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"104807","DOI":"10.1016\/j.knosys.2019.06.015","article-title":"Adaptive melanoma diagnosis using evolving clustering, ensemble and deep neural networks","volume":"187","author":"Tan","year":"2020","journal-title":"Knowl.-Based Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1016\/j.future.2021.03.022","article-title":"High-quality face image generation using particle swarm optimization-based generative adversarial networks","volume":"122","author":"Zhang","year":"2021","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1016\/j.compstruc.2014.03.007","article-title":"Symbiotic Organisms Search: A new metaheuristic optimization algorithm","volume":"139","author":"Cheng","year":"2014","journal-title":"Comput. Struct."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1995","DOI":"10.1007\/s00521-015-1923-y","article-title":"Monarch butterfly optimization","volume":"31","author":"Wang","year":"2019","journal-title":"Neural Comput. Appl."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Bacanin, N., Bezdan, T., Tuba, E., Strumberger, I., and Tuba, M. (2020). Monarch Butterfly Optimization Based Convolutional Neural Network Design. Mathematics, 8.","DOI":"10.3390\/math8060936"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Madureira, A.M., Abraham, A., Gandhi, N., and Varela, M.L. (2020). Modified and Hybridized Monarch Butterfly Algorithms for Multi-Objective Optimization. Hybrid Intelligent Systems, Springer.","DOI":"10.1007\/978-3-030-14347-3"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"137","DOI":"10.24846\/v21i2y201203","article-title":"Artificial Bee Colony (ABC) Algorithm for Constrained Optimization Improved with Genetic Operators","volume":"21","author":"Bacanin","year":"2012","journal-title":"Stud. Inform. Control"},{"key":"ref_38","unstructured":"Yang, X. (2010). Firefly Algorithm, Nature Inspired Metaheuristic Algorithms, Luniver Press."},{"key":"ref_39","unstructured":"Chen, D., Li, X., and Li, S. (2021). A Novel Convolutional Neural Network Model Based on Beetle Antennae Search Optimization Algorithm for Computerized Tomography Diagnosis. IEEE Trans. Neural Netw. Learn. Syst., 1\u201312."},{"key":"ref_40","unstructured":"Wang, J., and Chen, H. (2018). BSAS: Beetle swarm antennae search algorithm for optimization problems. arXiv."},{"key":"ref_41","unstructured":"Lee, C.H., Lai, W.Y., and Lin, Y.C. (2003, January 9\u201312). A TSK-type fuzzy neural network (TFNN) systems for dynamic systems identification. Proceedings of the 42nd IEEE International Conference on Decision and Control (IEEE Cat. No. 03CH37475), Maui, HI, USA."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2289","DOI":"10.1109\/TMI.2020.2968472","article-title":"SACNN: Self-attention convolutional neural network for low-dose CT denoising with self-supervised perceptual loss network","volume":"39","author":"Li","year":"2020","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"13051","DOI":"10.1007\/s00521-020-04749-2","article-title":"Evolving deep neural networks using coevolutionary algorithms with multi-population strategy","volume":"32","author":"Tirumala","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_44","first-page":"459","article-title":"Self-adaptive 2D-3D ensemble of fully convolutional networks for medical image segmentation","volume":"Volume 11313","author":"Landman","year":"2020","journal-title":"Medical Imaging 2020: Image Processing"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1109\/TEVC.2007.892759","article-title":"MOEA\/D: A Multiobjective Evolutionary Algorithm Based on Decomposition","volume":"11","author":"Zhang","year":"2007","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/j.media.2013.12.002","article-title":"Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challenge","volume":"18","author":"Litjens","year":"2014","journal-title":"Med. Image Anal."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"100650","DOI":"10.1016\/j.swevo.2020.100650","article-title":"Evolutionary LSTM-FCN networks for pattern classification in industrial processes","volume":"54","author":"Ortego","year":"2020","journal-title":"Swarm Evol. Comput."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"161519","DOI":"10.1109\/ACCESS.2020.3021527","article-title":"Evolving CNN-LSTM models for time series prediction using enhanced grey wolf optimizer","volume":"8","author":"Xie","year":"2020","journal-title":"IEEE Access"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","article-title":"Array programming with NumPy","volume":"585","author":"Harris","year":"2020","journal-title":"Nature"},{"key":"ref_50","unstructured":"Kingma, D., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations, Banff National Park."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"14369","DOI":"10.1109\/ACCESS.2021.3052489","article-title":"Particle Swarm Optimization for Automatically Evolving Convolutional Neural Networks for Image Classification","volume":"9","author":"Lawrence","year":"2021","journal-title":"IEEE Access"},{"key":"ref_52","unstructured":"LeCun, Y., Cortes, C., and Burges, C. (2021, November 21). The MNIST Database. Available online: http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Larochelle, H., Erhan, D., Courville, A., Bergstra, J., and Bengio, Y. (2007, January 20\u201324). An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation. Proceedings of the 24th International Conference on MACHINE Learning, ICML\u201907, Corvalis, OR, USA.","DOI":"10.1145\/1273496.1273556"},{"key":"ref_54","unstructured":"Larochelle, H., Erhan, D., and Courville, A. (2021, November 21). icml2007data. Available online: http:\/\/www.iro.umontreal.ca\/~lisa\/icml2007data\/."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1016\/j.neucom.2017.07.014","article-title":"A region-based image caption generator with refined descriptions","volume":"272","author":"Kinghorn","year":"2018","journal-title":"Neurocomputing"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/7936\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:36:54Z","timestamp":1760168214000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/7936"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,28]]},"references-count":55,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["s21237936"],"URL":"https:\/\/doi.org\/10.3390\/s21237936","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,11,28]]}}}