{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T06:06:35Z","timestamp":1791525995201,"version":"4.3.4"},"reference-count":77,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2017,5,24]],"date-time":"2017-05-24T00:00:00Z","timestamp":1495584000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Recent advances in Convolutional Neural Networks (CNNs) have obtained promising results in difficult deep learning tasks. However, the success of a CNN depends on finding an architecture to fit a given problem. A hand-crafted architecture is a challenging, time-consuming process that requires expert knowledge and effort, due to a large number of architectural design choices. In this article, we present an efficient framework that automatically designs a high-performing CNN architecture for a given problem. In this framework, we introduce a new optimization objective function that combines the error rate and the information learnt by a set of feature maps using deconvolutional networks (deconvnet). The new objective function allows the hyperparameters of the CNN architecture to be optimized in a way that enhances the performance by guiding the CNN through better visualization of learnt features via deconvnet. The actual optimization of the objective function is carried out via the Nelder-Mead Method (NMM). Further, our new objective function results in much faster convergence towards a better architecture. The proposed framework has the ability to explore a CNN architecture\u2019s numerous design choices in an efficient way and also allows effective, distributed execution and synchronization via web services. Empirically, we demonstrate that the CNN architecture designed with our approach outperforms several existing approaches in terms of its error rate. Our results are also competitive with state-of-the-art results on the MNIST dataset and perform reasonably against the state-of-the-art results on CIFAR-10 and CIFAR-100 datasets. Our approach has a significant role in increasing the depth, reducing the size of strides, and constraining some convolutional layers not followed by pooling layers in order to find a CNN architecture that produces a high recognition performance.<\/jats:p>","DOI":"10.3390\/e19060242","type":"journal-article","created":{"date-parts":[[2017,5,24]],"date-time":"2017-05-24T07:29:42Z","timestamp":1495610982000},"page":"242","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":229,"title":["A Framework for Designing the Architectures of Deep Convolutional Neural Networks"],"prefix":"10.3390","volume":"19","author":[{"given":"Saleh","family":"Albelwi","sequence":"first","affiliation":[{"name":"Computer Science and Engineering Department, University of Bridgeport, Bridgeport, CT 06604, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ausif","family":"Mahmood","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering Department, University of Bridgeport, Bridgeport, CT 06604, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2017,5,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1915","DOI":"10.1109\/TPAMI.2012.231","article-title":"Learning hierarchical features for scene labeling","volume":"35","author":"Farabet","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","unstructured":"Simonyan, K., and Zisserman, A. (arXiv, 2014). Very deep convolutional networks for large-scale image recognition, arXiv."},{"key":"ref_3","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). Imagenet classification with deep convolutional neural networks. Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kalchbrenner, N., Grefenstette, E., and Blunsom, P. (arXiv, 2014). A convolutional neural network for modelling sentences, arXiv.","DOI":"10.3115\/v1\/P14-1062"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Kim, Y. (arXiv, 2014). Convolutional neural networks for sentence classification, arXiv.","DOI":"10.3115\/v1\/D14-1181"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Conneau, A., Schwenk, H., LeCun, Y., and Barrault, L. (arXiv, 2016). Very deep convolutional networks for text classification, arXiv.","DOI":"10.18653\/v1\/E17-1104"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1113\/jphysiol.1968.sp008455","article-title":"Receptive fields and functional architecture of monkey striate cortex","volume":"195","author":"Hubel","year":"1968","journal-title":"J. Physiol."},{"key":"ref_8","unstructured":"Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. (arXiv, 2013). Overfeat: Integrated recognition, localization and detection using convolutional networks, arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/2200000006","article-title":"Learning deep architectures for AI","volume":"2","author":"Bengio","year":"2009","journal-title":"Found. Trends Mach. Learn."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_11","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_12","unstructured":"Liu, Y., Racah, E., Correa, J., Khosrowshahi, A., Lavers, D., Kunkel, K., Wehner, M., and Collins, W. (arXiv, 2016). Application of deep convolutional neural networks for detecting extreme weather in climate datasets, arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2014, January 6\u201312). Visualizing and understanding convolutional networks. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (arXiv, 2015). Deep residual learning for image recognition, arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_16","unstructured":"Srivastava, R.K., Greff, K., and Schmidhuber, J. (2015, January 7\u201312). Training very deep networks. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"He, K., and Sun, J. (2015, January 7\u201312). Convolutional neural networks at constrained time cost. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299173"},{"key":"ref_18","unstructured":"Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., and Wang, G. (arXiv, 2015). Recent advances in convolutional neural networks, arXiv."},{"key":"ref_19","unstructured":"De Andrade, A. (2014). Best Practices for Convolutional Neural Networks Applied to Object Recognition in Images, University of Toronto. Technical Report."},{"key":"ref_20","unstructured":"Zheng, A.X., and Bilenko, M. (2013, January 3\u20139). Lazy paired hyper-parameter tuning. Proceedings of the 23rd International Joint Conference on Artificial Intelligence, Beijing, China."},{"key":"ref_21","unstructured":"Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (arXiv, 2016). Hyperband: A novel bandit-based approach to hyperparameter optimization, arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Young, S.R., Rose, D.C., Karnowski, T.P., Lim, S.-H., and Patton, R.M. (2015, January 15\u201320). Optimizing deep learning hyper-parameters through an evolutionary algorithm. Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, Austin, TX, USA.","DOI":"10.1145\/2834892.2834896"},{"key":"ref_23","unstructured":"Bergstra, J.S., Bardenet, R., Bengio, Y., and K\u00e9gl, B. (2011, January 12\u201315). Algorithms for hyper-parameter optimization. Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain."},{"key":"ref_24","unstructured":"Snoek, J., Larochelle, H., and Adams, R.P. (2012, January 3\u20136). Practical bayesian optimization of machine learning algorithms. Proceedings of the 25th International Conference on Neural Information Processing System, Lake Tahoe, NV, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Wang, B., Pan, H., and Du, H. (2017). Motion sequence decomposition-based hybrid entropy feature and its application to fault diagnosis of a high-speed automatic mechanism. Entropy, 19.","DOI":"10.3390\/e19030086"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Albelwi, S., and Mahmood, A. (2016, January 18\u201320). Automated optimal architecture of deep convolutional neural networks for image recognition. Proceedings of the IEEE International Conference on Machine Learning and Applications, Anaheim, CA, USA.","DOI":"10.1109\/ICMLA.2016.0018"},{"key":"ref_27","unstructured":"Kohavi, R. (1995, January 20\u201325). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence, Montreal, QC, Canada."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Schaer, R., M\u00fcller, H., and Depeursinge, A. (2016). Optimized distributed hyperparameter search and simulation for lung texture classification in CT using hadoop. J. Imaging, 2.","DOI":"10.3390\/jimaging2020019"},{"key":"ref_29","first-page":"281","article-title":"Random search for hyper-parameter optimization","volume":"13","author":"Bergstra","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2011, January 17\u201321). Sequential model-based optimization for general algorithm configuration. Proceedings of the 5th International Conference on Learning and Intelligent Optimization, Rome, Italy.","DOI":"10.1007\/978-3-642-25566-3_40"},{"key":"ref_31","unstructured":"Murray, I., and Adams, R.P. (2010, January 6\u20139). Slice sampling covariance hyperparameters of latent gaussian models. Proceedings of the 24th Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_32","unstructured":"Gelbart, M.A. (2015). Constrained Bayesian Optimization and Applications. [Ph.D. Thesis, Harvard University]."},{"key":"ref_33","unstructured":"Loshchilov, I., and Hutter, F. (arXiv, 2016). CMA-ES for hyperparameter optimization of deep neural networks, arXiv."},{"key":"ref_34","unstructured":"Luketina, J., Berglund, M., Greff, K., and Raiko, C.T. (arXiv, 2015). Scalable gradient-based tuning of continuous regularization hyperparameters, arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/0885-2308(87)90009-X","article-title":"An adaptive training algorithm for back propagation networks","volume":"2","author":"Chan","year":"1987","journal-title":"Comput. Speech Lang."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Larsen, J., Svarer, C., Andersen, L.N., and Hansen, L.K. (1998). Adaptive Regularization in Neural Network Modeling. Neural Networks: Tricks of the Trade, Springer.","DOI":"10.1007\/3-540-49430-8_6"},{"key":"ref_37","unstructured":"Pedregosa, F. (arXiv, 2016). Hyperparameter optimization with approximate gradient, arXiv."},{"key":"ref_38","unstructured":"Yu, C., and Liu, B. (2002, January 12\u201317). A backpropagation algorithm with adaptive learning rate and momentum coefficient. Proceedings of the 2002 International Joint Conference on Neural Networks, Piscataway, NJ, USA."},{"key":"ref_39","unstructured":"Zeiler, M.D. (arXiv, 2012). Adadelta: An adaptive learning rate method, arXiv."},{"key":"ref_40","unstructured":"Caruana, R., Lawrence, S., and Giles, L. (2001, January 3\u20138). Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Proceedings of the 2001 Neural Information Processing Systems Conference, Vancouver, BC, Canada."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Graves, A., Mohamed, A., and Hinton, G. (2013, January 26\u201331). Speech recognition with deep recurrent neural networks. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.","DOI":"10.1109\/ICASSP.2013.6638947"},{"key":"ref_42","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the International Conference on Artificial Intelligence and Statistics, Sardinia, Italy."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1155\/2015\/369298","article-title":"Designing artificial neural networks using particle swarm optimization algorithms","volume":"2015","author":"Garro","year":"2015","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"458","DOI":"10.2166\/hydro.2010.032","article-title":"A hybrid model coupled with singular spectrum analysis for daily rainfall prediction","volume":"12","author":"Chau","year":"2010","journal-title":"J. Hydroinform."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"2655","DOI":"10.1007\/s11269-015-0962-6","article-title":"Improving forecasting accuracy of annual runoff time series using arima based on eemd decomposition","volume":"29","author":"Wang","year":"2015","journal-title":"Water Resour. Manag."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1617","DOI":"10.1016\/j.jhydrol.2015.08.022","article-title":"Data-driven input variable selection for rainfall\u2013runoff modeling using binary-coded particle swarm optimization and extreme learning machines","volume":"529","author":"Taormina","year":"2015","journal-title":"J. Hydrol."},{"key":"ref_47","first-page":"840","article-title":"Multilayer ensemble pruning via novel multi-sub-swarm particle swarm optimization","volume":"15","author":"Zhang","year":"2009","journal-title":"J. UCS"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Kulkarni, P., Zepeda, J., Jurie, F., P\u00e9rez, P., and Chevallier, L. (2015, January 7\u201310). Learning the structure of deep architectures using L1 regularization. Proceedings of the British Machine Vision Conference, Swansea, UK.","DOI":"10.5244\/C.29.23"},{"key":"ref_49","unstructured":"Zoph, B., and Le, Q.V. (arXiv, 2016). Neural architecture search with reinforcement learning, arXiv."},{"key":"ref_50","unstructured":"Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Navruzyan, A., Duffy, N., and Hodjat, B. (arXiv, 2017). Evolving deep neural networks, arXiv."},{"key":"ref_51","unstructured":"Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Le, Q., and Kurakin, A. (arXiv, 2017). Large-scale evolution of image classifiers, arXiv."},{"key":"ref_52","unstructured":"Baker, B., Gupta, O., Naik, N., and Raskar, R. (arXiv, 2016). Designing neural network architectures using reinforcement learning, arXiv."},{"key":"ref_53","unstructured":"Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lipson, H. (arXiv, 2015). Understanding neural networks through deep visualization, arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"6007","DOI":"10.3390\/e17096007","article-title":"A gloss composition and context clustering based distributed word sense representation model","volume":"17","author":"Chen","year":"2015","journal-title":"Entropy"},{"key":"ref_56","unstructured":"Nair, V., and Hinton, G.E. (2010, January 21\u201324). Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Aldhaheri, A., and Lee, J. (2017, January 9\u201311). Event detection on large social media using temporal analysis. Proceedings of the Computing and Communication Workshop and Conference, Las Vegas, NV, USA.","DOI":"10.1109\/CCWC.2017.7868467"},{"key":"ref_58","unstructured":"Hijazi, S., Kumar, R., and Rowen, C. (2017, May 20). Using Convolutional Neural Networks for Image Recognition. Available online: https:\/\/ip.cadence.com\/uploads\/901\/cnn_wp-pdf."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1007\/s10462-010-9165-y","article-title":"A review of instance selection methods","volume":"34","author":"Kittler","year":"2010","journal-title":"Artif. Intell. Rev."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Albelwi, S., and Mahmood, A. (2016, January 29). Analysis of instance selection algorithms on large datasets with deep convolutional neural networks. Proceedings of the IEEE Long Island Systems, Applications and Technology Conference, Farmingdale, NY, USA.","DOI":"10.1109\/LISAT.2016.7494142"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Skalak, D.B. (1994, January 10\u201313). Prototype and feature selection by sampling and random mutation hill climbing algorithms. Proceedings of the 11th International Conference on Machine Learning, New Brunswick, NJ, USA.","DOI":"10.1016\/B978-1-55860-335-6.50043-X"},{"key":"ref_62","unstructured":"Karpathy, A., Johnson, J., and Li, F.F. (arXiv, 2015). Visualizing and understanding recurrent networks, arXiv."},{"key":"ref_63","unstructured":"Erhan, D., Bengio, Y., Courville, A., and Vincent, P. (2009). Visualizing Higher-Layer Features of a Deep Network, University of Montreal."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1002\/asi.10242","article-title":"Requirements for a cocitation similarity measure, with special reference to pearson\u2019s correlation coefficient","volume":"54","author":"Ahlgren","year":"2003","journal-title":"J. Am. Soc. Inf. Sci. Technol."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Dragomir, A., Post, A., Akay, Y.M., Jneid, H., Paniagua, D., Denktas, A., Bozkurt, B., and Akay, M. (2016). Acoustic detection of coronary occlusions before and after stent placement using an electronic stethoscope. Entropy, 18.","DOI":"10.3390\/e18080281"},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"3059","DOI":"10.1093\/nar\/gkf436","article-title":"Mafft: A novel method for rapid multiple sequence alignment based on fast fourier transform","volume":"30","author":"Katoh","year":"2002","journal-title":"Nucleic Acids Res."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1093\/comjnl\/7.4.308","article-title":"A simplex method for function minimization","volume":"7","author":"Nelder","year":"1965","journal-title":"Comput. J."},{"key":"ref_68","unstructured":"Erl, T. (2004). Service-Oriented Architecture. A Field Guide to Integrating XML and Web Services, Prentice Hall PTR."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"3806","DOI":"10.3390\/e17063806","article-title":"On the detection of fake certificates via attribute correlation","volume":"17","author":"Gu","year":"2015","journal-title":"Entropy"},{"key":"ref_70","doi-asserted-by":"crossref","unstructured":"Alshinina, R., and Elleithy, K. (2017). Performance and challenges of service-oriented architecture for wireless sensor networks. Sensors, 17.","DOI":"10.3390\/s17030536"},{"key":"ref_71","unstructured":"Fielding, R.T. (2000). Architectural Styles and the Design of Network-Based Software Architectures. [Ph.D. Thesis, University of California]."},{"key":"ref_72","unstructured":"Krizhevsky, A., and Hinton, G. (2017, May 20). Learning Multiple Layers of Features from Tiny Images. Available online: https:\/\/www.cs.toronto.edu\/~kriz\/learning-features-2009-TR.pdf."},{"key":"ref_73","unstructured":"Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (July, January 30). Theano: A CPU and GPU math compiler. Proceedings of the Python for Scientific Computing Conference, Austin, TX, USA."},{"key":"ref_74","unstructured":"Domhan, T., Springenberg, J.T., and Hutter, F. (2015, January 25\u201331). Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. Proceedings of the 24th International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina."},{"key":"ref_75","unstructured":"Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (arXiv, 2013). Maxout networks, arXiv."},{"key":"ref_76","unstructured":"Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., and Fergus, R. (2013, January 16\u201321). Regularization of neural networks using dropconnect. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA."},{"key":"ref_77","unstructured":"Lee, C.Y., Xie, S., Gallagher, P.W., Zhang, Z., and Tu, Z. (2015, January 9\u201312). Deeply-supervised nets. Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, San Diego, CA, USA."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/19\/6\/242\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:36:45Z","timestamp":1760207805000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/19\/6\/242"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,5,24]]},"references-count":77,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2017,6]]}},"alternative-id":["e19060242"],"URL":"https:\/\/doi.org\/10.3390\/e19060242","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,5,24]]}}}