{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,16]],"date-time":"2025-05-16T05:24:33Z","timestamp":1747373073111},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,11,16]],"date-time":"2021-11-16T00:00:00Z","timestamp":1637020800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,16]],"date-time":"2021-11-16T00:00:00Z","timestamp":1637020800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Discov Artif Intell"],"published-print":{"date-parts":[[2021,12]]},"abstract":"Abstract<\/jats:title>This paper introduces two deep convolutional neural network training techniques that lead to more robust feature subspace separation in comparison to traditional training. Assume that dataset hasM<\/jats:italic>labels. The first method createsM<\/jats:italic>deep convolutional neural networks called$$\\{\\text {DCNN}_i\\}_{i=1}^{M}$$<\/jats:tex-math>{<\/mml:mo>DCNN<\/mml:mtext>i<\/mml:mi><\/mml:msub>}<\/mml:mo><\/mml:mrow>i<\/mml:mi>=<\/mml:mo>1<\/mml:mn><\/mml:mrow>M<\/mml:mi><\/mml:msubsup><\/mml:math><\/jats:alternatives><\/jats:inline-formula>. Each of the networks$$\\text {DCNN}_i$$<\/jats:tex-math>DCNN<\/mml:mtext>i<\/mml:mi><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>is composed of a convolutional neural network ($$\\text {CNN}_i$$<\/jats:tex-math>CNN<\/mml:mtext>i<\/mml:mi><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>) and a fully connected neural network ($$\\text {FCNN}_i$$<\/jats:tex-math>FCNN<\/mml:mtext>i<\/mml:mi><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>). In training, a set of projection matrices$$\\{\\mathbf {P}_i\\}_{i=1}^M$$<\/jats:tex-math>{<\/mml:mo>P<\/mml:mi>i<\/mml:mi><\/mml:msub>}<\/mml:mo><\/mml:mrow>i<\/mml:mi>=<\/mml:mo>1<\/mml:mn><\/mml:mrow>M<\/mml:mi><\/mml:msubsup><\/mml:math><\/jats:alternatives><\/jats:inline-formula>are created and adaptively updated as representations for feature subspaces$$\\{\\mathcal {S}_i\\}_{i=1}^M$$<\/jats:tex-math>{<\/mml:mo>S<\/mml:mi>i<\/mml:mi><\/mml:msub>}<\/mml:mo><\/mml:mrow>i<\/mml:mi>=<\/mml:mo>1<\/mml:mn><\/mml:mrow>M<\/mml:mi><\/mml:msubsup><\/mml:math><\/jats:alternatives><\/jats:inline-formula>. A rejection value is computed for each training based on its projections on feature subspaces. Each$$\\text {FCNN}_i$$<\/jats:tex-math>FCNN<\/mml:mtext>i<\/mml:mi><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>acts as a binary classifier with a cost function whose main parameter is rejection values. A threshold value$$t_i$$<\/jats:tex-math>t<\/mml:mi>i<\/mml:mi><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>is determined for$$i^{th}$$<\/jats:tex-math>i<\/mml:mi>th<\/mml:mi><\/mml:mrow><\/mml:msup><\/mml:math><\/jats:alternatives><\/jats:inline-formula>network$$\\text {DCNN}_i$$<\/jats:tex-math>DCNN<\/mml:mtext>i<\/mml:mi><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>. A testing strategy utilizing$$\\{t_i\\}_{i=1}^M$$<\/jats:tex-math>{<\/mml:mo>t<\/mml:mi>i<\/mml:mi><\/mml:msub>}<\/mml:mo><\/mml:mrow>i<\/mml:mi>=<\/mml:mo>1<\/mml:mn><\/mml:mrow>M<\/mml:mi><\/mml:msubsup><\/mml:math><\/jats:alternatives><\/jats:inline-formula>is also introduced. The second method creates a single DCNN and it computes a cost function whose parameters depend on subspace separations using the geodesic distance on the Grasmannian manifold of subspaces$$\\mathcal {S}_i$$<\/jats:tex-math>S<\/mml:mi>i<\/mml:mi><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>and the sum of all remaining subspaces$$\\{\\mathcal {S}_j\\}_{j=1,j\\ne i}^M$$<\/jats:tex-math>{<\/mml:mo>S<\/mml:mi>j<\/mml:mi><\/mml:msub>}<\/mml:mo><\/mml:mrow>j<\/mml:mi>=<\/mml:mo>1<\/mml:mn>,<\/mml:mo>j<\/mml:mi>\u2260<\/mml:mo>i<\/mml:mi><\/mml:mrow>M<\/mml:mi><\/mml:msubsup><\/mml:math><\/jats:alternatives><\/jats:inline-formula>. The proposed methods are tested using multiple network topologies. It is shown that while the first method works better for smaller networks, the second method performs better for complex architectures.<\/jats:p>","DOI":"10.1007\/s44163-021-00013-1","type":"journal-article","created":{"date-parts":[[2021,11,16]],"date-time":"2021-11-16T11:06:36Z","timestamp":1637060796000},"update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Robust feature space separation for deep convolutional neural network training"],"prefix":"10.1007","volume":"1","author":[{"given":"Ali","family":"Sekmen","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mustafa","family":"Parlaktuna","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ayad","family":"Abdul-Malek","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Erdem","family":"Erdemir","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmet Bugra","family":"Koku","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,11,16]]},"reference":[{"key":"13_CR1","unstructured":"Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th international conference on neural information processing systems, vol 1. Curran Associates Inc.; 2012. p. 1097-1105."},{"key":"13_CR2","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: 2015 IEEE international conference on computer vision (ICCV).","DOI":"10.1109\/ICCV.2015.123"},{"key":"13_CR3","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). 2016. p. 770\u2013778.","DOI":"10.1109\/CVPR.2016.90"},{"key":"13_CR4","doi-asserted-by":"crossref","unstructured":"Litjens GJ, Kooi T, Bejnordi BE, Setio AA, Ciompi F, Ghafoorian M, van der Laak JA, van Ginneken B, S\u00e1nchez CI A survey on deep learning in medical image analysis. CoRR, arXiv:abs\/1702.05747. 2017.","DOI":"10.1016\/j.media.2017.07.005"},{"key":"13_CR5","doi-asserted-by":"crossref","unstructured":"Angelova A, Krizhevsky A, Vanhoucke V, Ogale A, Ferguson D. Real-time pedestrian detection with deep network cascades. In: Proceedings of BMVC 2015. 2015.","DOI":"10.5244\/C.29.32"},{"key":"13_CR6","doi-asserted-by":"crossref","unstructured":"Parkhi OM, Vedaldi A, Zisserman A. Deep face recognition. In: Proceedings of the British machine vision conference (BMVC). 2015.","DOI":"10.5244\/C.29.41"},{"key":"13_CR7","unstructured":"Young T, Hazarika D, Poria S, Cambria E. Recent trends in deep learning based natural language processing. CoRR, arXiv:abs\/1708.02709. 2017."},{"key":"13_CR8","unstructured":"Lample G, Charton F. Deep learning for symbolic mathematics. CoRR, arXiv:abs\/1912.01412. 2019."},{"issue":"7639","key":"13_CR9","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542(7639):115\u20138.","journal-title":"Nature"},{"issue":"22","key":"13_CR10","doi-asserted-by":"publisher","first-page":"2402","DOI":"10.1001\/jama.2016.17216","volume":"316","author":"V Gulshan","year":"2016","unstructured":"Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson PQ, Mega J, Webster D. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316(22):2402\u201310.","journal-title":"JAMA"},{"key":"13_CR11","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1038\/s41586-021-03819-2","volume":"596","author":"J Jumper","year":"2021","unstructured":"Jumper J, Evans R, Pritzel A, et al. Highly accurate protein structure prediction with alphafold. Nature. 2021;596:583\u20139.","journal-title":"Nature"},{"issue":"3","key":"13_CR12","doi-asserted-by":"publisher","first-page":"1496","DOI":"10.1073\/pnas.1914677117","volume":"117","author":"J Yang","year":"2020","unstructured":"Yang J, Anishchenko I, Park H, Peng Z, Ovchinnikov S, Baker D. Improved protein structure prediction using predicted interresidue orientations. Proc Natl Acad Sci. 2020;117(3):1496\u2013503.","journal-title":"Proc Natl Acad Sci"},{"issue":"2065","key":"13_CR13","first-page":"20150203","volume":"374","author":"M St\u00e9phane","year":"2016","unstructured":"St\u00e9phane M. Understanding deep convolutional networks. Philos Trans R Soc Lond A Math Phys Eng Sci. 2016;374(2065):20150203.","journal-title":"Philos Trans R Soc Lond A Math Phys Eng Sci"},{"key":"13_CR14","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1016\/j.neunet.2020.01.018","volume":"124","author":"D-X Zhou","year":"2020","unstructured":"Zhou D-X. Theory of deep convolutional neural networks: Downsampling. Neural Netw. 2020;124:319\u201327.","journal-title":"Neural Netw"},{"key":"13_CR15","doi-asserted-by":"crossref","unstructured":"Berner J, Grohs P, Kutyniok G, Petersen P. The modern mathematics of deep learning. CoRR, arXiv:abs\/2105.04026. 2021.","DOI":"10.1017\/9781009025096.002"},{"key":"13_CR16","unstructured":"Kanatani K, Matsunaga C. Estimating the number of independent motions for multibody motion segmentation. In: 5th Asian conference on computer vision. 2002. p. 7\u20139."},{"issue":"6","key":"13_CR17","doi-asserted-by":"publisher","first-page":"643","DOI":"10.1109\/34.927464","volume":"23","author":"AS Georghiades","year":"2001","unstructured":"Georghiades AS, Belhumeur PN, Kriegman DJ. From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell. 2001;23(6):643\u201360.","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"issue":"2","key":"13_CR18","doi-asserted-by":"publisher","first-page":"425","DOI":"10.1016\/j.acha.2017.08.006","volume":"45","author":"A Aldroubi","year":"2018","unstructured":"Aldroubi A, Sekmen A, Koku AB, Cakmak AF. Similarity matrix framework for data from union of subspaces. Appl Comput Harmon Anal. 2018;45(2):425\u201335.","journal-title":"Appl Comput Harmon Anal"},{"key":"13_CR19","doi-asserted-by":"publisher","first-page":"65","DOI":"10.3389\/fams.2018.00065","volume":"4","author":"A Aldroubi","year":"2019","unstructured":"Aldroubi A, Hamm K, Koku AB, Sekmen A. Cur decompositions, similarity matrices, and subspace clustering. Front Appl Math Stat. 2019;4:65.","journal-title":"Front Appl Math Stat"},{"issue":"10","key":"13_CR20","doi-asserted-by":"publisher","first-page":"704","DOI":"10.1109\/LSP.2012.2214211","volume":"19","author":"A Aldroubi","year":"2012","unstructured":"Aldroubi A, Sekmen A. Nearness to local subspace algorithm for subspace and motion segmentation. IEEE Signal Process Lett. 2012;19(10):704\u20137.","journal-title":"IEEE Signal Process Lett"},{"key":"13_CR21","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1109\/MSP.2010.939739","volume":"28","author":"R Vidal","year":"2010","unstructured":"Vidal R. A tutorial on subspace clustering. IEEE Signal Process Mag. 2010;28:52\u201368.","journal-title":"IEEE Signal Process Mag"},{"key":"13_CR22","doi-asserted-by":"publisher","first-page":"340","DOI":"10.1016\/j.neucom.2020.04.120","volume":"404","author":"Q Huang","year":"2020","unstructured":"Huang Q, Zhang Y, Peng H, Dan T, Weng W, Cai H. Deep subspace clustering to achieve jointly latent feature extraction and discriminative learning. Neurocomputing. 2020;404:340\u201350.","journal-title":"Neurocomputing"},{"key":"13_CR23","doi-asserted-by":"crossref","unstructured":"Lv J, Kang Z, Lu X, Xu Z. Pseudo-supervised deep subspace clustering. CoRR, arXiv:abs\/2104.03531, 2021.","DOI":"10.1109\/TIP.2021.3079800"},{"issue":"2","key":"13_CR24","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","volume":"60","author":"DG Lowe","year":"2004","unstructured":"Lowe DG. Distinctive image features from scale-invariant keypoints. Int J Comput Vis. 2004;60(2):91\u2013110.","journal-title":"Int J Comput Vis"},{"key":"13_CR25","doi-asserted-by":"crossref","unstructured":"Davis SB, Mermelstein P. Readings in speech recognition. Chapter comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. San Francisco, CA: Morgan Kaufmann Publishers Inc.; 1990. p. 65\u201374.","DOI":"10.1016\/B978-0-08-051584-7.50010-3"},{"key":"13_CR26","unstructured":"Kingma D, Ba J. Adam: a method for stochastic optimization. In: International conference on learning representations, 12 2014."},{"key":"13_CR27","doi-asserted-by":"publisher","first-page":"107425","DOI":"10.1016\/j.patcog.2020.107425","volume":"107","author":"R Zhu","year":"2020","unstructured":"Zhu R, Dornaika F, Ruichek Y. Semi-supervised elastic manifold embedding with deep learning architecture. Pattern Recognit. 2020;107:107425.","journal-title":"Pattern Recognit"},{"issue":"9","key":"13_CR28","doi-asserted-by":"publisher","first-page":"3938","DOI":"10.1109\/TNNLS.2017.2740318","volume":"29","author":"X Chen","year":"2018","unstructured":"Chen X, Weng J, Wei L, Jiaming X, Weng J-S. Deep manifold learning combined with convolutional neural networks for action recognition. IEEE Trans Neural Netw Learn Syst. 2018;29(9):3938\u201352.","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"13_CR29","unstructured":"Dorfer M, Kelz R, Widmer G. Deep linear discriminant analysis. CoRR, arXiv:abs\/1511.04707. 2015."},{"key":"13_CR30","unstructured":"Chan T-H, Jia K, Gao S, Lu J, Zeng Z, Ma Y. Pcanet: A simple deep learning baseline for image classification? CoRR. arXiv:abs\/1404.3606. 2014."},{"key":"13_CR31","doi-asserted-by":"crossref","unstructured":"Parlaktuna M, Sekmen A, Koku AB, Abdul-Malek A. Enhanced deep learning with improved feature subspace separation. In: 2018 international conference on artificial intelligence and data processing (IDAP). 2018. p. 1\u20135.","DOI":"10.1109\/IDAP.2018.8620856"},{"key":"13_CR32","doi-asserted-by":"crossref","unstructured":"Parlaktune M. Enhanced deep learning with improved feature subspace separation. Master\u2019s thesis, Tennessee State University, 2018.","DOI":"10.1109\/IDAP.2018.8620856"},{"key":"13_CR33","unstructured":"Abdul-Malek A. Deep learning and subspace segmentation: theory and applications. PhD thesis, Tennessee State University, 2019."},{"issue":"12","key":"13_CR34","doi-asserted-by":"publisher","first-page":"1945","DOI":"10.1109\/TPAMI.2005.244","volume":"27","author":"R Vidal","year":"2005","unstructured":"Vidal R, Ma Y, Sastry S. Generalized principal component analysis (GPCA). IEEE Trans Pattern Anal Mach Intell. 2005;27(12):1945\u201359.","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"13_CR35","unstructured":"Roy O, Vetterli M. The effective rank: a measure of effective dimensionality. In: 2007 15th European signal processing conference. 2007. p. 606\u2013610"},{"key":"13_CR36","doi-asserted-by":"crossref","unstructured":"Zhang J, Zhu G, Heath Jr. RW, Huang K. Grassmannian learning: embedding geometry awareness in shallow and deep learning. CoRR, arXiv:abs\/1808.02229. 2018.","DOI":"10.1109\/CAC.2018.8623112"},{"issue":"11","key":"13_CR37","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","volume":"86","author":"Y Lecun","year":"1998","unstructured":"Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE. 1998;86(11):2278\u2013324.","journal-title":"Proc IEEE"},{"key":"13_CR38","unstructured":"Xiao H, Rasul K, Vollgraf R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. 2017."},{"key":"13_CR39","doi-asserted-by":"crossref","unstructured":"Cohen G, Afshar S, Tapson J, van Schaik A. EMNIST: an extension of MNIST to handwritten letters. CoRR, arXiv:abs\/1702.05373. 2017.","DOI":"10.1109\/IJCNN.2017.7966217"}],"container-title":["Discover Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44163-021-00013-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44163-021-00013-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44163-021-00013-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,12]],"date-time":"2024-09-12T08:00:08Z","timestamp":1726128008000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44163-021-00013-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,16]]},"references-count":39,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["13"],"URL":"https:\/\/doi.org\/10.1007\/s44163-021-00013-1","relation":{},"ISSN":["2731-0809"],"issn-type":[{"type":"electronic","value":"2731-0809"}],"subject":[],"published":{"date-parts":[[2021,11,16]]},"assertion":[{"value":"30 September 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 November 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 November 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"12"}}