{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T12:31:13Z","timestamp":1772800273096,"version":"3.50.1"},"reference-count":39,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2020,8,31]],"date-time":"2020-08-31T00:00:00Z","timestamp":1598832000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001807","name":"FAPESP","doi-asserted-by":"publisher","award":["017\/08817-7,  2015\/08118-6"],"award-info":[{"award-number":["017\/08817-7,  2015\/08118-6"]}],"id":[{"id":"10.13039\/501100001807","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Deep Neural Networks are known for impressive results in a wide range of applications, being responsible for many advances in technology over the past few years. However, debugging and understanding neural networks models\u2019 inner workings is a complex task, as there are several parameters and variables involved in every decision. Multidimensional projection techniques have been successfully adopted to display neural network hidden layer outputs in an explainable manner, but comparing different outputs often means overlapping projections or observing them side-by-side, presenting hurdles for users in properly conveying data flow. In this paper, we introduce a novel approach for comparing projections obtained from multiple stages in a neural network model and visualizing differences in data perception. Changes among projections are transformed into trajectories that, in turn, generate vector fields used to represent the general flow of information. This representation can then be used to create layouts that highlight new information about abstract structures identified by neural networks.<\/jats:p>","DOI":"10.3390\/info11090426","type":"journal-article","created":{"date-parts":[[2020,8,31]],"date-time":"2020-08-31T11:53:49Z","timestamp":1598874829000},"page":"426","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Exploring Neural Network Hidden Layer Activity Using Vector Fields"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5257-4925","authenticated-orcid":false,"given":"Gabriel D.","family":"Cantareira","sequence":"first","affiliation":[{"name":"Instituto de Ci\u00eancias Matem\u00e1ticas e de Computac\u00e3o, Universidade de S\u00e3o Paulo, S\u00e3o Paulo 13566-590, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elham","family":"Etemad","sequence":"additional","affiliation":[{"name":"Faculty of Computer Science, Dalhousie University, Halifax, NS B3H 4R2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2316-760X","authenticated-orcid":false,"given":"Fernando V.","family":"Paulovich","sequence":"additional","affiliation":[{"name":"Instituto de Ci\u00eancias Matem\u00e1ticas e de Computac\u00e3o, Universidade de S\u00e3o Paulo, S\u00e3o Paulo 13566-590, Brazil"},{"name":"Faculty of Computer Science, Dalhousie University, Halifax, NS B3H 4R2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,8,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2650","DOI":"10.1109\/TVCG.2018.2846735","article-title":"Multidimensional Projection for Visual Analytics: Linking Techniques with Distortions, Tasks, and Layout Enrichment","volume":"25","author":"Nonato","year":"2018","journal-title":"IEEE Trans. Vis. Comput. Graphics"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1109\/TVCG.2016.2598838","article-title":"Visualizing the hidden activity of artificial neural networks","volume":"23","author":"Rauber","year":"2017","journal-title":"IEEE Trans. Vis. Comput. Graphics"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Mahendran, A., and Vedaldi, A. (2015, January 7\u201312). Understanding deep image representations by inverting them. Proceedings of the IEEE IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299155"},{"key":"ref_5","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Cantareira, G.D., Paulovich, F.V., and Etemad, E. (2020, January 27\u201329). Visualizing Learning Space in Neural Network Hidden Layers. Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Valetta, Malta.","DOI":"10.5220\/0009168901100121"},{"key":"ref_7","unstructured":"Turkay, C., and Vrotsou, K. (2020). A Generic Model for Projection Alignment Applied to Neural Network Visualization. EuroVis Workshop on Visual Analytics (EuroVA), The Eurographics Association."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Scherer, D., M\u00fcller, A., and Behnke, S. (2010). Evaluation of pooling operations in convolutional architectures for object recognition. Proceedings of the International Conference on Artificial Neural Networks, Springer.","DOI":"10.1007\/978-3-642-15825-4_10"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2674","DOI":"10.1109\/TVCG.2018.2843369","article-title":"Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers","volume":"25","author":"Hohman","year":"2018","journal-title":"IEEE Trans. Vis. Comput. Graphics"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2014). Visualizing and understanding convolutional networks. Proceedings of the European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_12","unstructured":"Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv."},{"key":"ref_13","unstructured":"Zintgraf, L.M., Cohen, T.S., Adel, T., and Welling, M. (2017). Visualizing deep neural network decisions: Prediction difference analysis. arXiv."},{"key":"ref_14","first-page":"1","article-title":"Visualizing higher-layer features of a deep network","volume":"1341","author":"Erhan","year":"2009","journal-title":"Univ. Montr."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1007\/s11263-016-0911-8","article-title":"Visualizing deep convolutional neural networks using natural pre-images","volume":"120","author":"Mahendran","year":"2016","journal-title":"Int. J. Comput. Vis."},{"key":"ref_16","unstructured":"Babiker, H.K.B., and Goebel, R. (2017). An Introduction to Deep Visual Explanation. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1109\/TVCG.2016.2598831","article-title":"Towards better analysis of deep convolutional neural networks","volume":"23","author":"Liu","year":"2017","journal-title":"IEEE Trans. Vis. Comput. Graphics"},{"key":"ref_18","unstructured":"Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lipson, H. (2015). Understanding neural networks through deep visualization. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1109\/TVCG.2017.2744358","article-title":"Deepeyes: Progressive visual analytics for designing deep neural networks","volume":"24","author":"Pezzotti","year":"2018","journal-title":"IEEE Trans. Vis. Comput. Graphics"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1109\/TVCG.2017.2744718","article-title":"A cti V is: Visual Exploration of Industry-Scale Deep Neural Network Models","volume":"24","author":"Kahng","year":"2018","journal-title":"IEEE Trans. Vis. Comput. Graphics"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"McInnes, L., Healy, J., and Melville, J. (2018). Umap: Uniform manifold approximation and projection for dimension reduction. arXiv.","DOI":"10.21105\/joss.00861"},{"key":"ref_22","unstructured":"Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. (2014, January 21\u201326). Decaf: A deep convolutional activation feature for generic visual recognition. Proceedings of the International conference on machine learning, Beijing, China."},{"key":"ref_23","first-page":"339","article-title":"Learning features from music audio with deep belief networks","volume":"10","author":"Hamel","year":"2010","journal-title":"ISMIR"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Mohamed, A.r., Hinton, G., and Penn, G. (2012). Understanding how deep belief networks perform acoustic modelling. Neural Netw., 6\u20139.","DOI":"10.1109\/ICASSP.2012.6288863"},{"key":"ref_25","first-page":"26","article-title":"Visualizing high-dimensional data using t-sne. journal of machine learning research","volume":"9","author":"Hinton","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2563","DOI":"10.1109\/TVCG.2011.220","article-title":"Local affine multidimensional projection","volume":"17","author":"Joia","year":"2011","journal-title":"IEEE Trans. Vis. Comput. Graphics"},{"key":"ref_27","unstructured":"Rauber, P.E., Falc\u00e3o, A.X., and Telea, A.C. (2016). Visualizing time-dependent data using dynamic t-SNE. Eurographics\/IEEE VGTC Conference on Visualization, Eurographics Association."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Hilasaca, G., and Paulovich, F. (2019). Visual Feature Fusion and its Application to Support Unsupervised Clustering Tasks. arXiv.","DOI":"10.1177\/1473871619891062"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"e2","DOI":"10.23915\/distill.00002","article-title":"How to use t-sne effectively","volume":"1","author":"Wattenberg","year":"2016","journal-title":"Distill"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1111\/cgf.12107","article-title":"Vector Field k-Means: Clustering Trajectories by Fitting Multiple Vector Fields","volume":"32","author":"Ferreira","year":"2013","journal-title":"Comput. Graph. Forum."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1007\/s12650-016-0348-8","article-title":"From numerics to combinatorics: A survey of topological methods for vector field visualization","volume":"19","author":"Wang","year":"2016","journal-title":"J. Vis."},{"key":"ref_32","first-page":"2000","article-title":"A Survey on Visualization of Vector Fields By Texture-Based Methods","volume":"1","author":"Sanna","year":"2000","journal-title":"Devel. Pattern Rec."},{"key":"ref_33","unstructured":"Peng, Z., and Laramee, R.S. (2009). Higher Dimensional Vector Field Visualization: A Survey. TPCG, 149\u2013163."},{"key":"ref_34","unstructured":"Bian, J., Tian, D., Tang, Y., and Tao, D. (2018). A survey on trajectory clustering analysis. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"58295","DOI":"10.1109\/ACCESS.2018.2873779","article-title":"Big Trajectory Data: A Survey of Applications and Services","volume":"6","author":"Kong","year":"2018","journal-title":"IEEE Access"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-Based Learning Applied to Document Recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1141","DOI":"10.1109\/TVCG.2013.246","article-title":"Bundled visualization of dynamicgraph and trail data","volume":"20","author":"Hurter","year":"2013","journal-title":"IEEE Trans. Vis. Comput. Graphics"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201322). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/9\/426\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:05:16Z","timestamp":1760177116000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/9\/426"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,31]]},"references-count":39,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["info11090426"],"URL":"https:\/\/doi.org\/10.3390\/info11090426","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,31]]}}}