{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,27]],"date-time":"2025-10-27T16:16:42Z","timestamp":1761581802450,"version":"build-2065373602"},"reference-count":27,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2019,2,21]],"date-time":"2019-02-21T00:00:00Z","timestamp":1550707200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Single sensor systems and standard optical\u2014usually RGB CCTV video cameras\u2014fail to provide adequate observations, or the amount of spectral information required to build rich, expressive, discriminative features for object detection and tracking tasks in challenging outdoor and indoor scenes under various environmental\/illumination conditions. Towards this direction, we have designed a multisensor system based on thermal, shortwave infrared, and hyperspectral video sensors and propose a processing pipeline able to perform in real-time object detection tasks despite the huge amount of the concurrently acquired video streams. In particular, in order to avoid the computationally intensive coregistration of the hyperspectral data with other imaging modalities, the initially detected targets are projected through a local coordinate system on the hypercube image plane. Regarding the object detection, a detector-agnostic procedure has been developed, integrating both unsupervised (background subtraction) and supervised (deep learning convolutional neural networks) techniques for validation purposes. The detected and verified targets are extracted through the fusion and data association steps based on temporal spectral signatures of both target and background. The quite promising experimental results in challenging indoor and outdoor scenes indicated the robust and efficient performance of the developed methodology under different conditions like fog, smoke, and illumination changes.<\/jats:p>","DOI":"10.3390\/rs11040446","type":"journal-article","created":{"date-parts":[[2019,2,22]],"date-time":"2019-02-22T03:49:44Z","timestamp":1550807384000},"page":"446","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Fusing Multimodal Video Data for Detecting Moving Objects\/Targets in Challenging Indoor and Outdoor Scenes"],"prefix":"10.3390","volume":"11","author":[{"given":"Zacharias","family":"Kandylakis","sequence":"first","affiliation":[{"name":"Remote Sensing Laboratory, National Technical University of Athens, 15780 Zographos, Greece"}]},{"given":"Konstantinos","family":"Vasili","sequence":"additional","affiliation":[{"name":"Remote Sensing Laboratory, National Technical University of Athens, 15780 Zographos, Greece"}]},{"given":"Konstantinos","family":"Karantzalos","sequence":"additional","affiliation":[{"name":"Remote Sensing Laboratory, National Technical University of Athens, 15780 Zographos, Greece"}]}],"member":"1968","published-online":{"date-parts":[[2019,2,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1109\/MSP.2013.2278915","article-title":"Detection Algorithms in Hyperspectral Imaging Systems: An Overview of Practical Algorithms","volume":"31","author":"Manolakis","year":"2014","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Pieper, M., Manolakis, D., Cooley, T., Brueggeman, M., Weisner, A., and Jacobson, J. (2015, January 26\u201331). New insights and practical considerations in hyperspectral change detection. Proceedings of the 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Milan, Italy.","DOI":"10.1109\/IGARSS.2015.7326742"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4567","DOI":"10.1109\/TGRS.2017.2694159","article-title":"Object Tracking by Hierarchical Decomposition of Hyperspectral Video Sequences: Application to Chemical Gas Plume Tracking","volume":"55","author":"Tochon","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kandylakis, Z., Karantzalos, K., Doulamis, A., and Doulamis, N. (2015, January 2\u20135). Multiple Object Tracking with Background Estimation in Hyperspectral Video Sequences. Proceedings of the IEEE Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), Tokyo, Japan.","DOI":"10.1109\/WHISPERS.2015.8075367"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1967","DOI":"10.1109\/TCYB.2014.2362959","article-title":"Semi-Supervised Multitask Learning for Scene Recognition","volume":"45","author":"Lu","year":"2015","journal-title":"IEEE Trans. Cybern."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"3652","DOI":"10.1109\/TIP.2017.2695887","article-title":"A General Framework for Edited Video and Raw Video Summarization","volume":"26","author":"Li","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"884","DOI":"10.1109\/TCYB.2016.2531179","article-title":"Joint Dictionary Learning for Multispectral Change Detection","volume":"47","author":"Lu","year":"2017","journal-title":"IEEE Trans. Cybern."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1109\/TSMC.2016.2531671","article-title":"Heterogeneous Information Fusion and Visualization for a Large-Scale Intelligent Video Surveillance System","volume":"47","author":"Fan","year":"2017","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"19","DOI":"10.5194\/isprs-annals-IV-1-W1-19-2017","article-title":"Security Event Recognition for Visual Surveillance","volume":"4","author":"Liao","year":"2017","journal-title":"ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci."},{"key":"ref_10","first-page":"5575","article-title":"Beyond Group: Multiple Person Tracking via Minimal Topology-Energy-Variation","volume":"26","author":"Gao","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1109\/TPAMI.2005.102","article-title":"Effective Gaussian mixture learning for video background subtraction","volume":"27","author":"Lee","year":"2005","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1109\/TPAMI.2006.68","article-title":"A texture-based method for modeling the background and detecting moving objects","volume":"28","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"5035","DOI":"10.1109\/TIP.2016.2598680","article-title":"Foreground Detection with Simultaneous Dictionary Learning and Historical Pixel Maintenance","volume":"25","author":"Dong","year":"2016","journal-title":"IEEE Trans. Image Process."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2260","DOI":"10.1109\/TCSVT.2016.2581660","article-title":"A Low-Complexity Pedestrian Detection Framework for Smart Video Surveillance Systems","volume":"27","author":"Bilal","year":"2017","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1109\/LGRS.2018.2797538","article-title":"Multiscale Fully Convolutional Network for Foreground Object Detection in Infrared Videos","volume":"15","author":"Zeng","year":"2018","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 24\u201327). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 27\u201329). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Redmon, J., and Farhadi, A. (2017, January 11). YOLO9000: Better, Faster, Stronger. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Copenhagen, Denmark.","DOI":"10.1109\/CVPR.2017.690"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Makantasis, K., Karantzalos, K., Doulamis, A., and Doulamis, N. (2015, January 10\u201315). Deep supervised learning for hyperspectral data classification through convolutional neural networks. Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Beijing, China.","DOI":"10.1109\/IGARSS.2015.7326945"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kandylakis, Z., Karantzalos, K., Doulamis, A., and Karagiannidis, L. (2017, January 28\u201329). Multimodal Data Fusion for Effective Surveillance of Critical Infrastructures. Proceedings of the ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Hamburg, Germany.","DOI":"10.5194\/isprs-archives-XLII-3-W3-87-2017"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Karantzalos, K. (2009, January 5\u20137). Intrinsic dimensionality estimation and dimensionality reduction through scale space filtering. Proceedings of the 16th International Conference on Digital Signal Processing, Santorini-Hellas, Santorini, Greece.","DOI":"10.1109\/ICDSP.2009.5201196"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Karantzalos, K., Sotiras, A., and Paragios, N. (2014, January 6\u20137). Efficient and Automated Multi-Modal Satellite Data Registration through MRFs and Linear Programming. Proceedings of the Computer Vision and Pattern Recognition Workshops, Zurich, Switzerland.","DOI":"10.1109\/CVPRW.2014.57"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3409","DOI":"10.3390\/rs6043409","article-title":"Automatic Descriptor-based Co-registration of Frame Hyperspectral Data","volume":"6","author":"Vakalopoulou","year":"2014","journal-title":"Remote Sens. Environ."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2090","DOI":"10.1109\/TCSVT.2017.2711259","article-title":"Data-driven background subtraction algorithm for in-camera acceleration in thermal imagery","volume":"28","author":"Makantasis","year":"2018","journal-title":"IEEE Trans. Circuits Syst. Video Technol."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/11\/4\/446\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:33:46Z","timestamp":1760186026000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/11\/4\/446"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,2,21]]},"references-count":27,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2019,2]]}},"alternative-id":["rs11040446"],"URL":"https:\/\/doi.org\/10.3390\/rs11040446","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2019,2,21]]}}}