{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T06:52:37Z","timestamp":1781074357793,"version":"3.54.1"},"reference-count":45,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2021,12,18]],"date-time":"2021-12-18T00:00:00Z","timestamp":1639785600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the Ministry of Science and Technology of the People\u2019s Republic of China","award":["2016YFC0800401"],"award-info":[{"award-number":["2016YFC0800401"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Passive millimeter wave has been employed in security inspection owing to a good penetrability to clothing and harmlessness. However, the passive millimeter wave images (PMMWIs) suffer from low resolution and inherent noise. The published methods have rarely improved the quality of images for PMMWI and performed the detection only based on PMMWI with bounding box, which cause a high rate of false alarm. Moreover, it is difficult to identify the low-reflective non-metallic threats by the differences in grayscale. In this paper, a method of detecting concealed threats in human body is proposed. We introduce the GAN architecture to reconstruct high-quality images from multi-source PMMWIs. Meanwhile, we develop a novel detection pipeline involving semantic segmentation, image registration, and comprehensive analyzer. The segmentation network exploits multi-scale features to merge local and global information together in both PMMWIs and visible images to obtain precise shape and location information in the images, and the registration network is proposed for privacy concerns and the elimination of false alarms. With the grayscale and contour features, the detection for metallic and non-metallic threats can be conducted, respectively. After that, a synthetic strategy is applied to integrate the detection results of each single frame. In the numerical experiments, we evaluate the effectiveness of each module and the performance of the proposed method. Experimental results demonstrate that the proposed method outperforms the existing methods with 92.35% precision and 90.3% recall in our dataset, and also has a fast detection rate.<\/jats:p>","DOI":"10.3390\/s21248456","type":"journal-article","created":{"date-parts":[[2021,12,20]],"date-time":"2021-12-20T02:40:32Z","timestamp":1639968032000},"page":"8456","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Real-Time Detection of Concealed Threats with Passive Millimeter Wave and Visible Images via Deep Neural Networks"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2103-451X","authenticated-orcid":false,"given":"Hao","family":"Yang","sequence":"first","affiliation":[{"name":"School of Electronics and Information Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dinghao","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Electronics and Information Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shiyin","family":"Qin","sequence":"additional","affiliation":[{"name":"School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tie Jun","family":"Cui","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Millimeter Waves, School of Information Science and Engineering, Southeast University, Nanjing 210096, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jungang","family":"Miao","sequence":"additional","affiliation":[{"name":"School of Electronics and Information Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"148336","DOI":"10.1109\/ACCESS.2019.2946736","article-title":"Review of Active Millimeter Wave Imaging Techniques for Personnel Security Screening","volume":"7","author":"Wang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Xing, W., Zhang, J., and Liang, G. (2018, January 10\u201315). A Fast Detection Method Based on Deep Learning of Millimeter Wave Human Image. Proceedings of the the 2018 International Conference, Houston, TX, USA.","DOI":"10.1145\/3293663.3293678"},{"key":"ref_3","unstructured":"Singh, M.K., Park, H., Kim, S.H., Tiwary, U.S., and Kim, Y.H. (2005, January 29). Linear and nonlinear methods for passive millimeter-wave image deblurring. Proceedings of the IEEE International Geoscience & Remote Sensing Symposium, Seoul, Korea."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1167","DOI":"10.1109\/TMTT.2018.2880757","article-title":"Combining Commercially Available Active and Passive Sensors Into a Millimeter-Wave Imager for Concealed Weapon Detection","volume":"67","author":"Montesano","year":"2019","journal-title":"IEEE Trans. Microw. Theory Tech."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Petkie, D.T., Lucia, F., Casto, C., Helminger, P., and Franck, C. (2005). Active and passive millimeter and sub-millimeter-wave imaging. Technologies for Optical Countermeasures II; Femtosecond Phenomena II; and Passive Millimetre-Wave and Terahertz Imaging II, International Society for Optics and Photonics.","DOI":"10.1117\/12.638982"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Babacan, S.D., Luessi, M., Spinoulas, L., Katsaggelos, A.K., and Raptis, A. (2011, January 11\u201314). Compressive passive millimeter-wave imaging. Proceedings of the 18th IEEE International Conference on Image Processing, ICIP 2011, Brussels, Belgium.","DOI":"10.1109\/ICIP.2011.6116227"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kozacik, S., Paolini, A., Bonnett, J., Harrity, C., and Prather, D.W. (2015). Real-time image processing for passive mmW imagery. Passive and Active Millimeter-Wave Imaging XVIII, International Society for Optics and Photonics.","DOI":"10.1117\/12.2177395"},{"key":"ref_8","first-page":"3135","article-title":"Video rate passive millimeter-wave imager utilizing optical upconversion with improved size, weight, and power","volume":"Volume 9462","author":"Martin","year":"2015","journal-title":"Passive and Active Millimeter-Wave Imaging XVIII"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Salmon, N.A., Beale, J., Hayward, S., Hall, P., Macpherson, R., Metcalfe, R., and Harvey, A. (2008). Compact and light-weight digital beam-forming passive millimetre-wave imagers. Millimetre Wave and Terahertz Sensors and Technology, International Society for Optics and Photonics.","DOI":"10.1117\/12.801236"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"379","DOI":"10.2528\/PIERB12101505","article-title":"A passive millimeter-wave imager used for concealed weapon detection","volume":"46","author":"Zheng","year":"2013","journal-title":"Prog. Electromagn. Res."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"151","DOI":"10.2528\/PIERC13062801","article-title":"Initial results of a passive millimeter-wave imager used for concealed weapon detection BHU-2D-U","volume":"43","author":"Zheng","year":"2013","journal-title":"Prog. Electromagn. Res."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Yao, X., Liu, K., Hu, A., and Miao, J. (2015). Improved design of a passive millimeter-wave synthetic aperture interferometric imager for indoor applications. Millimetre Wave and Terahertz Sensors and Technology VIII, International Society for Optics and Photonics.","DOI":"10.1117\/12.2194749"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"5673","DOI":"10.1109\/TIM.2019.2957894","article-title":"A GaAs Power Detector Design for C-band Wideband Complex Cross-Correlation Measurement","volume":"69","author":"Chen","year":"2019","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Guo, X., Asif, M., Hu, A., Li, Z., and Miao, J. (2019). A 1-GHz 64-Channel Cross-Correlation System for Real-Time Interferometric Aperture Synthesis Imaging. Sensors, 19.","DOI":"10.3390\/s19071739"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"51933","DOI":"10.1109\/ACCESS.2019.2910888","article-title":"1-Bit\/2-Level Analog-to-Digital Conversion Based on Comparator and FPGA for Aperture Synthesis Passive Millimeter-Wave Imager","volume":"7","author":"Guo","year":"2019","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Guo, X., Asif, M., Hu, A., and Miao, J. (2018). Design of a low-cost cross-correlation system for aperture synthesis passive millimeter wave imager. Millimetre Wave and Terahertz Sensors and Technology XI, International Society for Optics and Photonics.","DOI":"10.1117\/12.2325517"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1145\/3422622","article-title":"Generative adversarial networks","volume":"63","author":"Goodfellow","year":"2020","journal-title":"Commun. ACM"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2663","DOI":"10.1109\/TMI.2018.2845918","article-title":"H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes","volume":"37","author":"Li","year":"2018","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_19","unstructured":"Iglovikov, V., and Shvets, A. (2018). Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation. arXiv."},{"key":"ref_20","unstructured":"Bengio, Y. (2012, January 2). Deep learning of representations for unsupervised and transfer learning. Proceedings of the ICML Workshop on Unsupervised and Transfer Learning, Bellevue, WA, USA."},{"key":"ref_21","unstructured":"Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. (2018). How does batch normalization help optimization?. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2580","DOI":"10.1109\/TCSVT.2017.2774927","article-title":"Deep CNNs for Object Detection Using Passive Millimeter Sensors","volume":"29","author":"Molina","year":"2019","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1515\/freq-2018-0255","article-title":"A multi-thresholding method based on Otsu\u2019s algorithm for the detection of concealed threats in passive millimeter-wave images","volume":"73","year":"2019","journal-title":"Frequenz"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1007\/s10762-015-0146-8","article-title":"Segmentation of concealed objects in passive millimeter-wave images based on the Gaussian mixture model","volume":"36","author":"Yu","year":"2015","journal-title":"J. Infrared Millim. Terahertz Waves"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Chen, Y., Pang, L., Liu, H., and Xu, X. (2018, January 7\u201310). Wavelet fusion for concealed object detection using passive millimeter wave sequence images. Proceedings of the International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, Beijing, China.","DOI":"10.5194\/isprs-archives-XLII-3-193-2018"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhu, S., and Li, Y. (2018, January 6\u20139). A multi-class classification system for passive millimeter-wave image. Proceedings of the 2018 International Conference on Microwave and Millimeter Wave Technology (ICMMT), Chengdu, China.","DOI":"10.1109\/ICMMT.2018.8563597"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Mateos, J., L\u00f3pez, A., Vega, M., Molina, R., and Katsaggelos, A.K. (2016, January 25\u201328). Multiframe blind deconvolution of passive millimeter wave images using variational dirichlet blur kernel estimation. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AR, USA.","DOI":"10.1109\/ICIP.2016.7532845"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wang, X., Gou, S., Li, J., Zhao, Y., and Mao, S. (2021). Self-paced feature attention fusion network for concealed object detection in millimeter-wave image. IEEE Trans. Circuits Syst. Video Technol.","DOI":"10.1109\/TCSVT.2021.3058246"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Pang, L., Liu, H., Chen, Y., and Miao, J. (2020). Real-time concealed object detection from passive millimeter wave images based on the YOLOv3 algorithm. Sensors, 20.","DOI":"10.3390\/s20061678"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Cheng, Z., Xiong, J., Liu, Y., Zhang, Y., and Yang, J. (2019, January 23\u201326). Research on Fast Target Detection And Classification Algorithm for Passive Millimeter Wave Imaging. Proceedings of the 2019 International Conference on Control, Automation and Information Sciences (ICCAIS), Chengdu, China.","DOI":"10.1109\/ICCAIS46528.2019.9074548"},{"key":"ref_31","first-page":"4205308","article-title":"A visible and passive millimeter wave image fusion algorithm based on pulse-coupled neural network in Tetrolet domain for early risk warning","volume":"2018","author":"Li","year":"2018","journal-title":"Math. Probl. Eng."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1007\/s10762-018-0558-3","article-title":"High-performance detection of concealed forbidden objects on human body with deep neural networks based on passive millimeter wave and visible imagery","volume":"40","author":"Guo","year":"2019","journal-title":"J. Infrared Millim. Terahertz Waves"},{"key":"ref_33","unstructured":"Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21\u201326). Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"4980","DOI":"10.1109\/TIP.2020.2977573","article-title":"DDcGAN: A Dual-discriminator Conditional Generative Adversarial Network for Multi-resolution Image Fusion","volume":"29","author":"Ma","year":"2020","journal-title":"IEEE Trans Image Process"},{"key":"ref_35","unstructured":"Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Dai, B., Fidler, S., Urtasun, R., and Lin, D. (2017, January 22\u201329). Towards diverse and natural image descriptions via a conditional gan. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.323"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., and Pal, C. (2016). The importance of skip connections in biomedical image segmentation. Deep Learning and Data Labeling for Medical Applications, Springer.","DOI":"10.1007\/978-3-319-46976-8_19"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"5147","DOI":"10.1007\/s00521-018-04007-6","article-title":"Removing ring artifacts in CBCT images via generative adversarial networks with unidirectional relative total variation loss","volume":"31","author":"Wang","year":"2019","journal-title":"Neural Comput. Appl."},{"key":"ref_39","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"95","DOI":"10.3389\/fnins.2019.00095","article-title":"Going deeper in spiking neural networks: VGG and residual architectures","volume":"13","author":"Sengupta","year":"2019","journal-title":"Front. Neurosci."},{"key":"ref_41","unstructured":"Ayyachamy, S., Alex, V., Khened, M., and Krishnamurthi, G. (2019). Medical image retrieval using Resnet-18. Medical Imaging 2019: Imaging Informatics for Healthcare, Research, and Applications, International Society for Optics and Photonics."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1109\/TCI.2020.2965304","article-title":"VIF-Net: An Unsupervised Framework for Infrared and Visible Image Fusion","volume":"6","author":"Hou","year":"2020","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"023522","DOI":"10.1117\/1.2945910","article-title":"Assessment of image fusion procedures using entropy, image quality, and multispectral classification","volume":"2","author":"Roberts","year":"2008","journal-title":"J. Appl. Remote Sens."},{"key":"ref_44","unstructured":"Wang, Y., Li, J., Lu, Y., Fu, Y., and Jiang, Q. (2003, January 14\u201317). Image quality evaluation based on image weighted separating block peak signal to noise ratio. Proceedings of the International Conference on Neural Networks & Signal Processing, Nanjing, China."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/24\/8456\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:51:15Z","timestamp":1760169075000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/24\/8456"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,18]]},"references-count":45,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["s21248456"],"URL":"https:\/\/doi.org\/10.3390\/s21248456","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,18]]}}}