{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T15:08:39Z","timestamp":1782313719037,"version":"3.54.5"},"reference-count":57,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2022,8,3]],"date-time":"2022-08-03T00:00:00Z","timestamp":1659484800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000181","name":"AFOSR","doi-asserted-by":"publisher","award":["FA9550-21-1-0224"],"award-info":[{"award-number":["FA9550-21-1-0224"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Human monitoring applications in indoor environments depend on accurate human identification and activity recognition (HIAR). Single modality sensor systems have shown to be accurate for HIAR, but there are some shortcomings to these systems, such as privacy, intrusion, and costs. To combat these shortcomings for a long-term monitoring solution, an interpretable, passive, multi-modal, sensor fusion system PRF-PIR is proposed in this work. PRF-PIR is composed of one software-defined radio (SDR) device and one novel passive infrared (PIR) sensor system. A recurrent neural network (RNN) is built as the HIAR model for this proposed solution to handle the temporal dependence of passive information captured by both modalities. We validate our proposed PRF-PIR system for a potential human monitoring system through the data collection of eleven activities from twelve human subjects in an academic office environment. From our data collection, the efficacy of the sensor fusion system is proven via an accuracy of 0.9866 for human identification and an accuracy of 0.9623 for activity recognition. The results of the system are supported with explainable artificial intelligence (XAI) methodologies to serve as a validation for sensor fusion over the deployment of single sensor solutions. PRF-PIR provides a passive, non-intrusive, and highly accurate system that allows for robustness in uncertain, highly similar, and complex at-home activities performed by a variety of human subjects.<\/jats:p>","DOI":"10.3390\/s22155787","type":"journal-article","created":{"date-parts":[[2022,8,3]],"date-time":"2022-08-03T23:33:01Z","timestamp":1659569581000},"page":"5787","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":45,"title":["Interpretable Passive Multi-Modal Sensor Fusion for Human Identification and Activity Recognition"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9994-6773","authenticated-orcid":false,"given":"Liangqi","family":"Yuan","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jack","family":"Andrews","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huaizheng","family":"Mu","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Asad","family":"Vakil","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Ewing","sequence":"additional","affiliation":[{"name":"Sensors Directorate, Air Force Research Laboratory, WPAFB, Dayton, OH 45433, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6894-6108","authenticated-orcid":false,"given":"Erik","family":"Blasch","sequence":"additional","affiliation":[{"name":"Information Directorate, Air Force Research Laboratory, Rome, NY 13441, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3443-4651","authenticated-orcid":false,"given":"Jia","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1109\/JBHI.2019.2951230","article-title":"Fusion of clinical, self-reported, and multisensor data for predicting falls","volume":"24","author":"Silva","year":"2019","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2368","DOI":"10.1109\/JBHI.2019.2963388","article-title":"Highly accurate bathroom activity recognition using infrared proximity sensors","volume":"24","author":"Chapron","year":"2019","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"106788","DOI":"10.1016\/j.asoc.2020.106788","article-title":"Jointly optimization for activity recognition in secure IoT-enabled elderly care applications","volume":"99","author":"Tao","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Yuan, L., and Li, J. (November, January 31). Smart cushion based on pressure sensor array for human sitting posture recognition. Proceedings of the 2021 IEEE Sensors, Sydney, Australia.","DOI":"10.1109\/SENSORS47087.2021.9639463"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Tan, J.-S., Beheshti, B.K., Binnie, T., Davey, P., Caneiro, J., Kent, P., Smith, A., O\u2019Sullivan, P., and Campbell, A. (2021). Human Activity Recognition for People with Knee Osteoarthritis\u2014A Proof-of-Concept. Sensors, 21.","DOI":"10.3390\/s21103381"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1049\/iet-cvi.2017.0062","article-title":"Skeleton-based human activity recognition for elderly monitoring systems","volume":"12","author":"Hbali","year":"2018","journal-title":"IET Comput. Vis."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Manjarres, J., Narvaez, P., Gasser, K., Percybrooks, W., and Pardo, M. (2019). Physical workload tracking using human activity recognition with wearable devices. Sensors, 20.","DOI":"10.3390\/s20010039"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"10953","DOI":"10.1109\/JIOT.2021.3051247","article-title":"Passive RFID Module With LSTM Recurrent Neural Network Activity Classification Algorithm for Ambient-Assisted Living","volume":"8","author":"Oguntala","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"8553","DOI":"10.1109\/JIOT.2019.2920283","article-title":"IoT wearable sensor and deep learning: An integrated approach for personalized human activity recognition in a smart home environment","volume":"6","author":"Bianchi","year":"2019","journal-title":"IEEE Internet Things J."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Luo, X., Guan, Q., Tan, H., Gao, L., Wang, Z., and Luo, X. (2017). Simultaneous indoor tracking and activity recognition using pyroelectric infrared sensors. Sensors, 17.","DOI":"10.3390\/s17081738"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"6014","DOI":"10.1007\/s10489-020-02134-z","article-title":"Sequential neural networks for multi-resident activity recognition in ambient sensing smart homes","volume":"51","author":"Natani","year":"2021","journal-title":"Appl. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Uddin, M.Z., Khaksar, W., and Torresen, J. (2018). Ambient sensors for elderly care and independent living: A survey. Sensors, 18.","DOI":"10.3390\/s18072027"},{"key":"ref_13","first-page":"693","article-title":"Unobtrusive activity recognition of elderly people living alone using anonymous binary sensors and DCNN","volume":"23","author":"Gochoo","year":"2018","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Casaccia, S., Braccili, E., Scalise, L., and Revel, G.M. (2019). Experimental assessment of sleep-related parameters by passive infrared sensors: Measurement setup, feature extraction, and uncertainty analysis. Sensors, 19.","DOI":"10.3390\/s19173773"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Yan, J., Lou, P., Li, R., Hu, J., and Xiong, J. (2018). Research on the multiple factors influencing human identification based on pyroelectric infrared sensors. Sensors, 18.","DOI":"10.3390\/s18020604"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1109\/TASE.2018.2874487","article-title":"A sensor fusion approach to indoor human localization based on environmental and wearable sensors","volume":"16","author":"Pham","year":"2018","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Andrews, J., Kowsika, M., Vakil, A., and Li, J. (2020, January 20\u201323). A motion induced passive infrared (PIR) sensor for stationary human occupancy detection. Proceedings of the 2020 IEEE\/ION Position, Location and Navigation Symposium (PLANS), Portland, OR, USA.","DOI":"10.1109\/PLANS46316.2020.9109909"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Andrews, J., and Li, J. (2021). Human Detection and Biometric Authentication with Ambient Sensors. Signal Process. Med. Biol. Cham. Springer Int. Publ., accepted.","DOI":"10.1007\/978-3-030-99383-2_2"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Vena, A., Samat, N., Sorli, B., and Podlecki, J. (2021). Ultralow Power and Compact Backscatter Wireless Device and Its SDR-Based Reading System for Environment Monitoring in UHF Band. IEEE Sens. Lett., 5.","DOI":"10.1109\/LSENS.2021.3074187"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"131102","DOI":"10.1109\/ACCESS.2019.2940386","article-title":"WiGrus: A WiFi-based gesture recognition system using software-defined radio","volume":"7","author":"Zhang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_21","unstructured":"Yuan, L., Chen, H., Ewing, R., Blasch, E., and Li, J. Three Dimensional Indoor Positioning Based on Passive Radio Frequency Signal Strength Distribution. IEEE Trans. Signal Process., 2022. submitted."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Liu, J., Vakil, A., Ewing, R., Shen, X., and Li, J. (2019, January 15\u201319). Human Presence Detection via Deep Learning of Passive Radio Frequency Data. Proceedings of the 2019 IEEE National Aerospace and Electronics Conference (NAECON), Dayton, OH, USA.","DOI":"10.1109\/NAECON46414.2019.9058116"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Mu, H., Liu, J., Ewing, R., and Li, J. (2019, January 24\u201326). Human Indoor Positioning via Passive Spectrum Monitoring. Proceedings of the 2021 55th Annual Conference on Information Sciences and Systems (CISS 2021), Baltimore, MD, USA.","DOI":"10.1109\/CISS50987.2021.9400291"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Mu, H., Ewing, R., Blasch, E., and Li, J. (2021, January 16\u201319). Human Subject Identification via Passive Spectrum Monitoring. Proceedings of the 2021 IEEE National Aerospace and Electronics Conference (NAECON), Dayton, MD, USA.","DOI":"10.1109\/NAECON49338.2021.9696427"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.patrec.2018.02.010","article-title":"Deep learning for sensor-based activity recognition: A survey","volume":"119","author":"Wang","year":"2019","journal-title":"Pattern Recognit. Lett."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"5775","DOI":"10.1109\/JSEN.2019.2904595","article-title":"Cooperative sensing and wearable computing for sequential hand gesture recognition","volume":"19","author":"Zhang","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1016\/j.neucom.2020.08.079","article-title":"Wearables-based multi-task gait and activity segmentation using recurrent neural networks","volume":"432","author":"Martindale","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1038\/s41586-021-03506-2","article-title":"High-performance brain-to-text communication via handwriting","volume":"593","author":"Willett","year":"2021","journal-title":"Nature"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1142\/S0218488598000094","article-title":"The vanishing gradient problem during learning recurrent neural nets and problem solutions","volume":"6","author":"Hochreiter","year":"1998","journal-title":"Int. J. Uncertain. Fuzziness Knowl. Based Syst."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.inffus.2020.06.004","article-title":"Multi-user activity recognition: Challenges and opportunities","volume":"63","author":"Li","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1109\/MWC.2019.1800405","article-title":"When RFID meets deep learning: Exploring cognitive intelligence for activity identification","volume":"26","author":"Fan","year":"2019","journal-title":"IEEE Wirel. Commun."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Denis, S., Berkvens, R., and Weyn, M. (2019). A survey on detection, tracking and identification in radio frequency-based device-free localization. Sensors, 19.","DOI":"10.3390\/s19235329"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"136777","DOI":"10.1109\/ACCESS.2019.2940834","article-title":"Optimized algorithm for RFID-based activity recognition of the elderly","volume":"7","author":"Yang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1109\/TMC.2017.2706282","article-title":"Compressive representation for device-free activity recognition with passive RFID signal strength","volume":"17","author":"Yao","year":"2017","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Liu, J., Mu, H., Vakil, A., Ewing, R., Shen, X., Blasch, E., and Li, J. (2020). Human Occupancy Detection via Passive Cognitive Radio. Sensors, 20.","DOI":"10.3390\/s20154248"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Liu, X., Yang, T., Tang, S., Guo, P., and Niu, J. (2020, January 21\u201325). From relative azimuth to absolute location: Pushing the limit of pir sensor based localization. Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, London, UK.","DOI":"10.1145\/3372224.3380878"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/LSENS.2020.2976801","article-title":"SLEEPIR: Synchronized low-energy electronically chopped PIR sensor for true presence detection","volume":"4","author":"Wu","year":"2020","journal-title":"IEEE Sens. Lett."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1109\/JSEN.2018.2875659","article-title":"A low-power electric-mechanical driving approach for true occupancy detection using a shuttered passive infrared sensor","volume":"19","author":"Wu","year":"2018","journal-title":"IEEE Sens. J."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"8656","DOI":"10.1109\/JSEN.2018.2869555","article-title":"Occupancy detection and localization by monitoring nonlinear energy flow of a shuttered passive infrared sensor","volume":"18","author":"Wu","year":"2018","journal-title":"IEEE Sens. J."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"47","DOI":"10.14257\/ijmue.2016.11.7.05","article-title":"Development of a sensing module for standing and moving human body using a shutter and PIR sensor","volume":"11","author":"Juan","year":"2016","journal-title":"Int. J. Multimed. Ubiquitous Eng."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Andrews, J., Vakil, A., and Li, J. (2020, January 5). Biometric Authentication and Stationary Detection of Human Subjects by Deep Learning of Passive Infrared (PIR) Sensor Data. Proceedings of the 2020 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), Philadelphia, PA, USA.","DOI":"10.1109\/SPMB50085.2020.9353613"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Kashimoto, Y., Fujiwara, M., Fujimoto, M., Suwa, H., Arakawa, Y., and Yasumoto, K. (2020, January 27\u201329). ALPAS: Analog-PIR-sensor-based activity recognition system in smarthome. Proceedings of the 2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA), Taiwan, China.","DOI":"10.1109\/AINA.2017.33"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.inffus.2020.03.011","article-title":"Baptizo: A sensor fusion based model for tracking the identity of human poses","volume":"62","author":"Bazo","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Hsu, F.-S., Chang, T.-C., Su, Z.-J., Huang, S.-J., and Chen, C.-C. (2021). Smart fall detection framework using hybridized video and ultrasonic sensors. Micromachines, 12.","DOI":"10.3390\/mi12050508"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Li, H., Le Kernec, J., Mehul, A., Gurbuz, S.Z., and Fioranelli, F. (2020, January 21\u201325). Distributed radar information fusion for gait recognition and fall detection. Proceedings of the 2020 IEEE Radar Conference (RadarConf20), Florence, Italy.","DOI":"10.1109\/RadarConf2043947.2020.9266319"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Aguileta, A.A., Brena, R.F., Mayora, O., Molino-Minero-Re, E., and Trejo, L.A. (2019). Multi-sensor fusion for activity recognition\u2014A survey. Sensors, 19.","DOI":"10.3390\/s19173808"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.inffus.2016.09.005","article-title":"Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges","volume":"35","author":"Gravina","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Yadav, S.K., Tiwari, K., Pandey, H.M., and Akbar, S.A. (2021). A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions. Knowl. Based Syst., 223.","DOI":"10.1016\/j.knosys.2021.106970"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1109\/5.554206","article-title":"Sensor fusion potential exploitation-innovative architectures and illustrative applications","volume":"85","author":"Dasarathy","year":"1997","journal-title":"Proc. IEEE"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Blasch, E., Vakil, A., Li, J., and Ewing, R. (2021, January 6\u201313). Multimodal Data Fusion Using Canonical Variates Analysis Confusion Matrix Fusion. Proceedings of the 2021 IEEE Aerospace Conference, Big Sky, MT, USA.","DOI":"10.1109\/AERO50100.2021.9438445"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Moosavi-Dezfooli, S.-M., Fawzi, A., and Frossard, P. (2016, January 27\u201330). Deepfool: A simple and accurate method to fool deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.282"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1145\/3236386.3241340","article-title":"The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery","volume":"16","author":"Lipton","year":"2018","journal-title":"Queue"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"1350","DOI":"10.1214\/15-AOAS848","article-title":"Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model","volume":"9","author":"Letham","year":"2015","journal-title":"Ann. Appl. Stat."},{"key":"ref_54","unstructured":"Lundberg, S.M., and Lee, S.-I. (2017, January 4\u20139). A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Red Hook, NY, USA."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"749","DOI":"10.1038\/s41551-018-0304-0","article-title":"Explainable machine-learning predictions for the prevention of hypoxaemia during surgery","volume":"2","author":"Lundberg","year":"2018","journal-title":"Nat. Biomed. Eng."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1109\/MAP.2015.2437274","article-title":"Movement Detection of Human Body Segments: Passive radio-frequency identification and machine-learning technologies","volume":"57","author":"Amendola","year":"2015","journal-title":"IEEE Antennas Propag. Mag."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Singh, A., and Lubecke, V. (2010, January 23\u201328). A heterodyne receiver for harmonic Doppler radar cardiopulmonary monitoring with body-worn passive RF tags. Proceedings of the 2010 IEEE MTT-S International Microwave Symposium, Anaheim, CA, USA.","DOI":"10.1109\/MWSYM.2010.5517732"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/15\/5787\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:01:43Z","timestamp":1760140903000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/15\/5787"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,3]]},"references-count":57,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["s22155787"],"URL":"https:\/\/doi.org\/10.3390\/s22155787","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,3]]}}}