{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T16:35:21Z","timestamp":1781714121102,"version":"3.54.5"},"reference-count":50,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2023,7,1]],"date-time":"2023-07-01T00:00:00Z","timestamp":1688169600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Education (MOE)","award":["MOST 109-2221-E-194-035-MY3"],"award-info":[{"award-number":["MOST 109-2221-E-194-035-MY3"]}]},{"name":"Ministry of Science and Technology (MOST) in Taiwan","award":["MOST 109-2221-E-194-035-MY3"],"award-info":[{"award-number":["MOST 109-2221-E-194-035-MY3"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents an RGB-NIR (Near Infrared) dual-modality technique to analyze the remote photoplethysmogram (rPPG) signal and hence estimate the heart rate (in beats per minute), from a facial image sequence. Our main innovative contribution is the introduction of several denoising techniques such as Modified Amplitude Selective Filtering (MASF), Wavelet Decomposition (WD), and Robust Principal Component Analysis (RPCA), which take advantage of RGB and NIR band characteristics to uncover the rPPG signals effectively through this Independent Component Analysis (ICA)-based algorithm. Two datasets, of which one is the public PURE dataset and the other is the CCUHR dataset built with a popular Intel RealSense D435 RGB-D camera, are adopted in our experiments. Facial video sequences in the two datasets are diverse in nature with normal brightness, under-illumination (i.e., dark), and facial motion. Experimental results show that the proposed method has reached competitive accuracies among the state-of-the-art methods even at a shorter video length. For example, our method achieves MAE = 4.45 bpm (beats per minute) and RMSE = 6.18 bpm for RGB-NIR videos of 10 and 20 s in the CCUHR dataset and MAE = 3.24 bpm and RMSE = 4.1 bpm for RGB videos of 60-s in the PURE dataset. Our system has the advantages of accessible and affordable hardware, simple and fast computations, and wide realistic applications.<\/jats:p>","DOI":"10.3390\/s23136079","type":"journal-article","created":{"date-parts":[[2023,7,3]],"date-time":"2023-07-03T00:53:16Z","timestamp":1688345596000},"page":"6079","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Heart Rate Estimation from Facial Image Sequences of a Dual-Modality RGB-NIR Camera"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8166-2844","authenticated-orcid":false,"given":"Wen-Nung","family":"Lie","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, Center for Innovative Research on Aging Society (CIRAS), and Advanced Institute of Manufacturing with High-Tech Innovations (AIM-HI), National Chung Cheng University, Chia-Yi 621, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dao-Quang","family":"Le","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Center for Innovative Research on Aging Society (CIRAS), and Advanced Institute of Manufacturing with High-Tech Innovations (AIM-HI), National Chung Cheng University, Chia-Yi 621, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5683-6878","authenticated-orcid":false,"given":"Chun-Yu","family":"Lai","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Center for Innovative Research on Aging Society (CIRAS), and Advanced Institute of Manufacturing with High-Tech Innovations (AIM-HI), National Chung Cheng University, Chia-Yi 621, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yu-Shin","family":"Fang","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Center for Innovative Research on Aging Society (CIRAS), and Advanced Institute of Manufacturing with High-Tech Innovations (AIM-HI), National Chung Cheng University, Chia-Yi 621, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Chen, Y.H., Zhuang, J.L., Li, B., and Zhang, Y. (2023). Remote Blood Pressure Estimation via the Spatiotemporal Mapping of Facial Videos. Sensors, 23.","DOI":"10.3390\/s23062963"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Yu, S.G., Kim, S.E., Kim, N.H., Suh, K.H., and Lee, E.C. (2021). Pulse Rate Variability Analysis Using Remote Photoplethysmography Signals. Sensors, 21.","DOI":"10.3390\/s21186241"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Chiang, J.C., Lie, W.N., Huang, H.C., Chen, K.T., Liang, J.Y., Lo, Y.C., and Huang, W.H. (2022). Posture Monitoring for Health Care of Bedridden Elderly Patients Using 3D Human Skeleton Analysis via Machine Learning Approach. Appl. Sci., 12.","DOI":"10.3390\/app12063087"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Jeanne, V., Asselman, M., Brinker, B.D., and Bulut, M. (2013, January 2\u20136). Camera-based Heart Rate Monitoring in Highly Dynamic Light Conditions. Proceedings of the International Conference on Connected Vehicles and Expo (ICCVE), Las Vegas, NV, USA.","DOI":"10.1109\/ICCVE.2013.6799899"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Kado, S., Monno, Y., Moriwaki, K., Yoshizaki, K., Tanaka, M., and Okutomi, M. (2018, January 18\u201321). Remote Heart Rate Measurement from RGB-NIR Video Based on Spatial and Spectral Face Patch Selection. Proceedings of the 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.","DOI":"10.1109\/EMBC.2018.8513464"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kurihara, K., Sugimura, D., and Hamamoto, T. (2019, January 22\u201325). Adaptive Fusion of RGB\/NIR Signals Based on Face\/Background Cross-Spectral Analysis for Heart Rate Estimation. Proceedings of the IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8803673"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"6528","DOI":"10.1109\/TIP.2021.3094739","article-title":"Non-Contact Heart Rate Estimation via Adaptive RGB\/NIR Signal Fusion","volume":"30","author":"Kurihara","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Dosso, Y.S., Bekele, A., and Green, J.R. (2018, January 11\u201313). Eulerian Magnification of Multi-Modal RGB-D Video for Heart Rate Estimation. Proceedings of the IEEE International Symposium on Medical Measurements and Applications (MeMeA), Rome, Italy.","DOI":"10.1109\/MeMeA.2018.8438741"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1109\/TBME.2010.2086456","article-title":"Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam","volume":"58","author":"Poh","year":"2010","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Chen, J., Chang, Z., Qiu, Q., Li, X., Sapiro, G., Bronstein, A., and Pietik\u00e4inen, M. (2016, January 12\u201315). RealSense = Real Heart Rate: Illumination Invariant Heart Rate Estimation from Videos. Proceedings of the 2016 Sixth International Conference on Image Processing Theory, Tools and Applications (IPTA), Oulu, Finland.","DOI":"10.1109\/IPTA.2016.7820970"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"11688","DOI":"10.1109\/JSEN.2020.2997785","article-title":"Spatial-Spectral-Temporal Fusion for Remote Heart Rate Estimation","volume":"20","author":"Kado","year":"2020","journal-title":"IEEE Sens. J."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Regev, N., and Wulich, D. (2020). Multi-Modal, Remote Breathing Monitor. Sensors, 20.","DOI":"10.3390\/s20041229"},{"key":"ref_13","unstructured":"Lewandowska, M., Rumi\u0144ski, J., Kocejko, T., and Nowak, J. (2011, January 18\u201321). Measuring Pulse Rate with a Webcam\u2014A Non-contact Method for Evaluating Cardiac Activity. Proceedings of the Federated Conference on Computer Science and Information Systems (FedCSIS), Szczecin, Poland."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1625","DOI":"10.1109\/TMM.2017.2672198","article-title":"Estimating Heart Rate and Rhythm via 3D Motion Tracking in Depth Video","volume":"19","author":"Yang","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Nagamatsu, G., Nowara, E.M., Pai, A., Veeraraghavan, A., and Kawasaki, H. (2020, January 20\u201324). PPG3D: Does 3D Head Tracking Improve Camera-based PPG Estimation? In Proceedings of the 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Montreal, QC, Canada.","DOI":"10.1109\/EMBC44109.2020.9176065"},{"key":"ref_16","first-page":"1","article-title":"Remote Heart Rate Measurement from Near-Infrared Videos Based on Joint Blind Source Separation with Delay-Coordinate Transformation","volume":"70","author":"Cheng","year":"2020","journal-title":"IEEE Trans Instrum. Meas."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Morawski, I., Lie, W.N., Aing, L., Chiang, J.C., and Chen, K.T. (2022). Deep-Learning Technique for Risk-Based Action Prediction Using Extremely Low-Resolution Thermopile Sensor Array. IEEE Trans. Circuits Syst. Video Technol.","DOI":"10.1109\/ICIP42928.2021.9506024"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Le, D.Q., Chiang, J.C., and Lie, W.N. (2022, January 27). Remote PPG Estimation from RGB-NIR Facial Image Sequence for Heart Rate Estimation. Proceedings of the 2022 IEEE International Symposium on Circuits and Systems (ISCAS), Austin, TX, USA.","DOI":"10.1109\/ISCAS48785.2022.9937641"},{"key":"ref_19","unstructured":"Viola, P., and Jones, M. (2001, January 8\u201314). Rapid Object Detection Using a Boosted Cascade of Simple Features. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Kauai, HI, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Babenko, B., Yang, M.H., and Belongie, S. (2009, January 20\u201325). Visual Tracking with Online Multiple Instance Learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206737"},{"key":"ref_21","unstructured":"(2023, May 01). Nasir6. Available online: https:\/\/github.com\/nasir6\/face-segmentation."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kim, D.Y., Lee, K., and Sohn, C.B. (2021). Assessment of ROI Selection for Facial Video-Based rPPG. Sensors, 21.","DOI":"10.3390\/s21237923"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Molinaro, N., Schena, E., Silvestri, S., and Massaroni, C. (2022). Multi-ROI Spectral Approach for the Continuous Remote Cardio-Respiratory Monitoring from Mobile Device Built-In Cameras. Sensors, 22.","DOI":"10.3390\/s22072539"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1965","DOI":"10.1364\/BOE.8.001965","article-title":"Amplitude-selective filtering for remote-PPG","volume":"8","author":"Wang","year":"2017","journal-title":"Biomed Opt. Express"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2878","DOI":"10.1109\/TBME.2013.2266196","article-title":"Robust Pulse Rate from Chrominance-Based rPPG","volume":"60","author":"Haan","year":"2013","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1479","DOI":"10.1109\/TBME.2016.2609282","article-title":"Algorithmic Principles of Remote PPG","volume":"64","author":"Wang","year":"2017","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1913","DOI":"10.1088\/0967-3334\/35\/9\/1913","article-title":"Improved Motion Robustness of Remote-PPG by Using the Blood Volume Pulse Signature","volume":"35","author":"Haan","year":"2014","journal-title":"Physiol. Meas."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Le, D.Q., Lie, W.N., Nguyen, Q.N., and Nguyen, T.T.A. (2020, January 27\u201328). Heart Rate Estimation Based on Facial Image Sequence. Proceedings of the 5th International Conference on Green Technology and Sustainable Development (GTSD), Ho Chi Minh City, Vietnam.","DOI":"10.1109\/GTSD50082.2020.9303142"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1109\/TBME.2012.2228482","article-title":"Adaptive Wavelet Wiener Filtering of ECG Signals","volume":"60","author":"Smital","year":"2012","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., and Sebe, N. (2016, January 27\u201330). Self-Adaptive Matrix Completion for Heart Rate Estimation from Face Videos under Realistic Conditions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.263"},{"key":"ref_31","first-page":"1","article-title":"Robust Principal Component Analysis?","volume":"58","author":"Li","year":"2011","journal-title":"J. ACM"},{"key":"ref_32","first-page":"1","article-title":"Sparse and Low Rank Matrix Decomposition via Alternating Direction Method","volume":"9","author":"Yuan","year":"2009","journal-title":"Pac. J. Optim."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1162\/089976699300016863","article-title":"High-Order Contrasts for Independent Component Analysis","volume":"11","author":"Cardoso","year":"1999","journal-title":"Neural Comput."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Stricker, R., M\u00fcller, S., and Gross, H.M. (2014, January 25\u201329). Non-contact Video-based Pulse Rate Measurement on a Mobile Service Robot. Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication, Edinburgh, UK.","DOI":"10.1109\/ROMAN.2014.6926392"},{"key":"ref_35","unstructured":"(2023, May 01). Visit the CCUHR Dataset. Available online: https:\/\/ccu365-my.sharepoint.com\/:u:\/g\/personal\/ieewnl_office365_ccu_edu_tw\/EcBZOCHSeKNMi1gMGkREqk8BnHcT7pjpnzCPprCsHekSUA?e=aKFjyn."},{"key":"ref_36","unstructured":"(2023, May 01). \u201cBIOPAC\u2019s Pulse Plethysmogram Amplifier\u201d, BIOPAC System, Inc., [Online]. Available online: https:\/\/www.biopac.com\/product\/pulse-plethysmogram-amplifier\/."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Li, X., Chen, J., Zhao, G., and Pietik\u00e4inen, M. (2014, January 23\u201328). Remote Heart Rate Measurement from Face Videos Under Realistic Situations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.543"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1974","DOI":"10.1109\/TBME.2015.2508602","article-title":"A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation","volume":"63","author":"Wang","year":"2016","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_39","unstructured":"Spetlik, R., Franc, V., Cech, J., and Matas, J. (2018, January 3\u20136). Visual Heart Rate Estimation with Convolutional Neural Network. Proceedings of the 29th British Machine Vision Conference, Newcastle, UK."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1373","DOI":"10.1109\/JBHI.2021.3051176","article-title":"PulseGAN: Learning to Generate Realistic Pulse Waveforms in Remote Photoplethys-mography","volume":"25","author":"Song","year":"2021","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Comas, J., Ruiz, A., and Sukno, F. (2022, January 19\u201320). Efficient Remote Photoplethysmography with Temporal Derivative Modules and Time-Shift Invariant Loss. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA.","DOI":"10.1109\/CVPRW56347.2022.00237"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Demirezen, H., and Erdem, C.E. (2018, January 15\u201320). Remote Photoplethysmography Using Nonlinear Mode Decomposition. Proceedings of the 2018 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), Calgary, AB, Canada.","DOI":"10.1109\/ICASSP.2018.8462538"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"e5859","DOI":"10.7717\/peerj.5859","article-title":"A Wavelet-based Decomposition Method for a Robust Extraction of Pulse Rate Signal from Video Recordings","volume":"6","author":"Podraj","year":"2018","journal-title":"Peer J."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Zhao, C., Mei, P., Xu, S., Li, Y.Q., and Feng, Y. (2019, January 27\u201328). Performance Evaluation of Visual Object Detection and Tracking Algorithms Used in Remote Photoplethysmography. Proceedings of the 2019 IEEE International Conference on Computer Vision (ICCV) Workshops, Seoul, Republic of Korea.","DOI":"10.1109\/ICCVW.2019.00204"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"131656","DOI":"10.1109\/ACCESS.2022.3229977","article-title":"ReViSe: Remote Vital Signs Measurement Using Smartphone Camera","volume":"10","author":"Qiao","year":"2022","journal-title":"IEEE Access"},{"key":"ref_46","first-page":"1","article-title":"A Compensation Network with Error Mapping for Robust Remote Photoplethysmography in Noise-Heavy Conditions","volume":"71","author":"Wu","year":"2022","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Niu, X., Han, H., Shan, S., and Chen, X. (2018, January 2\u20136). VIPL-HR: A Multi-modal Database for Pulse Estimation from Less-constrained Face Video. Proceedings of the Asian Conference on Computer Vision (ACCV), Perth, Australia.","DOI":"10.1007\/978-3-030-20873-8_36"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Wang, Z.K., Kao, Y., and Hsu, C.T. (2019, January 22\u201325). Vision-Based Heart Rate Estimation Via A Two-Stream CNN. Proceedings of the IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8803649"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Napolean, Y., Marwade, A., Tomen, N., Alkemade, P., Eijsvogels, T., and van Gemert, J. (2022, January 16\u201319). Heart Rate Estimation in Intense Exercise Videos. Proceedings of the IEEE International Conference on Image Processing (ICIP), Bordeaux, France.","DOI":"10.1109\/ICIP46576.2022.9897604"},{"key":"ref_50","unstructured":"Lingvall, K.U. (2017). Remote Heart Rate Estimation by Evaluating Measurements from Multiple Signals. [Master\u2019s Thesis, School of Computer Science and Communication, KTH Royal Institute of Technology]."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/13\/6079\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:04:36Z","timestamp":1760126676000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/13\/6079"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,1]]},"references-count":50,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["s23136079"],"URL":"https:\/\/doi.org\/10.3390\/s23136079","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,1]]}}}