{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:46:17Z","timestamp":1760240777610,"version":"build-2065373602"},"reference-count":31,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2019,9,24]],"date-time":"2019-09-24T00:00:00Z","timestamp":1569283200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>We present a system that utilizes a range of image processing algorithms to allow fully automated thermal face analysis under both laboratory and real-world conditions. We implement methods for face detection, facial landmark detection, face frontalization and analysis, combining all of these into a fully automated workflow. The system is fully modular and allows implementing own additional algorithms for improved performance or specialized tasks. Our suggested pipeline contains a histogtam of oriented gradients support vector machine (HOG-SVM) based face detector and different landmark detecion methods implemented using feature-based active appearance models, deep alignment networks and a deep shape regression network. Face frontalization is achieved by utilizing piecewise affine transformations. For the final analysis, we present an emotion recognition system that utilizes HOG features and a random forest classifier and a respiratory rate analysis module that computes average temperatures from an automatically detected region of interest. Results show that our combined system achieves a performance which is comparable to current stand-alone state-of-the-art methods for thermal face and landmark datection and a classification accuracy of 65.75% for four basic emotions.<\/jats:p>","DOI":"10.3390\/s19194135","type":"journal-article","created":{"date-parts":[[2019,9,25]],"date-time":"2019-09-25T03:51:18Z","timestamp":1569383478000},"page":"4135","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["A Modular System for Detection, Tracking and Analysis of Human Faces in Thermal Infrared Recordings"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3695-4696","authenticated-orcid":false,"given":"Marcin","family":"Kopaczka","sequence":"first","affiliation":[{"name":"Institute of Imaging and Computer Vision, RWTH Aachen University, 52062 Aachen, Germany"}]},{"given":"Lukas","family":"Breuer","sequence":"additional","affiliation":[{"name":"Institute of Imaging and Computer Vision, RWTH Aachen University, 52062 Aachen, Germany"}]},{"given":"Justus","family":"Schock","sequence":"additional","affiliation":[{"name":"Institute of Imaging and Computer Vision, RWTH Aachen University, 52062 Aachen, Germany"}]},{"given":"Dorit","family":"Merhof","sequence":"additional","affiliation":[{"name":"Institute of Imaging and Computer Vision, RWTH Aachen University, 52062 Aachen, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2019,9,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Kopaczka, M., Schock, J., Nestler, J., Kielholz, K., and Merhof, D. (2018, January 16\u201318). A combined modular system for face detection, head pose estimation, face tracking and emotion recognition in thermal infrared images. Proceedings of the 2018 IEEE International Conference on Imaging Systems and Techniques (IST), Krakow, Poland.","DOI":"10.1109\/IST.2018.8577124"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Friedrich, G., and Yeshurun, Y. (2002, January 22\u201324). Seeing people in the dark: Face recognition in infrared images. Presented at the International Workshop on Biologically Motivated Computer Vision, Tuebingen, Germany.","DOI":"10.1007\/3-540-36181-2_35"},{"key":"ref_3","unstructured":"Reese, K., Zheng, Y., and Elmaghraby, A. (2019, September 11). A Comparison of Face Detection Algorithms in Visible and Thermal Spectrums. Available online: https:\/\/pdfs.semanticscholar.org\/cd58\/d7f2672fedf71d4ac6f7fcd71621612b2d25.pdf."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kopaczka, M., Nestler, J., and Merhof, D. (2017). Face detection in thermal infrared images: A comparison of algorithm-and machine-learning-based approaches. International Conference on Advanced Concepts for Intelligent Vision Systems, Springer.","DOI":"10.1007\/978-3-319-70353-4_44"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1689","DOI":"10.1016\/j.cviu.2013.07.010","article-title":"Face recognition in low resolution thermal images","volume":"117","author":"Mostafa","year":"2013","journal-title":"Comput. Vision Image Underst."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Ma, C., Trung, N., Uchiyama, H., Nagahara, H., Shimada, A., and Taniguchi, R. (2017). Adapting local features for face detection in thermal image. Sensors, 17.","DOI":"10.3390\/s17122741"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1389","DOI":"10.1109\/TIM.2018.2884364","article-title":"A Thermal Infrared Face Database with Facial Landmarks and Emotion Labels","volume":"68","author":"Kopaczka","year":"2018","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_8","first-page":"403","article-title":"High-resolution thermal face dataset for face and expression recognition","volume":"25","author":"Kowalski","year":"2018","journal-title":"Metrol. Meas. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1109\/TBME.2008.2003265","article-title":"Imaging facial signs of neurophysiological responses","volume":"56","author":"Shastri","year":"2008","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Pereira, C.B., Czaplik, M., Blazek, V., Leonhardt, S., and Teichmann, D. (2018). Monitoring of cardiorespiratory signals using thermal imaging: A pilot study on healthy human subjects. Sensors, 18.","DOI":"10.3390\/s18051541"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/j.cviu.2006.08.011","article-title":"Coalitional tracking","volume":"106","author":"Dowdall","year":"2007","journal-title":"Comput. Vision Image Underst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"4378","DOI":"10.1364\/BOE.6.004378","article-title":"Remote monitoring of breathing dynamics using infrared thermography","volume":"6","author":"Pereira","year":"2015","journal-title":"Biomed. Opt. Express"},{"key":"ref_13","first-page":"199","article-title":"Automatic eye corners detection and tracking algorithm in sequence of thermal medical images","volume":"61","year":"2015","journal-title":"Meas. Autom. Monit."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Tzeng, H.W., Lee, H.C., and Chen, M.Y. (2011, January 8\u201310). The design of isotherm face recognition technique based on nostril localization. Proceedings of the 2011 International Conference on System Science and Engineering, Macao, China.","DOI":"10.1109\/ICSSE.2011.5961878"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Poster, D., Hu, S., Nasrabadi, N., and Riggan, B. (2019, January 16\u201320). An Examination of Deep-Learning Based Landmark Detection Methods on Thermal Face Imagery. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00129"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Kopaczka, M., Kolk, R., and Merhof, D. (2018, January 14\u201317). A fully annotated thermal face database and its application for thermal facial expression recognition. Proceedings of the 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), Houston, TX, USA.","DOI":"10.1109\/I2MTC.2018.8409768"},{"key":"ref_17","unstructured":"Kopaczka, M., Acar, K., and Merhof, D. (2019, September 11). Robust Facial Landmark Detection and Face Tracking in Thermal Infrared Images Using Active Appearance Models. Available online: https:\/\/pdfs.semanticscholar.org\/50a0\/930cb8cc353e15a5cb4d2f41b365675b5ebf.pdf."},{"key":"ref_18","unstructured":"Kopaczka, M., Schock, J., and Merhof, D. (2019). Super-realtime facial landmark detection and shape fitting by deep regression of shape model parameters. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Cardone, D., and Merla, A. (2017). New frontiers for applications of thermal infrared imaging devices: Computational psychopshysiology in the neurosciences. Sensors, 17.","DOI":"10.3390\/s17051042"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1007\/s00138-013-0570-5","article-title":"Thermal cameras and applications: A survey","volume":"25","author":"Gade","year":"2014","journal-title":"Mach. Vision Appl."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"951","DOI":"10.1111\/psyp.12243","article-title":"Thermal infrared imaging in psychophysiology: Potentialities and limits","volume":"51","author":"Ioannou","year":"2014","journal-title":"Psychophysiology"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Proch\u00e1zka, A., Charv\u00e1tov\u00e1, H., Vy\u0161ata, O., Kopal, J., and Chambers, J. (2017). Breathing analysis using thermal and depth imaging camera video records. Sensors, 17.","DOI":"10.3390\/s17061408"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Hu, M., Zhai, G., Li, D., Fan, Y., Duan, H., Zhu, W., and Yang, X. (2018). Combination of near-infrared and thermal imaging techniques for the remote and simultaneous measurements of breathing and heart rates under sleep situation. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0190466"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Gault, T., and Farag, A. (2013, January 23\u201328). A fully automatic method to extract the heart rate from thermal video. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Portland, OR, USA.","DOI":"10.1109\/CVPRW.2013.57"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1038\/srep00305","article-title":"Fast by nature-how stress patterns define human experience and performance in dexterous tasks","volume":"2","author":"Pavlidis","year":"2012","journal-title":"Sci. Rep."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kosonogov, V., De Zorzi, L., Honor\u00e9, J., Mart\u00ednez-Vel\u00e1zquez, E.S., Nandrino, J.L., Martinez-Selva, J.M., and Sequeira, H. (2017). Facial thermal variations: A new marker of emotional arousal. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0183592"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"4729","DOI":"10.1038\/s41598-019-41172-7","article-title":"Detecting changes in facial temperature induced by a sudden auditory stimulus based on deep learning-assisted face tracking","volume":"9","author":"Sonkusare","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_28","first-page":"97","article-title":"Human face recognition using thermal image","volume":"22","author":"Wang","year":"2002","journal-title":"J. Med. Biol. Eng."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Kri\u0161to, M., and Ivasic-Kos, M. (2018, January 21\u201325). An overview of thermal face recognition methods. Proceedings of the 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, Croatia.","DOI":"10.23919\/MIPRO.2018.8400200"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1788","DOI":"10.1016\/j.imavis.2009.05.007","article-title":"The painful face\u2013pain expression recognition using active appearance models","volume":"27","author":"Ashraf","year":"2009","journal-title":"Image Vision Comput."},{"key":"ref_31","unstructured":"Akgul, F. (2013). ZeroMQ, Packt Publishing Ltd."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/19\/4135\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:23:41Z","timestamp":1760189021000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/19\/4135"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,24]]},"references-count":31,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2019,10]]}},"alternative-id":["s19194135"],"URL":"https:\/\/doi.org\/10.3390\/s19194135","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2019,9,24]]}}}