{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T21:25:16Z","timestamp":1772573116072,"version":"3.50.1"},"reference-count":54,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T00:00:00Z","timestamp":1694649600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Appl. Math. Stat."],"abstract":"<jats:p>Microexpressions are fast and spatially small facial expressions that are difficult to detect. Therefore, motion magnification techniques, which aim at amplifying and hence revealing subtle motion in videos, appear useful for handling such expressions. There are basically two main approaches, namely, via Eulerian or Lagrangian techniques. While the first one magnifies motion implicitly by operating directly on image pixels, the Lagrangian approach uses optical flow (OF) techniques to extract and magnify pixel trajectories. In this study, we propose a novel approach for local Lagrangian motion magnification of facial micro-motions. Our contribution is 3-fold: first, we fine tune the recurrent all-pairs field transforms (RAFT) for OFs deep learning approach for faces by adding ground truth obtained from the variational dense inverse search (DIS) for the OF algorithm applied to the CASME II video set of facial micro expressions. This enables us to produce OFs of facial videos in an efficient and sufficiently accurate way. Second, since facial micro-motions are both local in space and time, we propose to approximate the OF field by sparse components both in space and time leading to a double sparse decomposition. Third, we use this decomposition to magnify micro-motions in specific areas of the face, where we introduce a new forward warping strategy using a triangular splitting of the image grid and barycentric interpolation of the RGB vectors at the corners of the transformed triangles. We demonstrate the feasibility of our approach by various examples.<\/jats:p>","DOI":"10.3389\/fams.2023.1164491","type":"journal-article","created":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T20:23:52Z","timestamp":1694723032000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Lagrangian motion magnification with double sparse optical flow decomposition"],"prefix":"10.3389","volume":"9","author":[{"given":"Philipp","family":"Flotho","sequence":"first","affiliation":[]},{"given":"Cosmas","family":"Heiss","sequence":"additional","affiliation":[]},{"given":"Gabriele","family":"Steidl","sequence":"additional","affiliation":[]},{"given":"Daniel J.","family":"Strauss","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,9,14]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2185520.2185561","article-title":"Eulerian video magnification for revealing subtle changes in the world","volume":"31","author":"Wu","year":"2012","journal-title":"ACM Trans Graph"},{"key":"B2","doi-asserted-by":"publisher","first-page":"4119","DOI":"10.1109\/CVPR.2015.7299039","article-title":"Video magnification in presence of large motions","author":"Elgharib","year":"2015","journal-title":"Proceedings of the IEEE Computer Social Conference Computer Vision Pattern Recognition"},{"key":"B3","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1145\/2461912.2461966","article-title":"Phase-based video motion processing","volume":"32","author":"Wadhwa","year":"2013","journal-title":"ACM Trans Graph"},{"key":"B4","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1109\/CVPR.2017.61","article-title":"Video acceleration magnification","author":"Zhang","year":"2017","journal-title":"Proceedings of the IEEE Computer Social Conference Computer Vision Pattern Recognition"},{"key":"B5","doi-asserted-by":"publisher","first-page":"300","DOI":"10.1016\/j.jsv.2018.01.050","article-title":"Vibration-based damage detection in wind turbine blades using Phase-based Motion Estimation and motion magnification","volume":"421","author":"Sarrafi","year":"2018","journal-title":"J Sound Vib"},{"key":"B6","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1007\/s40799-020-00393-6","article-title":"Modal parameter estimation of a compliant panel using phase-based motion magnification and stereoscopic digital image correlation","volume":"45","author":"Eitner","year":"2021","journal-title":"Exper Tech"},{"key":"B7","doi-asserted-by":"publisher","first-page":"30789","DOI":"10.1007\/s11042-020-09147-3","article-title":"Exposing AI-generated videos with motion magnification","volume":"80","author":"Fei","year":"2021","journal-title":"Multimed Tools Appl"},{"key":"B8","first-page":"1780","article-title":"Respiratory rate monitoring by video processing using local motion magnification","volume-title":"Proceedings of the European Signal Processing Conference EUSIPCO","author":"Alinovi","year":"2018"},{"key":"B9","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1145\/2185520.2185562","article-title":"Selectively de-animating video","volume":"31","author":"Bai","year":"2012","journal-title":"ACM Trans Graph"},{"key":"B10","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1145\/1073204.1073223","article-title":"Motion magnification","volume":"24","author":"Liu","year":"2005","journal-title":"ACM Trans Graph"},{"key":"B11","first-page":"3586","article-title":"Lagrangian motion magnification revisited: Continuous, magnitude driven motion scaling for psychophysiological experiments","volume-title":"Annual International Conference IEEE Engineering Medical Biology Social.","author":"Flotho","year":"2018"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1002\/jbio.202100330","article-title":"Software for non-parametric image registration of 2-photon imaging data","author":"Flotho","year":"2022","journal-title":"J Biophotonics"},{"key":"B13","doi-asserted-by":"publisher","first-page":"116872","DOI":"10.1016\/j.jmatprotec.2020.116872","article-title":"Measurement of strain, strain rate and crack evolution in shear cutting","volume":"288","author":"Hartmann","year":"2021","journal-title":"J Mater Process Technol"},{"key":"B14","doi-asserted-by":"publisher","first-page":"e54536","DOI":"10.7554\/eLife.54536","article-title":"Vestigial auriculomotor activity indicates the direction of auditory attention in humans","volume":"9","author":"Strauss","year":"2020","journal-title":"Elife"},{"key":"B15","doi-asserted-by":"publisher","first-page":"2443","DOI":"10.1109\/CVPR.2013.316","article-title":"Large displacement optical flow from nearest neighbor fields","author":"Chen","year":"2013","journal-title":"Proceedings of the IEEE Computer Social Conference Computer Vision Pattern Recognition"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s11263-010-0390-2","article-title":"database and evaluation methodology for optical flow","volume":"92","author":"Baker","year":"2011","journal-title":"Int J Comput Vis"},{"key":"B17","doi-asserted-by":"crossref","first-page":"2432","DOI":"10.1109\/CVPR.2010.5539939","article-title":"Secrets of optical flow estimation and their principles","volume-title":"2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition","author":"Sun","year":"2010"},{"key":"B18","first-page":"471","article-title":"Fast optical flow using dense inverse search","volume-title":"Computer Vision ECCV","author":"Kroeger","year":"2016"},{"key":"B19","first-page":"25","article-title":"High accuracy optical flow estimation based on a theory for warping","volume-title":"Computer Vision ECCV","author":"Brox","year":"2004"},{"key":"B20","doi-asserted-by":"publisher","first-page":"4040","DOI":"10.1109\/CVPR.2016.438","article-title":"A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation","author":"Mayer","year":"2016","journal-title":"Proceedings of the IEEE Computer Social Conference Computer Vision Pattern Recognition"},{"key":"B21","first-page":"402","article-title":"Raft: Recurrent all-pairs field transforms for optical flow","volume-title":"Computer Vision ECCV","author":"Teed","year":"2020"},{"key":"B22","first-page":"611","article-title":"A naturalistic open source movie for optical flow evaluation","volume-title":"European Conference on Computer Vision (ECCV) Part IV, LNCS 7577.","author":"Butler","year":"2012"},{"key":"B23","first-page":"165","article-title":"Disentangling architecture and training for optical flow","volume-title":"European Conference on Computer Vision (ECCV)","author":"Sun","year":"2022"},{"key":"B24","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1007\/s10919-013-0159-8","article-title":"How fast are the leaked facial expressions: The duration of micro-expressions","volume":"37","author":"Yan","year":"2013","journal-title":"J Nonverbal Behav"},{"key":"B25","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1109\/TAFFC.2017.2713359","article-title":"Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition","volume":"10","author":"Huang","year":"2017","journal-title":"IEEE Trans Affect Comput"},{"key":"B26","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1109\/TAFFC.2016.2518162","article-title":"Microexpression identification and categorization using a facial dynamics map","volume":"8","author":"Xu","year":"2017","journal-title":"IEEE Trans Affect Comput"},{"key":"B27","first-page":"650","article-title":"Micro-expression motion magnification: Global Lagrangian vs. local Eulerian approaches.","volume-title":"IEEE International Conference Automation Face Gesture Recognition Workshops","author":"N","year":"2018"},{"key":"B28","doi-asserted-by":"publisher","first-page":"414","DOI":"10.3390\/make3020021","article-title":"Review of automatic microexpression recognition in the past decade","volume":"3","author":"Zhang","year":"2021","journal-title":"Mach Learn Knowl Extr"},{"key":"B29","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1109\/CISP.2013.6743978","article-title":"Video-based heart rate measurement using head motion tracking and ICA","volume-title":"2013 6th International Congress on Image and Signal Processing (CISP)","author":"Shan","year":"2013"},{"key":"B30","first-page":"2130","article-title":"Deep multimodal learning for audio-visual speech recognition","volume-title":"Proceedings of the IEEE International Conference Acoustic Speech Signal Process","author":"Mroueh","year":"2015"},{"key":"B31","doi-asserted-by":"crossref","first-page":"689","DOI":"10.1145\/1553374.1553463","article-title":"Online dictionary learning for sparse coding","volume-title":"Annual International Conference on Machine Learning ICML '09","author":"Mairal","year":"2009"},{"key":"B32","doi-asserted-by":"publisher","first-page":"626","DOI":"10.1007\/s10851-021-01019-1","article-title":"Robust PCA via regularized REAPER with a matrix-free proximal algorithm","volume":"63","author":"Beinert","year":"2021","journal-title":"J Mathem Imag Vis."},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.309","article-title":"Unifying nuclear norm and bilinear factorization approaches for low-rank matrix decomposition","author":"Cabral","year":"2013","journal-title":"Proceedings of the 2013 IEEE International Conference on Computer Vision"},{"key":"B34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1970392.1970395","article-title":"Robust principal component analysis?","volume":"58","author":"Cand\u00e9s","year":"2011","journal-title":"J ACM"},{"key":"B35","doi-asserted-by":"publisher","first-page":"2258","DOI":"10.1137\/130948793","article-title":"On infimal convolution of TV-type functionals and applications to video and image reconstruction","volume":"7","author":"Holler","year":"2014","journal-title":"SIAM J Imag Sci"},{"key":"B36","volume-title":"Digital Image Warping","author":"Wolberg","year":"1990"},{"key":"B37","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1016\/0004-3702(81)90024-2","article-title":"Determining optical flow","volume":"17","author":"Horn","year":"1981","journal-title":"Art Intelligence"},{"key":"B38","doi-asserted-by":"crossref","first-page":"1945","DOI":"10.1007\/978-1-4939-0790-8_38","article-title":"Optical Flow","volume-title":"Handbook of Mathematical Methods in Imaging","author":"Becker","year":"2015"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.316","article-title":"FlowNet: learning optical flow with convolutional networks","author":"Dosovitskiy","year":"2015","journal-title":"IEEE International Conference on Computer Vision (ICCV)"},{"key":"B40","doi-asserted-by":"publisher","first-page":"3061","DOI":"10.1109\/CVPR.2015.7298925","article-title":"Object scene flow for autonomous vehicles","author":"Menze","year":"2015","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B41","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1109\/CVPRW.2016.10","article-title":"The HCI benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving","author":"Kondermann","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops"},{"key":"B42","doi-asserted-by":"publisher","first-page":"e86041","DOI":"10.1371\/journal.pone.0086041","article-title":"CASME II: An improved spontaneous micro-expression database and the baseline evaluation","volume":"9","author":"Yan WJ Li","year":"2014","journal-title":"PLoS ONE"},{"key":"B43","first-page":"1","article-title":"A spontaneous micro-expression database: Inducement, collection and baseline","volume-title":"IEEE International Conference Automation Face Gesture Recognition Workshops","author":"Li","year":"2013"},{"key":"B44","first-page":"122","article-title":"The OpenCV library","volume":"120","author":"Bradski","year":"2000","journal-title":"Dr Dobb's J Softw Tools"},{"key":"B45","doi-asserted-by":"crossref","DOI":"10.1137\/1.9781611974997","volume-title":"First-Order Methods in Optimization","author":"Beck","year":"2017"},{"key":"B46","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1007\/978-3-319-41589-5_10","article-title":"First Order algorithms in variational image processing","volume-title":"Splitting Methods in Communication, Imaging, Science, and Engineering","author":"Burger","year":"2016"},{"key":"B47","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1093\/imanum\/20.3.389","article-title":"new approach to variable selection in least squares problems","volume":"20","author":"Osborne","year":"2000","journal-title":"IMA J Numer Anal"},{"key":"B48","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1214\/009053604000000067","article-title":"Least angle regression","volume":"32","author":"Efron","year":"2004","journal-title":"Ann Stat"},{"key":"B49","volume-title":"Nonlinear Programming","author":"Bertsekas","year":"1999"},{"key":"B50","doi-asserted-by":"publisher","first-page":"e202000512","DOI":"10.1002\/jbio.202000512","article-title":"Multimodal data acquisition at SARS-CoV-2 drive through screening centers: Setup description and experiences in Saarland, Germany","volume":"14","author":"Flotho","year":"2021","journal-title":"J Biophotonics"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19790-1_40","article-title":"FlowFormer: a transformer architecture for optical flow","author":"Huang","year":"2022"},{"key":"B52","first-page":"2215","article-title":"Lagrangian motion magnification with landmark-prior and sparse PCA for facial microexpressions and micromovements","volume-title":"Annual International Conference IEEE Engineering Medical Biology Social","author":"Flotho","year":"2022"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2022.05.077","article-title":"A comparative study on optical flow for facial expression analysis","author":"Allaert","year":"2022","journal-title":"Neurocomputing"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2022.3197622","article-title":"Self-supervised approach for facial movement based optical flow","author":"Alkaddour","year":"2021","journal-title":"arXiv:2105.01256"}],"container-title":["Frontiers in Applied Mathematics and Statistics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fams.2023.1164491\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T20:24:07Z","timestamp":1694723047000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fams.2023.1164491\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,14]]},"references-count":54,"alternative-id":["10.3389\/fams.2023.1164491"],"URL":"https:\/\/doi.org\/10.3389\/fams.2023.1164491","relation":{},"ISSN":["2297-4687"],"issn-type":[{"value":"2297-4687","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,14]]},"article-number":"1164491"}}