{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T21:57:20Z","timestamp":1769551040777,"version":"3.49.0"},"publisher-location":"Cham","reference-count":32,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783030876630","type":"print"},{"value":"9783030876647","type":"electronic"}],"license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,1,31]],"date-time":"2022-01-31T00:00:00Z","timestamp":1643587200000},"content-version":"vor","delay-in-days":30,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Falsified\u00a0media threatens key areas of our society, ranging from politics to journalism to economics. Simple and inexpensive tools available today enable easy, credible manipulations of multimedia assets. Some even utilize advanced artificial intelligence\u00a0concepts to manipulate media, resulting in videos known as <jats:italic>deepfakes<\/jats:italic>. Social media platforms and their \u201cecho chamber\u201d effect propagate fabricated digital content at scale, sometimes with dire consequences in real-world situations. However, ensuring semantic consistency across falsified media assets of different modalities is still very challenging for current deepfake\u00a0tools. Therefore, cross-modal analysis (e.g.,\u00a0 video-based and audio-based analysis) provides forensic analysts an opportunity to identify inconsistencies with higher accuracy. In this chapter, we introduce several approaches to detect deepfakes. These approaches leverage different data modalities, including video and audio. We show that the presented methods achieve accurate detection for various large-scale datasets.<\/jats:p>","DOI":"10.1007\/978-3-030-87664-7_11","type":"book-chapter","created":{"date-parts":[[2022,1,31]],"date-time":"2022-01-31T09:03:06Z","timestamp":1643619786000},"page":"235-254","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Deepfake Detection Using Multiple Data Modalities"],"prefix":"10.1007","author":[{"given":"Hanxiang","family":"Hao","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emily R.","family":"Bartusiak","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"G\u00fcera","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"Mas Montserrat","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sriram","family":"Baireddy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ziyue","family":"Xiang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sri Kalyan","family":"Yarlagadda","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ruiting","family":"Shao","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"J\u00e1nos","family":"Horv\u00e1th","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Justin","family":"Yang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fengqing","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edward J.","family":"Delp","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,1,31]]},"reference":[{"key":"11_CR1","doi-asserted-by":"crossref","unstructured":"Yang X, Li Y, Lyu S (2019) Exposing deep fakes using inconsistent head poses. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing, May 2019","DOI":"10.1109\/ICASSP.2019.8683164"},{"key":"11_CR2","doi-asserted-by":"crossref","unstructured":"Yang X, Li Y, Qi H, Lyu S (2019) Exposing gan-synthesized faces using landmark locations. In: Proceedings of the international workshop on information hiding and multimedia security, July 2019","DOI":"10.1145\/3335203.3335724"},{"key":"11_CR3","doi-asserted-by":"crossref","unstructured":"Marra F, Gragnaniello D, Cozzolino D, Verdoliva L (2018) Detection of gan-generated fake images over social networks. In: Proceedings of the IEEE conference on multimedia information processing and retrieval, April 2018","DOI":"10.1109\/MIPR.2018.00084"},{"key":"11_CR4","doi-asserted-by":"crossref","unstructured":"Gragnaniello D, Cozzolino D, Marra F, Poggi G, Verdoliva L (2021) Are GAN generated images easy to detect? A critical analysis of the state-of-the-art. In: Proceedings of the IEEE international conference on multimedia and expo, July 2021","DOI":"10.1109\/ICME51207.2021.9428429"},{"key":"11_CR5","doi-asserted-by":"crossref","unstructured":"Cozzolino D, R\u00f6ssler A, Thies J, Nie\u00dfner M, Verdoliva L (2021) Id-reveal: Identity-aware deepfake video detection. In: arXiv preprint arXiv:2012.02512, December 2021","DOI":"10.1109\/ICCV48922.2021.01483"},{"issue":"5","key":"11_CR6","doi-asserted-by":"publisher","first-page":"910","DOI":"10.1109\/JSTSP.2020.3002101","volume":"14","author":"L Verdoliva","year":"2020","unstructured":"Verdoliva L (2020) Media forensics and deepfakes: an overview. IEEE J Select Topics Signal Process 14(5):910\u2013932","journal-title":"IEEE J Select Topics Signal Process"},{"key":"11_CR7","doi-asserted-by":"crossref","unstructured":"Tolosana R, Vera-Rodr\u00edguez R, Fi\u00e9rrez J, Morales A, Ortega-Garcia J (2020) Deepfakes and beyond: a survey of face manipulation and fake detection. In: arXiv preprint arXiv:2001.00179, January 2020","DOI":"10.1016\/j.inffus.2020.06.014"},{"key":"11_CR8","doi-asserted-by":"crossref","unstructured":"Mirsky Y, Lee W (2021) The creation and detection of deepfakes: a survey. In: ACM Computing survey, vol 54, No 1, January 2021","DOI":"10.1145\/3425780"},{"key":"11_CR9","doi-asserted-by":"crossref","unstructured":"Nguyen TT, Nguyen CM, Nguyen DT, Nguyen DT, Nahavandi S (2021) Deep learning for deepfakes creation and detection. In: arXiv preprint arXiv:1909.11573, April 2021","DOI":"10.2139\/ssrn.4030341"},{"key":"11_CR10","first-page":"1","volume-title":"IEEE international conference on advanced video and signal based surveillance, November 2018","author":"D G\u00fcera","year":"2018","unstructured":"G\u00fcera D, Delp EJ (2018) Deepfake video detection using recurrent neural networks. IEEE international conference on advanced video and signal based surveillance, November 2018. Auckland, New Zealand, pp 1\u20136"},{"key":"11_CR11","doi-asserted-by":"crossref","unstructured":"Montserrat D, Hao H, Yarlagadda S, Baireddy S, Shao R, Horvath J, Bartusiak ER, Yang J, Guera D, Zhu F, Delp E (2020) Deepfakes detection with automatic face weighting. In: IEEE conference on computer vision and pattern recognition workshops, June 2020, pp 2851\u20132859","DOI":"10.1109\/CVPRW50498.2020.00342"},{"key":"11_CR12","doi-asserted-by":"crossref","unstructured":"Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, June 2016, Las Vegas, pp 2818\u20132826","DOI":"10.1109\/CVPR.2016.308"},{"key":"11_CR13","unstructured":"Tan M, Le QV (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: arXiv preprint arXiv:1905.11946"},{"key":"11_CR14","doi-asserted-by":"crossref","unstructured":"Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, July 2017, Honolulu, pp 1251\u20131258","DOI":"10.1109\/CVPR.2017.195"},{"issue":"3","key":"11_CR15","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky O, Deng J, Hao S, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211\u2013252","journal-title":"Int J Comput Vis"},{"key":"11_CR16","doi-asserted-by":"crossref","unstructured":"Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(7)","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"11_CR17","doi-asserted-by":"crossref","unstructured":"Cho K, van Merri\u00ebnboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder\u2013decoder for statistical machine translation. In: Proceedings of the conference on empirical methods in natural language processing, October 2014","DOI":"10.3115\/v1\/D14-1179"},{"key":"11_CR18","doi-asserted-by":"crossref","unstructured":"Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. In: IEEE signal processing letters, vol 23, April 2016","DOI":"10.1109\/LSP.2016.2603342"},{"key":"11_CR19","doi-asserted-by":"crossref","unstructured":"Deng J, Guo J, Xue N, Zafeiriou S (2019) ArcFace: additive angular margin loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, June 2019, Long Beach","DOI":"10.1109\/CVPR.2019.00482"},{"key":"11_CR20","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I (2017) Attention is all you need. In: Proceedings of advances in neural information processing systems, December 2017, Long Beach, pp 5998\u20136008"},{"key":"11_CR21","unstructured":"Dolhansky B, Howes R, Pflaum B, Baram N, Ferrer CC (2019) The deepfake detection challenge (dfdc) preview dataset. In: arXiv preprint arXiv:1910.08854"},{"key":"11_CR22","unstructured":"Kingma D,\u00a0Ba J (2015) Adam: A method for stochastic optimization. In: Proceedings of the IEEE conference on international conference for learning representations, May 2015"},{"key":"11_CR23","doi-asserted-by":"crossref","unstructured":"Suwajanakorn S, Seitz SM, Kemelmacher-Shlizerman I (2017) Synthesizing obama: learning lip sync from audio. ACM Trans Graph 36(4)","DOI":"10.1145\/3263919"},{"key":"11_CR24","volume-title":"Asvspoof 2019: The 3rd automatic speaker verification spoofing and countermeasures challenge database","author":"J Yamagishi","year":"2019","unstructured":"Yamagishi J, Todisco M, Sahidullah M, Delgado H, Wang X, Evans N, Kinnunen T, Lee K, Vestman V, Nautsch A (2019) Asvspoof 2019: The 3rd automatic speaker verification spoofing and countermeasures challenge database. University of Edinburgh, The Centre for Speech Technology Research"},{"key":"11_CR25","unstructured":"Todisco M, Yamagishi J, Sahidullah M, Delgado H, Wang X, Evans N, Kinnunen T, Lee K, Vestman V, Nautsch A (2019) Asvspoof 2019: Automatic speaker verification spoofing and countermeasures challenge evaluation plan. In: ASVspoof consortium, January 2019"},{"key":"11_CR26","unstructured":"Kingma D,\u00a0Ba J (2015) Adam: a method for stochastic optimization. In: Proceedings of the international conference for learning representations, May 2015, San Diego"},{"key":"11_CR27","doi-asserted-by":"crossref","unstructured":"Korshunov P,\u00a0Marcel S (2018) Speaker inconsistency detection in tampered videos. In: Proceedings of the IEEE European signal processing conference, September 2018, pp 2375\u20132379","DOI":"10.23919\/EUSIPCO.2018.8553270"},{"key":"11_CR28","doi-asserted-by":"crossref","unstructured":"Agarwal S, Farid H, Fried O, Agrawala M (2020) Detecting deep-fake videos from phoneme-viseme mismatches. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, June 2020, pp 2814\u20132822","DOI":"10.1109\/CVPRW50498.2020.00338"},{"key":"11_CR29","doi-asserted-by":"crossref","unstructured":"Mittal T, Bhattacharya U, Chandra R, Bera A, Manocha D (2020) Emotions don\u2019t lie: an audio-visual deepfake detection method using affective cues. In: Proceedings of the ACM international conference on multimedia, October 2020, Seattle, pp 2823\u20132832","DOI":"10.1145\/3394171.3413570"},{"key":"11_CR30","doi-asserted-by":"crossref","unstructured":"Zadeh A, Liang PP, Mazumder N, Poria S, Cambria E, Morency L-P (2018) Memory fusion network for multi-view sequential learning. In: Proceedings of the AAAI conference on artificial intelligence","DOI":"10.1609\/aaai.v32i1.12021"},{"key":"11_CR31","doi-asserted-by":"crossref","unstructured":"Schroff F, Kalenichenko D, Philbin J (2015) FaceNet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE computer vision and pattern recognition. Boston, pp 815\u2013823","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"11_CR32","unstructured":"G\u00fcera D, Baireddy S, Bestagini P, Tubaro S, Delp EJ (2019) We need no pixels: Video manipulation detection using stream descriptors. In: Proceedings of the international conference on machine learning, synthetic-realities: deep learning for detecting audiovisual fakes workshop, June 2019, Long Beach"}],"container-title":["Advances in Computer Vision and Pattern Recognition","Handbook of Digital Face Manipulation and Detection"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-87664-7_11","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,9]],"date-time":"2024-02-09T07:04:44Z","timestamp":1707462284000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-030-87664-7_11"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"ISBN":["9783030876630","9783030876647"],"references-count":32,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-87664-7_11","relation":{},"ISSN":["2191-6586","2191-6594"],"issn-type":[{"value":"2191-6586","type":"print"},{"value":"2191-6594","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022]]},"assertion":[{"value":"31 January 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}}]}}