{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T16:39:26Z","timestamp":1792082366534,"version":"4.4.0"},"reference-count":51,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2023,4,29]],"date-time":"2023-04-29T00:00:00Z","timestamp":1682726400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"EU\u2014NGEU","award":["951911"],"award-info":[{"award-number":["951911"]}]},{"name":"project AI4Media","award":["951911"],"award-info":[{"award-number":["951911"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>The increasing use of deep learning techniques to manipulate images and videos, commonly referred to as \u201cdeepfakes\u201d, is making it more challenging to differentiate between real and fake content, while various deepfake detection systems have been developed, they often struggle to detect deepfakes in real-world situations. In particular, these methods are often unable to effectively distinguish images or videos when these are modified using novel techniques which have not been used in the training set. In this study, we carry out an analysis of different deep learning architectures in an attempt to understand which is more capable of better generalizing the concept of deepfake. According to our results, it appears that Convolutional Neural Networks (CNNs) seem to be more capable of storing specific anomalies and thus excel in cases of datasets with a limited number of elements and manipulation methodologies. The Vision Transformer, conversely, is more effective when trained with more varied datasets, achieving more outstanding generalization capabilities than the other methods analysed. Finally, the Swin Transformer appears to be a good alternative for using an attention-based method in a more limited data regime and performs very well in cross-dataset scenarios. All the analysed architectures seem to have a different way to look at deepfakes, but since in a real-world environment the generalization capability is essential, based on the experiments carried out, the attention-based architectures seem to provide superior performances.<\/jats:p>","DOI":"10.3390\/jimaging9050089","type":"journal-article","created":{"date-parts":[[2023,5,1]],"date-time":"2023-05-01T13:54:46Z","timestamp":1682949286000},"page":"89","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["On the Generalization of Deep Learning Models in Video Deepfake Detection"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0755-6154","authenticated-orcid":false,"given":"Davide Alessandro","family":"Coccomini","sequence":"first","affiliation":[{"name":"Istituto di Scienza e Tecnologie dell\u2019Informazione, 56124 Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3471-1196","authenticated-orcid":false,"given":"Roberto","family":"Caldelli","sequence":"additional","affiliation":[{"name":"National Inter-University Consortium for Telecommunications (CNIT), 50134 Florence, Italy"},{"name":"Faculty of Economics, Universitas Mercatorum, 00186 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6258-5313","authenticated-orcid":false,"given":"Fabrizio","family":"Falchi","sequence":"additional","affiliation":[{"name":"Istituto di Scienza e Tecnologie dell\u2019Informazione, 56124 Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3715-149X","authenticated-orcid":false,"given":"Claudio","family":"Gennaro","sequence":"additional","affiliation":[{"name":"Istituto di Scienza e Tecnologie dell\u2019Informazione, 56124 Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,4,29]]},"reference":[{"key":"ref_1","unstructured":"Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv."},{"key":"ref_2","unstructured":"Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8\u201313). Generative Adversarial Networks. Proceedings of the Advances in Neural Information Processing Systems 27, Montreal, QC, Canada."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Thies, J., Zollh\u00f6fer, M., Stamminger, M., Theobalt, C., and Nie\u00dfner, M. (2016, January 27\u201330). Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.262"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Korshunova, I., Shi, W., Dambre, J., and Theis, L. (2017, January 21\u201326). Fast Face-Swap Using Convolutional Neural Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Honolulu, HI, USA.","DOI":"10.1109\/ICCV.2017.397"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/JPROC.2021.3049196","article-title":"Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications","volume":"109","author":"Liu","year":"2021","journal-title":"Proc. IEEE"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Karras, T., Laine, S., and Aila, T. (2018, January 18\u201322). A Style-Based Generator Architecture for Generative Adversarial Networks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition 2018, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2019.00453"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Fagni, T., Falchi, F., Gambini, M., Martella, A., and Tesconi, M. (2021). TweepFake: About detecting deepfake tweets. PLoS ONE, 16.","DOI":"10.1371\/journal.pone.0251415"},{"key":"ref_8","unstructured":"Coccomini, D.A., Zilos, G.K., Amato, G., Caldelli, R., Falchi, F., Papadopoulos, S., and Gennaro, C. (2022). MINTIME: Multi-Identity Size-Invariant Video Deepfake Detection. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Baxevanakis, S., Kordopatis-Zilos, G., Galopoulos, P., Apostolidis, L., Levacher, K., Schlicht, I.B., Teyssou, D., Kompatsiaris, I., and Papadopoulos, S. (2022, January 27\u201330). The MeVer DeepFake Detection Service: Lessons Learnt from Developing and Deploying in the Wild. Proceedings of the 1st International Workshop on Multimedia AI against Disinformation, Newark, NJ, USA.","DOI":"10.1145\/3512732.3533587"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zheng, Y., Bao, J., Chen, D., Zeng, M., and Wen, F. (2021, January 11\u201317). Exploring Temporal Coherence for More General Video Face Forgery Detection. Proceedings of the ICCV, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.01477"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.patrec.2021.03.005","article-title":"Optical Flow based CNN for detection of unlearnt deepfake manipulations","volume":"146","author":"Caldelli","year":"2021","journal-title":"Pattern Recognit. Lett."},{"key":"ref_12","unstructured":"Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., and Ferrer, C.C. (2020). The deepfake detection challenge (dfdc) dataset. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Guarnera, L., Giudice, O., Guarnera, F., Ortis, A., Puglisi, G., Paratore, A., Bui, L.M.Q., Fontani, M., Coccomini, D.A., and Caldelli, R. (2022). The Face Deepfake Detection Challenge. J. Imaging, 8.","DOI":"10.3390\/jimaging8100263"},{"key":"ref_14","unstructured":"Korshunov, P., and Marcel, S. (2018). Deepfakes: A new threat to face recognition? assessment and detection. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Yang, X., Li, Y., and Lyu, S. (2019, January 12\u201317). Exposing deep fakes using inconsistent head poses. Proceedings of the ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8683164"},{"key":"ref_16","unstructured":"Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., and Nie\u00dfner, M. (November, January 27). Faceforensics++: Learning to detect manipulated facial images. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Li, Y., Yang, X., Sun, P., Qi, H., and Lyu, S. (2020, January 13\u201319). Celeb-df: A large-scale challenging dataset for deepfake forensics. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00327"},{"key":"ref_18","unstructured":"Dufour, N., and Gully, A. (2023, April 25). Contributing Data to Deep-Fake Detection Research. Available online: https:\/\/ai.googleblog.com\/2019\/09\/contributing-data-to-deepfake-detection.html."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Jiang, L., Li, R., Wu, W., Qian, C., and Loy, C.C. (2020, January 13\u201319). Deeperforensics-1.0: A large-scale dataset for real-world face forgery detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00296"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"He, Y., Gan, B., Chen, S., Zhou, Y., Yin, G., Song, L., Sheng, L., Shao, J., and Liu, Z. (2021, January 20\u201325). ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00434"},{"key":"ref_21","unstructured":"Seferbekov, S. (2023, April 25). DFDC 1st Place Solution. Available online: https:\/\/github.com\/selimsef\/dfdc_deepfake_challenge."},{"key":"ref_22","unstructured":"Wodajo, D., and Atnafu, S. (2021). Deepfake Video Detection Using Convolutional Vision Transformer. arXiv."},{"key":"ref_23","unstructured":"Heo, Y.J., Choi, Y.J., Lee, Y.W., and Kim, B.G. (2021). Deepfake Detection Scheme Based on Vision Transformer and Distillation. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Chen, C.F., Fan, Q., and Panda, R. (2021). Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv.","DOI":"10.1109\/ICCV48922.2021.00041"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Coccomini, D., Messina, N., Gennaro, C., and Falchi, F. (2022). Combining EfficientNet and Vision Transformers for Video Deepfake Detection. arXiv.","DOI":"10.1007\/978-3-031-06433-3_19"},{"key":"ref_26","unstructured":"Tan, M., and Le, Q.V. (2021). EfficientNetV2: Smaller Models and Faster Training. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chen, L., Maddox, R.K., Duan, Z., and Xu, C. (2019, January 15\u201320). Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00802"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Deng, Y., Yang, J., Chen, D., Wen, F., and Tong, X. (2020, January 13\u201319). Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00520"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Jo, Y., and Park, J. (November, January 27). SC-FEGAN: Face Editing Generative Adversarial Network with User\u2019s Sketch and Color. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00183"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020, January 13\u201319). Analyzing and Improving the Image Quality of StyleGAN. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00813"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Lee, C.H., Liu, Z., Wu, L., and Luo, P. (2020, January 13\u201319). MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00559"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Li, L., Bao, J., Yang, H., Chen, D., and Wen, F. (2020, January 13\u201319). Advancing High Fidelity Identity Swapping for Forgery Detection. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00512"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Nirkin, Y., Keller, Y., and Hassner, T. (November, January 27). FSGAN: Subject Agnostic Face Swapping and Reenactment. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00728"},{"key":"ref_34","unstructured":"Siarohin, A., Lathuiliere, S., Tulyakov, S., Ricci, E., and Sebe, N. (2019). First Order Motion Model for Image Animation. arXiv."},{"key":"ref_35","unstructured":"Petrov, I., Gao, D., Chervoniy, N., Liu, K., Marangonda, S., Um\u00e9, C., Dpfks, M., Luis, R., Jiang, J., and Zhang, S. (2020). DeepFaceLab: A simple, flexible and extensible face swapping framework. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3306346.3323028","article-title":"Text-based Editing of Talking-head Video","volume":"38","author":"Fried","year":"2019","journal-title":"ACM Trans. Graph."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1499","DOI":"10.1109\/LSP.2016.2603342","article-title":"Joint face detection and alignment using multitask cascaded convolutional networks","volume":"23","author":"Zhang","year":"2016","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_38","unstructured":"Buslaev, A., Parinov, A., Khvedchenya, E., Iglovikov, V.I., and Kalinin, A.A. (2018). Albumentations: Fast and flexible image augmentations. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Coccomini, D.A., Caldelli, R., Falchi, F., Gennaro, C., and Amato, G. (2022, January 27\u201330). Cross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection. Proceedings of the 1st International Workshop on Multimedia AI against Disinformation, New York, NY, USA.","DOI":"10.1145\/3512732.3533582"},{"key":"ref_40","unstructured":"Kolesnikov, A., Dosovitskiy, A., Weissenborn, D., Heigold, G., Uszkoreit, J., Beyer, L., Minderer, M., Dehghani, M., Houlsby, N., and Gelly, S. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 20\u201325). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., and Guo, B. (2020, January 13\u201319). Face X-ray for More General Face Forgery Detection. Proceedings of the CVPR, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00505"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021, January 10\u201317). Emerging Properties in Self-Supervised Vision Transformers. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00951"},{"key":"ref_44","unstructured":"Li, Y., and Lyu, S. (2019, January 16\u201320). Exposing DeepFake Videos By Detecting Face Warping Artifacts. Proceedings of the CVPR Workshops, Long Beach, CA, USA."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Haliassos, A., Mira, R., Petridis, S., and Pantic, M. (2022, January 18\u201324). Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection. Proceedings of the CVPR, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01453"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Nguyen, H.H., Fang, F., Yamagishi, J., and Echizen, I. (2019, January 23\u201326). Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos. Proceedings of the 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS), Tampa, FL, USA.","DOI":"10.1109\/BTAS46853.2019.9185974"},{"key":"ref_47","unstructured":"Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., and Natarajan, P. (2019, January 16\u201320). Recurrent Convolutional Strategies for Face Manipulation Detection in Videos. Proceedings of the CVPR Workshops, Long Beach, CA, USA."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Wang, S.Y., Wang, O., Zhang, R., Owens, A., and Efros, A.A. (2020, January 14\u201319). CNN-Generated Images Are Surprisingly Easy to Spot\u2026 for Now. Proceedings of the CVPR, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00872"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Haliassos, A., Vougioukas, K., Petridis, S., and Pantic, M. (2021, January 20\u201325). Lips Don\u2019t Lie: A Generalisable and Robust Approach to Face Forgery Detection. Proceedings of the CVPR, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00500"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Khalil, S.S., Youssef, S.M., and Saleh, S.N. (2021). iCaps-Dfake: An Integrated Capsule-Based Model for Deepfake Image and Video Detection. Future Internet, 13.","DOI":"10.3390\/fi13040093"}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/5\/89\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:26:34Z","timestamp":1760124394000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/5\/89"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,29]]},"references-count":51,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["jimaging9050089"],"URL":"https:\/\/doi.org\/10.3390\/jimaging9050089","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,29]]}}}