{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T10:21:03Z","timestamp":1771064463859,"version":"3.50.1"},"reference-count":47,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T00:00:00Z","timestamp":1671062400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003093","name":"Ministry of Higher Education, Malaysia","doi-asserted-by":"publisher","award":["FRGS\/1\/2021\/ICT07\/UMP\/02\/1"],"award-info":[{"award-number":["FRGS\/1\/2021\/ICT07\/UMP\/02\/1"]}],"id":[{"id":"10.13039\/501100003093","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Deep learning is notably successful in data analysis, computer vision, and human control. Nevertheless, this approach has inevitably allowed the development of DeepFake video sequences and images that could be altered so that the changes are not easily or explicitly detectable. Such alterations have been recently used to spread false news or disinformation. This study aims to identify <jats:italic>Deepfaked<\/jats:italic> videos and images and alert viewers to the possible falsity of the information. The current work presented a novel means of revealing fake face videos by cascading the convolution network with recurrent neural networks and fully connected network (FCN) models. The system detection approach utilizes the eye-blinking state in temporal video frames. Notwithstanding, it is deemed challenging to precisely depict (i) artificiality in fake videos and (ii) spatial information within the individual frame through this physiological signal. Spatial features were extracted using the VGG16 network and trained with the ImageNet dataset. The temporal features were then extracted in every 20 sequences through the LSTM network. On another note, the pre-processed eye-blinking state served as a probability to generate a novel BPD dataset. This newly-acquired dataset was fed to three models for training purposes with each entailing four, three, and six hidden layers, respectively. Every model constitutes a unique architecture and specific dropout value. Resultantly, the model optimally and accurately identified tampered videos within the dataset. The study model was assessed using the current BPD dataset based on one of the most complex datasets (FaceForensic++) with 90.8% accuracy. Such precision was successfully maintained in datasets that were not used in the training process. The training process was also accelerated by lowering the computation prerequisites.<\/jats:p>","DOI":"10.1371\/journal.pone.0278989","type":"journal-article","created":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T18:48:07Z","timestamp":1671130087000},"page":"e0278989","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":18,"title":["Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence"],"prefix":"10.1371","volume":"17","author":[{"given":"Muhammad Salihin","family":"Saealal","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0795-4096","authenticated-orcid":true,"given":"Mohd Zamri","family":"Ibrahim","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David. J.","family":"Mulvaney","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohd Ibrahim","family":"Shapiai","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Norasyikin","family":"Fadilah","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2022,12,15]]},"reference":[{"issue":"11","key":"pone.0278989.ref001","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1145\/3422622","article-title":"Generative Adversarial Networks","volume":"63","author":"I Goodfellow","year":"2020","journal-title":"Communications of the ACM"},{"key":"pone.0278989.ref002","first-page":"701","volume-title":"Advances in Neural Information Processing Systems","author":"MY Liu","year":"2017"},{"key":"pone.0278989.ref003","unstructured":"He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. vol. 2016-December. IEEE Computer Society; 2016. p. 770\u2013778. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-84986274465&doi=10.1109%2fCVPR.2016.90&partnerID=40&md5=f67e8d2a623bac88aad535d2c0a6d374."},{"issue":"8","key":"pone.0278989.ref004","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"S Hochreiter","year":"1997","journal-title":"Neural Computation"},{"key":"pone.0278989.ref005","unstructured":"Isola P, Zhu JY, Zhou T, Efros AA. Image-to-Image Translation with Conditional Adversarial Networks. In: Proceedings\u201430th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. vol. 2017-January. Institute of Electrical and Electronics Engineers Inc.; 2017. p. 5967\u20135976. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85030759098&doi=10.1109%2fCVPR.2017.632&partnerID=40&md5=fb2a5fe5a1479af939bffe544bd49dcd."},{"issue":"7","key":"pone.0278989.ref006","article-title":"A Study of Deep CNN-Based Classification of Open and Closed Eyes Using a Visible Light Camera Sensor","volume":"17","author":"KW Kim","year":"2017","journal-title":"Sensors (Switzerland)"},{"key":"pone.0278989.ref007","unstructured":"Kingma DP, Lei Ba J. Adam: A Method for Stochastic Optimization. In: 3rd International Conference on Learning Representations, ICLR 2015\u2014Conference Track Proceedings. International Conference on Learning Representations, ICLR; 2015. Available from: https:\/\/arxiv.org\/abs\/1412.6980."},{"issue":"5","key":"pone.0278989.ref008","doi-asserted-by":"crossref","first-page":"910","DOI":"10.1109\/JSTSP.2020.3002101","article-title":"Media Forensics and DeepFakes: An Overview","volume":"14","author":"L Verdoliva","year":"2020","journal-title":"IEEE Journal on Selected Topics in Signal Processing"},{"key":"pone.0278989.ref009","doi-asserted-by":"crossref","unstructured":"Wang SY, Wang O, Zhang R, Owens A, Efros AA. CNN-Generated Images Are Surprisingly Easy to Spot\u2026 for Now. In: 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2020. p. 8692\u20138701.","DOI":"10.1109\/CVPR42600.2020.00872"},{"key":"pone.0278989.ref010","unstructured":"Li Y, Lyu S. Exposing DeepFake Videos By Detecting Face Warping Artifacts. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops; 2019."},{"key":"pone.0278989.ref011","unstructured":"Li Y, Chang MC, Lyu S. In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking. In: 10th IEEE International Workshop on Information Forensics and Security, WIFS 2018. Institute of Electrical and Electronics Engineers Inc.; 2019. p. 1\u20137. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85062882418&doi=10.1109%2fWIFS.2018.8630787&partnerID=40&md5=3dec5b984609ccca745fca39a2bfe4fb."},{"issue":"2","key":"pone.0278989.ref012","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1109\/TIFS.2011.2106121","article-title":"Detecting Recompression of JPEG Images via Periodicity Analysis of Compression Artifacts for Tampering Detection","volume":"6","author":"YL Chen","year":"2011","journal-title":"IEEE Transactions on Information Forensics and Security"},{"issue":"5","key":"pone.0278989.ref013","doi-asserted-by":"crossref","first-page":"1566","DOI":"10.1109\/TIFS.2012.2202227","article-title":"Image Forgery Localization via Fine-Grained Analysis of CFA Artifacts","volume":"7","author":"P Ferrara","year":"2012","journal-title":"IEEE Transactions on Information Forensics and Security"},{"issue":"2","key":"pone.0278989.ref014","doi-asserted-by":"crossref","DOI":"10.3390\/e22020249","article-title":"A Novel Counterfeit Feature Extraction Technique for Exposing Face-Swap Images Based on Deep Learning and Error Level Analysis","volume":"22","author":"W Zhang","year":"2020","journal-title":"Entropy"},{"key":"pone.0278989.ref015","article-title":"Anti-Forensics of Image Contrast Enhancement Based on Generative Adversarial Network","volume":"2021","author":"H Zou","year":"2021","journal-title":"Security and Communication Networks"},{"key":"pone.0278989.ref016","unstructured":"Bestagini P, Milani S, Tagliasacchi M, Tubaro S. Local Tampering Detection in Video Sequences. In: 2013 IEEE International Workshop on Multimedia Signal Processing, MMSP 2013; 2013. p. 488\u2013493. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-84892505453&doi=10.1109%2fMMSP.2013.6659337&partnerID=40&md5=00f115de87a906874b2d1ff4228aa4cb."},{"issue":"1","key":"pone.0278989.ref017","doi-asserted-by":"crossref","first-page":"470","DOI":"10.1364\/OE.22.000470","article-title":"Forensic Use of Photo Response Non-uniformity of Imaging Sensors and a Counter Method","volume":"22","author":"AE Dirik","year":"2014","journal-title":"Optics Express"},{"key":"pone.0278989.ref018","unstructured":"Sameer VU, Naskar R, Modalavalasa S. Mitigating Adaptive PRNU Denoising in Camera Model Identification: An Anti\u2013Counter Forensic Approach. In: IEEE Region 10 Annual International Conference, Proceedings\/TENCON. vol. 2019-October. Institute of Electrical and Electronics Engineers Inc.; 2019. p. 903\u2013907. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85077713715&doi=10.1109%2fTENCON.2019.8929355&partnerID=40&md5=e8868763f6d83a84ba75073be88ef061."},{"key":"pone.0278989.ref019","unstructured":"Xie C, Tan M, Gong B, Wang J, Yuille AL, Le QV. Adversarial Examples Improve Image Recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society; 2020. p. 816\u2013825. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85093074996&doi=10.1109%2fCVPR42600.2020.00090&partnerID=40&md5=f0d08e1eacc4fb6a452becba5868efca."},{"issue":"7","key":"pone.0278989.ref020","doi-asserted-by":"crossref","first-page":"3286","DOI":"10.1109\/TIP.2019.2895466","article-title":"Hybrid LSTM and Encoder\u2013Decoder Architecture for Detection of Image Forgeries","volume":"28","author":"JH Bappy","year":"2019","journal-title":"IEEE Transactions on Image Processing"},{"key":"pone.0278989.ref021","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/j.jvcir.2017.09.003","article-title":"Aligned and Non-aligned Double JPEG Detection Using Convolutional Neural Networks","volume":"49","author":"M Barni","year":"2017","journal-title":"Journal of Visual Communication and Image Representation"},{"key":"pone.0278989.ref022","unstructured":"Wu J, Liu L, Kang X, Sun W. A Generative Adversarial Network Framework for JPEG Anti-Forensics. In: 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2020\u2014Proceedings. Institute of Electrical and Electronics Engineers Inc.; 2020. p. 1442\u20131447. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85100919989&partnerID=40&md5=dd7789183499aa6683b62c0c5c977229."},{"key":"pone.0278989.ref023","unstructured":"Tariq S, Lee S, Kim H, Shin Y, Woo SS. GAN is a Friend or Foe? A Framework to Detect Various Fake Face Images. In: Proceedings of the ACM Symposium on Applied Computing. vol. Part F147772. Association for Computing Machinery; 2019. p. 1296\u20131303. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85065671310&doi=10.1145%2f3297280.3297410&partnerID=40&md5=edbd72fc9f298c514102ecb02187645a."},{"key":"pone.0278989.ref024","unstructured":"Maksutov AA, Morozov VO, Lavrenov AA, Smirnov AS. Methods of Deepfake Detection Based on Machine Learning. In: Proceedings of the 2020 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering, EIConRus 2020. Institute of Electrical and Electronics Engineers Inc.; 2020. p. 408\u2013411. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85082991408&doi=10.1109%2fEIConRus49466.2020.9039057&partnerID=40&md5=8b87feae61afa027b45dca27840204b3."},{"key":"pone.0278989.ref025","unstructured":"Pan G, Sun L, Wu Z, Lao S. Eyeblink-based Anti-Spoofing in Face Recognition from a Generic Webcamera. In: Proceedings of the IEEE International Conference on Computer Vision; 2007. p. 1\u20138. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-50949086465&doi=10.1109%2fICCV.2007.4409068&partnerID=40&md5=39692c7cae0151182976a40fa0077e91."},{"key":"pone.0278989.ref026","doi-asserted-by":"crossref","first-page":"83144","DOI":"10.1109\/ACCESS.2020.2988660","article-title":"DeepVision: Deepfakes Detection Using Human Eye Blinking Pattern","volume":"8","author":"T Jung","year":"2020","journal-title":"IEEE Access"},{"key":"pone.0278989.ref027","unstructured":"Yang F, Yu X, Huang J, Yang P, Metaxas D. Robust Eyelid Tracking for Fatigue Detection. In: Proceedings\u2014International Conference on Image Processing, ICIP; 2012. p. 1829\u20131832. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-84875865452&doi=10.1109%2fICIP.2012.6467238&partnerID=40&md5=a1f96e4e7685478f02fb36cb5ce48720."},{"key":"pone.0278989.ref028","doi-asserted-by":"crossref","DOI":"10.1016\/j.cviu.2022.103525","article-title":"Deep Learning for Deepfakes Creation and Detection: A Survey","volume":"223","author":"TT Nguyen","year":"2022","journal-title":"Computer Vision and Image Understanding"},{"key":"pone.0278989.ref029","first-page":"1486","volume-title":"Advances in Neural Information Processing Systems","author":"E Denton","year":"2015"},{"key":"pone.0278989.ref030","first-page":"671","volume-title":"Readings in Computer Vision","author":"PJ Burt","year":"1987"},{"key":"pone.0278989.ref031","unstructured":"Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In: 4th International Conference on Learning Representations, ICLR 2016\u2014Conference Track Proceedings. International Conference on Learning Representations, ICLR; 2016. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85083950271&partnerID=40&md5=d56c1a588cadb82fd3fc0f705a144a73."},{"key":"pone.0278989.ref032","unstructured":"Arjovsky M, Chintala S, Bottou L. Wasserstein Generative Adversarial Networks. In: 34th International Conference on Machine Learning, ICML 2017. vol. 1. International Machine Learning Society (IMLS); 2017. p. 298\u2013321. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85047016172&partnerID=40&md5=7fde6f9714ef8880afa04f3ed13c7234."},{"key":"pone.0278989.ref033","unstructured":"Shrivastava A, Pfister T, Tuzel O, Susskind J, Wang W, Webb R. Learning from Simulated and Unsupervised Images through Adversarial Training. In: Proceedings\u201430th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. vol. 2017-January. Institute of Electrical and Electronics Engineers Inc.; 2017. p. 2242\u20132251. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85041912723&doi=10.1109%2fCVPR.2017.241&partnerID=40&md5=67b60a2b6177dba9328c2026ea55a252."},{"key":"pone.0278989.ref034","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.asoc.2014.01.020","article-title":"Fully Automated Real Time Fatigue Detection of Drivers Through Fuzzy Expert Systems","volume":"18","author":"T Azim","year":"2014","journal-title":"Applied Soft Computing Journal"},{"issue":"3","key":"pone.0278989.ref035","doi-asserted-by":"crossref","first-page":"545","DOI":"10.1109\/TITS.2016.2582900","article-title":"Towards Detection of Bus Driver Fatigue Based on Robust Visual Analysis of Eye State","volume":"18","author":"B Mandal","year":"2017","journal-title":"IEEE Transactions on Intelligent Transportation Systems"},{"key":"pone.0278989.ref036","first-page":"33","article-title":"Automatic Assessment of Eye Blinking Patterns through Statistical Shape Models","volume":"5815","author":"FM Sukno","year":"2009","journal-title":"Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"issue":"1","key":"pone.0278989.ref037","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1006\/cviu.1995.1004","article-title":"Active Shape Models\u2014Their Training and Application","volume":"61","author":"TF Cootes","year":"1995","journal-title":"Computer Vision and Image Understanding"},{"key":"pone.0278989.ref038","first-page":"1","article-title":"Real-time Eye Blink Detection using Facial Landmarks","author":"J Cech","year":"2016","journal-title":"Cent Mach Perception, Dep Cybern Fac Electr Eng Czech Tech Univ Prague"},{"key":"pone.0278989.ref039","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.ijcce.2021.02.002","article-title":"Facial Expression Recognition via ResNet-50","volume":"2","author":"B Li","year":"2021","journal-title":"International Journal of Cognitive Computing in Engineering"},{"issue":"4","key":"pone.0278989.ref040","doi-asserted-by":"crossref","first-page":"677","DOI":"10.1109\/TPAMI.2016.2599174","article-title":"Long-Term Recurrent Convolutional Networks for Visual Recognition and Description","volume":"39","author":"J Donahue","year":"2017","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"pone.0278989.ref041","unstructured":"Zhou T, Wang W, Liang Z, Shen J. Face Forensics in the Wild. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society; 2021. p. 5774\u20135784. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-85115136497&doi=10.1109%2fCVPR46437.2021.00572&partnerID=40&md5=2481dada26c9801025e7a337f7891eb8."},{"key":"pone.0278989.ref042","unstructured":"Kazemi V, Sullivan J. One Millisecond Face Alignment with an Ensemble of Regression Trees. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Cvpr\u201914. Usa: IEEE Computer Society; 2014. p. 1867\u20131874. Available from: https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-84911391543&doi=10.1109%2fCVPR.2014.241&partnerID=40&md5=a3223af23757fd145907bf1b229c16bb."},{"key":"pone.0278989.ref043","doi-asserted-by":"crossref","first-page":"103170","DOI":"10.1016\/j.cviu.2021.103170","article-title":"Fake Face Detection via Adaptive Manipulation Traces Extraction Network","volume":"204","author":"Z Guo","year":"2021","journal-title":"Computer Vision and Image Understanding"},{"issue":"3","key":"pone.0278989.ref044","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1109\/TBIOM.2021.3065735","article-title":"Fighting Fake News: Two Stream Network for Deepfake Detection via Learnable SRM","volume":"3","author":"B Han","year":"2021","journal-title":"IEEE Transactions on Biometrics, Behavior, and Identity Science"},{"key":"pone.0278989.ref045","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.patrec.2021.03.005","article-title":"Optical Flow based CNN for Detection of Unlearnt Deepfake Manipulations","volume":"146","author":"R Caldelli","year":"2021","journal-title":"Pattern Recognition Letters"},{"key":"pone.0278989.ref046","doi-asserted-by":"crossref","unstructured":"Li M, Liu B, Hu Y, Zhang L, Wang S. Deepfake Detection Using Robust Spatial and Temporal Features from Facial Landmarks. In: Proceedings\u20149th International Workshop on Biometrics and Forensics, IWBF 2021. Institute of Electrical and Electronics Engineers Inc.; 2021. p. 1\u20136.","DOI":"10.1109\/IWBF50991.2021.9465076"},{"issue":"3","key":"pone.0278989.ref047","doi-asserted-by":"crossref","first-page":"1089","DOI":"10.1109\/TCSVT.2021.3074259","article-title":"Detecting Compressed Deepfake Videos in Social Networks Using Frame-Temporality Two-Stream Convolutional Network","volume":"32","author":"J Hu","year":"2022","journal-title":"IEEE Transactions on Circuits and Systems for Video Technology"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0278989","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T18:48:39Z","timestamp":1671130119000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0278989"}},"subtitle":[],"editor":[{"given":"Sathishkumar","family":"V E","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2022,12,15]]},"references-count":47,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,12,15]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0278989","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,15]]}}}