{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T07:39:10Z","timestamp":1773128350199,"version":"3.50.1"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,11,15]],"date-time":"2022-11-15T00:00:00Z","timestamp":1668470400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,11,15]],"date-time":"2022-11-15T00:00:00Z","timestamp":1668470400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["PRT\/BD\/152202\/2021"],"award-info":[{"award-number":["PRT\/BD\/152202\/2021"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Seizure semiology is a well-established method to classify epileptic seizure types, but requires a significant amount of resources as long-term Video-EEG monitoring needs to be visually analyzed. Therefore, computer vision based diagnosis support tools are a promising approach. In this article, we utilize infrared (IR) and depth (3D) videos to show the feasibility of a 24\/7 novel object and action recognition based deep learning (DL) monitoring system to differentiate between epileptic seizures in frontal lobe epilepsy (FLE), temporal lobe epilepsy (TLE) and non-epileptic events. Based on the largest 3Dvideo-EEG database in the world (115 seizures\/+680,000 video-frames\/427GB), we achieved a promising cross-subject validation f1-score of 0.833\u00b10.061 for the 2 class (FLE vs. TLE) and 0.763 \u00b1 0.083 for the 3 class (FLE vs. TLE vs. non-epileptic) case, from 2 s samples, with an automated semi-specialized depth (Acc.95.65%) and Mask R-CNN (Acc.96.52%) based cropping pipeline to pre-process the videos, enabling a near-real-time seizure type detection and classification tool. Our results demonstrate\u00a0the feasibility of our novel DL approach to support 24\/7 epilepsy monitoring, outperforming all previously published methods.<\/jats:p>","DOI":"10.1038\/s41598-022-23133-9","type":"journal-article","created":{"date-parts":[[2022,11,15]],"date-time":"2022-11-15T10:02:49Z","timestamp":1668506569000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Novel 3D video action recognition deep learning approach for near real time epileptic seizure classification"],"prefix":"10.1038","volume":"12","author":[{"given":"Tam\u00e1s","family":"Kar\u00e1csony","sequence":"first","affiliation":[]},{"given":"Anna Mira","family":"Loesch-Biffar","sequence":"additional","affiliation":[]},{"given":"Christian","family":"Vollmar","sequence":"additional","affiliation":[]},{"given":"Jan","family":"R\u00e9mi","sequence":"additional","affiliation":[]},{"given":"Soheyl","family":"Noachtar","sequence":"additional","affiliation":[]},{"given":"Jo\u00e3o Paulo Silva","family":"Cunha","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,11,15]]},"reference":[{"key":"23133_CR1","doi-asserted-by":"publisher","first-page":"837","DOI":"10.1016\/j.ncl.2016.06.015","volume":"34","author":"A Singh","year":"2016","unstructured":"Singh, A. & Trevick, S. The Epidemiology of Global Epilepsy. Neurol. Clin. 34, 837\u2013847. https:\/\/doi.org\/10.1016\/j.ncl.2016.06.015 (2016).","journal-title":"Neurol. Clin."},{"key":"23133_CR2","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1016\/j.yebeh.2009.02.028","volume":"15","author":"S Noachtar","year":"2009","unstructured":"Noachtar, S. & Borggraefe, I. Epilepsy surgery: A critical review. Epilepsy Behav. 15, 66\u201372. https:\/\/doi.org\/10.1016\/j.yebeh.2009.02.028 (2009).","journal-title":"Epilepsy Behav."},{"key":"23133_CR3","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1016\/j.yebeh.2009.02.029","volume":"15","author":"S Noachtar","year":"2009","unstructured":"Noachtar, S. & Peters, A. S. Semiology of epileptic seizures: A critical review. Epilepsy Behav. 15, 2\u20139. https:\/\/doi.org\/10.1016\/j.yebeh.2009.02.029 (2009).","journal-title":"Epilepsy Behav."},{"key":"23133_CR4","doi-asserted-by":"publisher","first-page":"1683","DOI":"10.1093\/brain\/124.9.1683","volume":"124","author":"F Rosenow","year":"2001","unstructured":"Rosenow, F. Presurgical evaluation of epilepsy. Brain 124, 1683\u20131700. https:\/\/doi.org\/10.1093\/brain\/124.9.1683 (2001).","journal-title":"Brain"},{"key":"23133_CR5","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1111\/j.1528-1157.1997.tb01093.x","volume":"38","author":"A Bleasel","year":"1997","unstructured":"Bleasel, A., Kotagal, P., Kankirawatana, P. & Rybicki, L. Lateralizing value and semiology of ictal limb posturing and version in temporal lobe and extratemporal epilepsy. Epilepsia 38, 168\u2013174. https:\/\/doi.org\/10.1111\/j.1528-1157.1997.tb01093.x (1997).","journal-title":"Epilepsia"},{"key":"23133_CR6","doi-asserted-by":"publisher","first-page":"1124","DOI":"10.1016\/j.clinph.2014.09.023","volume":"126","author":"F F\u00fcrbass","year":"2015","unstructured":"F\u00fcrbass, F. et al. Prospective multi-center study of an automatic online seizure detection system for epilepsy monitoring units. Clin. Neurophysiol. 126, 1124\u20131131. https:\/\/doi.org\/10.1016\/j.clinph.2014.09.023 (2015).","journal-title":"Clin. Neurophysiol."},{"key":"23133_CR7","doi-asserted-by":"publisher","unstructured":"do Carmo Vilas-Boas, M. & Cunha, J. P. S. Movement quantification in neurological diseases: Methods and applications. IEEE Rev. Biomed. Eng. 9, 15\u201331. https:\/\/doi.org\/10.1109\/rbme.2016.2543683 (2016).","DOI":"10.1109\/rbme.2016.2543683"},{"key":"23133_CR8","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1080\/21681163.2016.1141062","volume":"6","author":"F Achilles","year":"2016","unstructured":"Achilles, F. et al. Convolutional neural networks for real-time epileptic seizure detection. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. 6, 264\u2013269. https:\/\/doi.org\/10.1080\/21681163.2016.1141062 (2016).","journal-title":"Comput. Methods Biomech. Biomed. Eng. Imaging Vis."},{"key":"23133_CR9","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1016\/j.yebeh.2018.07.028","volume":"87","author":"D Ahmedt-Aristizabal","year":"2018","unstructured":"Ahmedt-Aristizabal, D. et al. A hierarchical multimodal system for motion analysis in patients with epilepsy. Epilepsy Behav. 87, 46\u201358. https:\/\/doi.org\/10.1016\/j.yebeh.2018.07.028 (2018).","journal-title":"Epilepsy Behav."},{"key":"23133_CR10","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/j.yebeh.2018.02.010","volume":"82","author":"D Ahmedt-Aristizabal","year":"2018","unstructured":"Ahmedt-Aristizabal, D. et al. Deep facial analysis: A new phase I epilepsy evaluation using computer vision. Epilepsy Behav. 82, 17\u201324. https:\/\/doi.org\/10.1016\/j.yebeh.2018.02.010 (2018).","journal-title":"Epilepsy Behav."},{"key":"23133_CR11","doi-asserted-by":"publisher","unstructured":"Ahmedt-Aristizabal, D. et\u00a0al. Deep Motion Analysis for Epileptic Seizure Classification. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, vol. 2018-July, 3578\u20133581, https:\/\/doi.org\/10.1109\/EMBC.2018.8513031 (Institute of Electrical and Electronics Engineers Inc., 2018).","DOI":"10.1109\/EMBC.2018.8513031"},{"key":"23133_CR12","doi-asserted-by":"publisher","unstructured":"Ahmedt-Aristizabal, D. et\u00a0al. Vision-Based Mouth Motion Analysis in Epilepsy: A 3D Perspective. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 1625\u20131629, https:\/\/doi.org\/10.1109\/EMBC.2019.8857656 (Institute of Electrical and Electronics Engineers Inc., 2019).","DOI":"10.1109\/EMBC.2019.8857656"},{"key":"23133_CR13","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1016\/j.seizure.2018.12.017","volume":"65","author":"D Ahmedt-Aristizabal","year":"2019","unstructured":"Ahmedt-Aristizabal, D. et al. Aberrant epileptic seizure identification: A computer vision perspective. Seizure 65, 65\u201371. https:\/\/doi.org\/10.1016\/j.seizure.2018.12.017 (2019).","journal-title":"Seizure"},{"key":"23133_CR14","doi-asserted-by":"publisher","unstructured":"Maia, P., Hartl, E., Vollmar, C., Noachtar, S. & Cunha, J. P.\u00a0S. Epileptic seizure classification using the NeuroMov database. In 2019 IEEE 6th Portuguese Meeting on Bioengineering (ENBENG). https:\/\/doi.org\/10.1109\/enbeng.2019.8692465 (IEEE, 2019).","DOI":"10.1109\/enbeng.2019.8692465"},{"key":"23133_CR15","doi-asserted-by":"publisher","unstructured":"Karacsony, T., Loesch-Biffar, A.\u00a0M., Vollmar, C., Noachtar, S. & Cunha, J. P.\u00a0S. A Deep Learning Architecture for Epileptic Seizure Classification Based on Object and Action Recognition. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4117\u20134121, https:\/\/doi.org\/10.1109\/icassp40776.2020.9054649 (Institute of Electrical and Electronics Engineers (IEEE), Barcelona, Spain, 2020).","DOI":"10.1109\/icassp40776.2020.9054649"},{"key":"23133_CR16","doi-asserted-by":"publisher","first-page":"642","DOI":"10.1016\/j.yebeh.2011.01.020","volume":"20","author":"J R\u00e9mi","year":"2011","unstructured":"R\u00e9mi, J. et al. Quantitative movement analysis differentiates focal seizures characterized by automatisms. Epilepsy Behav. 20, 642\u2013647. https:\/\/doi.org\/10.1016\/j.yebeh.2011.01.020 (2011).","journal-title":"Epilepsy Behav."},{"key":"23133_CR17","doi-asserted-by":"publisher","unstructured":"Kar\u00e1csony, T., Loesch-Biffar, A.\u00a0M., Vollmar, C., Noachtar, S. & Cunha, J. P.\u00a0S. Deepepil: Towards an Epileptologist-Friendly AI Enabled Seizure Classification Cloud System based on Deep Learning Analysis of 3D videos. BHI 2021 - 2021 IEEE EMBS International Conference on Biomedical and Health Informatics, Proceedings. https:\/\/doi.org\/10.1109\/BHI50953.2021.9508555 (2021).","DOI":"10.1109\/BHI50953.2021.9508555"},{"key":"23133_CR18","doi-asserted-by":"publisher","DOI":"10.1016\/j.jns.2015.09.065","volume":"357","author":"F Achilles","year":"2015","unstructured":"Achilles, F. et al. Deep convolutional neural networks for automatic identification of epileptic seizures in infrared and depth images. J. Neurol. Sci. 357, e436. https:\/\/doi.org\/10.1016\/j.jns.2015.09.065 (2015).","journal-title":"J. Neurol. Sci."},{"key":"23133_CR19","unstructured":"Pereira\u00a0Choupina, H.\u00a0M. et\u00a0al. NeuroKinect 3.0: Multi-bed 3Dvideo-EEG system for epilepsy clinical motion monitoring. Stud. Health Technol. Inf. 247, 46\u201350 (2018)."},{"key":"23133_CR20","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0145669","volume":"11","author":"JPS Cunha","year":"2016","unstructured":"Cunha, J. P. S. et al. NeuroKinect: A novel low-cost 3dvideo-EEG system for epileptic seizure motion quantification. PLoS ONE 11, e0145669. https:\/\/doi.org\/10.1371\/journal.pone.0145669 (2016).","journal-title":"PLoS ONE"},{"key":"23133_CR21","unstructured":"Pereira\u00a0Choupina, H.\u00a0M. et\u00a0al. NeuroKinect 3.0: Multi-Bed 3Dvideo-EEG system for epilepsy clinical motion monitoring. Stud. Health Technol. Inf. 247, 46\u201350 (2018)."},{"key":"23133_CR22","doi-asserted-by":"publisher","first-page":"1817","DOI":"10.1111\/epi.13907","volume":"58","author":"D Ahmedt-Aristizabal","year":"2017","unstructured":"Ahmedt-Aristizabal, D. et al. Automated analysis of seizure semiology and brain electrical activity in presurgery evaluation of epilepsy: A focused survey. Epilepsia 58, 1817\u20131831. https:\/\/doi.org\/10.1111\/epi.13907 (2017).","journal-title":"Epilepsia"},{"key":"23133_CR23","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"23133_CR24","doi-asserted-by":"publisher","unstructured":"Carreira, J. & Zisserman, A. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https:\/\/doi.org\/10.1109\/cvpr.2017.502 (IEEE, 2017).","DOI":"10.1109\/cvpr.2017.502"},{"key":"23133_CR25","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1001\/jamaneurol.2019.4785","volume":"77","author":"WO Tatum","year":"2020","unstructured":"Tatum, W. O. et al. Assessment of the predictive value of outpatient smartphone videos for diagnosis of epileptic seizures. JAMA Neurol. 77, 593\u2013600. https:\/\/doi.org\/10.1001\/jamaneurol.2019.4785 (2020).","journal-title":"JAMA Neurol."},{"key":"23133_CR26","doi-asserted-by":"publisher","unstructured":"He, K., Gkioxari, G., Dollar, P. & Girshick, R. Mask r-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV). https:\/\/doi.org\/10.1109\/iccv.2017.322 (IEEE, 2017).","DOI":"10.1109\/iccv.2017.322"},{"key":"23133_CR27","unstructured":"Chollet, F. & others. Keras. https:\/\/keras.io (2015)."},{"key":"23133_CR28","unstructured":"Abdulla, W. Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow. https:\/\/github.com\/matterport\/Mask_RCNN (2017)."},{"key":"23133_CR29","doi-asserted-by":"publisher","unstructured":"Lin, T.-Y. et\u00a0al. Microsoft COCO: Common Objects in Context. In Computer Vision\u2014ECCV 2014, 740\u2013755. https:\/\/doi.org\/10.1007\/978-3-319-10602-1_48 (Springer International Publishing, 2014).","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"23133_CR30","doi-asserted-by":"publisher","first-page":"938","DOI":"10.1016\/j.medengphy.2011.10.013","volume":"34","author":"JPS Cunha","year":"2012","unstructured":"Cunha, J. P. S. et al. Movement quantification in epileptic seizures: A feasibility study for a new 3D approach. Med. Eng. Phys. 34, 938\u2013945. https:\/\/doi.org\/10.1016\/j.medengphy.2011.10.013 (2012).","journal-title":"Med. Eng. Phys."},{"key":"23133_CR31","unstructured":"Abdulla, W. Keras implementation of Inflated 3D from Quo Vardis paper + weights. https:\/\/github.com\/dlpbc\/keras-kinetics-i3d (2018)."},{"key":"23133_CR32","unstructured":"Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML\u201915, 448\u2013456 (JMLR.org, 2015)."},{"key":"23133_CR33","doi-asserted-by":"publisher","unstructured":"Deng, J. et\u00a0al. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. https:\/\/doi.org\/10.1109\/cvpr.2009.5206848 (IEEE, 2009).","DOI":"10.1109\/cvpr.2009.5206848"},{"key":"23133_CR34","unstructured":"Kay, W. et\u00a0al. The Kinetics Human Action Video Dataset. arXiv (2017). arXiv.org\/abs\/1705.06950v1."},{"key":"23133_CR35","unstructured":"Autonomio Talos. https:\/\/github.com\/autonomio\/talos (2020)."},{"key":"23133_CR36","doi-asserted-by":"crossref","unstructured":"Li, X., Chen, S., Hu, X. & Yang, J. Understanding the disharmony between dropout and batch normalization by variance shift. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019).","DOI":"10.1109\/CVPR.2019.00279"},{"key":"23133_CR37","doi-asserted-by":"publisher","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In 2015 IEEE International Conference on Computer Vision (ICCV), https:\/\/doi.org\/10.1109\/iccv.2015.123 (IEEE, 2015).","DOI":"10.1109\/iccv.2015.123"},{"key":"23133_CR38","unstructured":"Kingma, D.\u00a0P. & Ba, J. Adam: A method for stochastic optimization. ICLR 2015 (2014). arXiv.org\/abs\/1412.6980v9."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-23133-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-23133-9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-23133-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,15]],"date-time":"2022-11-15T11:24:12Z","timestamp":1668511452000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-23133-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,15]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["23133"],"URL":"https:\/\/doi.org\/10.1038\/s41598-022-23133-9","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,15]]},"assertion":[{"value":"1 August 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 October 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 November 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"19571"}}