{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T07:03:18Z","timestamp":1775977398711,"version":"3.50.1"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,4,20]],"date-time":"2020-04-20T00:00:00Z","timestamp":1587340800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,4,20]],"date-time":"2020-04-20T00:00:00Z","timestamp":1587340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Wu Tsai Neurotranslate Grant"},{"DOI":"10.13039\/100011040","name":"ODNI | Intelligence Community Postdoctoral Research Fellowship Program","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100011040","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Wu Tsai Neurotranslate Grant, LVIS LLC"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Automated seizure detection from electroencephalography (EEG) would improve the quality of patient care while reducing medical costs, but achieving reliably high performance across patients has proven difficult. Convolutional Neural Networks (CNNs) show promise in addressing this problem, but they are limited by a lack of large labeled training datasets. We propose using imperfect but plentiful archived annotations to train CNNs for automated, real-time EEG seizure detection across patients. While these weak annotations indicate possible seizures with precision scores as low as 0.37, they are commonly produced in large volumes within existing clinical workflows by a mixed group of technicians, fellows, students, and board-certified epileptologists. We find that CNNs trained using such weak annotations achieve Area Under the Receiver Operating Characteristic curve (AUROC) values of 0.93 and 0.94 for pediatric and adult seizure onset detection, respectively. Compared to currently deployed clinical software, our model provides a 31% increase (18 points) in F1-score for pediatric patients and a 17% increase (11 points) for adult patients. These results demonstrate that weak annotations, which are sustainably collected via existing clinical workflows, can be leveraged to produce clinically useful seizure detection models.<\/jats:p>","DOI":"10.1038\/s41746-020-0264-0","type":"journal-article","created":{"date-parts":[[2020,4,20]],"date-time":"2020-04-20T10:03:19Z","timestamp":1587376999000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":72,"title":["Weak supervision as an efficient approach for automated seizure detection in electroencephalography"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1427-0469","authenticated-orcid":false,"given":"Khaled","family":"Saab","sequence":"first","affiliation":[]},{"given":"Jared","family":"Dunnmon","sequence":"additional","affiliation":[]},{"given":"Christopher","family":"R\u00e9","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5057-4369","authenticated-orcid":false,"given":"Daniel","family":"Rubin","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2938-6184","authenticated-orcid":false,"given":"Christopher","family":"Lee-Messer","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,4,20]]},"reference":[{"key":"264_CR1","doi-asserted-by":"publisher","first-page":"1177","DOI":"10.3390\/jcm8081177","volume":"8","author":"M Strein","year":"2019","unstructured":"Strein, M., Holton-Burke, J. P., Smith, L. R. & Brophy, G. M. Prevention, treatment, and monitoring of seizures in the intensive care unit. J. Clin. Med. 8, 1177 (2019).","journal-title":"J. Clin. Med."},{"key":"264_CR2","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1016\/j.clinph.2014.05.037","volume":"126","author":"M Brandon Westover","year":"2015","unstructured":"Brandon Westover, M. et al. The probability of seizures during EEG monitoring in critically ill adults. Clin. Neurophysiol. 126, 463\u2013471 (2015).","journal-title":"Clin. Neurophysiol."},{"key":"264_CR3","doi-asserted-by":"publisher","first-page":"1130","DOI":"10.1111\/j.1528-1167.2011.03070.x","volume":"52","author":"K Williams","year":"2011","unstructured":"Williams, K., Jarrar, R. & Buchhalter, J. Continuous video-EEG monitoring in pediatric intensive care units. Epilepsia 52, 1130\u20131136 (2011).","journal-title":"Epilepsia"},{"key":"264_CR4","doi-asserted-by":"publisher","DOI":"10.1016\/S1474-4422(13)70246-1","author":"NS Abend","year":"2013","unstructured":"Abend, N. S., Wusthoff, C. J., Goldberg, E. M. & Dlugos, D. J. Electrographic seizures and status epilepticus in critically ill children and neonates with encephalopathy. Lancet Neurol. https:\/\/doi.org\/10.1016\/S1474-4422(13)70246-1 (2013).","journal-title":"Lancet Neurol."},{"key":"264_CR5","doi-asserted-by":"crossref","unstructured":"Worden, L. T. et al. The probability of seizures during continuous EEG monitoring in high-risk neonates. Epilepsia 60, 2508\u20132518 (2019).","DOI":"10.1111\/epi.16387"},{"key":"264_CR6","doi-asserted-by":"publisher","DOI":"10.1111\/epi.13294","author":"I Megiddo","year":"2016","unstructured":"Megiddo, I. et al. Health and economic benefits of public financing of epilepsy treatment in India: an agent-based simulation model. Epilepsia https:\/\/doi.org\/10.1111\/epi.13294 (2016).","journal-title":"Epilepsia"},{"key":"264_CR7","doi-asserted-by":"publisher","DOI":"10.1111\/epi.12550","author":"RS Fisher","year":"2014","unstructured":"Fisher, R. S. et al. ILAE Official Report: a practical clinical definition of epilepsy. Epilepsia https:\/\/doi.org\/10.1111\/epi.12550 (2014).","journal-title":"Epilepsia"},{"key":"264_CR8","doi-asserted-by":"publisher","DOI":"10.1046\/j.1528-1157.2003.25002.x","author":"D Ott","year":"2003","unstructured":"Ott, D. et al. Behavioral disorders in pediatric epilepsy: unmet psychiatric need. Epilepsia https:\/\/doi.org\/10.1046\/j.1528-1157.2003.25002.x (2003).","journal-title":"Epilepsia"},{"key":"264_CR9","doi-asserted-by":"publisher","DOI":"10.1111\/ane.12014","author":"MP Kerr","year":"2012","unstructured":"Kerr, M. P. The impact of epilepsy on patients\u2019 lives. Acta Neurol. Scand. Suppl. https:\/\/doi.org\/10.1111\/ane.12014 (2012).","journal-title":"Acta Neurol. Scand. Suppl."},{"key":"264_CR10","doi-asserted-by":"publisher","DOI":"10.1212\/01.wnl.0000436948.93399.2a","author":"JP Ney","year":"2013","unstructured":"Ney, J. P., Van Der Goes, D. N., Nuwer, M. R., Nelson, L. & Eccher, M. A. Continuous and routine eeg in intensive care. Neurology https:\/\/doi.org\/10.1212\/01.wnl.0000436948.93399.2a (2013).","journal-title":"Neurology"},{"key":"264_CR11","doi-asserted-by":"publisher","DOI":"10.1097\/WNP.0000000000000170","author":"NS Abend","year":"2015","unstructured":"Abend, N. S., Topjian, A. A. & Williams, S. How much does it cost to identify a critically ill child experiencing electrographic seizures? J. Clin. Neurophysiol. https:\/\/doi.org\/10.1097\/WNP.0000000000000170 (2015).","journal-title":"J. Clin. Neurophysiol."},{"key":"264_CR12","doi-asserted-by":"publisher","DOI":"10.1038\/nature21056","author":"A Esteva","year":"2017","unstructured":"Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature https:\/\/doi.org\/10.1038\/nature21056 (2017).","journal-title":"Nature"},{"key":"264_CR13","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2018181422","author":"JA Dunnmon","year":"2019","unstructured":"Dunnmon, J. A. et al. Assessment of convolutional neural networks for automated classification of chest radiographs. Radiology https:\/\/doi.org\/10.1148\/radiol.2018181422 (2019).","journal-title":"Radiology"},{"key":"264_CR14","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-018-0029-1","author":"A Rajkomar","year":"2018","unstructured":"Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. npj Digit. Med. https:\/\/doi.org\/10.1038\/s41746-018-0029-1 (2018).","journal-title":"npj Digit. Med."},{"key":"264_CR15","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0316-z","author":"A Esteva","year":"2019","unstructured":"Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. https:\/\/doi.org\/10.1038\/s41591-018-0316-z (2019).","journal-title":"Nat. Med."},{"key":"264_CR16","doi-asserted-by":"publisher","unstructured":"Madani, A., Ong, J. R., Tibrewal, A. & Mofrad, M. R. K. Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease. npj Digit. Med. https:\/\/doi.org\/10.1038\/s41746-018-0065-x (2018).","DOI":"10.1038\/s41746-018-0065-x"},{"key":"264_CR17","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0097-x","author":"DSW Ting","year":"2019","unstructured":"Ting, D. S. W. et al. Deep learning in estimating prevalence and systemic risk factors for diabetic retinopathy: a multi-ethnic study. npj Digit. Med. https:\/\/doi.org\/10.1038\/s41746-019-0097-x (2019).","journal-title":"npj Digit. Med."},{"key":"264_CR18","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0096-y","author":"P Khosravi","year":"2019","unstructured":"Khosravi, P. et al. Deep learning enables robust assessment and selection of human blastocysts after in vitro fertilization. npj Digit. Med. https:\/\/doi.org\/10.1038\/s41746-019-0096-y (2019).","journal-title":"npj Digit. Med."},{"key":"264_CR19","doi-asserted-by":"publisher","unstructured":"Arbabshirani, M. R. et al. Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. npj Digit. Med. https:\/\/doi.org\/10.1038\/s41746-017-0015-z (2018).","DOI":"10.1038\/s41746-017-0015-z"},{"key":"264_CR20","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0105-1","author":"MA Badgeley","year":"2019","unstructured":"Badgeley, M. A. et al. Deep learning predicts hip fracture using confounding patient and healthcare variables. npj Digit. Med. https:\/\/doi.org\/10.1038\/s41746-019-0105-1 (2019).","journal-title":"npj Digit. Med."},{"key":"264_CR21","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0104-2","author":"C-C Kuo","year":"2019","unstructured":"Kuo, C.-C. et al. Automation of the kidney function prediction and classification through ultrasound-based kidney imaging using deep learning. npj Digit. Med. https:\/\/doi.org\/10.1038\/s41746-019-0104-2 (2019).","journal-title":"npj Digit. Med."},{"key":"264_CR22","first-page":"178","volume":"56","author":"P Thodoroff","year":"2016","unstructured":"Thodoroff, P., Pineau, J. & Lim, A. Learning robust features using deep learning for automatic seizure detection. Mach. Learn. Healthc. Conf. 56, 178\u2013190 (2016).","journal-title":"Mach. Learn. Healthc. Conf"},{"key":"264_CR23","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002683","author":"JR Zech","year":"2018","unstructured":"Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLoS Med. https:\/\/doi.org\/10.1371\/journal.pmed.1002683 (2018).","journal-title":"PLoS Med."},{"key":"264_CR24","doi-asserted-by":"publisher","DOI":"10.1016\/j.yebeh.2015.03.010","author":"M Bandarabadi","year":"2015","unstructured":"Bandarabadi, M., Rasekhi, J., Teixeira, C. A., Karami, M. R. & Dourado, A. On the proper selection of preictal period for seizure prediction. Epilepsy Behav. https:\/\/doi.org\/10.1016\/j.yebeh.2015.03.010 (2015).","journal-title":"Epilepsy Behav."},{"key":"264_CR25","doi-asserted-by":"publisher","DOI":"10.1155\/2014\/572082","author":"N Ozdemir","year":"2014","unstructured":"Ozdemir, N. & Yildirim, E. Patient specific seizure prediction system using hilbert spectrum and Bayesian networks classifiers. Comput. Math. Methods Med. https:\/\/doi.org\/10.1155\/2014\/572082 (2014).","journal-title":"Comput. Math. Methods Med."},{"key":"264_CR26","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0121182","author":"K Gadhoumi","year":"2015","unstructured":"Gadhoumi, K., Gotman, J. & Lina, J. M. Scale invariance properties of intracerebral eeg improve seizure prediction in mesial temporal lobe epilepsy. PLoS ONE https:\/\/doi.org\/10.1371\/journal.pone.0121182 (2015).","journal-title":"PLoS ONE"},{"key":"264_CR27","doi-asserted-by":"publisher","DOI":"10.1111\/j.1528-1167.2011.03138.x","author":"Y Park","year":"2011","unstructured":"Park, Y., Luo, L., Parhi, K. K. & Netoff, T. Seizure prediction with spectral power of EEG using cost-sensitive support vector machines. Epilepsia https:\/\/doi.org\/10.1111\/j.1528-1167.2011.03138.x (2011).","journal-title":"Epilepsia"},{"key":"264_CR28","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2014.2358640","author":"N Wang","year":"2015","unstructured":"Wang, N. & Lyu, M. R. Extracting and selecting distinctive EEG features for efficient epileptic seizure prediction. IEEE J. Biomed. Heal. Inform. https:\/\/doi.org\/10.1109\/JBHI.2014.2358640 (2015).","journal-title":"IEEE J. Biomed. Heal. Inform."},{"key":"264_CR29","unstructured":"Ratner, A. J., De Sa, C. M., Wu, S., Selsam, D. & R\u00e9, C. Data Programming: Creating Large Training Sets, Quickly. In Advances in Neural Information Processing Systems. Curran Associates, Inc. 29, 3567\u20133575 (2016)."},{"key":"264_CR30","unstructured":"Alfonseca, E., Filippova, K., Delort, J. Y. & Garrido, G. Pattern learning for relation extraction with a hierarchical topic model. in 50th Annual Meeting of the Association for Computational Linguistics, ACL 2012\u2014Proceedings of the Conference (2012)."},{"key":"264_CR31","doi-asserted-by":"crossref","unstructured":"Yuen, M. C., King, I. & Leung, K. S. A survey of crowdsourcing systems. IEEE International Conference on Privacy, Security, Risk and Trust and IEEE International Conference on Social Computing, (2011).","DOI":"10.1109\/PASSAT\/SocialCom.2011.203"},{"key":"264_CR32","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1016\/j.cnp.2018.03.002","volume":"3","author":"J Brogger","year":"2018","unstructured":"Brogger, J. et al. Visual EEG reviewing times with SCORE EEG. Clin. Neurophysiol. Pract. 3, 59\u201364 (2018).","journal-title":"Clin. Neurophysiol. Pract."},{"key":"264_CR33","doi-asserted-by":"publisher","DOI":"10.1097\/WNP.0000000000000493","author":"CN Joshi","year":"2018","unstructured":"Joshi, C. N. et al. Semiautomated spike detection software persyst 13 is noninferior to human readers when calculating the spike-wave index in electrical status epilepticus in sleep. J. Clin. Neurophysiol. https:\/\/doi.org\/10.1097\/WNP.0000000000000493 (2018).","journal-title":"J. Clin. Neurophysiol."},{"key":"264_CR34","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2018.04.021","author":"I Ullah","year":"2018","unstructured":"Ullah, I., Hussain, M., Qazi, EulH. & Aboalsamh, H. An automated system for epilepsy detection using EEG brain signals based on deep learning approach. Expert Syst. Appl. https:\/\/doi.org\/10.1016\/j.eswa.2018.04.021 (2018).","journal-title":"Expert Syst. Appl."},{"key":"264_CR35","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2017.09.017","author":"UR Acharya","year":"2018","unstructured":"Acharya, U. R., Oh, S. L., Hagiwara, Y., Tan, J. H. & Adeli, H. Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals. Comput. Biol. Med. https:\/\/doi.org\/10.1016\/j.compbiomed.2017.09.017 (2018).","journal-title":"Comput. Biol. Med."},{"key":"264_CR36","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2018.00083","author":"V Shah","year":"2018","unstructured":"Shah, V. et al. The temple university hospital seizure detection corpus. Front. Neuroinform. https:\/\/doi.org\/10.3389\/fninf.2018.00083 (2018).","journal-title":"Front. Neuroinform."},{"key":"264_CR37","unstructured":"Asif, U., Roy, S., Tang, J. & Harrer, S. SeizureNet: A Deep Convolutional Neural Network for Accurate Seizure Type Classification and Seizure Detection. arXiv Prepr. arXiv1903.03232 (2019)."},{"key":"264_CR38","doi-asserted-by":"crossref","unstructured":"Schiratti, J. B., Le Douget, J. E., Le Van Quyen, M., Essid, S. & Gramfort, A. An Ensemble Learning Approach to Detect Epileptic Seizures from Long Intracranial EEG Recordings. IEEE International Conference on Acoustics, Speech and Signal Processing\u2014Proceedings (2018).","DOI":"10.1109\/ICASSP.2018.8461489"},{"key":"264_CR39","doi-asserted-by":"crossref","unstructured":"Tsiouris, K. M., Tzallas, A. T., Markoula, S., Koutsouris, D., Konitsiotis, S. & Fotiadis, D. I. A review of automated methodologies for the detection of epileptic episodes using long-term EEG signals. In Handbook of Research on Trends in the Diagnosis and Treatment of Chronic Conditions. IGI Global, 231\u2013261 (2018).","DOI":"10.4018\/978-1-4666-8828-5.ch011"},{"key":"264_CR40","doi-asserted-by":"crossref","unstructured":"Direito, B. et al. Feature selection in high dimensional EEG features spaces for epileptic seizure prediction. IFAC Proceedings Volumes (IFAC-PapersOnline) (Elsevier, 2011).","DOI":"10.3182\/20110828-6-IT-1002.03331"},{"key":"264_CR41","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2017.05.020","author":"RG Hefron","year":"2017","unstructured":"Hefron, R. G., Borghetti, B. J., Christensen, J. C. & Kabban, C. M. S. Deep long short-term memory structures model temporal dependencies improving cognitive workload estimation. Pattern Recognit. Lett. https:\/\/doi.org\/10.1016\/j.patrec.2017.05.020 (2017).","journal-title":"Pattern Recognit. Lett."},{"key":"264_CR42","doi-asserted-by":"crossref","unstructured":"Ratner, A. et al. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment (2017).","DOI":"10.14778\/3157794.3157797"},{"key":"264_CR43","doi-asserted-by":"crossref","unstructured":"Saab, K., Dunnmon, J., Goldman, R., Ratner, A., Sagreiya, H., R\u00e9, C. & Rubin, D. Doubly Weak Supervision of Deep Learning Models for Head CT. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 811\u2013819 (2019).","DOI":"10.1007\/978-3-030-32248-9_90"},{"key":"264_CR44","doi-asserted-by":"crossref","unstructured":"Scheffer, I. E. et al. ILAE classification of the epilepsies: position paper of the ILAE Commission for Classification and Terminology. Epilepsia 58, 512\u2013521 (2017).","DOI":"10.1111\/epi.13709"},{"key":"264_CR45","doi-asserted-by":"crossref","unstructured":"Roy, S., Kiral-Kornek, I. & Harrer, S. ChronoNet: a deep recurrent neural network for abnormal EEG identification. In Conference on Artificial Intelligence in Medicine in Europe. Springer, 47\u201356 (2019).","DOI":"10.1007\/978-3-030-21642-9_8"},{"key":"264_CR46","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V. & Alemi, A. A. Inception-v4, inception-ResNet and the impact of residual connections on learning. 31st AAAI Conference on Artificial Intelligence (2017).","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"264_CR47","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. 30th IEEE Conference on Computer Vision and Pattern Recognition (2017).","DOI":"10.1109\/CVPR.2017.243"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0264-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0264-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0264-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T02:10:02Z","timestamp":1670379002000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0264-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,20]]},"references-count":47,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["264"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-0264-0","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,20]]},"assertion":[{"value":"6 September 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 March 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 April 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"59"}}