{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T16:01:46Z","timestamp":1786464106730,"version":"build-2736575974"},"reference-count":26,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T00:00:00Z","timestamp":1786406400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurol."],"abstract":"<jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Generating reference standards to train and evaluate the accuracy of artificial intelligence (AI) algorithms poses a significant challenge. Particularly when interpreting complex signals like electroencephalography (EEG), where interrater variability is considerable. We aimed to characterize the impact of interrater variability when evaluating the performance of an AI algorithm for detecting electrographic status epilepticus in point-of-care (POC) limited-montage EEG.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We analyzed 604 EEGs collected using a POC EEG system (Ceribell Inc.). Each EEG was independently reviewed by 5\u20137 blinded experts, who annotated seizures and completed standardized assessments. The EEGs were later analyzed by the Clarity AI algorithm (version 7), trained on separate EEG dataset. Interrater agreement was assessed using Gwet\u2019s AC1. Sensitivity and specificity for detecting electrographic status epilepticus (ESE) were estimated for the reviewers and the AI algorithm. Multiple evaluation schemes were employed, including simple majority consensus of the full group (group majority), simple majority using leave-one-out analysis, and 2-of-3 majority with bootstrap sampling.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Of 604 POC EEGs, the group majority identified 8 cases (1.3%) meeting ACNS criteria for ESE and 14 (2.3%) with seizures, the rest were classified as normal\/slowing (80.1%), highly epileptiform patterns (6.6%), or other findings (4.3%). Twentynine cases lacked consensus interpretation. Interrater agreement among reviewers was 0.67\u20130.68. Compared to the group majority, the AI algorithm showed higher sensitivity (median 100%) with lower specificity (93.5%) than individual reviewers (median sensitivity 60%, specificity 98.7%). Using all possible 3-reviewer combinations to define majority agreement, the number of EEGs identified as ESE varied widely (4\u201318 cases, median\u202f=\u202f10). The AI algorithm consistently achieved significantly higher sensitivity than external human reviewers (71.4% vs. 50%,\n                      <jats:italic>p<\/jats:italic>\n                      \u202f&amp;lt;\u202f0.001). However, the AI\u2019s specificity, while still high (median\u202f=\u202f93.9%), was slightly lower than that of human reviewers (median\u202f=\u202f98.4%,\n                      <jats:italic>p<\/jats:italic>\n                      \u202f&amp;lt;\u202f0.001), though the AI\u2019s specificity had consistently narrower spread.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>This study highlights the challenges of defining the correct answer for EEG interpretation, especially for ESE, and its consequences for evaluating seizure detection AI tools. Future work should explore how AI assistance impacts human interpretation, particularly in reducing interrater variability across a broader range of EEG patterns.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fneur.2026.1833203","type":"journal-article","created":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T13:50:01Z","timestamp":1786456201000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Assessing the performance of artificial intelligence in detecting electrographic status epilepticus when human experts often disagree"],"prefix":"10.3389","volume":"17","author":[{"given":"Khalid","family":"Alsherbini","sequence":"first","affiliation":[{"name":"Department of Neurology, Banner Health, University of Arizona","place":["Phoenix, AZ, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michelle","family":"Armenta Salas","sequence":"additional","affiliation":[{"name":"Ceribell Inc.","place":["Sunnyvale, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Suganya","family":"Karunakaran","sequence":"additional","affiliation":[{"name":"Ceribell Inc.","place":["Sunnyvale, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Archit","family":"Gupta","sequence":"additional","affiliation":[{"name":"Ceribell Inc.","place":["Sunnyvale, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Baharan","family":"Kamousi","sequence":"additional","affiliation":[{"name":"Ceribell Inc.","place":["Sunnyvale, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Raymond","family":"Woo","sequence":"additional","affiliation":[{"name":"Ceribell Inc.","place":["Sunnyvale, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Veeresh","family":"Kumar N. Shivamurthy","sequence":"additional","affiliation":[{"name":"Department of Neurology, Trinity Health, University of Connecticut","place":["Hartford, CT, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"King-Stephens","sequence":"additional","affiliation":[{"name":"Department of Neurology, University of California\u2014Irvine","place":["Irvine, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lawrence J.","family":"Hirsch","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale School of Medicine","place":["New Haven, CT, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Stern","sequence":"additional","affiliation":[{"name":"Department of Neurology, University of California\u2014Los Angeles","place":["Los Angeles, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ioannis","family":"Karakis","sequence":"additional","affiliation":[{"name":"Department of Neurology, Emory University","place":["Atlanta, GA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vikram R.","family":"Rao","sequence":"additional","affiliation":[{"name":"Department of Neurology and Weill Institute for Neurosciences, University of California, San Francisco","place":["San Francisco, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Farid G.","family":"Sadaka","sequence":"additional","affiliation":[{"name":"Department of Adult Critical Care, Mercy Hospital St. Louis","place":["St. Louis, MO, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Parshaw J.","family":"Dorriz","sequence":"additional","affiliation":[{"name":"Department of Neurology, Providence Health, Mission Viejo Medical Center","place":["Mission Viejo, CA, United States"]},{"name":"Department of Neurology, Keck School of Medicine, University of Southern California","place":["Los Angeles, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Josef","family":"Parvizi","sequence":"additional","affiliation":[{"name":"Department of Neurology and Neurological Sciences, Stanford University","place":["Palo Alto, CA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,8,11]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1097\/01.WNP.0000158701.89576.4C","article-title":"The ACNS subcommittee on research terminology for continuous EEG monitoring: proposed standardized terminology for rhythmic and periodic EEG patterns encountered in critically ill patients","volume":"22","author":"Hirsch","year":"2005","journal-title":"J Clin Neurophysiol"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1097\/WNP.0b013e3182570f83","article-title":"Interrater reliability of ICU EEG research terminology","volume":"29","author":"Mani","year":"2012","journal-title":"J Clin Neurophysiol"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1097\/WNP.0b013e3182051123","article-title":"Interobserver reproducibility of electroencephalogram interpretation in critically ill children","volume":"28","author":"Abend","year":"2011","journal-title":"J Clin Neurophysiol"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"e1737","DOI":"10.1212\/WNL.0000000000201670","article-title":"Interrater reliability of expert Electroencephalographers identifying seizures and rhythmic and periodic patterns in EEGs","volume":"100","author":"Jing","year":"2023","journal-title":"Neurology"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1097\/wnp.0000000000000344","article-title":"Characteristics of EEG interpreters associated with higher interrater agreement","volume":"34","author":"Halford","year":"2017","journal-title":"J Clin Neurophysiol"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"1714","DOI":"10.1212\/WNL.35.12.1714","article-title":"Interobserver variability in EEG interpretation","volume":"35","author":"Williams","year":"1985","journal-title":"Neurology"},{"key":"ref7","doi-asserted-by":"publisher","first-page":"485","DOI":"10.1046\/j.1440-1819.2003.01152.x","article-title":"An intervention to improve the interrater reliability of clinical EEG interpretations","volume":"57","author":"Azuma","year":"2003","journal-title":"Psychiatry Clin Neurosci"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1001\/jamaneurol.2019.3531","article-title":"Interrater reliability of experts in identifying Interictal epileptiform discharges in electroencephalograms","volume":"77","author":"Jing","year":"2020","journal-title":"JAMA Neurol"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1097\/WNP.0000000000000709","article-title":"Seizure detection: Interreader agreement and detection algorithm assessments using a large dataset","volume":"38","author":"Scheuer","year":"2021","journal-title":"J Clin Neurophysiol"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1111\/j.1528-1167.2009.02040.x","article-title":"The history of status epilepticus and its treatment","volume":"50","author":"Neligan","year":"2009","journal-title":"Epilepsia"},{"key":"ref11","volume-title":"Status Epilepticus","author":"Wylie","year":"2023"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1097\/wnp.0000000000000806","article-title":"American clinical neurophysiology society's standardized critical care EEG terminology: 2021 version","volume":"38","author":"Hirsch","year":"2021","journal-title":"J Clin Neurophysiol"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"1515","DOI":"10.1111\/epi.13121","article-title":"A definition and classification of status epilepticus \u2013 report of the ILAE task force on classification of status epilepticus","volume":"56","author":"Trinka","year":"2015","journal-title":"Epilepsia"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1212\/CON.0000000000001103","article-title":"Management of status epilepticus, refractory status epilepticus, and super-refractory status epilepticus","volume":"28","author":"Trinka","year":"2022","journal-title":"Continuum (Minneap Minn)"},{"key":"ref15","doi-asserted-by":"publisher","first-page":"2351","DOI":"10.1111\/epi.17694","article-title":"Diagnosing nonconvulsive status epilepticus: defining electroencephalographic and clinical response to diagnostic intravenous antiseizure medication trials","volume":"64","author":"Leitinger","year":"2023","journal-title":"Epilepsia"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"e88","DOI":"10.1111\/epi.16289","article-title":"The difficulty of diagnosing NCSE in clinical practice; external validation of the Salzburg criteria","volume":"60","author":"Goselink","year":"2019","journal-title":"Epilepsia"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"908","DOI":"10.1007\/s12028-020-01120-0","article-title":"Monitoring the burden of seizures and highly epileptiform patterns in critical care with a novel machine learning method","volume":"34","author":"Kamousi","year":"2021","journal-title":"Neurocrit Care"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1186\/1471-2288-13-61","article-title":"A comparison of Cohen's kappa and Gwet\u2019s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples","volume":"13","author":"Wongpakaran","year":"2013","journal-title":"BMC Med Res Methodol"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"13406","DOI":"10.1038\/s41598-021-92827-3","article-title":"Prospective evaluation of interrater agreement between EEG technologists and neurophysiologists","volume":"11","author":"Beuchat","year":"2021","journal-title":"Sci Rep"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"2433","DOI":"10.1111\/epi.18365","article-title":"Validation of a discrete electrographic seizure detection algorithm for extended-duration, reduced-channel wearable EEG","volume":"66","author":"Newton","year":"2025","journal-title":"Epilepsia"},{"key":"ref21","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1016\/j.neucli.2022.08.002","article-title":"One EEG, one read\u2014a manifesto towards reducing interrater variability among experts","volume":"52","author":"Nascimento","year":"2022","journal-title":"Neurophysiologie Clinique"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"1366","DOI":"10.1111\/epi.12653","article-title":"Interrater agreement for critical care EEG terminology","volume":"55","author":"Gaspard","year":"2014","journal-title":"Epilepsia"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"1661","DOI":"10.1016\/j.clinph.2014.11.008","article-title":"Inter-rater agreement on identification of electrographic seizures and periodic discharges in ICU EEG recordings","volume":"126","author":"Halford","year":"2015","journal-title":"Clin Neurophysiol"},{"key":"ref24","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1016\/j.yebeh.2014.01.011","article-title":"EEG interpretation reliability and interpreter confidence: a large single-center study","volume":"32","author":"Grant","year":"2014","journal-title":"Epilepsy Behav"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"e210234","DOI":"10.1212\/WNL.0000000000210234","article-title":"Accuracy of a rapid-response EEG\u2019S automated seizure-burden estimator","volume":"104","author":"Sheikh","year":"2025","journal-title":"Neurology"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"394","DOI":"10.1016\/j.neucli.2022.08.002","article-title":"EEG reading with or without clinical information \u2013 a real-world practice study","volume":"52","author":"Nascimento","year":"2022","journal-title":"Neurophysiol Clin"}],"container-title":["Frontiers in Neurology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fneur.2026.1833203\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T13:50:02Z","timestamp":1786456202000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fneur.2026.1833203\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,11]]},"references-count":26,"alternative-id":["10.3389\/fneur.2026.1833203"],"URL":"https:\/\/doi.org\/10.3389\/fneur.2026.1833203","relation":{},"ISSN":["1664-2295"],"issn-type":[{"value":"1664-2295","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,11]]},"article-number":"1833203"}}