{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,4]],"date-time":"2026-08-04T14:57:11Z","timestamp":1785855431015,"version":"3.56.0"},"reference-count":37,"publisher":"Wiley","issue":"11","license":[{"start":{"date-parts":[[2025,8,6]],"date-time":"2025-08-06T00:00:00Z","timestamp":1754438400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"},{"start":{"date-parts":[[2025,8,6]],"date-time":"2025-08-06T00:00:00Z","timestamp":1754438400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2014431"],"award-info":[{"award-number":["2014431"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["RF1AG064312"],"award-info":[{"award-number":["RF1AG064312"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["RF1NS120947"],"award-info":[{"award-number":["RF1NS120947"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01AG073410"],"award-info":[{"award-number":["R01AG073410"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01HL161253"],"award-info":[{"award-number":["R01HL161253"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Epilepsia"],"published-print":{"date-parts":[[2025,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Detection of seizures and rhythmic or periodic patterns (SRPPs) on electroencephalography (EEG) is crucial for the diagnosis and management of patients with neurological critical illness. Although automated detection methods are advancing, they require large, high\u2010quality, expert\u2010annotated datasets for training. However, expert annotation is limited by the availability of trained neurophysiologists. Crowdsourcing, or soliciting contributions from a large group of people, may present a potential solution. This study evaluates the feasibility of crowdsourcing annotations of short epochs of EEG recordings by comparing the performance of experts and non\u2010experts in identifying six SRPPs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We conducted an EEG scoring contest using a mobile app, involving expert and non\u2010expert participants. Non\u2010experts in our studies include physicians\u2010MD, medical students\u2010MS, nurse\u2010practitioner\u2010NP\/physician\u2010assistant\u2010PA\/pharmacists, NP\u2010students\/PA\u2010students\/pharmacy\u2010students\/other\u2010healthcare\u2010students, and others. Performance was assessed using pairwise agreement and Fleiss' kappa between experts, and accuracy comparisons between experts and the crowd using individual and weighted majority votes.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>A total of 1542 participants (8 experts and 1534 non\u2010experts) answered 478\u2009834 questions across six SRPPs: seizures, generalized and lateralized periodic discharges (GPDs and LPDs), and generalized and lateralized rhythmic delta activity (GRDA LRDA), and \u201cOther.\u201d Using individual, non\u2010weighted votes, the crowd's performance was inferior to experts for overall and across six SRPP identification. Using weighted majority votes, the crowd was non\u2010inferior to experts for overall SRPP identification with accuracy of .70, 95% confidence interval [CI]: .69\u2013.70 compared to expert's accuracy of .68, 95% CI: .68\u2013.70. The crowd performed comparably or better than experts in identifying most SRPPs, except for LPDs and \u201cOther\u201d. No individual expert outperformed the crowd on overall metrics.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Significance<\/jats:title>\n                    <jats:p>This proof\u2010of\u2010concept highlights the promise of crowd reviewers for obtaining expert\u2010level annotations of SRPPs, which could potentially accelerate the development of large, diverse datasets for training automated detection algorithms. Challenges, such as varying calibration\/test splits across crowd participants in the study and the absence of gold standard labels in the real\u2010world settings, remain to be addressed.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1111\/epi.18547","type":"journal-article","created":{"date-parts":[[2025,8,6]],"date-time":"2025-08-06T16:49:07Z","timestamp":1754498947000},"page":"4366-4380","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating crowdsourcing for\n                    <scp>ICU EEG<\/scp>\n                    annotation: A comparison with expert performance"],"prefix":"10.1111","volume":"66","author":[{"given":"Wan\u2010Yee","family":"Kong","sequence":"first","affiliation":[{"name":"Beth Israel Deaconess Medical Center  Boston Massachusetts USA"},{"name":"Harvard Medical School  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"F\u00e1bio A.","family":"Nascimento","sequence":"additional","affiliation":[{"name":"Washington University School of Medicine  St. Louis Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron","family":"Struck","sequence":"additional","affiliation":[{"name":"University of Wisconsin Hospital and Clinics  Madison Wisconsin USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erik","family":"Duhaime","sequence":"additional","affiliation":[{"name":"Centaur Labs  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Srishti","family":"Kapur","sequence":"additional","affiliation":[{"name":"Centaur Labs  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edilberto","family":"Amorim","sequence":"additional","affiliation":[{"name":"University of California San Francisco  San Francisco California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gregory","family":"Kapinos","sequence":"additional","affiliation":[{"name":"Icahn School of Medicine at Mount Sinai  New York City New York USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andres","family":"Rodriguez","sequence":"additional","affiliation":[{"name":"Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brendan","family":"Thomas","sequence":"additional","affiliation":[{"name":"Massachusetts General Hospital  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masoom","family":"Desai","sequence":"additional","affiliation":[{"name":"University of new Mexico  Albuquerque New Mexico USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5283-7476","authenticated-orcid":false,"given":"Jong Woo","family":"Lee","sequence":"additional","affiliation":[{"name":"Harvard Medical School  Boston Massachusetts USA"},{"name":"Brigham and Women's Hospital  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4803-312X","authenticated-orcid":false,"given":"M. Brandon","family":"Westover","sequence":"additional","affiliation":[{"name":"Beth Israel Deaconess Medical Center  Boston Massachusetts USA"},{"name":"Harvard Medical School  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jin","family":"Jing","sequence":"additional","affiliation":[{"name":"Beth Israel Deaconess Medical Center  Boston Massachusetts USA"},{"name":"Harvard Medical School  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,8,6]]},"reference":[{"key":"e_1_2_11_2_1","doi-asserted-by":"publisher","DOI":"10.1097\/WNP.0000000000000806"},{"key":"e_1_2_11_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cnp.2022.11.004"},{"key":"e_1_2_11_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinph.2021.02.403"},{"key":"e_1_2_11_5_1","doi-asserted-by":"publisher","DOI":"10.3389\/fmolb.2023.1146606"},{"key":"e_1_2_11_6_1","doi-asserted-by":"publisher","DOI":"10.1111\/epi.17246"},{"key":"e_1_2_11_7_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamaneurol.2019.3485"},{"key":"e_1_2_11_8_1","doi-asserted-by":"crossref","unstructured":"ThomasJ ComorettoL JinJ DauwelsJ CashSS WestoverMB.EEG CLassification via convolutional neural network\u2010based interictal epileptiform event detection. in 2018 40th annual international conference of the IEEE engineering in medicine and biology society (EMBC) 2018 pp. 3148\u20133151.https:\/\/doi.org\/10.1109\/EMBC.2018.8512930","DOI":"10.1109\/EMBC.2018.8512930"},{"key":"e_1_2_11_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/epi4.12704"},{"key":"e_1_2_11_10_1","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.0000000000201670"},{"key":"e_1_2_11_11_1","doi-asserted-by":"publisher","DOI":"10.2307\/2348873"},{"key":"e_1_2_11_12_1","doi-asserted-by":"crossref","unstructured":"DuhaimeEP JinM MoultonT WeberJ KurtanskyNR HalpernA et\u00a0al.Nonexpert crowds outperform expert individuals in diagnostic accuracy on a skin lesion diagnosis task. in 2023 IEEE 20th international symposium on biomedical imaging (ISBI) Apr. 2023 pp. 1\u20135.https:\/\/doi.org\/10.1109\/ISBI53787.2023.10230646","DOI":"10.1109\/ISBI53787.2023.10230646"},{"key":"e_1_2_11_13_1","doi-asserted-by":"publisher","DOI":"10.2196\/38412"},{"key":"e_1_2_11_14_1","doi-asserted-by":"publisher","DOI":"10.2196\/48589"},{"key":"e_1_2_11_15_1","doi-asserted-by":"publisher","DOI":"10.2196\/51397"},{"key":"e_1_2_11_16_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010024\u201001095\u20108"},{"key":"e_1_2_11_17_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41597\u2010020\u20100533\u20104"},{"key":"e_1_2_11_18_1","unstructured":"t\u2010Test Chi\u2010Square ANOVA Regression Correlation. [cited 2025 Apr 04]. Available from:https:\/\/datatab.net\/tutorial\/fleiss\u2010kappa"},{"key":"e_1_2_11_19_1","doi-asserted-by":"publisher","DOI":"10.1186\/1471\u20102288\u201013\u201061"},{"key":"e_1_2_11_20_1","first-page":"185","volume-title":"International Series in Operations Research & Management Science, in Judgment in Predictive Analytics","author":"Collins RN","year":"2023"},{"key":"e_1_2_11_21_1","doi-asserted-by":"publisher","DOI":"10.1097\/WNP.0000000000000344"},{"key":"e_1_2_11_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinph.2017.06.252"},{"key":"e_1_2_11_23_1","doi-asserted-by":"publisher","DOI":"10.1186\/s41235\u2010024\u201000558\u20106"},{"key":"e_1_2_11_24_1","doi-asserted-by":"crossref","unstructured":"BaiS WangD MullerT ChengP ChenJ.Stability of weighted majority voting under estimated weights.2024arXiv: arXiv:2207.06118.https:\/\/doi.org\/10.48550\/arXiv.2207.06118","DOI":"10.65109\/AGJJ2319"},{"key":"e_1_2_11_25_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2014.1909"},{"key":"e_1_2_11_26_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0403723101"},{"key":"e_1_2_11_27_1","doi-asserted-by":"publisher","DOI":"10.1097\/WNP.0b013e318182ed67"},{"key":"e_1_2_11_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.seizure.2008.10.010"},{"key":"e_1_2_11_29_1","doi-asserted-by":"publisher","DOI":"10.1097\/WNP.0b013e3182570f83"},{"key":"e_1_2_11_30_1","doi-asserted-by":"publisher","DOI":"10.1111\/epi.12653"},{"key":"e_1_2_11_31_1","unstructured":"Centaur Labs\u2014Medical Data Labeling Solutions. [cited 2024 Aug 19]. Available from:https:\/\/www.centaurlabs.com\/"},{"key":"e_1_2_11_32_1","unstructured":"Amazon Mechanical Turk. [cited 2024 Aug 19]. Available from:https:\/\/www.mturk.com\/"},{"key":"e_1_2_11_33_1","unstructured":"Improves AI with Data\u2014Powering AI Innovation Appen. [cited 2024 Aug 19]. Available from:https:\/\/www.appen.com\/"},{"key":"e_1_2_11_34_1","unstructured":"Generative AI and Computer Vision Data Annotation Sama. [cited 2024 Aug 19]. Available from:https:\/\/www.sama.com\/"},{"key":"e_1_2_11_35_1","unstructured":"ImageNet. [cited 2024 Aug 19]. Available from:https:\/\/www.image\u2010net.org\/index.php"},{"key":"e_1_2_11_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/SPMB50085.2020.9353617"},{"key":"e_1_2_11_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cnp.2023.08.003"},{"key":"e_1_2_11_38_1","doi-asserted-by":"publisher","DOI":"10.1002\/epd2.20229"}],"container-title":["Epilepsia"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/epi.18547","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/epi.18547","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/epi.18547","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,4]],"date-time":"2026-08-04T14:38:53Z","timestamp":1785854333000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/epi.18547"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,6]]},"references-count":37,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2025,11]]}},"alternative-id":["10.1111\/epi.18547"],"URL":"https:\/\/doi.org\/10.1111\/epi.18547","archive":["Portico"],"relation":{},"ISSN":["0013-9580","1528-1167"],"issn-type":[{"value":"0013-9580","type":"print"},{"value":"1528-1167","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,6]]},"assertion":[{"value":"2024-12-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-06-25","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}