{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T12:59:44Z","timestamp":1772456384302,"version":"3.50.1"},"reference-count":31,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2022,2,22]],"date-time":"2022-02-22T00:00:00Z","timestamp":1645488000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000065","name":"National Institute of Neurological Disorders and Stroke","doi-asserted-by":"publisher","award":["1DP1 OD029758"],"award-info":[{"award-number":["1DP1 OD029758"]}],"id":[{"id":"10.13039\/100000065","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Mirowski Family Foundation; and by contributions from Jonathan and Bonnie Rothberg"},{"name":"National Institute of Neurological Disorders and Stroke of the National Institutes of Health","award":["K23NS121520"],"award-info":[{"award-number":["K23NS121520"]}]},{"name":"American Academy of Neurology Susan S. Spencer Clinical Research Training Scholarship"},{"DOI":"10.13039\/100017021","name":"Mirowski Family Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100017021","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Office of Naval Research Contract","award":["N00014-19-1-2620"],"award-info":[{"award-number":["N00014-19-1-2620"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,13]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Seizure frequency and seizure freedom are among the most important outcome measures for patients with epilepsy. In this study, we aimed to automatically extract this clinical information from unstructured text in clinical notes. If successful, this could improve clinical decision-making in epilepsy patients and allow for rapid, large-scale retrospective research.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We developed a finetuning pipeline for pretrained neural models to classify patients as being seizure-free and to extract text containing their seizure frequency and date of last seizure from clinical notes. We annotated 1000 notes for use as training and testing data and determined how well 3 pretrained neural models, BERT, RoBERTa, and Bio_ClinicalBERT, could identify and extract the desired information after finetuning.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The finetuned models (BERTFT, Bio_ClinicalBERTFT, and RoBERTaFT) achieved near-human performance when classifying patients as seizure free, with BERTFT and Bio_ClinicalBERTFT achieving accuracy scores over 80%. All 3 models also achieved human performance when extracting seizure frequency and date of last seizure, with overall F1 scores over 0.80. The best combination of models was Bio_ClinicalBERTFT for classification, and RoBERTaFT for text extraction. Most of the gains in performance due to finetuning required roughly 70 annotated notes.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion and Conclusion<\/jats:title>\n                  <jats:p>Our novel machine reading approach to extracting important clinical outcomes performed at or near human performance on several tasks. This approach opens new possibilities to support clinical practice and conduct large-scale retrospective clinical research. Future studies can use our finetuning pipeline with minimal training annotations to answer new clinical questions.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocac018","type":"journal-article","created":{"date-parts":[[2022,2,9]],"date-time":"2022-02-09T12:24:34Z","timestamp":1644409474000},"page":"873-881","source":"Crossref","is-referenced-by-count":56,"title":["Extracting seizure frequency from epilepsy clinic notes: a machine reading approach to natural language processing"],"prefix":"10.1093","volume":"29","author":[{"given":"Kevin","family":"Xie","sequence":"first","affiliation":[{"name":"Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Ryan S","family":"Gallagher","sequence":"additional","affiliation":[{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Department of Neurology, Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Erin C","family":"Conrad","sequence":"additional","affiliation":[{"name":"Department of Neurology, Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Chadric O","family":"Garrick","sequence":"additional","affiliation":[{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Steven N","family":"Baldassano","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"John M","family":"Bernabei","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Peter D","family":"Galer","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Department of Biomedical and Health Informatics, Children\u2019s Hospital of Philadelphia, Philadelphia, Pennsylvania, USA"}]},{"given":"Nina J","family":"Ghosn","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Adam S","family":"Greenblatt","sequence":"additional","affiliation":[{"name":"Department of Neurology, Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Tara","family":"Jennings","sequence":"additional","affiliation":[{"name":"Department of Neurology, Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Alana","family":"Kornspun","sequence":"additional","affiliation":[{"name":"Department of Neurology, Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Catherine V","family":"Kulick-Soper","sequence":"additional","affiliation":[{"name":"Department of Neurology, Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Jal M","family":"Panchal","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"The General Robotics, Automation, Sensing and Perception Laboratory, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Akash R","family":"Pattnaik","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Brittany H","family":"Scheid","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Danmeng","family":"Wei","sequence":"additional","affiliation":[{"name":"Department of Neurology, Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Micah","family":"Weitzman","sequence":"additional","affiliation":[{"name":"Department of Electrical and Systems Engineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Ramya","family":"Muthukrishnan","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Joongwon","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Brian","family":"Litt","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Department of Neurology, Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Colin A","family":"Ellis","sequence":"additional","affiliation":[{"name":"Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"},{"name":"Department of Neurology, Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"Dan","family":"Roth","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]}],"member":"286","published-online":{"date-parts":[[2022,2,22]]},"reference":[{"key":"2022041311585690800_ocac018-B1","author":"Ehrenstein","year":"2019"},{"key":"2022041311585690800_ocac018-B2","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1146\/annurev-publhealth-032315-021353","article-title":"Using electronic health records for population health research: a review of methods and applications","volume":"37","author":"Casey","year":"2016","journal-title":"Annu Rev Public Health"},{"issue":"1","key":"2022041311585690800_ocac018-B3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s00392-016-1025-6","article-title":"Electronic health records to facilitate clinical research","volume":"106","author":"Cowie","year":"2017","journal-title":"Clin Res Cardiol"},{"issue":"1","key":"2022041311585690800_ocac018-B4","first-page":"1123","article-title":"Unlocking the potential of electronic health records for health research","volume":"5","author":"Lee","year":"2020","journal-title":"Int J Popul Data Sci"},{"issue":"22","key":"2022041311585690800_ocac018-B5","doi-asserted-by":"crossref","first-page":"4081","DOI":"10.1200\/JCO.2003.08.972","article-title":"Researching the cost of research","volume":"21","author":"Wright","year":"2003","journal-title":"J Clin Oncol"},{"issue":"22","key":"2022041311585690800_ocac018-B6","doi-asserted-by":"crossref","first-page":"4145","DOI":"10.1200\/JCO.2003.08.156","article-title":"The costs of conducting clinical research","volume":"21","author":"Emanuel","year":"2003","journal-title":"J Clin Oncol"},{"issue":"6","key":"2022041311585690800_ocac018-B7","doi-asserted-by":"crossref","first-page":"2234","DOI":"10.1097\/PRS.0b013e3181f44abc","article-title":"Observational studies: cohort and case-control studies","volume":"126","author":"Song","year":"2010","journal-title":"Plast Reconstr Surg"},{"issue":"7","key":"2022041311585690800_ocac018-B8","doi-asserted-by":"crossref","first-page":"e0131521","DOI":"10.1371\/journal.pone.0131521","article-title":"How to establish and follow up a large prospective cohort study in the 21st century - lessons from UK COSMOS","volume":"10","author":"Toledano","year":"2015","journal-title":"PLoS One"},{"key":"2022041311585690800_ocac018-B9","first-page":"4171","author":"Devlin","year":"2019"},{"key":"2022041311585690800_ocac018-B10","first-page":"2898","author":"Chalkidis","year":"2020"},{"key":"2022041311585690800_ocac018-B11","first-page":"6000","author":"Vaswani"},{"key":"2022041311585690800_ocac018-B12","first-page":"72","author":"Alsentzer","year":"2019"},{"key":"2022041311585690800_ocac018-B13","first-page":"5","author":"Klie","year":"2018"},{"issue":"1","key":"2022041311585690800_ocac018-B14","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ Psychol Meas"},{"key":"2022041311585690800_ocac018-B15","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: the kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem Med"},{"key":"2022041311585690800_ocac018-B16","author":"Liu"},{"key":"2022041311585690800_ocac018-B17","first-page":"4238","author":"Han","year":"2019"},{"key":"2022041311585690800_ocac018-B18","first-page":"5532","author":"Soni","year":"2020"},{"key":"2022041311585690800_ocac018-B19","first-page":"4543","author":"Sulem","year":"2021"},{"key":"2022041311585690800_ocac018-B20","first-page":"784","author":"Rajpurkar","year":"2018"},{"key":"2022041311585690800_ocac018-B21","first-page":"38","author":"Wolf","year":"2020"},{"key":"2022041311585690800_ocac018-B22","first-page":"3363","author":"Zhou","year":"2019"},{"key":"2022041311585690800_ocac018-B23","first-page":"4070","author":"Vashishtha","year":"2020"},{"key":"2022041311585690800_ocac018-B24","author":"Yu","year":"2020"},{"key":"2022041311585690800_ocac018-B25","first-page":"3622","author":"Liu","year":"2020"},{"key":"2022041311585690800_ocac018-B26","first-page":"13388","article-title":"Natural language inference in context - investigating contextual reasoning over long texts","volume":"35","author":"Liu","year":"2021","journal-title":"Proc AAAI Conf Artif Intell"},{"key":"2022041311585690800_ocac018-B27","author":"Helwe","year":"2021"},{"key":"2022041311585690800_ocac018-B28","first-page":"1586","author":"Romanov","year":"2018"},{"issue":"12","key":"2022041311585690800_ocac018-B29","doi-asserted-by":"crossref","first-page":"1935","DOI":"10.1093\/jamia\/ocaa189","article-title":"Clinical concept extraction using transformers","volume":"27","author":"Yang","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2022041311585690800_ocac018-B30","doi-asserted-by":"crossref","first-page":"e023232","DOI":"10.1136\/bmjopen-2018-023232","article-title":"Using natural language processing to extract structured epilepsy data from unstructured clinic letters: development and validation of the ExECT (extraction of epilepsy clinical text) system","volume":"9","author":"Fonferko-Shadrach","year":"2019","journal-title":"BMJ Open"},{"key":"2022041311585690800_ocac018-B31","author":"Beltagy","year":"2020"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocac018\/42870037\/ocac018.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/5\/873\/43372456\/ocac018.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/5\/873\/43372456\/ocac018.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T12:20:19Z","timestamp":1649852419000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/5\/873\/6534112"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,22]]},"references-count":31,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,2,22]]},"published-print":{"date-parts":[[2022,4,13]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac018","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,5,1]]},"published":{"date-parts":[[2022,2,22]]}}}