{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T06:44:00Z","timestamp":1789454640558,"version":"build-2803163510"},"reference-count":26,"publisher":"American Society of Clinical Oncology (ASCO)","issue":"8","content-domain":{"domain":["ascopubs.org"],"crossmark-restriction":true},"short-container-title":["JCO Clin Cancer Inform"],"published-print":{"date-parts":[[2024,7]]},"abstract":"<jats:sec><jats:title>PURPOSE<\/jats:title><jats:p> Stage in multiple myeloma (MM) is an essential measure of disease risk, but its measurement in large databases is often lacking. We aimed to develop and validate a natural language processing (NLP) algorithm to extract oncologists' documentation of stage in the national Veterans Affairs (VA) Healthcare System. <\/jats:p><\/jats:sec><jats:sec><jats:title>METHODS<\/jats:title><jats:p> Using nationwide electronic health record (EHR) and cancer registry data from the VA Corporate Data Warehouse, we developed and validated a rule-based NLP algorithm to extract oncologist-determined MM stage. To that end, a clinician annotated MM stage within over 5,000 short snippets of clinical notes, and annotated MM stage at MM treatment initiation for 200 patients. These were allocated into snippet- and patient-level development and validation sets. We developed MM stage extraction and roll-up algorithms within the development sets. After the algorithms were finalized, we validated them using standard measures in held-out validation sets. <\/jats:p><\/jats:sec><jats:sec><jats:title>RESULTS<\/jats:title><jats:p> We developed algorithms for three different MM staging systems that have been in widespread use (Revised International Staging System [R-ISS], International Staging System [ISS], and Durie-Salmon [DS]) and for stage reported without a clearly defined system. Precision and recall were uniformly high for MM stage at the snippet level, ranging from 0.92 to 0.99 for the different MM staging systems. Performance in identifying for MM stage at treatment initiation at the patient level was also excellent, with precision of 0.92, 0.96, 0.90, and 0.86 and recall of 0.99, 0.98, 0.94, and 0.92 for R-ISS, ISS, DS, and unclear stage, respectively. <\/jats:p><\/jats:sec><jats:sec><jats:title>CONCLUSION<\/jats:title><jats:p> Our MM stage extraction algorithm uses rule-based NLP and data aggregation to accurately measure MM stage documented in oncology notes and pathology reports in VA's national EHR system. It may be adapted to other systems where MM stage is recorded in clinical notes. <\/jats:p><\/jats:sec>","DOI":"10.1200\/cci.23.00197","type":"journal-article","created":{"date-parts":[[2024,7,22]],"date-time":"2024-07-22T20:00:01Z","timestamp":1721678401000},"update-policy":"https:\/\/doi.org\/10.1200\/crossmark","source":"Crossref","is-referenced-by-count":3,"title":["Natural Language Processing Algorithm to Extract Multiple Myeloma Stage From Oncology Notes in the Veterans Affairs Healthcare System"],"prefix":"10.1200","author":[{"given":"Sergey D.","family":"Goryachev","sequence":"first","affiliation":[{"name":"Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), Boston, MA"},{"name":"VA Boston Healthcare System, Boston, MA"},{"name":"VA Boston Cooperative Studies Program, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cenk","family":"Yildirim","sequence":"additional","affiliation":[{"name":"Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), Boston, MA"},{"name":"VA Boston Healthcare System, Boston, MA"},{"name":"VA Boston Cooperative Studies Program, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2582-7816","authenticated-orcid":true,"given":"Clark","family":"DuMontier","sequence":"additional","affiliation":[{"name":"New England Geriatrics Research, Education and Clinical Center, VA Boston Healthcare System, Boston, MA"},{"name":"Division of Aging, Brigham and Women's Hospital, Boston, MA"},{"name":"Divison of Population Sciences, Dana-Farber Cancer Institute, Boston, MA"},{"name":"Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2266-0221","authenticated-orcid":true,"given":"Jennifer","family":"La","sequence":"additional","affiliation":[{"name":"Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), Boston, MA"},{"name":"VA Boston Healthcare System, Boston, MA"},{"name":"Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mayuri","family":"Dharne","sequence":"additional","affiliation":[{"name":"VA Boston Healthcare System, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J. Michael","family":"Gaziano","sequence":"additional","affiliation":[{"name":"Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), Boston, MA"},{"name":"VA Boston Healthcare System, Boston, MA"},{"name":"Division of Aging, Brigham and Women's Hospital, Boston, MA"},{"name":"Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2314-4368","authenticated-orcid":true,"given":"Mary T.","family":"Brophy","sequence":"additional","affiliation":[{"name":"Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), Boston, MA"},{"name":"VA Boston Healthcare System, Boston, MA"},{"name":"VA Boston Cooperative Studies Program, Boston, MA"},{"name":"Boston University School of Medicine, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7344-9795","authenticated-orcid":true,"given":"Nikhil C.","family":"Munshi","sequence":"additional","affiliation":[{"name":"VA Boston Healthcare System, Boston, MA"},{"name":"Harvard Medical School, Boston, MA"},{"name":"Department of Medical Oncology, Dana-Farber Cancer Institute, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jane A.","family":"Driver","sequence":"additional","affiliation":[{"name":"New England Geriatrics Research, Education and Clinical Center, VA Boston Healthcare System, Boston, MA"},{"name":"Division of Aging, Brigham and Women's Hospital, Boston, MA"},{"name":"Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6868-7011","authenticated-orcid":true,"given":"Nhan V.","family":"Do","sequence":"additional","affiliation":[{"name":"Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), Boston, MA"},{"name":"VA Boston Healthcare System, Boston, MA"},{"name":"VA Boston Cooperative Studies Program, Boston, MA"},{"name":"Boston University School of Medicine, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8058-3423","authenticated-orcid":true,"given":"Nathanael R.","family":"Fillmore","sequence":"additional","affiliation":[{"name":"Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), Boston, MA"},{"name":"VA Boston Healthcare System, Boston, MA"},{"name":"Harvard Medical School, Boston, MA"},{"name":"Department of Medical Oncology, Dana-Farber Cancer Institute, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"233","reference":[{"key":"b1","doi-asserted-by":"publisher","DOI":"10.1002\/cam4.2698"},{"key":"b2","doi-asserted-by":"publisher","DOI":"10.1182\/bloodadvances.2019000308"},{"key":"b3","volume-title":"TNM Classification","author":"Rosen RD","year":"2022"},{"key":"b4","doi-asserted-by":"publisher","DOI":"10.3390\/cancers13123053"},{"key":"b5","doi-asserted-by":"publisher","DOI":"10.1093\/jnci\/djab007"},{"key":"b6","doi-asserted-by":"publisher","DOI":"10.1097\/NAQ.0000000000000118"},{"key":"b7","first-page":"76","volume":"46","author":"Zullig LL","year":"2019","journal-title":"J Registry Manag"},{"key":"b8","doi-asserted-by":"publisher","DOI":"10.1053\/j.seminoncol.2019.10.001"},{"key":"b9","doi-asserted-by":"publisher","DOI":"10.1002\/pds.5579"},{"key":"b11","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324904003523"},{"key":"b12","first-page":"537","volume":"2013","author":"Gundlapalli AV","year":"2013","journal-title":"AMIA Annu Symp Proc"},{"key":"b13","first-page":"36","volume":"2008","author":"Gundlapalli AV","year":"2008","journal-title":"Summit Transl Bioinforma"},{"key":"b14","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocw097"},{"key":"b15","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooaa064"},{"key":"b16","author":"Malty AM","journal-title":"JCO Clin Cancer Inform"},{"key":"b17","doi-asserted-by":"publisher","DOI":"10.3390\/jcm8070999"},{"key":"b18","doi-asserted-by":"publisher","DOI":"10.1186\/s13326-020-00231-z"},{"key":"b19","author":"Ryu JH","journal-title":"JCO Clin Cancer Inform"},{"key":"b20","doi-asserted-by":"publisher","DOI":"10.1158\/1055-9965.EPI-19-0873"},{"key":"b21","doi-asserted-by":"publisher","DOI":"10.37549\/AR2102"},{"key":"b22","doi-asserted-by":"publisher","DOI":"10.1111\/j.1525-139X.2009.00584.x"},{"key":"b23","doi-asserted-by":"publisher","DOI":"10.1182\/bloodadvances.2023011019"},{"key":"b24","doi-asserted-by":"publisher","DOI":"10.2196\/12239"},{"key":"b25","doi-asserted-by":"publisher","DOI":"10.1200\/JCO.21.02614"},{"key":"b26","author":"Wang L","journal-title":"JCO Clin Cancer Inform"},{"key":"b27","doi-asserted-by":"publisher","DOI":"10.1002\/wics.1549"}],"container-title":["JCO Clinical Cancer Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ascopubs.org\/doi\/pdfdirect\/10.1200\/CCI.23.00197","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,22]],"date-time":"2024-07-22T20:00:10Z","timestamp":1721678410000},"score":1,"resource":{"primary":{"URL":"https:\/\/ascopubs.org\/doi\/10.1200\/CCI.23.00197"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7]]},"references-count":26,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2024,7]]}},"alternative-id":["10.1200\/CCI.23.00197"],"URL":"https:\/\/doi.org\/10.1200\/cci.23.00197","relation":{},"ISSN":["2473-4276"],"issn-type":[{"value":"2473-4276","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7]]},"assertion":[{"value":"2023-09-29","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-14","order":1,"name":"revised","label":"Revised","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-06","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2300197"}}