{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,3]],"date-time":"2025-07-03T16:29:23Z","timestamp":1751560163907},"reference-count":33,"publisher":"Georg Thieme Verlag KG","issue":"01\/02","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Methods Inf Med"],"published-print":{"date-parts":[[2022,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>\n          Objective\u2003Natural language processing (NLP) systems convert unstructured text into analyzable data. Here, we describe the performance measures of NLP to capture granular details on nodules from thyroid ultrasound (US) reports and reveal critical issues with reporting language.<\/jats:p><jats:p>\n          Methods\u2003We iteratively developed NLP tools using clinical Text Analysis and Knowledge Extraction System (cTAKES) and thyroid US reports from 2007 to 2013. We incorporated nine nodule features for NLP extraction. Next, we evaluated the precision, recall, and accuracy of our NLP tools using a separate set of US reports from an academic medical center (A) and a regional health care system (B) during the same period. Two physicians manually annotated each test-set report. A third physician then adjudicated discrepancies. The adjudicated \u201cgold standard\u201d was then used to evaluate NLP performance on the test-set.<\/jats:p><jats:p>\n          Results\u2003A total of 243 thyroid US reports contained 6,405 data elements. Inter-annotator agreement for all elements was 91.3%. Compared with the gold standard, overall recall of the NLP tool was 90%. NLP recall for thyroid lobe or isthmus characteristics was: laterality 96% and size 95%. NLP accuracy for nodule characteristics was: laterality 92%, size 92%, calcifications 76%, vascularity 65%, echogenicity 62%, contents 76%, and borders 40%. NLP recall for presence or absence of lymphadenopathy was 61%. Reporting style accounted for 18% errors. For example, the word \u201cheterogeneous\u201d interchangeably referred to nodule contents or echogenicity. While nodule dimensions and laterality were often described, US reports only described contents, echogenicity, vascularity, calcifications, borders, and lymphadenopathy, 46, 41, 17, 15, 9, and 41% of the time, respectively. Most nodule characteristics were equally likely to be described at hospital A compared with hospital B.<\/jats:p><jats:p>\n          Conclusions\u2003NLP can automate extraction of critical information from thyroid US reports. However, ambiguous and incomplete reporting language hinders performance of NLP systems regardless of institutional setting. Standardized or synoptic thyroid US reports could improve NLP performance.<\/jats:p>","DOI":"10.1055\/s-0041-1740493","type":"journal-article","created":{"date-parts":[[2022,1,6]],"date-time":"2022-01-06T23:45:00Z","timestamp":1641512700000},"page":"011-018","source":"Crossref","is-referenced-by-count":1,"title":["Ambiguous and Incomplete: Natural Language Processing Reveals Problematic Reporting Styles in Thyroid Ultrasound Reports"],"prefix":"10.1055","volume":"61","author":[{"given":"Priya H.","family":"Dedhia","sequence":"additional","affiliation":[{"name":"Department of Surgery, Division of Surgical Oncology, Ohio State University Comprehensive Cancer Center and Ohio State University Wexner Medical Center, Columbus, Ohio, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kallie","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Surgery, Division of Endocrine Surgery, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yiqiang","family":"Song","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Medical Informatics, Department of Pediatrics, University of Wisconsin-Madison, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eric","family":"LaRose","sequence":"additional","affiliation":[{"name":"Center for Precision Medicine Research, Marshfield Clinic Research Institute, Marshfield Clinic Health System, Marshfield, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joseph R.","family":"Imbus","sequence":"additional","affiliation":[{"name":"Department of Surgery, Division of Endocrine Surgery, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peggy L.","family":"Peissig","sequence":"additional","affiliation":[{"name":"Center for Precision Medicine Research, Marshfield Clinic Research Institute, Marshfield Clinic Health System, Marshfield, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eneida A.","family":"Mendonca","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Medical Informatics, Department of Pediatrics, University of Wisconsin-Madison, Madison, Wisconsin, United States"},{"name":"Department of Pediatrics, Department of Biostatistics and Health Data Sciences, Indiana University, Indianapolis, Indiana, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David F.","family":"Schneider","sequence":"additional","affiliation":[{"name":"Department of Surgery, Division of Endocrine Surgery, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"194","published-online":{"date-parts":[[2022,1,6]]},"reference":[{"issue":"01","key":"ref1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1089\/thy.2015.0020","article-title":"2015 American Thyroid Association Management Guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer","volume":"26","author":"B R Haugen","year":"2016","journal-title":"Thyroid"},{"key":"ref2","doi-asserted-by":"crossref","first-page":"1272","DOI":"10.1016\/j.jacr.2015.07.011","article-title":"Thyroid ultrasound reporting lexicon: white paper of the ACR Thyroid Imaging, Reporting and Data System (TIRADS) Committee","volume":"12","author":"E G Grant","year":"2015","journal-title":"J Am Coll Radiol"},{"issue":"05","key":"ref3","doi-asserted-by":"crossref","first-page":"587","DOI":"10.1016\/j.jacr.2017.01.046","article-title":"ACR Thyroid Imaging, Reporting and Data System (TI-RADS): white paper of the ACR TI-RADS Committee","volume":"14","author":"F N Tessler","year":"2017","journal-title":"J Am Coll Radiol"},{"key":"ref4","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.jbi.2017.07.012","article-title":"Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review","volume":"73","author":"K Kreimeyer","year":"2017","journal-title":"J Biomed Inform"},{"key":"ref5","volume-title":"The Digitization of the World from Edge to Core; 2018","author":"D Reinsel"},{"issue":"01","key":"ref6","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1055\/s-0038-1638592","article-title":"Extracting information from textual documents in the electronic health record: a review of recent research","volume":"17","author":"S M Meystre","year":"2008","journal-title":"Yearb Med Inform"},{"issue":"01","key":"ref7","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1186\/s12911-019-0843-7","article-title":"Development and validation of a pragmatic natural language processing approach to identifying falls in older adults in the emergency department","volume":"19","author":"B W Patterson","year":"2019","journal-title":"BMC Med Inform Decis Mak"},{"key":"ref8","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1016\/j.jbi.2017.04.011","article-title":"Automated annotation and classification of BI-RADS assessment from radiology reports","volume":"69","author":"S M Castro","year":"2017","journal-title":"J Biomed Inform"},{"issue":"05","key":"ref9","doi-asserted-by":"crossref","first-page":"989","DOI":"10.1007\/s10278-013-9616-5","article-title":"Automated extraction of BI-RADS final assessment categories from radiology reports with natural language processing","volume":"26","author":"D A Sippo","year":"2013","journal-title":"J Digit Imaging"},{"key":"ref10","first-page":"460","article-title":"Automating tissue bank annotation from pathology reports\u2014comparison to a gold standard expert annotation set","volume":"2005","author":"K Liu","year":"2005","journal-title":"AMIA Annu Symp Proc"},{"key":"ref11","first-page":"565","article-title":"Facilitating cancer research using natural language processing of pathology reports","volume":"107","author":"H Xu","year":"2004","journal-title":"Stud Health Technol Inform"},{"key":"ref12","first-page":"237","article-title":"Extracting structured medication event information from discharge summaries","volume":"2008","author":"S Gold","year":"2008","journal-title":"AMIA Annu Symp Proc"},{"issue":"06","key":"ref13","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1038\/nrg3208","article-title":"Mining electronic health records: towards better research applications and clinical care","volume":"13","author":"P B Jensen","year":"2012","journal-title":"Nat Rev Genet"},{"issue":"07","key":"ref14","doi-asserted-by":"crossref","first-page":"4125","DOI":"10.1007\/s00330-020-06721-z","article-title":"Identification of patients with carotid stenosis using natural language processing","volume":"30","author":"X Wu","year":"2020","journal-title":"Eur Radiol"},{"key":"ref15","first-page":"190","article-title":"Evaluating the portability of an NLP system for processing echocardiograms: a retrospective, multi-site observational study","volume":"2019","author":"P Adekkanattu","year":"2020","journal-title":"AMIA Annu Symp Proc"},{"key":"ref16","doi-asserted-by":"crossref","first-page":"89043","DOI":"10.1109\/ACCESS.2019.2923221","article-title":"Automatically structuring on Chinese ultrasound report of cerebrovascular diseases via natural language processing","volume":"7","author":"P Chen","year":"2019","journal-title":"IEEE Access"},{"key":"ref17","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/j.ijmedinf.2017.02.011","article-title":"Creation of a simple natural language processing tool to support an imaging utilization quality dashboard","volume":"101","author":"J Swartz","year":"2017","journal-title":"Int J Med Inform"},{"key":"ref18","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1016\/j.jss.2020.07.015","article-title":"Thyroid ultrasound reports: will the thyroid imaging, reporting, and data system improve natural language processing capture of critical thyroid nodule features?","volume":"256","author":"K J Chen","year":"2020","journal-title":"J Surg Res"},{"issue":"05","key":"ref19","doi-asserted-by":"crossref","first-page":"913","DOI":"10.1136\/amiajnl-2011-000607","article-title":"Automatic classification of mammography reports by BI-RADS breast tissue composition class","volume":"19","author":"B Percha","year":"2012","journal-title":"J Am Med Inform Assoc"},{"issue":"12","key":"ref20","doi-asserted-by":"crossref","first-page":"e22982","DOI":"10.2196\/22982","article-title":"Extracting family history of patients from clinical narratives: exploring an end-to-end solution with deep learning models","volume":"8","author":"X Yang","year":"2020","journal-title":"JMIR Med Inform"},{"issue":"02","key":"ref21","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1148\/radiol.16142770","article-title":"Natural language processing in radiology: a systematic review","volume":"279","author":"E Pons","year":"2016","journal-title":"Radiology"},{"issue":"01","key":"ref22","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1148\/rg.2016150080","article-title":"Natural language processing technologies in radiology research and clinical applications","volume":"36","author":"T Cai","year":"2016","journal-title":"Radiographics"},{"issue":"01","key":"ref24","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1186\/s12911-019-0908-7","article-title":"A validated natural language processing algorithm for brain imaging phenotypes from radiology reports in UK electronic health records","volume":"19","author":"E Wheater","year":"2019","journal-title":"BMC Med Inform Decis Mak"},{"issue":"04","key":"ref25","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1055\/s-0039-1695791","article-title":"Interactive NLP in clinical care: identifying incidental findings in radiology reports","volume":"10","author":"G Trivedi","year":"2019","journal-title":"Appl Clin Inform"},{"issue":"01","key":"ref26","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1186\/1471-2105-15-266","article-title":"Natural language processing of radiology reports for the detection of thromboembolic diseases and clinically relevant incidental findings","volume":"15","author":"A-D Pham","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"ref27","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1200\/CCI.19.00060","article-title":"Automating clinical chart review: an open-source natural language processing pipeline developed on free-text radiology reports from patients with glioblastoma","volume":"4","author":"J T Senders","year":"2020","journal-title":"JCO Clin Cancer Inform"},{"issue":"04","key":"ref28","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1001\/jamaoto.2020.5233","article-title":"Association of the implementation of a standardized thyroid ultrasonography reporting program with documentation of nodule characteristics","volume":"147","author":"A F Hamour","year":"2021","journal-title":"JAMA Otolaryngol Head Neck Surg"},{"issue":"03","key":"ref29","doi-asserted-by":"crossref","first-page":"233","DOI":"10.4236\/ojrad.2016.63031","article-title":"Electronic synoptic reporting of thyroid nodules: potential for reduction in number of patients undergoing thyroid nodule biopsies","volume":"06","author":"J T Wang","year":"2016","journal-title":"Open J Radiol"},{"issue":"02","key":"ref30","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1503\/cjs.010316","article-title":"The utility of thyroid ultrasonography in the management of thyroid nodules","volume":"60","author":"G Gamme","year":"2017","journal-title":"Can J Surg"},{"issue":"05","key":"ref31","doi-asserted-by":"crossref","first-page":"931","DOI":"10.1016\/j.amjsurg.2017.03.030","article-title":"Completeness of ultrasound reporting impacts time to biopsy for benign and malignant thyroid nodules","volume":"213","author":"A Inman","year":"2017","journal-title":"Am J Surg"},{"issue":"01","key":"ref32","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1186\/s12880-019-0325-5","article-title":"Structured reporting of head and neck ultrasound examinations","volume":"19","author":"B P Ernst","year":"2019","journal-title":"BMC Med Imaging"},{"issue":"05","key":"ref33","doi-asserted-by":"crossref","first-page":"743","DOI":"10.1016\/j.jacr.2018.01.024","article-title":"Improved quality of thyroid ultrasound reports after implementation of the ACR thyroid imaging reporting and data system nodule lexicon and risk stratification system","volume":"15","author":"A S Griffin","year":"2018","journal-title":"J Am Coll Radiol"},{"issue":"05","key":"ref34","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1530\/EJE-12-0936","article-title":"Prospective evaluation of thyroid imaging reporting and data system on 4550 nodules with and without elastography","volume":"168","author":"G Russ","year":"2013","journal-title":"Eur J Endocrinol"}],"container-title":["Methods of Information in Medicine"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/s-0041-1740493.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,19]],"date-time":"2022-08-19T23:02:13Z","timestamp":1660950133000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/s-0041-1740493"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,6]]},"references-count":33,"journal-issue":{"issue":"01\/02","published-online":{"date-parts":[[2022,8,19]]},"published-print":{"date-parts":[[2022,5]]}},"URL":"https:\/\/doi.org\/10.1055\/s-0041-1740493","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["0026-1270","2511-705X"],"issn-type":[{"value":"0026-1270","type":"print"},{"value":"2511-705X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,6]]}}}