{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T21:23:05Z","timestamp":1790716985253,"version":"4.1.0"},"reference-count":13,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,12,1]],"date-time":"2019-12-01T00:00:00Z","timestamp":1575158400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2019,12,30]],"date-time":"2019-12-30T00:00:00Z","timestamp":1577664000000},"content-version":"vor","delay-in-days":29,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000071","name":"National Institute of Child Health and Human Development","doi-asserted-by":"publisher","award":["NICHD 5K12HD043483-12"],"award-info":[{"award-number":["NICHD 5K12HD043483-12"]}],"id":[{"id":"10.13039\/100000071","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000069","name":"National Institute of Arthritis and Musculoskeletal and Skin Diseases","doi-asserted-by":"publisher","award":["1 K08AR072757-01"],"award-info":[{"award-number":["1 K08AR072757-01"]}],"id":[{"id":"10.13039\/100000069","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000097","name":"National Center for Research Resources","doi-asserted-by":"publisher","award":["UL1 RR024975"],"award-info":[{"award-number":["UL1 RR024975"]}],"id":[{"id":"10.13039\/100000097","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","award":["ULTR000445"],"award-info":[{"award-number":["ULTR000445"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Arthritis Res Ther"],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n<jats:title>Background<\/jats:title>\n<jats:p>Systemic sclerosis (SSc) is a rare disease with studies limited by small sample sizes. Electronic health records (EHRs) represent a powerful tool to study patients with rare diseases such as SSc, but validated methods are needed. We developed and validated EHR-based algorithms that incorporate billing codes and clinical data to identify SSc patients in the EHR.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Methods<\/jats:title>\n<jats:p>We used a de-identified EHR with over 3 million subjects and identified 1899 potential SSc subjects with at least 1 count of the SSc ICD-9 (710.1) or ICD-10-CM (M34*) codes. We randomly selected 200 as a training set for chart review. A subject was a case if diagnosed with SSc by a rheumatologist, dermatologist, or pulmonologist. We selected the following algorithm components based on clinical knowledge and available data: SSc ICD-9 and ICD-10-CM codes, positive antinuclear antibody (ANA) (titer \u2265\u20091:80), and a keyword of Raynaud\u2019s phenomenon (RP). We performed both rule-based and machine learning techniques for algorithm development. Positive predictive values (PPVs), sensitivities, and <jats:italic>F<\/jats:italic>-scores (which account for PPVs and sensitivities) were calculated for the algorithms.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>PPVs were low for algorithms using only 1 count of the SSc ICD-9 code. As code counts increased, the PPVs increased. PPVs were higher for algorithms using ICD-10-CM codes versus the ICD-9 code. Adding a positive ANA and RP keyword increased the PPVs of algorithms only using ICD billing codes. Algorithms using \u2265\u20093 or \u2265\u20094 counts of the SSc ICD-9 or ICD-10-CM codes and ANA positivity had the highest PPV at 100% but a low sensitivity at 50%. The algorithm with the highest <jats:italic>F<\/jats:italic>-score of 91% was \u2265\u20094 counts of the ICD-9 or ICD-10-CM codes with an internally validated PPV of 90%. A machine learning method using random forests yielded an algorithm with a PPV of 84%, sensitivity of 92%, and <jats:italic>F<\/jats:italic>-score of 88%. The most important feature was RP keyword.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Conclusions<\/jats:title>\n<jats:p>Algorithms using only ICD-9 codes did not perform well to identify SSc patients. The highest performing algorithms incorporated clinical data with billing codes. EHR-based algorithms can identify SSc patients across a healthcare system, enabling researchers to examine important outcomes.<\/jats:p>\n<\/jats:sec>","DOI":"10.1186\/s13075-019-2092-7","type":"journal-article","created":{"date-parts":[[2019,12,30]],"date-time":"2019-12-30T20:02:17Z","timestamp":1577736137000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":50,"title":["Rule-based and machine learning algorithms identify patients with systemic sclerosis accurately in the electronic health record"],"prefix":"10.1186","volume":"21","author":[{"given":"Lia","family":"Jamian","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lee","family":"Wheless","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leslie J.","family":"Crofford","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9573-4335","authenticated-orcid":false,"given":"April","family":"Barnado","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2019,12,30]]},"reference":[{"key":"2092_CR1","doi-asserted-by":"publisher","first-page":"362","DOI":"10.1038\/clpt.2008.89","volume":"84","author":"DM Roden","year":"2008","unstructured":"Roden DM, Pulley JM, Basford MA, Bernard GR, Clayton EW, Balser JR, et al. Development of a large-scale de-identified DNA biobank to enable personalized medicine. Clin Pharmacol Ther. 2008;84:362\u20139.","journal-title":"Clin Pharmacol Ther"},{"key":"2092_CR2","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1016\/j.compbiomed.2014.07.022","volume":"53","author":"D Redd","year":"2014","unstructured":"Redd D, Frech TM, Murtaugh MA, Rhiannon J, Zeng QT. Informatics can identify systemic sclerosis (SSc) patients at risk for scleroderma renal crisis. Comput Biol Med. 2014;53:203\u20135.","journal-title":"Comput Biol Med"},{"key":"2092_CR3","doi-asserted-by":"publisher","first-page":"253","DOI":"10.3109\/03009742.2015.1008038","volume":"44","author":"A Valenzuela","year":"2015","unstructured":"Valenzuela A, Yaqub A, Fiorentino D, Krishnan E, Chung L. Validation of the ICD-9-CM code for systemic sclerosis using updated ACR\/EULAR classification criteria. Scand J Rheumatol. 2015;44:253\u20135.","journal-title":"Scand J Rheumatol"},{"key":"2092_CR4","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1002\/acr.22989","volume":"69","author":"A Barnado","year":"2017","unstructured":"Barnado A, Casey C, Carroll RJ, Wheless L, Denny JC, Crofford LJ. Developing electronic health record algorithms that accurately identify patients with systemic lupus erythematosus. Arthritis Care Res (Hoboken). 2017;69:687\u201393.","journal-title":"Arthritis Care Res (Hoboken)"},{"issue":"Suppl 10","key":"2092_CR5","doi-asserted-by":"publisher","first-page":"K62","DOI":"10.1016\/j.vaccine.2013.06.104","volume":"31","author":"KG Moores","year":"2013","unstructured":"Moores KG, Sathe NA. A systematic review of validated methods for identifying systemic lupus erythematosus (SLE) using administrative or claims data. Vaccine. 2013;31(Suppl 10):K62\u201373.","journal-title":"Vaccine"},{"key":"2092_CR6","doi-asserted-by":"publisher","first-page":"648","DOI":"10.1097\/GIM.0b013e3181efe2df","volume":"12","author":"L Dumitrescu","year":"2010","unstructured":"Dumitrescu L, Ritchie MD, Brown-Gentry K, Pulley JM, Basford M, Denny JC, et al. Assessing the accuracy of observer-reported ancestry in a biorepository linked to electronic medical records. Genet Med. 2010;12:648\u201350.","journal-title":"Genet Med"},{"key":"2092_CR7","volume-title":"Classification and regression trees. The Wadsworth statistics\/probability series","author":"L Breiman","year":"1983","unstructured":"Breiman L, Friedman J, Olshen RA, Stone CJ. Classification and regression trees. The Wadsworth statistics\/probability series. 1st ed. Belmont: Wadsworth International Group; 1983.","edition":"1"},{"key":"2092_CR8","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L. Random Forests. Machine Learning. 2001;45:5\u201332.","journal-title":"Machine Learning"},{"key":"2092_CR9","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1016\/0197-2456(90)90005-M","volume":"11","author":"WD Dupont","year":"1990","unstructured":"Dupont WD, Plummer WD Jr. Power and sample size calculations. A review and computer program. Control Clin Trials. 1990;11:116\u201328.","journal-title":"Control Clin Trials"},{"key":"2092_CR10","doi-asserted-by":"publisher","first-page":"2737","DOI":"10.1002\/art.38098","volume":"65","author":"F van den Hoogen","year":"2013","unstructured":"van den Hoogen F, et al. 2013 classification criteria for systemic sclerosis: an American College of Rheumatology\/European League against Rheumatism collaborative initiative. Arthritis Rheum. 2013;65:2737\u201347.","journal-title":"Arthritis Rheum"},{"key":"2092_CR11","doi-asserted-by":"publisher","first-page":"85","DOI":"10.3899\/jrheum.171186","volume":"46","author":"SM Gordon","year":"2019","unstructured":"Gordon SM, et al. Risk factors for future scleroderma renal crisis at systemic sclerosis diagnosis. J Rheumatol. 2019;46:85\u201392.","journal-title":"J Rheumatol"},{"key":"2092_CR12","doi-asserted-by":"publisher","first-page":"754","DOI":"10.1136\/ard.2006.062901","volume":"66","author":"UA Walker","year":"2007","unstructured":"Walker UA, et al. Clinical risk assessment of organ manifestations in systemic sclerosis: a report from the EULAR Scleroderma Trials and Research group database. Ann Rheum Dis. 2007;66:754\u201363.","journal-title":"Ann Rheum Dis"},{"key":"2092_CR13","doi-asserted-by":"publisher","first-page":"e162","DOI":"10.1136\/amiajnl-2011-000583","volume":"19","author":"RJ Carroll","year":"2012","unstructured":"Carroll RJ, et al. Portability of an algorithm to identify rheumatoid arthritis in electronic health records. J Am Med Inform Assoc. 2012;19:e162\u20139.","journal-title":"J Am Med Inform Assoc"}],"container-title":["Arthritis Research &amp; Therapy"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13075-019-2092-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s13075-019-2092-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13075-019-2092-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,29]],"date-time":"2020-12-29T00:33:58Z","timestamp":1609202038000},"score":1,"resource":{"primary":{"URL":"https:\/\/arthritis-research.biomedcentral.com\/articles\/10.1186\/s13075-019-2092-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12]]},"references-count":13,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["2092"],"URL":"https:\/\/doi.org\/10.1186\/s13075-019-2092-7","relation":{},"ISSN":["1478-6362"],"issn-type":[{"value":"1478-6362","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12]]},"assertion":[{"value":"17 August 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 December 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 December 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Approval was obtained from the Institutional Review Board of Vanderbilt University Medical Center. As de-identified data was used, research was classified as non-human subject research, and informed consent was waived.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"305"}}