{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T09:50:16Z","timestamp":1780566616062,"version":"3.54.1"},"reference-count":34,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2025,5,16]],"date-time":"2025-05-16T00:00:00Z","timestamp":1747353600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/publicdomain\/zero\/1.0\/"}],"funder":[{"DOI":"10.13039\/100000030","name":"Centers for Disease Control and Prevention","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:sec id=\"sec019\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Post-COVID conditions (PCC) have proven difficult to diagnose. In this retrospective observational study, we aimed to characterize the level of variation in PCC diagnoses observed across clinicians from a number of methodological angles and to determine whether natural language classifiers trained on clinical notes can reconcile differences in diagnostic definitions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec020\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We used data from 519 primary care clinics around the United States who were in the American Family Cohort registry between October 1, 2021 (when the ICD-10 code for PCC was activated) and November 1, 2023. There were 6,116 patients with a diagnostic code for PCC (U09.9), and 5,020 with diagnostic codes for both PCC and COVID-19. We explored these data using 4 different outcomes: 1) Time between COVID-19 and PCC diagnostic codes; 2) Count of patients with PCC diagnostic codes per clinician; 3) Patient-specific probability of PCC diagnostic code based on patient and clinician characteristics; and 4) Performance of a natural language classifier trained on notes from 5,000 patients annotated by two physicians to indicate probable PCC.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec021\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Of patients with diagnostic codes for PCC and COVID-19, 61.3% were diagnosed with PCC less than 12 weeks after initial recorded COVID-19. Clinicians in the top 1% of diagnostic propensity accounted for more than a third of all PCC diagnoses (35.8%). Comparing LASSO logistic regressions predicting documentation of PCC diagnosis, a log-likelihood test showed significantly better fit when clinician and practice site indicators were included (p\u2009&lt;\u20090.0001). Inter-rater agreement between physician annotators on PCC diagnosis was moderate (Cohen\u2019s kappa: 0.60), and performance of the natural language classifiers was marginal (best AUC: 0.724, 95% credible interval: 0.555\u20130.878).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec022\">\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>We found evidence of substantial disagreement between clinicians on diagnostic criteria for PCC. The variation in diagnostic rates across clinicians points to the possibilities of under- and over-diagnosis for patients.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0324017","type":"journal-article","created":{"date-parts":[[2025,5,16]],"date-time":"2025-05-16T13:34:01Z","timestamp":1747402441000},"page":"e0324017","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":2,"title":["Heterogeneity of diagnosis and documentation of post-COVID conditions in primary care: A machine learning analysis"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8154-0276","authenticated-orcid":true,"given":"Nathaniel","family":"Hendrix","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rishi V.","family":"Parikh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Madeline","family":"Taskier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Grace","family":"Walter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ilia","family":"Rochlin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sharon","family":"Saydah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emilia H.","family":"Koumans","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oscar","family":"Rinc\u00f3n-Guevara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David H.","family":"Rehkopf","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert L.","family":"Phillips","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2025,5,16]]},"reference":[{"issue":"18","key":"pone.0324017.ref001","doi-asserted-by":"crossref","first-page":"1746","DOI":"10.1056\/NEJMsb2408466","article-title":"Long covid defined","volume":"391","author":"National Academies of Sciences, and Medicine Committee on Examining the Working Definition for Long Covid","year":"2024","journal-title":"N Engl J Med"},{"key":"pone.0324017.ref002","volume-title":"National research action plan on long COVID","author":"Department of Health and Human Services, Office of the Assistant Secretary for Health","year":"2022"},{"key":"pone.0324017.ref003","doi-asserted-by":"crossref","first-page":"104413","DOI":"10.1016\/j.ebiom.2022.104413","article-title":"Generalisable long COVID subtypes: findings from the NIH N3C and RECOVER programmes","volume":"87","author":"JT Reese","year":"2023","journal-title":"EBioMedicine"},{"issue":"7","key":"pone.0324017.ref004","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2022.24359","article-title":"Rates and factors associated with documentation of diagnostic codes for long COVID in the national veterans affairs health care system","volume":"5","author":"GN Ioannou","year":"2022","journal-title":"JAMA Netw Open"},{"key":"pone.0324017.ref005","article-title":"Characterizing the use of the ICD-10 code for long COVID in 3 US healthcare systems","author":"HG Zhang","year":"2023","journal-title":"medRxiv"},{"issue":"712","key":"pone.0324017.ref006","doi-asserted-by":"crossref","DOI":"10.3399\/BJGP.2021.0301","article-title":"Clinical coding of long COVID in English primary care: a federated analysis of 58 million patient records in situ using OpenSAFELY","volume":"71","author":"AJ Walker","year":"2021","journal-title":"Br J Gen Pract"},{"key":"pone.0324017.ref007","first-page":"1","volume-title":"Using natural language processing on free-text clinical notes to identify patients with long-term COVID effects. Proceedings of the 13th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics","author":"Y Zhu","year":"2022"},{"issue":"7","key":"pone.0324017.ref008","article-title":"Identifying who has long COVID in the USA: a machine learning approach using N3C data","volume":"4","author":"ER Pfaff","year":"2022","journal-title":"Lancet Digit Health"},{"issue":"12","key":"pone.0324017.ref009","doi-asserted-by":"crossref","DOI":"10.1093\/ofid\/ofac640","article-title":"An elastic net regression model for identifying long COVID patients using health administrative data: a population-based study","volume":"9","author":"M Binka","year":"2022","journal-title":"Open Forum Infect Dis"},{"key":"pone.0324017.ref010","volume-title":"The American Family Cohort (v12.5)","author":"A Vala","year":"2023"},{"key":"pone.0324017.ref011","first-page":"667","volume-title":"How Redundant are Redundant Encodings? Blindness in the Wild and Racial Disparity when Race is Unobserved. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency","author":"L Cheng","year":"2023"},{"key":"pone.0324017.ref012","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: synthetic minority over-sampling technique","volume":"16","author":"NV Chawla","year":"2002","journal-title":"jair"},{"issue":"10","key":"pone.0324017.ref013","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2022.38804","article-title":"Prevalence and correlates of long COVID symptoms among US adults","volume":"5","author":"RH Perlis","year":"2022","journal-title":"JAMA Netw Open"},{"key":"pone.0324017.ref014","doi-asserted-by":"crossref","first-page":"104736","DOI":"10.1016\/j.ijmedinf.2022.104736","article-title":"A hybrid model to identify fall occurrence from electronic health records","volume":"162","author":"S Fu","year":"2022","journal-title":"Int J Med Inform"},{"key":"pone.0324017.ref015","article-title":"Post-COVID conditions.","volume-title":"Centers for disease control and prevention [Internet]","author":"Centers for Disease Control and Prevention","year":"16"},{"key":"pone.0324017.ref016","first-page":"785","volume-title":"XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"T Chen","year":"2016"},{"issue":"2","key":"pone.0324017.ref017","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.jointm.2021.09.002","article-title":"Machine learning using the extreme gradient boosting (XGBoost) algorithm predicts 5-day delta of SOFA score at ICU admission in COVID-19 patients","volume":"1","author":"J Montomoli","year":"2021","journal-title":"J Intensive Med"},{"key":"pone.0324017.ref018","doi-asserted-by":"crossref","first-page":"81","DOI":"10.18653\/v1\/2022.louhi-1.10","article-title":"BioSimCSE: biomedical sentence embeddings using contrastive learning.","volume-title":"Proceedings of the 13th international workshop on health text mining and information analysis (LOUHI)","author":"K Kanakarajan","year":"2022"},{"key":"pone.0324017.ref019","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.inffus.2021.11.011","article-title":"Tabular data: deep learning is not all you need","volume":"81","author":"R Shwartz-Ziv","year":"2022","journal-title":"Inf Fusion"},{"key":"pone.0324017.ref020","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.jbi.2017.07.006","article-title":"Recurrent neural networks for classifying relations in clinical notes","volume":"72","author":"Y Luo","year":"2017","journal-title":"J Biomed Inform"},{"key":"pone.0324017.ref021","volume-title":"Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks. arXiv","author":"RC Staudemeyer","year":"2019"},{"key":"pone.0324017.ref022","volume-title":"Clinical-longformer and clinical-bigbird: transformers for long clinical sequences. arXiv","author":"Y Li","year":"2022"},{"issue":"5","key":"pone.0324017.ref023","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1109\/TAI.2021.3114390","article-title":"Toward Text Data Augmentation for Sentiment Analysis","volume":"3","author":"HQ Abonizio","year":"2022","journal-title":"IEEE Trans Artif Intell"},{"key":"pone.0324017.ref024","first-page":"2623","volume-title":"Optuna: a next-generation hyperparameter optimization framework. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining","author":"T Akiba","year":"2019"},{"key":"pone.0324017.ref025","first-page":"92","volume-title":"Statsmodels: econometric and statistical modeling with python","author":"S Seabold","year":"2010"},{"key":"pone.0324017.ref026","first-page":"2825","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"F Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"pone.0324017.ref027","volume-title":"HuggingFace\u2019s transformers: state-of-the-art natural language processing. arXiv","author":"T Wolf","year":"2020"},{"issue":"20","key":"pone.0324017.ref028","doi-asserted-by":"crossref","first-page":"1727","DOI":"10.1001\/jama.2023.6604","article-title":"How primary care physicians can recognize and treat long COVID","volume":"329","author":"EW-Y Landhuis","year":"2023","journal-title":"JAMA"},{"key":"pone.0324017.ref029","author":"National Institute for Health and Care Research"},{"key":"pone.0324017.ref030","article-title":"NICE guideline on long covid","volume":"371","author":"M Sivan","year":"2020","journal-title":"BMJ"},{"issue":"5","key":"pone.0324017.ref031","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1016\/j.medj.2021.03.003","article-title":"Defining long COVID: going back to the start","volume":"2","author":"NA Alwan","year":"2021","journal-title":"Med"},{"key":"pone.0324017.ref032","doi-asserted-by":"crossref","first-page":"110164","DOI":"10.1016\/j.ejrad.2022.110164","article-title":"Long-COVID diagnosis: From diagnostic to advanced AI-driven models","volume":"148","author":"R Cau","year":"2022","journal-title":"Eur J Radiol"},{"issue":"3","key":"pone.0324017.ref033","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1038\/s41579-022-00846-2","article-title":"Long COVID: major findings, mechanisms and recommendations","volume":"21","author":"HE Davis","year":"2023","journal-title":"Nat Rev Microbiol"},{"issue":"5","key":"pone.0324017.ref034","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1097\/INF.0000000000003502","article-title":"The challenge of studying long COVID: an updated review","volume":"41","author":"P Zimmermann","year":"2022","journal-title":"Pediatr Infect Dis J"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0324017","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,16]],"date-time":"2025-05-16T13:34:09Z","timestamp":1747402449000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0324017"}},"subtitle":[],"editor":[{"given":"Jiawen","family":"Deng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2025,5,16]]},"references-count":34,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,5,16]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0324017","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.02.07.24302451","asserted-by":"object"}]},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,16]]}}}