{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T10:38:13Z","timestamp":1791283093098,"version":"4.1.0"},"reference-count":38,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2025,1,16]],"date-time":"2025-01-16T00:00:00Z","timestamp":1736985600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This paper introduces a novel framework for evaluating phenotype algorithms (PAs) using the open-source tool, Cohort Diagnostics.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>The method is based on several diagnostic criteria to evaluate a patient cohort returned by a PA. Diagnostics include estimates of incidence rate, index date entry code breakdown, and prevalence of all observed clinical events prior to, on, and after index date. We test our framework by evaluating one PA for systemic lupus erythematosus (SLE) and two PAs for Alzheimer\u2019s disease (AD) across 10 different observational data sources.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>By utilizing CohortDiagnostics, we found that the population-level characteristics of individuals in the cohort of SLE closely matched the disease\u2019s anticipated clinical profile. Specifically, the incidence rate of SLE was consistently higher in occurrence among females. Moreover, expected clinical events like laboratory tests, treatments, and repeated diagnoses were also observed. For AD, although one PA identified considerably fewer patients, absence of notable differences in clinical characteristics between the two cohorts suggested similar specificity.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec004\">\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>We provide a practical and data-driven approach to evaluate PAs, using two clinical diseases as examples, across a network of OMOP data sources. Cohort Diagnostics can ensure the subjects identified by a specific PA align with those intended for inclusion in a research study.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec005\">\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Diagnostics based on large-scale population-level characterization can offer insights into the misclassification errors of PAs.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0310634","type":"journal-article","created":{"date-parts":[[2025,1,16]],"date-time":"2025-01-16T13:37:41Z","timestamp":1737034661000},"page":"e0310634","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":13,"title":["CohortDiagnostics: Phenotype evaluation across a network of observational data sources using population-level characterization"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4949-7236","authenticated-orcid":true,"given":"Gowtham A.","family":"Rao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6976-2594","authenticated-orcid":true,"given":"Azza","family":"Shoaibi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rupa","family":"Makadia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2682-2187","authenticated-orcid":true,"given":"Jill","family":"Hardin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joel","family":"Swerdel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0755-5191","authenticated-orcid":true,"given":"James","family":"Weaver","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erica A.","family":"Voss","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mitchell M.","family":"Conover","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen","family":"Fortin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8630-5347","authenticated-orcid":true,"given":"Anthony G.","family":"Sena","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chris","family":"Knoll","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nigel","family":"Hughes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2294-3459","authenticated-orcid":true,"given":"James P.","family":"Gilbert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clair","family":"Blacketer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alan","family":"Andryc","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frank","family":"DeFalco","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anthony","family":"Molinaro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2970-0778","authenticated-orcid":true,"given":"Jenna","family":"Reps","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0817-5361","authenticated-orcid":true,"given":"Martijn J.","family":"Schuemie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick B.","family":"Ryan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2025,1,16]]},"reference":[{"issue":"e2","key":"pone.0310634.ref001","doi-asserted-by":"crossref","first-page":"e243","DOI":"10.1136\/amiajnl-2013-001930","article-title":"A collaborative approach to developing an electronic health record phenotyping algorithm for drug-induced liver injury","volume":"20","author":"CL Overby","year":"2013","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"3","key":"pone.0310634.ref002","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1093\/jamia\/ocx110","article-title":"High-fidelity phenotyping: richness and freedom from bias","volume":"25","author":"G Hripcsak","year":"2017","journal-title":"Journal of the American Medical Informatics Association"},{"key":"pone.0310634.ref003","unstructured":"Weaver J, Potvien A, Swerdel J, Voss EA, Hester L, Shoaibi A, et al., editors. Best Practices for Creating the Standardized Content of an Entry in the OHDSI Phenotype Library. 5th OHDSI Annual Symposium; 2019."},{"issue":"9","key":"pone.0310634.ref004","doi-asserted-by":"crossref","first-page":"giab059","DOI":"10.1093\/gigascience\/giab059","article-title":"Desiderata for the development of next-generation electronic health record phenotype libraries","volume":"10","author":"M Chapman","year":"2021","journal-title":"GigaScience"},{"key":"pone.0310634.ref005","author":"J Kuha","journal-title":"Misclassification Error. Encyclopedia of Biostatistics2005"},{"issue":"6","key":"pone.0310634.ref006","doi-asserted-by":"crossref","first-page":"1251","DOI":"10.1093\/jamia\/ocv070","article-title":"Review and evaluation of electronic health records-driven phenotype algorithm authoring tools for clinical and translational research","volume":"22","author":"J Xu","year":"2015","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"6","key":"pone.0310634.ref007","doi-asserted-by":"crossref","first-page":"1046","DOI":"10.1093\/jamia\/ocv202","article-title":"PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability","volume":"23","author":"JC Kirby","year":"2016","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"2","key":"pone.0310634.ref008","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1002\/pds.3374","article-title":"Cancer recording and mortality in the General Practice Research Database and linked cancer registries","volume":"22","author":"R Boggon","year":"2013","journal-title":"Pharmacoepidemiology and Drug Safety"},{"key":"pone.0310634.ref009","doi-asserted-by":"crossref","first-page":"12","DOI":"10.3352\/jeehp.2013.10.12","article-title":"The retrospective chart review: important methodological considerations","volume":"10","author":"M Vassar","year":"2013","journal-title":"J Educ Eval Health Prof"},{"issue":"2","key":"pone.0310634.ref010","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.1553-2712.2004.tb01433.x","article-title":"Advanced statistics: understanding medical record review (MRR) studies","volume":"11","author":"A Worster","year":"2004","journal-title":"Acad Emerg Med"},{"issue":"12","key":"pone.0310634.ref011","doi-asserted-by":"crossref","first-page":"1545","DOI":"10.1093\/jamia\/ocz105","article-title":"UK phenomics platform for developing and validating electronic health record phenotypes: CALIBER","volume":"26","author":"S Denaxas","year":"2019","journal-title":"Journal of the American Medical Informatics Association"},{"key":"pone.0310634.ref012","doi-asserted-by":"crossref","first-page":"103258","DOI":"10.1016\/j.jbi.2019.103258","article-title":"PheValuator: development and evaluation of a phenotype algorithm evaluator","volume":"97","author":"JN Swerdel","year":"2019","journal-title":"Journal of biomedical informatics"},{"issue":"e1","key":"pone.0310634.ref013","doi-asserted-by":"crossref","first-page":"e147","DOI":"10.1136\/amiajnl-2012-000896","article-title":"Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network","volume":"20","author":"KM Newton","year":"2013","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"1","key":"pone.0310634.ref014","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1136\/amiajnl-2011-000376","article-title":"Validation of a common data model for active safety surveillance research","volume":"19","author":"JM Overhage","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"pone.0310634.ref015","article-title":"PHOEBE 2.0: selecting the right concept sets for the right patients using lexical, semantic, and data-driven recommendations","author":"A Ostropolets"},{"key":"pone.0310634.ref016","article-title":"editor Phenotyping in distributed data networks: selecting the right codes for the right patients","author":"RP Ostropolets A","year":"2022","journal-title":"AMIA Annual Symposium Proceedings"},{"issue":"2","key":"pone.0310634.ref017","doi-asserted-by":"crossref","first-page":"e0281929","DOI":"10.1371\/journal.pone.0281929","article-title":"Using a data-driven approach for the development and evaluation of phenotype algorithms for systemic lupus erythematosus","volume":"18","author":"JN Swerdel","year":"2023","journal-title":"PLOS ONE"},{"key":"pone.0310634.ref018","volume-title":"Harrison\u2019s Manual of Medicine:","author":"JL Jameson","year":"2020"},{"issue":"4","key":"pone.0310634.ref019","doi-asserted-by":"crossref","first-page":"807","DOI":"10.3233\/JAD-2009-1099","article-title":"The accuracy of Medicare claims as an epidemiological tool: the case of dementia revisited","volume":"17","author":"DH Taylor","year":"2009","journal-title":"J Alzheimers Dis"},{"issue":"4","key":"pone.0310634.ref020","doi-asserted-by":"crossref","first-page":"700","DOI":"10.1111\/epi.12045","article-title":"Seizures in patients with Alzheimer\u2019s disease or vascular dementia: a population-based nested case-control analysis","volume":"54","author":"P Imfeld","year":"2013","journal-title":"Epilepsia"},{"issue":"3","key":"pone.0310634.ref021","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1093\/jamia\/ocu023","article-title":"Feasibility and utility of applications of the common data model to multiple, disparate observational health databases","volume":"22","author":"EA Voss","year":"2015","journal-title":"Journal of the American Medical Informatics Association"},{"key":"pone.0310634.ref022","first-page":"574","article-title":"Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers","volume":"216","author":"G Hripcsak","year":"2015","journal-title":"Stud Health Technol Inform"},{"key":"pone.0310634.ref023","author":"C. Blacketer","year":"2023","journal-title":"ETL-LambdaBuilder"},{"issue":"10","key":"pone.0310634.ref024","doi-asserted-by":"crossref","first-page":"2251","DOI":"10.1093\/jamia\/ocab132","article-title":"Increasing trust in real-world evidence through evaluation of observational data quality","volume":"28","author":"C Blacketer","year":"2021","journal-title":"Journal of the American Medical Informatics Association"},{"key":"pone.0310634.ref025","author":"Team RC","year":"2013","journal-title":"R: A language and environment for statistical computing"},{"key":"pone.0310634.ref026","author":"P Samarati","year":"1998","journal-title":"Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression"},{"key":"pone.0310634.ref027","author":"G. Rao","year":"2023","journal-title":"Cohort Diagnostics: An R package"},{"key":"pone.0310634.ref028","author":"G. Rao","year":"2023","journal-title":"Evaluation of Systemic Lupus Erythematous and Alzheimers Disease using Cohort Diagnostics"},{"key":"pone.0310634.ref029","article-title":"Characterizing the incidence of adverse events of special interest for COVID-19 vaccines across eight countries: a multinational network cohort study","author":"X Li","year":"2021","journal-title":"Medrxiv"},{"key":"pone.0310634.ref030","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1385\/CRIAI:25:1:3","article-title":"The epidemiology of systemic lupus erythematosus","volume":"25","author":"S Jim\u00e9nez","year":"2003","journal-title":"Clinical reviews in allergy & immunology"},{"issue":"6","key":"pone.0310634.ref031","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1093\/gerona\/glab373","article-title":"Validation of Claims Algorithms to Identify Alzheimer\u2019s Disease and Related Dementias","volume":"77","author":"EP McCarthy","year":"2022","journal-title":"J Gerontol A Biol Sci Med Sci"},{"issue":"6","key":"pone.0310634.ref032","doi-asserted-by":"crossref","first-page":"685","DOI":"10.1007\/s40264-022-01187-y","article-title":"Phenotype algorithms for the identification and characterization of vaccine-induced thrombotic thrombocytopenia in real world data: a multinational network cohort study","volume":"45","author":"A Shoaibi","year":"2022","journal-title":"Drug Safety"},{"key":"pone.0310634.ref033","doi-asserted-by":"crossref","first-page":"369","DOI":"10.2147\/CLEP.S323292","article-title":"Unraveling COVID-19: a large-scale characterization of 4.5 million COVID-19 cases using CHARYBDIS","author":"K Kostka","year":"2022","journal-title":"Clinical Epidemiology"},{"key":"pone.0310634.ref034","article-title":"Phenotype Development and Evaluation of Heart Failure: A Case Study in using Patient Level Prediction to Improve Phenotype Validity","author":"PM Desai"},{"key":"pone.0310634.ref035","article-title":"Epidemiology of vasomotor symptoms (VMS) in menopausal women (EpiVaSym): a multi-country, large-scale OHDSI network analytic study","author":"R Herrera"},{"key":"pone.0310634.ref036","article-title":"Standardised and reproducible phenotyping using distributed analytics and tools in the Data Analysis and Real World Interrogation Network (DARWIN EU\u00ae)","author":"DP Alhambra","year":"2024","journal-title":"Authorea Preprints"},{"issue":"5","key":"pone.0310634.ref037","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1093\/jamia\/ocad009","article-title":"Reproducible variability: assessing investigator discordance across 9 research teams attempting to reproduce the same observational study","volume":"30","author":"A Ostropolets","year":"2023","journal-title":"Journal of the American Medical Informatics Association"},{"key":"pone.0310634.ref038","doi-asserted-by":"crossref","first-page":"104177","DOI":"10.1016\/j.jbi.2022.104177","article-title":"PheValuator 2.0: Methodological improvements for the PheValuator approach to semi-automated phenotype algorithm evaluation","volume":"135","author":"JN Swerdel","year":"2022","journal-title":"Journal of Biomedical Informatics"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0310634","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,16]],"date-time":"2025-01-16T13:37:52Z","timestamp":1737034672000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0310634"}},"subtitle":[],"editor":[{"given":"Ernesto","family":"Iadanza","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2025,1,16]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1,16]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0310634","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.06.28.23291982","asserted-by":"object"}]},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,16]]}}}