{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T11:29:49Z","timestamp":1769599789718,"version":"3.49.0"},"reference-count":50,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2025,11,10]],"date-time":"2025-11-10T00:00:00Z","timestamp":1762732800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000051","name":"National Human Genome Research Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000051","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01HG011902"],"award-info":[{"award-number":["R01HG011902"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R35HG010714"],"award-info":[{"award-number":["R35HG010714"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,2,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To build natural language processing (NLP) strategies to characterize measures of genetic counseling (GC) efficiency and classify measures according to phase of GC (pre- or post-genetic testing).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>This study selected and annotated 800 GC notes from 7 clinical specialties in a large academic medical center for NLP model development and validation. The NLP approaches extracted GC efficiency measures, including direct and indirect time and GC phase. The models were then applied to 24\u00a0102 GC notes collected from January 2016 through December 2023.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>NLP approaches performed well (F1 scores of 0.95 and 0.90 for direct time in GC and GC phase classification, respectively). Our findings showed median direct time in GC of 50 minutes, with significant differences in direct time distributions observed across clinical specialties, time periods (2016-2019 or 2020-2023), delivery modes (in person or telehealth), and GC phase.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>As referrals to GC increase, there is increasing pressure to improve efficiency. Our NLP strategy was used to generate and summarize real-world evidence of GC time for 7 clinical specialties. These approaches enable future research on the impact of interventions intended to improve GC efficiency.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>This work demonstrated the practical value of NLP to provide a useful and scalable strategy to generate real world evidence of GC efficiency. Principles presented in this work may also be valuable for health services research in other practice areas.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf190","type":"journal-article","created":{"date-parts":[[2025,11,10]],"date-time":"2025-11-10T11:43:17Z","timestamp":1762774997000},"page":"295-303","source":"Crossref","is-referenced-by-count":1,"title":["Assessing genetic counseling efficiency with natural language processing"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7689-646X","authenticated-orcid":false,"given":"Michelle H","family":"Nguyen","sequence":"first","affiliation":[{"name":"Institute for Computational Medicine, Johns Hopkins Whiting School of Engineering , Baltimore, MD 21218,","place":["United States"]},{"name":"Department of Biomedical Engineering, The Johns Hopkins University School of Medicine , Baltimore, MD 21205,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carolyn D","family":"Applegate","sequence":"additional","affiliation":[{"name":"Department of Genetic Medicine, The Johns Hopkins University School of Medicine , Baltimore, MD 21205,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7870-8584","authenticated-orcid":false,"given":"Brittney","family":"Murray","sequence":"additional","affiliation":[{"name":"Division of Cardiology, Department of Medicine, The Johns Hopkins University School of Medicine , Baltimore, MD 21205,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ayah","family":"Zirikly","sequence":"additional","affiliation":[{"name":"Center for Language and Speech Processing, Johns Hopkins Whiting School of Engineering , Baltimore, MD 21218,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3465-294X","authenticated-orcid":false,"given":"Crystal","family":"Tichnell","sequence":"additional","affiliation":[{"name":"Division of Cardiology, Department of Medicine, The Johns Hopkins University School of Medicine , Baltimore, MD 21205,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Catherine","family":"Gordon","sequence":"additional","affiliation":[{"name":"Division of Cardiology, Department of Medicine, The Johns Hopkins University School of Medicine , Baltimore, MD 21205,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lisa R","family":"Yanek","sequence":"additional","affiliation":[{"name":"Department of Medicine, Division of General Internal Medicine, The Johns Hopkins University School of Medicine , Baltimore, MD 21205,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0854-0418","authenticated-orcid":false,"given":"Cynthia A","family":"James","sequence":"additional","affiliation":[{"name":"Division of Cardiology, Department of Medicine, The Johns Hopkins University School of Medicine , Baltimore, MD 21205,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9302-5968","authenticated-orcid":false,"given":"Casey Overby","family":"Taylor","sequence":"additional","affiliation":[{"name":"Institute for Computational Medicine, Johns Hopkins Whiting School of Engineering , Baltimore, MD 21218,","place":["United States"]},{"name":"Department of Biomedical Engineering, The Johns Hopkins University School of Medicine , Baltimore, MD 21205,","place":["United States"]},{"name":"Department of Medicine, Division of General Internal Medicine, The Johns Hopkins University School of Medicine , Baltimore, MD 21205,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2025,11,10]]},"reference":[{"key":"2026012716173413700_ocaf190-B1","doi-asserted-by":"publisher","first-page":"1437","DOI":"10.1038\/s41436-020-0825-2","article-title":"The composition and capacity of the clinical genetics workforce in high-income countries: a scoping review","volume":"22","author":"Dragojlovic","year":"2020","journal-title":"Genet Med"},{"key":"2026012716173413700_ocaf190-B2","first-page":"158","article-title":"Does supply equal demand? The workforce of direct patient care genetic counselors in Wisconsin","volume":"119","author":"Dawson","year":"2020","journal-title":"WMJ Off Publ State Med Soc Wis"},{"key":"2026012716173413700_ocaf190-B3","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1038\/s41436-019-0628-5","article-title":"Genetics workforce: distribution of genetics services and challenges to health care in California","volume":"22","author":"Penon-Portmann","year":"2020","journal-title":"Genet Med"},{"key":"2026012716173413700_ocaf190-B4","doi-asserted-by":"publisher","first-page":"1458","DOI":"10.1038\/s41436-021-01162-5","article-title":"The 2019 US medical genetics workforce: a focus on clinical genetics","volume":"23","author":"Jenkins","year":"2021","journal-title":"Genet Med"},{"key":"2026012716173413700_ocaf190-B5","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1007\/s10897-017-0158-8","article-title":"Projecting the supply and demand for certified genetic counselors: a Workforce Study","volume":"27","author":"Hoskovec","year":"2018","journal-title":"J Genet Couns"},{"key":"2026012716173413700_ocaf190-B6","doi-asserted-by":"publisher","first-page":"1348","DOI":"10.1038\/s41436-020-0797-2","article-title":"Positive impact of genetic counseling assistants on genetic counseling efficiency, patient volume, and cost in a cancer genetics clinic","volume":"22","author":"Hallquist","year":"2020","journal-title":"Genet Med"},{"key":"2026012716173413700_ocaf190-B7","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1007\/s12687-019-00423-w","article-title":"Developing a culturally targeted video to enhance the use of genetic counseling in Latina women at increased risk for hereditary breast and ovarian cancer","volume":"11","author":"Hurtado-de-Mendoza","year":"2020","journal-title":"J Community Genet"},{"key":"2026012716173413700_ocaf190-B8","doi-asserted-by":"publisher","first-page":"e940","DOI":"10.1002\/mgg3.940","article-title":"The uptake of family screening in hypertrophic cardiomyopathy and an online video intervention to facilitate family communication","volume":"7","author":"Harris","year":"2019","journal-title":"Mol Genet Genomic Med"},{"key":"2026012716173413700_ocaf190-B9","year":"2024"},{"key":"2026012716173413700_ocaf190-B10","doi-asserted-by":"publisher","first-page":"1002","DOI":"10.1055\/s-0041-1736625","article-title":"Characterizing multitasking and workflow fragmentation in electronic health records among emergency department clinicians: using time-motion data to understand documentation burden","volume":"12","author":"Moy","year":"2021","journal-title":"Appl Clin Inform"},{"key":"2026012716173413700_ocaf190-B11","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1007\/s12687-022-00598-9","article-title":"Comparison of willingness and preference for genetic counseling via telemedicine: before vs during the COVID-19 pandemic","volume":"13","author":"Allison","year":"2022","journal-title":"J Community Genet"},{"key":"2026012716173413700_ocaf190-B12","doi-asserted-by":"publisher","first-page":"1214","DOI":"10.1002\/jgc4.1516","article-title":"Clinical genetic counselor experience in the adoption of telehealth in the United States and Canada during the COVID-19 pandemic","volume":"30","author":"Ma","year":"2021","journal-title":"J Genet Couns"},{"key":"2026012716173413700_ocaf190-B13","doi-asserted-by":"publisher","first-page":"103526","DOI":"10.1016\/j.jbi.2020.103526","article-title":"Clinical concept extraction: a methodology review","volume":"109","author":"Fu","year":"2020","journal-title":"J Biomed Inform"},{"key":"2026012716173413700_ocaf190-B14","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1016\/j.jbi.2017.07.012","article-title":"Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review","volume":"73","author":"Kreimeyer","year":"2017","journal-title":"J Biomed Inform"},{"key":"2026012716173413700_ocaf190-B15","doi-asserted-by":"publisher","first-page":"104139","DOI":"10.1016\/j.drudis.2024.104139","article-title":"Machine learning and natural language processing in clinical trial eligibility criteria parsing: a scoping review","volume":"29","author":"Kantor","year":"2024","journal-title":"Drug Discov Today"},{"key":"2026012716173413700_ocaf190-B16","doi-asserted-by":"publisher","first-page":"1163","DOI":"10.1093\/jamia\/ocz163","article-title":"Cohort selection for clinical trials: n2c2 2018 shared task track 1","volume":"26","author":"Stubbs","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2026012716173413700_ocaf190-B17","doi-asserted-by":"publisher","first-page":"2287","DOI":"10.1093\/jamia\/ocab143","article-title":"A systematic review of automatic text summarization for biomedical literature and EHRs","volume":"28","author":"Wang","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2026012716173413700_ocaf190-B18","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1016\/j.ajhg.2018.05.010","article-title":"Deep phenotyping on electronic health records facilitates genetic diagnosis by clinical exomes","volume":"103","author":"Son","year":"2018","journal-title":"Am J Hum Genet"},{"key":"2026012716173413700_ocaf190-B19","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1109\/TCBB.2018.2849968","article-title":"Natural language processing for EHR-based computational phenotyping","volume":"16","author":"Zeng","year":"2019","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2026012716173413700_ocaf190-B20","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"2026012716173413700_ocaf190-B21","first-page":"3735","volume-title":"Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC\u201912).","author":"Chang","year":"2024"},{"key":"2026012716173413700_ocaf190-B22","author":"Blechschmidt","year":"2024"},{"key":"2026012716173413700_ocaf190-B23","volume-title":"Natural Language Processing with Python","author":"Bird","year":"2009","edition":"1st"},{"key":"2026012716173413700_ocaf190-B24","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1038\/s41746-020-0258-y","article-title":"Protected Health Information filter (Philter): accurately and securely de-identifying free-text clinical notes","volume":"3","author":"Norgeot","year":"2020","journal-title":"NPJ Digit Med"},{"key":"2026012716173413700_ocaf190-B25","unstructured":"Montani I, Honnibal M, Honnibal M, Boyd A, Landeghem SV, Peters H. \u00a0explosion\/spaCy: v3.7.2: fixes for APIs and requirements. Published online October 16, 2023. Accessed July 14, 2025. 10.5281\/zenodo.10009823"},{"key":"2026012716173413700_ocaf190-B26","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"2026012716173413700_ocaf190-B27","doi-asserted-by":"publisher","first-page":"72","DOI":"10.18653\/v1\/W19-1909","volume-title":"Proceedings of the 2nd Clinical Natural Language Processing Workshop","author":"Alsentzer","year":"2019"},{"key":"2026012716173413700_ocaf190-B28","author":"Liu","year":"2019"},{"key":"2026012716173413700_ocaf190-B29","author":"Sanh","year":"2020"},{"key":"2026012716173413700_ocaf190-B30","doi-asserted-by":"publisher","first-page":"54","DOI":"10.18653\/v1\/N19-4010","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations).","author":"Akbik"},{"key":"2026012716173413700_ocaf190-B31","volume-title":"Advances in Neural Information Processing Systems","author":"Lundberg","year":"2017"},{"key":"2026012716173413700_ocaf190-B32","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","article-title":"SciPy 1.0: fundamental algorithms for scientific computing in Python","volume":"17","author":"Virtanen","year":"2020","journal-title":"Nat Methods"},{"key":"2026012716173413700_ocaf190-B33","doi-asserted-by":"publisher","first-page":"1169","DOI":"10.21105\/joss.01169","article-title":"scikit-posthocs: pairwise multiple comparison tests in Python","volume":"4","author":"Terpilowski","year":"2019","journal-title":"J Open Source Softw"},{"key":"2026012716173413700_ocaf190-B34","doi-asserted-by":"publisher","first-page":"92","DOI":"10.25080\/Majora","author":"Seabold","year":"2010"},{"key":"2026012716173413700_ocaf190-B35","doi-asserted-by":"publisher","first-page":"3021","DOI":"10.21105\/joss.03021","article-title":"seaborn: statistical data visualization","volume":"6","author":"Waskom","year":"2021","journal-title":"J Open Source Softw"},{"key":"2026012716173413700_ocaf190-B36","unstructured":"Charlier F, Weber M, Proost S, et al \u00a0Statannotations. Published online November 25, 2024. Accessed November 4, 2025. 10.5281\/zenodo.14258156"},{"key":"2026012716173413700_ocaf190-B37","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1187\/cbe.13-04-0082","article-title":"The other half of the story: effect size analysis in quantitative research","volume":"12","author":"Maher","year":"2013","journal-title":"CBE Life Sci Educ"},{"key":"2026012716173413700_ocaf190-B38","doi-asserted-by":"publisher","first-page":"1069","DOI":"10.1002\/jgc4.1717","article-title":"Mode of delivery preference in prenatal genetic counseling between English- and Spanish-speaking patients at two US medical institutions","volume":"32","author":"Steyer","year":"2023","journal-title":"J Genet Couns"},{"key":"2026012716173413700_ocaf190-B39","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1186\/s13053-023-00250-8","article-title":"Comparing telemedicine and in-person gastrointestinal cancer genetic appointment outcomes during the COVID-19 pandemic","volume":"21","author":"Williams","year":"2023","journal-title":"Hered Cancer Clin Pract"},{"key":"2026012716173413700_ocaf190-B40","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s12687-023-00693-5","article-title":"Patient experiences in receiving telegenetics care for inherited cardiovascular diseases","volume":"15","author":"Temares","year":"2024","journal-title":"J Community Genet"},{"key":"2026012716173413700_ocaf190-B41","doi-asserted-by":"publisher","first-page":"118","DOI":"10.1089\/tmj.2023.0021","article-title":"Patient and provider experiences and views on the use of telehealth in genetics clinics in response to the COVID-19 pandemic","volume":"30","author":"Martinez","year":"2024","journal-title":"Telemed J E-Health"},{"key":"2026012716173413700_ocaf190-B42","doi-asserted-by":"publisher","first-page":"834","DOI":"10.1002\/jgc4.1792","article-title":"The impact of converting to telehealth for cancer genetic counseling and testing during the COVID-19 pandemic","volume":"33","author":"Proussaloglou","year":"2024","journal-title":"J Genet Couns"},{"key":"2026012716173413700_ocaf190-B43","doi-asserted-by":"publisher","first-page":"e1946","DOI":"10.1002\/mgg3.1946","article-title":"The use of telemedicine in cardiogenetics clinical practice during the COVID-19 pandemic","volume":"10","author":"Liang","year":"2022","journal-title":"Mol Genet Genomic Med"},{"key":"2026012716173413700_ocaf190-B44","doi-asserted-by":"publisher","first-page":"958","DOI":"10.1002\/jgc4.1462","article-title":"The COVID-19 pandemic and reproductive genetic counseling: changes in access and service delivery at an academic medical center in the United States","volume":"30","author":"Mann","year":"2021","journal-title":"J Genet Couns"},{"key":"2026012716173413700_ocaf190-B45","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1007\/s12553-022-00670-x","article-title":"Rapid implementation of telegenetic services during the COVID-19 pandemic allowed continuing patient access, but not equally for all","volume":"12","author":"Noss","year":"2022","journal-title":"Health Technol"},{"key":"2026012716173413700_ocaf190-B46","doi-asserted-by":"publisher","first-page":"e1999","DOI":"10.1002\/jgc4.1999","article-title":"Evaluating genetic counseling session duration: a scoping review of patient care time, influencing factors, and impact on patient outcomes","volume":"34","author":"Glanton","year":"2025","journal-title":"J Genet Couns"},{"key":"2026012716173413700_ocaf190-B47","doi-asserted-by":"publisher","first-page":"878","DOI":"10.1002\/pd.6713","article-title":"Unequal uptake: insurance-related disparities in prenatal genetic counseling and screening at a Quaternary Medical Center","volume":"45","author":"Mallampati","year":"2025","journal-title":"Prenat Diagn"},{"key":"2026012716173413700_ocaf190-B48","doi-asserted-by":"publisher","first-page":"448","DOI":"10.1002\/pd.5447","article-title":"The impact of insurance coverage on the prenatal genetic counseling process: an exploration of genetic counselors\u2019 experiences with TRICARE","volume":"39","author":"Brown","year":"2019","journal-title":"Prenat Diagn"},{"key":"2026012716173413700_ocaf190-B49","doi-asserted-by":"publisher","first-page":"1263","DOI":"10.3390\/jpm12081263","article-title":"Association between health insurance type and genetic testing and\/or counseling for breast and ovarian cancer","volume":"12","author":"Mansur","year":"2022","journal-title":"J Pers Med"},{"key":"2026012716173413700_ocaf190-B50","doi-asserted-by":"publisher","first-page":"pkae018","DOI":"10.1093\/jncics\/pkae018","article-title":"Randomized trial promoting cancer genetic risk assessment when genetic counseling cost removed: 1-year follow-up","volume":"8","author":"An","year":"2024","journal-title":"JNCI Cancer Spectr"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/2\/295\/65252417\/ocaf190.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/2\/295\/65252417\/ocaf190.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T21:17:40Z","timestamp":1769548660000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/2\/295\/8317866"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,10]]},"references-count":50,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,11,10]]},"published-print":{"date-parts":[[2026,2,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf190","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,2]]},"published":{"date-parts":[[2025,11,10]]}}}