{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T15:00:32Z","timestamp":1784646032299,"version":"3.55.0"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"S7","license":[{"start":{"date-parts":[[2021,11,9]],"date-time":"2021-11-09T00:00:00Z","timestamp":1636416000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,9]],"date-time":"2021-11-09T00:00:00Z","timestamp":1636416000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000060","name":"National Institute of Allergy and Infectious Diseases","doi-asserted-by":"publisher","award":["R21 AI142702"],"award-info":[{"award-number":["R21 AI142702"]}],"id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["R01 HL126667"],"award-info":[{"award-number":["R01 HL126667"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["R01 AG068007"],"award-info":[{"award-number":["R01 AG068007"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>There are significant variabilities in guideline-concordant documentation in asthma care. However, assessing clinician\u2019s documentation is not feasible using only structured data but requires labor-intensive chart review of electronic health records (EHRs). A certain guideline element in asthma control factors, such as review inhaler techniques, requires context understanding to correctly capture from EHR free text.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>The study data consist of two sets: (1) manual chart reviewed data\u20141039 clinical notes of 300 patients with asthma diagnosis, and (2) weakly labeled data (distant supervision)\u201427,363 clinical notes from 800 patients with asthma diagnosis. A context-aware language model, Bidirectional Encoder Representations from Transformers (BERT) was developed to identify inhaler techniques in EHR free text. Both original BERT and clinical BioBERT (cBERT) were applied with a cost-sensitivity to deal with imbalanced data. The distant supervision using weak labels by rules was also incorporated to augment the training set and alleviate a costly manual labeling process in the development of a deep learning algorithm. A hybrid approach using post-hoc rules was also explored to fix BERT model errors. The performance of BERT with\/without distant supervision, hybrid, and rule-based models were compared in precision, recall, F-score, and accuracy.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The BERT models on the original data performed similar to a rule-based model in F1-score (0.837, 0.845, and 0.838 for rules, BERT, and cBERT, respectively). The BERT models with distant supervision produced higher performance (0.853 and 0.880 for BERT and cBERT, respectively) than without distant supervision and a rule-based model. The hybrid models performed best in F1-score of 0.877 and 0.904 over the distant supervision on BERT and cBERT.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>The proposed BERT models with distant supervision demonstrated its capability to identify inhaler techniques in EHR free text, and outperformed both the rule-based model and BERT models trained on the original data. With a distant supervision approach, we may alleviate costly manual chart review to generate the large training data required in most deep learning-based models. A hybrid model was able to fix BERT model errors and further improve the performance.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-021-01633-4","type":"journal-article","created":{"date-parts":[[2021,11,9]],"date-time":"2021-11-09T13:03:12Z","timestamp":1636462992000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Identification of asthma control factor in clinical notes using a hybrid deep learning model"],"prefix":"10.1186","volume":"21","author":[{"given":"Bhavani Singh","family":"Agnikula Kshatriya","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elham","family":"Sagheb","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chung-Il","family":"Wi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jungwon","family":"Yoon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hee Yun","family":"Seol","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Young","family":"Juhn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8256-2602","authenticated-orcid":false,"given":"Sunghwan","family":"Sohn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,11,9]]},"reference":[{"issue":"17","key":"1633_CR1","first-page":"547","volume":"60","author":"Centers for Disease Control and Prevention","year":"2011","unstructured":"Centers for Disease Control and Prevention. Vital signs: asthma prevalence, disease characteristics, and self-management education: United States, 2001\u20132009. MMWR Morb Mortal Wkly Rep. 2011;60(17):547.","journal-title":"MMWR Morb Mortal Wkly Rep"},{"key":"1633_CR2","unstructured":"Lethbridge-\u00c7ejku M, Vickerie JL. Summary health statistics for US adults; National health interview survey. 2003."},{"key":"1633_CR3","volume-title":"The high concentration of US health care expenditures","author":"MW Stanton","year":"2006","unstructured":"Stanton MW, Rutherford M. The high concentration of US health care expenditures. Rockville: Agency for Healthcare Research and Quality Rockville; 2006."},{"key":"1633_CR4","unstructured":"National Heart, Lung, Blood Institute. Expert panel report III: guidelines for the diagnosis and management of asthma. 2007."},{"issue":"2","key":"1633_CR5","first-page":"357","volume":"36","author":"LT Piecoro","year":"2001","unstructured":"Piecoro LT, Potoski M, Talbert JC, Doherty DE. Asthma prevalence, cost, and adherence with expert guidelines on the utilization of health care services and costs in a state Medicaid population. Health Serv Res. 2001;36(2):357.","journal-title":"Health Serv Res"},{"issue":"4","key":"1633_CR6","first-page":"1242","volume":"127","author":"A Senthilselvan","year":"2005","unstructured":"Senthilselvan A, Lawson JA, Rennie DC, Dosman JA. Regular use of corticosteroids and low use of short-acting \u03b22-agonists can reduce asthma hospitalization. Chest. 2005;127(4):1242\u201351.","journal-title":"Chest"},{"issue":"5","key":"1633_CR7","doi-asserted-by":"publisher","first-page":"591","DOI":"10.1016\/j.jpeds.2004.12.017","volume":"146","author":"MM Cloutier","year":"2005","unstructured":"Cloutier MM, Hall CB, Wakefield DB, Bailit H. Use of asthma guidelines by primary care providers to reduce hospitalizations and emergency department visits in poor, minority, urban children. J Pediatr. 2005;146(5):591\u20137.","journal-title":"J Pediatr"},{"issue":"4","key":"1633_CR8","doi-asserted-by":"publisher","first-page":"e902","DOI":"10.1542\/peds.2006-1805","volume":"120","author":"P Fox","year":"2007","unstructured":"Fox P, Porter PG, Lob SH, Boer JH, Rocha DA, Adelson JW. Improving asthma-related health outcomes among low-income, multiethnic, school-aged children: results of a demonstration project that combined continuous quality improvement and community health worker strategies. Pediatrics. 2007;120(4):e902\u201311.","journal-title":"Pediatrics"},{"issue":"3","key":"1633_CR9","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1370\/afm.1624","volume":"12","author":"JW Mold","year":"2014","unstructured":"Mold JW, et al. Implementing asthma guidelines using practice facilitation and local learning collaboratives: a randomized controlled trial. Ann Fam Med. 2014;12(3):233\u201340.","journal-title":"Ann Fam Med"},{"issue":"2","key":"1633_CR10","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1016\/j.acap.2012.10.009","volume":"13","author":"AB Yee","year":"2013","unstructured":"Yee AB, Fagnano M, Halterman JS. Preventive asthma care delivery in the primary care office: missed opportunities for children with persistent asthma symptoms. Acad Pediatr. 2013;13(2):98\u2013104.","journal-title":"Acad Pediatr"},{"issue":"4","key":"1633_CR11","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1016\/j.mayocp.2016.01.010","volume":"91","author":"BP Yawn","year":"2016","unstructured":"Yawn BP, Rank MA, Cabana MD, Wollan PC, Juhn YJ. Adherence to asthma guidelines in children, tweens, and adults in primary care settings: a practice-based network assessment. Mayo Clin Proc. 2016;91(4):411\u201321.","journal-title":"Mayo Clin Proc"},{"issue":"5","key":"1633_CR12","doi-asserted-by":"publisher","first-page":"e64933","DOI":"10.1371\/journal.pone.0064933","volume":"8","author":"GM Weber","year":"2013","unstructured":"Weber GM, Kohane IS. Extracting physician group intelligence from electronic health records to support evidence based medicine. PLoS ONE. 2013;8(5):e64933.","journal-title":"PLoS ONE"},{"issue":"5","key":"1633_CR13","doi-asserted-by":"publisher","first-page":"412","DOI":"10.1001\/archpediatrics.2011.49","volume":"165","author":"A Shapiro","year":"2011","unstructured":"Shapiro A, Gracy D, Quinones W, Applebaum J, Sarmiento A. Putting guidelines into practice: improving documentation of pediatric asthma management using a decision-making tool. Arch Pediatr Adolesc Med. 2011;165(5):412\u20138.","journal-title":"Arch Pediatr Adolesc Med"},{"key":"1633_CR14","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. 2018. arXiv preprint https:\/\/arxiv.org\/1810.04805."},{"key":"1633_CR15","unstructured":"Wang Y et al. A deep representation empowered distant supervision paradigm for clinical information extraction. 2018. arXiv preprint https:\/\/arciv.org\/1804.07814."},{"issue":"7","key":"1633_CR16","doi-asserted-by":"publisher","first-page":"e0216913","DOI":"10.1371\/journal.pone.0216913","volume":"14","author":"P Su","year":"2019","unstructured":"Su P, Li G, Wu C, Vijay-Shanker K. Using distant supervision to augment manually annotated data for relation extraction. PLoS ONE. 2019;14(7):e0216913.","journal-title":"PLoS ONE"},{"issue":"1","key":"1633_CR17","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1016\/j.jaip.2017.04.041","volume":"6","author":"C-I Wi","year":"2018","unstructured":"Wi C-I, et al. Natural language processing for asthma ascertainment in different practice settings. J Allergy Clin Immunol Pract. 2018;6(1):126\u201331.","journal-title":"J Allergy Clin Immunol Pract"},{"issue":"4","key":"1633_CR18","doi-asserted-by":"publisher","first-page":"430","DOI":"10.1164\/rccm.201610-2006OC","volume":"196","author":"C-I Wi","year":"2017","unstructured":"Wi C-I, et al. Application of a natural language processing algorithm to asthma ascertainment: an automated chart review. Am J Respir Crit Care Med. 2017;196(4):430\u20137.","journal-title":"Am J Respir Crit Care Med"},{"issue":"1","key":"1633_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12890-018-0593-9","volume":"18","author":"H Kaur","year":"2018","unstructured":"Kaur H, et al. Automated chart review utilizing natural language processing algorithm for asthma predictive index. BMC Pulm Med. 2018;18(1):1\u20139.","journal-title":"BMC Pulm Med"},{"issue":"3","key":"1633_CR20","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1093\/jamia\/ocx138","volume":"25","author":"S Sohn","year":"2018","unstructured":"Sohn S, et al. Clinical documentation variations and NLP system portability: a case study in asthma birth cohorts across institutions. J Am Med Inform Assoc. 2018;25(3):353\u20139.","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"1633_CR21","doi-asserted-by":"publisher","first-page":"2292","DOI":"10.1016\/j.jaci.2017.12.1003","volume":"141","author":"S Sohn","year":"2018","unstructured":"Sohn S, et al. Ascertainment of asthma prognosis using natural language processing from electronic medical records. J Allergy Clin Immunol. 2018;141(6):2292\u20134.","journal-title":"J Allergy Clin Immunol"},{"issue":"2","key":"1633_CR22","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1016\/j.jaci.2019.12.897","volume":"145","author":"Y Juhn","year":"2020","unstructured":"Juhn Y, Liu H. Artificial intelligence approaches using natural language processing to advance EHR-based clinical research. J Allergy Clin Immunol. 2020;145(2):463\u20139.","journal-title":"J Allergy Clin Immunol"},{"issue":"1","key":"1633_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12911-018-0723-6","volume":"19","author":"Y Wang","year":"2019","unstructured":"Wang Y, et al. A clinical text classification paradigm using weak supervision and deep representation. BMC Med Inform Decis Mak. 2019;19(1):1.","journal-title":"BMC Med Inform Decis Mak"},{"issue":"5","key":"1633_CR24","doi-asserted-by":"publisher","first-page":"1589","DOI":"10.1109\/JBHI.2017.2767063","volume":"22","author":"B Shickel","year":"2017","unstructured":"Shickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform. 2017;22(5):1589\u2013604.","journal-title":"IEEE J Biomed Health Inform"},{"key":"1633_CR25","doi-asserted-by":"publisher","first-page":"102008","DOI":"10.1016\/j.artmed.2021.102008","volume":"112","author":"KS Kalyan","year":"2021","unstructured":"Kalyan KS, Sangeetha S. Bertmcn: mapping colloquial phrases to standard medical concepts using bert and highway network. Artif Intell Med. 2021;112:102008.","journal-title":"Artif Intell Med"},{"issue":"11","key":"1633_CR26","doi-asserted-by":"publisher","first-page":"1297","DOI":"10.1093\/jamia\/ocz096","volume":"26","author":"Y Si","year":"2019","unstructured":"Si Y, Wang J, Xu H, Roberts K. Enhancing clinical concept extraction with contextual embeddings. J Am Med Inform Assoc. 2019;26(11):1297\u2013304.","journal-title":"J Am Med Inform Assoc"},{"key":"1633_CR27","unstructured":"Wei Q et al. Relation extraction from clinical narratives using pre-trained language models. In: AMIA annual symposium proceedings, 2019. American Medical Informatics Association, 2019; p. 1236."},{"key":"1633_CR28","doi-asserted-by":"publisher","DOI":"10.1093\/database\/baz116","author":"T Chen","year":"2019","unstructured":"Chen T, Mingfen W, Li H. A general approach for improving deep learning-based medical relation extraction using a pre-trained model and fine-tuning. Database. 2019. https:\/\/doi.org\/10.1093\/database\/baz116.","journal-title":"Database"},{"key":"1633_CR29","unstructured":"Costa AD et al. Multiple sclerosis severity classification from clinical text. 2020. arXiv preprint https:\/\/arxiv.org\/2010.15316."},{"key":"1633_CR30","unstructured":"Mulyar A, Schumacher E, Rouhizadeh M, Dredze M. Phenotyping of clinical notes with improved document classification models using contextualized neural language models. 2019. arXiv preprint https:\/\/arxiv.org\/1910.13664."},{"issue":"4","key":"1633_CR31","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","volume":"36","author":"J Lee","year":"2020","unstructured":"Lee J, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234\u201340.","journal-title":"Bioinformatics"},{"key":"1633_CR32","unstructured":"Huang K, Altosaar J, Ranganath R. Clinicalbert: modeling clinical notes and predicting hospital readmission. 2019. arXiv preprint https:\/\/arxiv.org\/1904.05342."},{"key":"1633_CR33","unstructured":"Alsentzer E et al. Publicly available clinical BERT embeddings. 2019. arXiv preprint https:\/\/arxiv.org\/1904.03323."},{"key":"1633_CR34","unstructured":"Liu H et al. An information extraction framework for cohort identification using electronic health records. Presented at the AMIA Summits Transl Sci Proc, San Francisco, CA, 2013."},{"key":"1633_CR35","unstructured":"Madabushi HT, Kochkina E, Castelle M. Cost-sensitive BERT for generalisable sentence classification with imbalanced data. 2020. arXiv preprint https:\/\/arxiv.org\/2003.11563."}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01633-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-021-01633-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01633-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,12]],"date-time":"2023-11-12T11:58:29Z","timestamp":1699790309000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-021-01633-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,9]]},"references-count":35,"journal-issue":{"issue":"S7","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["1633"],"URL":"https:\/\/doi.org\/10.1186\/s12911-021-01633-4","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,9]]},"assertion":[{"value":"13 September 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 November 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This study was approved by the Mayo Clinic internal review board. The study was a retrospective study utilizing study subjects\u2019 EHRs. We only included those who granted authorization for accessing their EHRs for research. Therefore, we did not require the study-specific consent to participate in the study.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"272"}}