{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T10:56:35Z","timestamp":1740135395530,"version":"3.37.3"},"reference-count":55,"publisher":"Springer Science and Business Media LLC","issue":"S9","license":[{"start":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T00:00:00Z","timestamp":1635724800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,5]],"date-time":"2022-04-05T00:00:00Z","timestamp":1649116800000},"content-version":"vor","delay-in-days":155,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000145","name":"Division of Information and Intelligent Systems","doi-asserted-by":"publisher","award":["1633370"],"award-info":[{"award-number":["1633370"]}],"id":[{"id":"10.13039\/100000145","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"U.S. National Library of Medicine","doi-asserted-by":"publisher","award":["2R01LM010681-05"],"award-info":[{"award-number":["2R01LM010681-05"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004543","name":"China Scholarship Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004543","id-type":"DOI","asserted-by":"publisher"}]},{"name":"UNC SILS Kilgour Faculty Research Grant"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2021,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Natural language processing (NLP) tasks in the health domain often deal with limited amount of labeled data due to high annotation costs and naturally rare observations. To compensate for the lack of training data, health NLP researchers often have to leverage knowledge and resources external to a task at hand. Recently, pretrained large-scale language models such as the Bidirectional Encoder Representations from Transformers (BERT) have been proven to be a powerful way of learning rich linguistic knowledge from massive unlabeled text and transferring that knowledge to downstream tasks. However, previous downstream tasks often used training data at such a large scale that is unlikely to obtain in the health domain. In this work, we aim to study whether BERT can still benefit downstream tasks when training data are relatively small in the context of health NLP.<\/jats:p><\/jats:sec><jats:sec><jats:title>Method<\/jats:title><jats:p>We conducted a learning curve analysis to study the behavior of BERT and baseline models as training data size increases. We observed the classification performance of these models on two disease diagnosis data sets, where some diseases are naturally rare and have very limited observations (fewer than 2 out of 10,000). The baselines included commonly used text classification models such as sparse and dense bag-of-words models, long short-term memory networks, and their variants that leveraged external knowledge. To obtain learning curves, we incremented the amount of training examples per disease from small to large, and measured the classification performance in macro-averaged<jats:inline-formula><jats:alternatives><jats:tex-math>$$F_{1}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:msub><mml:mi>F<\/mml:mi><mml:mn>1<\/mml:mn><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>score.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>On the task of classifying all diseases, the learning curves of BERT were consistently above all baselines, significantly outperforming them across the spectrum of training data sizes. But under extreme situations where only one or two training documents per disease were available, BERT was outperformed by linear classifiers with carefully engineered bag-of-words features.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>As long as the amount of training documents is not extremely few, fine-tuning a pretrained BERT model is a highly effective approach to health NLP tasks like disease classification. However, in extreme cases where each class has only one or two training documents and no more will be available, simple linear models using bag-of-words features shall be considered.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-022-01829-2","type":"journal-article","created":{"date-parts":[[2022,4,5]],"date-time":"2022-04-05T11:09:36Z","timestamp":1649156976000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["When BERT meets Bilbo: a learning curve analysis of pretrained language model on disease classification"],"prefix":"10.1186","volume":"21","author":[{"given":"Xuedong","family":"Li","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Walter","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dezhong","family":"Peng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiaozhu","family":"Mei","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0278-2347","authenticated-orcid":false,"given":"Yue","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,4,5]]},"reference":[{"key":"1829_CR1","unstructured":"Amazon Web Services. Amazon SageMaker Ground Truth pricing. https:\/\/aws.amazon.com\/sagemaker\/groundtrut. Accessed July 2020."},{"key":"1829_CR2","doi-asserted-by":"publisher","first-page":"S20","DOI":"10.1016\/j.jbi.2015.07.020","volume":"58","author":"A Stubbs","year":"2015","unstructured":"Stubbs A, Uzuner \u00d6. Annotating longitudinal clinical narratives for de-identification: the 2014 i2b2\/UTHealth corpus. J Biomed Inform. 2015;58:S20\u20139.","journal-title":"J Biomed Inform"},{"key":"1829_CR3","unstructured":"United States Department of Health and Human Services. National Organization for Rare Disorders (NORD); Last Updated June 23, 2020. https:\/\/www.nidcd.nih.gov\/directory\/national-organization-rare-disorders-nord. Accessed 23 June 2020."},{"key":"1829_CR4","unstructured":"European Commission. Rare Diseases. https:\/\/ec.europa.eu\/health\/non-communicable-diseases\/steering-group\/rare-diseases_en. Accessed 16 July 2020."},{"issue":"3","key":"1829_CR5","doi-asserted-by":"publisher","first-page":"e17984","DOI":"10.2196\/17984","volume":"8","author":"I Spasic","year":"2020","unstructured":"Spasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inform. 2020;8(3):e17984.","journal-title":"JMIR Med Inform"},{"issue":"4","key":"1829_CR6","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1197\/jamia.M1157","volume":"10","author":"AB Wilcox","year":"2003","unstructured":"Wilcox AB, Hripcsak G. The role of domain knowledge in automating medical text report classification. J Am Med Inform Assoc. 2003;10(4):330\u20138.","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"1829_CR7","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1016\/j.jbi.2010.02.005","volume":"43","author":"D Demner-Fushman","year":"2010","unstructured":"Demner-Fushman D, Mork JG, Shooshan SE, Aronson AR. UMLS content views appropriate for NLP processing of the biomedical literature vs. clinical text. J Biomed Inform. 2010;43(4):587\u201394.","journal-title":"J Biomed Inform"},{"issue":"1","key":"1829_CR8","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1093\/jamia\/ocz169","volume":"27","author":"PI Dissanayake","year":"2020","unstructured":"Dissanayake PI, Colicchio TK, Cimino JJ. Using clinical reasoning ontologies to make smarter clinical decision support systems: a systematic review and data synthesis. J Am Med Inform Assoc. 2020;27(1):159\u201374.","journal-title":"J Am Med Inform Assoc"},{"key":"1829_CR9","first-page":"182","volume":"252","author":"E Zhang","year":"2018","unstructured":"Zhang E, Thurier Q, Boyle L. Improving clinical named-entity recognition with transfer learning. Stud Health Technol Inform. 2018;252:182\u20137.","journal-title":"Stud Health Technol Inform"},{"issue":"1","key":"1829_CR10","doi-asserted-by":"publisher","first-page":"368","DOI":"10.1186\/s12859-017-1776-8","volume":"18","author":"G Crichton","year":"2017","unstructured":"Crichton G, Pyysalo S, Chiu B, Korhonen A. A neural network multi-task learning approach to biomedical named entity recognition. BMC Bioinform. 2017;18(1):368.","journal-title":"BMC Bioinform"},{"issue":"1","key":"1829_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12911-018-0723-6","volume":"19","author":"Y Wang","year":"2019","unstructured":"Wang Y, Sohn S, Liu S, Shen F, Wang L, Atkinson EJ, et al. A clinical text classification paradigm using weak supervision and deep representation. BMC Med Inform Decis Mak. 2019;19(1):1\u201313.","journal-title":"BMC Med Inform Decis Mak"},{"key":"1829_CR12","doi-asserted-by":"publisher","first-page":"103522","DOI":"10.1016\/j.jbi.2020.103522","volume":"109","author":"N Pattisapu","year":"2020","unstructured":"Pattisapu N, Anand V, Patil S, Palshikar G, Varma V. Distant supervision for medical concept normalization. J Biomed Inform. 2020;109:103522.","journal-title":"J Biomed Inform"},{"issue":"5","key":"1829_CR13","doi-asserted-by":"publisher","first-page":"809","DOI":"10.1136\/amiajnl-2011-000648","volume":"19","author":"RL Figueroa","year":"2012","unstructured":"Figueroa RL, Zeng-Treitler Q, Ngo LH, Goryachev S, Wiechmann EP. Active learning for clinical text classification: is it better than random sampling? J Am Med Inform Assoc. 2012;19(5):809\u201316.","journal-title":"J Am Med Inform Assoc"},{"key":"1829_CR14","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K. Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. 2018."},{"key":"1829_CR15","unstructured":"Adhikari A, Ram A, Tang R, Lin J. Docbert: Bert for document classification. arXiv:1904.08398. 2019."},{"key":"1829_CR16","doi-asserted-by":"crossref","unstructured":"Wang A, Singh A, Michael J, Hill F, Levy O, Bowman SR. Glue: a multi-task benchmark and analysis platform for natural language understanding. arXiv:1804.07461. 2018.","DOI":"10.18653\/v1\/W18-5446"},{"issue":"4","key":"1829_CR17","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","volume":"36","author":"J Lee","year":"2020","unstructured":"Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234\u201340.","journal-title":"Bioinformatics"},{"key":"1829_CR18","doi-asserted-by":"crossref","unstructured":"Alsentzer E, Murphy JR, Boag W, Weng WH, Jin D, Naumann T, et al. Publicly available clinical BERT embeddings. arXiv:1904.03323. 2019.","DOI":"10.18653\/v1\/W19-1909"},{"key":"1829_CR19","doi-asserted-by":"crossref","unstructured":"Peng Y, Yan S, Lu Z. Transfer learning in biomedical natural language processing: an evaluation of bert and elmo on ten benchmarking datasets. arXiv:1906.05474. 2019.","DOI":"10.18653\/v1\/W19-5006"},{"issue":"12","key":"1829_CR20","doi-asserted-by":"publisher","first-page":"1632","DOI":"10.1093\/jamia\/ocz164","volume":"26","author":"L Yao","year":"2019","unstructured":"Yao L, Jin Z, Mao C, Zhang Y, Luo Y. Traditional Chinese medicine clinical records classification with BERT and domain specific corpora. J Am Med Inform Assoc. 2019;26(12):1632\u20136.","journal-title":"J Am Med Inform Assoc"},{"key":"1829_CR21","doi-asserted-by":"publisher","first-page":"103422","DOI":"10.1016\/j.jbi.2020.103422","volume":"107","author":"X Li","year":"2020","unstructured":"Li X, Zhang H, Zhou XH. Chinese clinical named entity recognition with variant neural structures based on BERT methods. J Biomed Inform. 2020;107:103422.","journal-title":"J Biomed Inform"},{"key":"1829_CR22","unstructured":"Zhang N, Jia Q, Yin K, Dong L, Gao F, Hua N. Conceptualized representation learning for chinese biomedical text mining. arXiv:2008.10813. 2020."},{"key":"1829_CR23","unstructured":"Hestness J, Narang S, Ardalani N, Diamos G, Jun H, Kianinejad H, et al. Deep learning scaling is predictable, empirically. arXiv:1712.00409. 2017."},{"issue":"6","key":"1829_CR24","doi-asserted-by":"publisher","first-page":"646","DOI":"10.1136\/jamia.2009.001024","volume":"17","author":"MH Stanfill","year":"2010","unstructured":"Stanfill MH, Williams M, Fenton SH, Jenders RA, Hersh WR. A systematic literature review of automated clinical coding and classification systems. J Am Med Inform Assoc. 2010;17(6):646\u201351.","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"1829_CR25","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1197\/jamia.M2408","volume":"15","author":"\u00d6 Uzuner","year":"2008","unstructured":"Uzuner \u00d6, Goldstein I, Luo Y, Kohane I. Identifying patient smoking status from medical discharge records. J Am Med Inform Assoc. 2008;15(1):14\u201324.","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"1829_CR26","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1197\/jamia.M3115","volume":"16","author":"\u00d6 Uzuner","year":"2009","unstructured":"Uzuner \u00d6. Recognizing obesity and comorbidities in sparse data. J Am Med Inform Assoc. 2009;16(4):561\u201370.","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"1829_CR27","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1186\/s12911-019-0781-4","volume":"19","author":"L Yao","year":"2019","unstructured":"Yao L, Mao C, Luo Y. Clinical text classification with rule-based features and knowledge-guided convolutional neural networks. BMC Med Inform Decis Mak. 2019;19(3):71.","journal-title":"BMC Med Inform Decis Mak"},{"issue":"6","key":"1829_CR28","doi-asserted-by":"publisher","first-page":"998","DOI":"10.1016\/j.jbi.2013.08.011","volume":"46","author":"J Huh","year":"2013","unstructured":"Huh J, Yetisgen-Yildiz M, Pratt W. Text classification for assisting moderators in online health communities. J Biomed Inform. 2013;46(6):998\u20131005.","journal-title":"J Biomed Inform"},{"key":"1829_CR29","doi-asserted-by":"publisher","DOI":"10.1007\/s12652-019-01399-8","author":"DC Edara","year":"2019","unstructured":"Edara DC, Vanukuri LP, Sistla V, Kolli VKK. Sentiment analysis and text categorization of cancer medical records with LSTM. J Ambient Intell Humaniz Comput. 2019. https:\/\/doi.org\/10.1007\/s12652-019-01399-8.","journal-title":"J Ambient Intell Humaniz Comput"},{"issue":"5","key":"1829_CR30","doi-asserted-by":"publisher","first-page":"631","DOI":"10.1136\/amiajnl-2010-000022","volume":"18","author":"T Botsis","year":"2011","unstructured":"Botsis T, Nguyen MD, Woo EJ, Markatou M, Ball R. Text mining for the vaccine adverse event reporting system: medical text classification using informative feature selection. J Am Med Inform Assoc. 2011;18(5):631\u20138.","journal-title":"J Am Med Inform Assoc"},{"key":"1829_CR31","first-page":"162","volume-title":"Industrial conference on data mining","author":"J Zhong","year":"2018","unstructured":"Zhong J, Yi X, Xuan D, Xie Y. Categorization of patient diseases for chinese electronic health record analysis: a case study. In: Perner P, editor. Industrial conference on data mining. Cham: Springer; 2018. p. 162\u201372."},{"issue":"4","key":"1829_CR32","doi-asserted-by":"publisher","first-page":"222","DOI":"10.1016\/S1532-0464(03)00012-1","volume":"35","author":"C Friedman","year":"2002","unstructured":"Friedman C, Kra P, Rzhetsky A. Two biomedical sublanguages: a description based on the theories of Zellig Harris. J Biomed Inform. 2002;35(4):222\u201335.","journal-title":"J Biomed Inform"},{"issue":"3","key":"1829_CR33","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1093\/jamia\/ocz200","volume":"27","author":"S Wu","year":"2020","unstructured":"Wu S, Roberts K, Datta S, Du J, Ji Z, Si Y, et al. Deep learning in clinical natural language processing: a methodical review. J Am Med Inform Assoc. 2020;27(3):457\u201370.","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"1829_CR34","doi-asserted-by":"publisher","first-page":"992","DOI":"10.1016\/j.jbi.2012.04.010","volume":"45","author":"VN Garla","year":"2012","unstructured":"Garla VN, Brandt C. Ontology-guided feature engineering for clinical text classification. J Biomed Inform. 2012;45(5):992\u20138.","journal-title":"J Biomed Inform"},{"issue":"5","key":"1829_CR35","doi-asserted-by":"publisher","first-page":"238","DOI":"10.1186\/s12911-019-0938-1","volume":"19","author":"X Li","year":"2019","unstructured":"Li X, Wang Y, Wang D, Yuan W, Peng D, Mei Q. Improving rare disease classification using imperfect knowledge graph. BMC Med Inform Decis Mak. 2019;19(5):238.","journal-title":"BMC Med Inform Decis Mak"},{"key":"1829_CR36","doi-asserted-by":"crossref","unstructured":"Choi E, Bahadori MT, Song L, Stewart WF, Sun J. GRAM: graph-based attention model for healthcare representation learning. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining; 2017. p. 787\u201395.","DOI":"10.1145\/3097983.3098126"},{"key":"1829_CR37","doi-asserted-by":"crossref","unstructured":"Zhang Z, Han X, Liu Z, Jiang X, Sun M, Liu Q. ERNIE: enhanced language representation with informative entities. arXiv:1905.07129. 2019.","DOI":"10.18653\/v1\/P19-1139"},{"key":"1829_CR38","unstructured":"Liu W, Zhou P, Zhao Z, Wang Z, Ju Q, Deng H, et al. K-bert: enabling language representation with knowledge graph. arXiv:1909.07606. 2019."},{"key":"1829_CR39","unstructured":"Jieba Chinese text segmentation. https:\/\/github.com\/fxsjy\/jieba. Accessed 26 Mar 2019."},{"key":"1829_CR40","first-page":"1048","volume":"5","author":"E Gabrilovich","year":"2005","unstructured":"Gabrilovich E, Markovitch S. Feature generation for text categorization using world knowledge. IJCAI. 2005;5:1048\u201353.","journal-title":"IJCAI"},{"key":"1829_CR41","doi-asserted-by":"crossref","unstructured":"Guo J, Che W, Wang H, Liu T. Revisiting embedding features for simple semi-supervised learning. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP); 2014. p. 110\u201320.","DOI":"10.3115\/v1\/D14-1012"},{"key":"1829_CR42","unstructured":"Wu Y, Xu J, Jiang M, Zhang Y, Xu H. A study of neural word embeddings for named entity recognition in clinical text. In: AMIA annual symposium proceedings, vol. 2015. American Medical Informatics Association; 2015. p. 1326."},{"key":"1829_CR43","doi-asserted-by":"crossref","unstructured":"Tang J, Qu M, Wang M, Zhang M, Yan J, Mei Q. Line: large-scale information network embedding. In: Proceedings of the 24th international conference on world wide web; 2015. p. 1067\u201377.","DOI":"10.1145\/2736277.2741093"},{"key":"1829_CR44","first-page":"3111","volume":"26","author":"T Mikolov","year":"2013","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst. 2013;26:3111\u20139.","journal-title":"Adv Neural Inf Process Syst"},{"key":"1829_CR45","unstructured":"Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. arXiv:1301.3781. 2013."},{"key":"1829_CR46","first-page":"5998","volume":"30","author":"A Vaswani","year":"2017","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:5998\u20136008.","journal-title":"Adv Neural Inf Process Syst"},{"key":"1829_CR47","unstructured":"Smith NA. Contextual word representations: a contextual introduction. arXiv:1902.06006. 2019."},{"issue":"Mar","key":"1829_CR48","first-page":"1289","volume":"3","author":"G Forman","year":"2003","unstructured":"Forman G. An extensive empirical study of feature selection metrics for text classification. J Mach Learn Res. 2003;3(Mar):1289\u2013305.","journal-title":"J Mach Learn Res"},{"key":"1829_CR49","unstructured":"Su J. Pretrained Word2Vector. https:\/\/kexue.fm\/archives\/4304. Accessed 03 Apr 2017."},{"key":"1829_CR50","doi-asserted-by":"crossref","unstructured":"Xu B, Xu Y, Liang J, Xie C, Liang B, Cui W, et al. CN-DBpedia: a never-ending Chinese knowledge extraction system. In: International conference on industrial, engineering and other applications of applied intelligent systems. Springer; 2017. p. 428\u201338.","DOI":"10.1007\/978-3-319-60045-1_44"},{"key":"1829_CR51","unstructured":"Wikipedia. F1 Score. https:\/\/en.wikipedia.org\/wiki\/F1_score. Accessed 26 Mar 2019."},{"key":"1829_CR52","unstructured":"Guyon I, Cawley GC, Dror G, Lemaire V. Results of the active learning challenge. In: Active learning and experimental design workshop in conjunction with AISTATS 2010; 2011. p. 19\u201345."},{"key":"1829_CR53","doi-asserted-by":"crossref","unstructured":"Lilleberg J, Zhu Y, Zhang Y. Support vector machines and word2vec for text classification with semantic features. In: 2015 IEEE 14th international conference on cognitive informatics & cognitive computing (ICCI*CC). IEEE; 2015. p. 136\u201340.","DOI":"10.1109\/ICCI-CC.2015.7259377"},{"key":"1829_CR54","doi-asserted-by":"crossref","unstructured":"Gururangan S, Marasovic A, Swayamdipta S, Lo K, Beltagy I, Downey D, et al. Don\u2019t stop pretraining: adapt language models to domains and tasks. In: Proceedings of the 58th annual meeting of the association for computational linguistics. Online: Association for Computational Linguistics; 2020. p. 8342\u201360. https:\/\/aclanthology.org\/2020.acl-main.740\/.","DOI":"10.18653\/v1\/2020.acl-main.740"},{"key":"1829_CR55","doi-asserted-by":"crossref","unstructured":"Zhang H, Lu AX, Abdalla M, et al. Hurtful words: quantifying biases in clinical contextual word embeddings. In: Proceedings of the ACM conference on health, inference, and learning. 2020. p. 110\u201320.","DOI":"10.1145\/3368555.3384448"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01829-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-022-01829-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01829-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T16:54:29Z","timestamp":1675184069000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-022-01829-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11]]},"references-count":55,"journal-issue":{"issue":"S9","published-print":{"date-parts":[[2021,11]]}},"alternative-id":["1829"],"URL":"https:\/\/doi.org\/10.1186\/s12911-022-01829-2","relation":{},"ISSN":["1472-6947"],"issn-type":[{"type":"electronic","value":"1472-6947"}],"subject":[],"published":{"date-parts":[[2021,11]]},"assertion":[{"value":"6 March 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 March 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 April 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"377"}}