{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T16:21:10Z","timestamp":1792254070972,"version":"4.4.0"},"reference-count":46,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T00:00:00Z","timestamp":1687910400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000038","name":"NSERC","doi-asserted-by":"publisher","award":["2017-04647"],"award-info":[{"award-number":["2017-04647"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Digit. Health"],"abstract":"<jats:sec><jats:title>Background<\/jats:title><jats:p>The use of social media data provides an opportunity to complement traditional influenza and COVID-19 surveillance methods for the detection and control of outbreaks and informing public health interventions.<\/jats:p><\/jats:sec><jats:sec><jats:title>Objective<\/jats:title><jats:p>The first aim of this study is to investigate the degree to which Twitter users disclose health experiences related to influenza and COVID-19 that could be indicative of recent plausible influenza cases or symptomatic COVID-19 infections. Second, we seek to use the Twitter datasets to train and evaluate the classification performance of Bidirectional Encoder Representations from Transformers (BERT) and variant language models in the context of influenza and COVID-19 infection detection.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We constructed two Twitter datasets using a keyword-based filtering approach on English-language tweets collected from December 2016 to December 2022 in Saskatchewan, Canada. The influenza-related dataset comprised tweets filtered with influenza-related keywords from December 13, 2016, to March 17, 2018, while the COVID-19 dataset comprised tweets filtered with COVID-19 symptom-related keywords from January 1, 2020, to June 22, 2021. The Twitter datasets were cleaned, and each tweet was annotated by at least two annotators as to whether it suggested recent plausible influenza cases or symptomatic COVID-19 cases. We then assessed the classification performance of pre-trained transformer-based language models, including BERT-base, BERT-large, RoBERTa-base, RoBERT-large, BERTweet-base, BERTweet-covid-base, BERTweet-large, and COVID-Twitter-BERT (CT-BERT) models, on each dataset. To address the notable class imbalance, we experimented with both oversampling and undersampling methods.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The influenza dataset had 1129 out of 6444 (17.5%) tweets annotated as suggesting recent plausible influenza cases. The COVID-19 dataset had 924 out of 11939 (7.7%) tweets annotated as inferring recent plausible COVID-19 cases. When compared against other language models on the COVID-19 dataset, CT-BERT performed the best, supporting the highest scores for recall (94.8%), F1(94.4%), and accuracy (94.6%). For the influenza dataset, BERTweet models exhibited better performance. Our results also showed that applying data balancing techniques such as oversampling or undersampling method did not lead to improved model performance.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Utilizing domain-specific language models for monitoring users\u2019 health experiences related to influenza and COVID-19 on social media shows improved classification performance and has the potential to supplement real-time disease surveillance.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fdgth.2023.1203874","type":"journal-article","created":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T12:05:45Z","timestamp":1687953945000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Comparison of pretrained transformer-based models for influenza and COVID-19 detection using social media text data in Saskatchewan, Canada"],"prefix":"10.3389","volume":"5","author":[{"given":"Yuan","family":"Tian","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenjing","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lujie","family":"Duan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wade","family":"McDonald","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nathaniel","family":"Osgood","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,6,28]]},"reference":[{"key":"B1","year":""},{"key":"B2","year":""},{"key":"B3","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1146\/annurev-publhealth-040119-094402","article-title":"Social media-, internet-based disease surveillance for public health","volume":"41","author":"Aiello","year":"2020","journal-title":"Annu Rev Public Health"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare10081478","article-title":"Comparison of pretraining models, strategies for Health-Related social media text classification","volume":"10","author":"Guo","year":"2022","journal-title":"Healthcare (Basel)"},{"key":"B5","doi-asserted-by":"publisher","first-page":"994949","DOI":"10.3389\/fpubh.2022.994949","article-title":"Modeling approaches for early warning, monitoring of pandemic situations as well as decision support","volume":"10","author":"Botz","year":"2022","journal-title":"Front Public Health"},{"key":"B6","author":"Byrd","year":""},{"key":"B7","doi-asserted-by":"publisher","first-page":"e65","DOI":"10.2196\/publichealth.8627","article-title":"Twitter-based influenza detection after flu peak via tweets with indirect information: text mining study","volume":"4","author":"Wakamiya","year":"2018","journal-title":"JMIR Public Health Surveill"},{"key":"B8","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1007\/s13755-021-00158-4","article-title":"Monitoring COVID-19 pandemic through the lens of social media using natural language processing and machine learning","volume":"9","author":"Liu","year":"2021","journal-title":"Health Inf Sci Syst"},{"key":"B9","doi-asserted-by":"publisher","first-page":"e27670","DOI":"10.2196\/27670","article-title":"Social media monitoring of the COVID-19 pandemic and influenza epidemic with adaptation for informal language in arabic twitter data: Qualitative study","volume":"9","author":"Alsudias","year":"2021","journal-title":"JMIR Med Inform"},{"key":"B10","author":"Lin","year":""},{"key":"B11","doi-asserted-by":"publisher","first-page":"e14118","DOI":"10.1371\/journal.pone.0014118","article-title":"Pandemics in the age of twitter: content analysis of tweets during the 2009 H1N1 outbreak","volume":"5","author":"Chew","year":"2010","journal-title":"PLoS ONE"},{"key":"B12","doi-asserted-by":"publisher","first-page":"e25431","DOI":"10.2196\/25431","article-title":"Tracking COVID-19 discourse on twitter in north america: Infodemiology study using topic modeling and Aspect-Based sentiment analysis","volume":"23","author":"Jang","year":"2021","journal-title":"J Med Internet Res"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1016\/j.jjimei.2021.100021","article-title":"Leveraging twitter data to understand public sentiment for the COVID-19 outbreak in singapore","author":"Ridhwan","year":"2021","journal-title":"Int J Inf Manag Data Insights"},{"key":"B14","doi-asserted-by":"publisher","first-page":"e21978","DOI":"10.2196\/21978","article-title":"Public perception of the COVID-19 pandemic on twitter: Sentiment analysis, topic modeling study","volume":"6","author":"Boon-Itt","year":"2020","journal-title":"JMIR Public Health Surveill"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph18010218","article-title":"A sentiment analysis approach to predict an individual\u2019s awareness of the precautionary procedures to prevent COVID-19 outbreaks in saudi arabia","volume":"18","author":"Aljameel","year":"2020","journal-title":"Int J Environ Res Public Health"},{"key":"B16","author":"Sitaula","year":"2022"},{"key":"B17","doi-asserted-by":"publisher","first-page":"5681574","DOI":"10.1155\/2022\/5681574","article-title":"A hybrid feature extraction method for nepali COVID-19-Related tweets classification","volume":"2022","author":"Shahi","year":"2022","journal-title":"Comput Intell Neurosci"},{"key":"B18","doi-asserted-by":"publisher","first-page":"2158184","DOI":"10.1155\/2021\/2158184","article-title":"Deep learning-based methods for sentiment analysis on nepali COVID-19-related tweets","volume":"2021","author":"Sitaula","year":"2021","journal-title":"Comput Intell Neurosci"},{"key":"B19","doi-asserted-by":"publisher","first-page":"e19509","DOI":"10.2196\/19509","article-title":"Machine learning to detect self-reporting of symptoms, testing access,, recovery associated with COVID-19 on twitter: retrospective big data infoveillance study","volume":"6","author":"Mackey","year":"2020","journal-title":"JMIR Public Health Surveill"},{"key":"B20","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1017\/S0008423920000311","article-title":"A chronological and geographical analysis of personal reports of COVID-19 on twitter from the UK","volume":"53","author":"Golder","year":"2022","journal-title":"Digit Health"},{"key":"B21","author":"Klein","year":""},{"key":"B22","author":"Devlin","year":""},{"key":"B23","author":"Liu","year":""},{"key":"B24","author":"Radford","year":""},{"key":"B25","article-title":"Xlnet: Generalized autoregressive pretraining for language understanding","author":"Yang","year":""},{"key":"B26","doi-asserted-by":"publisher","first-page":"103982","DOI":"10.1016\/j.jbi.2021.103982","article-title":"AMMU: A survey of transformer-based biomedical pretrained language models","volume":"126","author":"Kalyan","year":"2022","journal-title":"J Biomed Inform"},{"key":"B27","doi-asserted-by":"publisher","first-page":"e37984","DOI":"10.2196\/37984","article-title":"Discovering long COVID symptom patterns: association rule mining and sentiment analysis in social media tweets","volume":"6","author":"Matharaarachchi","year":"2022","journal-title":"JMIR Form Res"},{"key":"B28","doi-asserted-by":"publisher","first-page":"e33678","DOI":"10.2196\/33678","article-title":"Classification of twitter vaping discourse using BERTweet: comparative deep learning study","volume":"10","author":"Baker","year":"2022","journal-title":"JMIR Med Inform"},{"key":"B29","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1007\/s13278-023-01025-8","article-title":"Identifying covid-19 english informative tweets using limited labelled data","volume":"13","author":"Kothuru","year":"2023","journal-title":"Soc Netw Anal Min"},{"key":"B30","author":"Kaufmann","year":""},{"key":"B31","author":"\u00d6zberk","year":""},{"key":"B32","author":"Wolf","year":""},{"key":"B33","first-page":"1","article-title":"Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning","volume":"18","author":"Lema\u00eetre","year":"2017","journal-title":"J Mach Learn Res"},{"key":"B34","year":""},{"key":"B35","article-title":"Attention is all you need","author":"Vaswani","year":""},{"key":"B36","author":"Nguyen","year":""},{"key":"B37","doi-asserted-by":"publisher","first-page":"1023281","DOI":"10.3389\/frai.2023.1023281","article-title":"COVID-Twitter-BERT: A natural language processing model to analyse COVID-19 content on twitter","volume":"6","author":"M\u00fcller","year":"2023","journal-title":"Front Artif Intell"},{"key":"B38","author":"Mapes","year":""},{"key":"B39","author":"Kingma","year":""},{"key":"B40","author":"Mosbach","year":""},{"key":"B41","doi-asserted-by":"publisher","first-page":"524","DOI":"10.14745\/ccdr.v47i12a03","article-title":"Among sheeples and antivaxxers: Social media responses to COVID-19 vaccine news posted by canadian news organizations, and recommendations to counter vaccine hesitancy","volume":"47","author":"Tang","year":"2021","journal-title":"Can Commun Dis Rep"},{"key":"B42","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1017\/S0008423920000311","article-title":"A rare moment of cross-partisan consensus: elite and public response to the COVID-19 pandemic in Canada","volume":"53","author":"Merkley","year":"2020","journal-title":"Can J Polit Sci"},{"key":"B43","doi-asserted-by":"publisher","first-page":"e21646","DOI":"10.2196\/21646","article-title":"COVID-19 and the gendered use of emojis on twitter: infodemiology study","volume":"22","author":"Al-Rawi","year":"2020","journal-title":"J Med Internet Res"},{"key":"B44","doi-asserted-by":"publisher","first-page":"e2137257","DOI":"10.1001\/jamanetworkopen.2021.37257","article-title":"Global percentage of asymptomatic SARS-CoV-2 infections among the tested population and individuals with confirmed COVID-19 diagnosis: a systematic review and meta-analysis","volume":"4","author":"Ma","year":"2021","journal-title":"JAMA Netw Open"},{"key":"B45","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1007\/s40726-020-00161-5","article-title":"Making waves perspectives of modelling and monitoring of SARS-CoV-2 in aquatic environment for COVID-19 pandemic","volume":"6","author":"Kumar","year":"2020","journal-title":"Curr Pollut Rep"},{"key":"B46","doi-asserted-by":"publisher","DOI":"10.20944\/preprints202302.0078.v1","article-title":"Real-time epidemiology and acute care need monitoring and forecasting for COVID-19 via Bayesian Sequential Monte Carlo-leveraged transmission models","author":"Li","year":"2023","journal-title":"Preprints"}],"container-title":["Frontiers in Digital Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2023.1203874\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T12:05:52Z","timestamp":1687953952000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2023.1203874\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,28]]},"references-count":46,"alternative-id":["10.3389\/fdgth.2023.1203874"],"URL":"https:\/\/doi.org\/10.3389\/fdgth.2023.1203874","relation":{},"ISSN":["2673-253X"],"issn-type":[{"value":"2673-253X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,28]]},"article-number":"1203874"}}