{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T19:51:58Z","timestamp":1779306718296,"version":"3.51.4"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,2,18]],"date-time":"2020-02-18T00:00:00Z","timestamp":1581984000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,2,18]],"date-time":"2020-02-18T00:00:00Z","timestamp":1581984000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007192","name":"Rijksinstituut voor Volksgezondheid en Milieu","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100007192","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>We developed a system to automatically classify stance towards vaccination in Twitter messages, with a focus on messages with a negative stance. Such a system makes it possible to monitor the ongoing stream of messages on social media, offering actionable insights into public hesitance with respect to vaccination. At the moment, such monitoring is done by means of regular sentiment analysis with a poor performance on detecting negative stance towards vaccination. For Dutch Twitter messages that mention vaccination-related key terms, we annotated their stance and feeling in relation to vaccination (provided that they referred to this topic). Subsequently, we used these coded data to train and test different machine learning set-ups. With the aim to best identify messages with a negative stance towards vaccination, we compared set-ups at an increasing dataset size and decreasing reliability, at an increasing number of categories to distinguish, and with different classification algorithms.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>We found that Support Vector Machines trained on a combination of strictly and laxly labeled data with a more fine-grained labeling yielded the best result, at an F1-score of 0.36 and an Area under the ROC curve of 0.66, considerably outperforming the currently used sentiment analysis that yielded an F1-score of 0.25 and an Area under the ROC curve of 0.57. We also show that the recall of our system could be optimized to 0.60 at little loss of precision.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>The outcomes of our study indicate that stance prediction by a computerized system only is a challenging task. Nonetheless, the model showed sufficient recall on identifying negative tweets so as to reduce the manual effort of reviewing messages. Our analysis of the data and behavior of our system suggests that an approach is needed in which the use of a larger training dataset is combined with a setting in which a human-in-the-loop provides the system with feedback on its predictions.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-020-1046-y","type":"journal-article","created":{"date-parts":[[2020,2,18]],"date-time":"2020-02-18T18:03:01Z","timestamp":1582048981000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["Monitoring stance towards vaccination in twitter messages"],"prefix":"10.1186","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1932-3200","authenticated-orcid":false,"given":"Florian","family":"Kunneman","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mattijs","family":"Lambooij","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Albert","family":"Wong","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Antal van den","family":"Bosch","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liesbeth","family":"Mollema","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,2,18]]},"reference":[{"issue":"11","key":"1046_CR1","doi-asserted-by":"publisher","first-page":"14118","DOI":"10.1371\/journal.pone.0014118","volume":"5","author":"C Chew","year":"2010","unstructured":"Chew C, Eysenbach G. Pandemics in the age of twitter: content analysis of tweets during the 2009 h1n1 outbreak. PLoS ONE. 2010; 5(11):14118.","journal-title":"PLoS ONE"},{"issue":"10","key":"1046_CR2","doi-asserted-by":"publisher","first-page":"1002199","DOI":"10.1371\/journal.pcbi.1002199","volume":"7","author":"M Salath\u00e9","year":"2011","unstructured":"Salath\u00e9 M, Khandelwal S. Assessing vaccination sentiments with online social media: implications for infectious disease dynamics and control. PLoS Comput Biol. 2011; 7(10):1002199.","journal-title":"PLoS Comput Biol"},{"key":"1046_CR3","doi-asserted-by":"publisher","unstructured":"Du J, Xu J, Song H, Liu X, Tao C. Optimization on machine learning based approaches for sentiment analysis on hpv vaccines related tweets. J Biomed Semant. 2017; 8(1). https:\/\/doi.org\/10.1186\/s13326-017-0120-6.","DOI":"10.1186\/s13326-017-0120-6"},{"issue":"12","key":"1046_CR4","doi-asserted-by":"publisher","first-page":"318","DOI":"10.2196\/jmir.6670","volume":"18","author":"PM Massey","year":"2016","unstructured":"Massey PM, Leader A, Yom-Tov E, Budenz A, Fisher K, Klassen AC. Applying multiple data collection tools to quantify human papillomavirus vaccine communication on twitter. J Med Internet Res. 2016; 18(12):318.","journal-title":"J Med Internet Res"},{"issue":"7","key":"1046_CR5","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1016\/S1473-3099(13)70108-7","volume":"13","author":"HJ Larson","year":"2013","unstructured":"Larson HJ, Smith DM, Paterson P, Cumming M, Eckersberger E, Freifeld CC, Ghinai I, Jarrett C, Paushter L, Brownstein JS, et al. Measuring vaccine confidence: analysis of data obtained by a media surveillance system used to analyse public concerns about vaccines. The Lancet Infect Dis. 2013; 13(7):606\u201313.","journal-title":"The Lancet Infect Dis"},{"key":"1046_CR6","doi-asserted-by":"crossref","unstructured":"Linge JP, Steinberger R, Weber TP, Yangarber R, van der Goot E, Al Khudhairy DH, Stilianakis NI. Internet surveillance systems for early alerting of health threats. Eurosurveillance. 2009; 14(13).","DOI":"10.2807\/ese.14.13.19162-en"},{"issue":"5","key":"1046_CR7","doi-asserted-by":"publisher","first-page":"1553","DOI":"10.1016\/j.foodres.2010.04.009","volume":"43","author":"A Rortais","year":"2010","unstructured":"Rortais A, Belyaeva J, Gemo M, Van der Goot E, Linge JP. Medisys: An early-warning system for the detection of (re-) emerging food-and feed-borne hazards. Food Res Int. 2010; 43(5):1553\u20136.","journal-title":"Food Res Int"},{"issue":"50","key":"1046_CR8","doi-asserted-by":"publisher","first-page":"6166","DOI":"10.1016\/j.vaccine.2016.11.007","volume":"34","author":"BFH Becker","year":"2016","unstructured":"Becker BFH, Larson HJ, Bonhoeffer J, van Mulligen EM, Kors JA, Sturkenboom MCJM. Evaluation of a multinational, multilingual vaccine debate on twitter. Vaccine. 2016; 34(50):6166\u201371.","journal-title":"Vaccine"},{"key":"1046_CR9","volume-title":"Proceedings of the AAAI Joint Workshop on Health Intelligence (W3PHIAI)","author":"X Huang","year":"2017","unstructured":"Huang X, Smith MC, Paul MJ, Ryzhkov D, Quinn SC, Broniatowski DA, Dredze M. Examining patterns of influenza vaccination in social media. In: Proceedings of the AAAI Joint Workshop on Health Intelligence (W3PHIAI). San Francisco: AAAI: 2017."},{"key":"1046_CR10","doi-asserted-by":"publisher","first-page":"4494","DOI":"10.1016\/j.vaccine.2017.07.029","volume":"35","author":"F Aquino","year":"2017","unstructured":"Aquino F, Donzelli G, De Franco E, Privitera G, Lopalco PL, Carducci A. The web and public confidence in mmr vaccination in Italy. Vaccine. 2017; 35:4494\u20138.","journal-title":"Vaccine"},{"issue":"1","key":"1046_CR11","doi-asserted-by":"publisher","first-page":"13963","DOI":"10.1038\/s41598-018-32029-6","volume":"8","author":"M Wagner","year":"2018","unstructured":"Wagner M, Lampos V, Cox IJ, Pebody R. The added value of online user-generated content in traditional methods for influenza surveillance. Sci Rep. 2018; 8(1):13963.","journal-title":"Sci Rep"},{"key":"1046_CR12","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1007\/978-3-642-15939-8_42","volume-title":"Machine Learning and Knowledge Discovery in Databases","author":"Vasileios Lampos","year":"2010","unstructured":"Lampos V, De Bie T, Cristianini N. Flu detector-tracking epidemics on twitter. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer: 2010. p. 599\u2013602. https:\/\/doi.org\/10.1007\/978-3-642-15939-8_42."},{"issue":"10","key":"1046_CR13","doi-asserted-by":"publisher","first-page":"e236","DOI":"10.2196\/jmir.3416","volume":"16","author":"Ruchit Nagar","year":"2014","unstructured":"Nagar R, Yuan Q, Freifeld CC, Santillana M, Nojima A, Chunara R, Brownstein JS. A case study of the New York City 2012-2013 influenza season with daily geocoded twitter data from temporal and spatiotemporal perspectives. J Med Internet Res. 2014; 16(10). https:\/\/doi.org\/10.2196\/jmir.3416.","journal-title":"Journal of Medical Internet Research"},{"issue":"7","key":"1046_CR14","doi-asserted-by":"publisher","first-page":"69305","DOI":"10.1371\/journal.pone.0069305","volume":"8","author":"E-K Kim","year":"2013","unstructured":"Kim E-K, Seok JH, Oh JS, Lee HW, Kim KH. Use of hangeul twitter to track and predict human influenza infection. PLoS ONE. 2013; 8(7):69305.","journal-title":"PLoS ONE"},{"issue":"5","key":"1046_CR15","doi-asserted-by":"publisher","first-page":"19467","DOI":"10.1371\/journal.pone.0019467","volume":"6","author":"A Signorini","year":"2011","unstructured":"Signorini A, Segre AM, Polgreen PM. The use of twitter to track levels of disease activity and public concern in the us during the influenza a h1n1 pandemic. PLoS ONE. 2011; 6(5):19467.","journal-title":"PLoS ONE"},{"issue":"4","key":"1046_CR16","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1177\/0267323113486235","volume":"28","author":"PLM Vasterman","year":"2013","unstructured":"Vasterman PLM, Ruigrok N. Pandemic alarm in the dutch media: Media coverage of the 2009 influenza a (h1n1) pandemic and the role of the expert sources. Eur J Commun. 2013; 28(4):436\u201353.","journal-title":"Eur J Commun"},{"issue":"5","key":"1046_CR17","doi-asserted-by":"publisher","first-page":"e128","DOI":"10.2196\/jmir.3863","volume":"17","author":"Liesbeth Mollema","year":"2015","unstructured":"Mollema L, Harmsen IA, Broekhuizen E, Clijnk R, De Melker H, Paulussen T, Kok G, Ruiter R, Das E. Disease detection or public opinion reflection? content analysis of tweets, other social media, and online newspapers during the measles outbreak in the netherlands in 2013. J Med Internet Res. 2015; 17(5). https:\/\/doi.org\/10.2196\/jmir.3863.","journal-title":"Journal of Medical Internet Research"},{"key":"1046_CR18","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1016\/j.future.2016.06.032","volume":"66","author":"G Bello-Orgaz","year":"2017","unstructured":"Bello-Orgaz G, Hernandez-Castro J, Camacho D. Detecting discussion communities on vaccination in twitter. Future Gener Comput Syst. 2017; 66:125\u201336.","journal-title":"Future Gener Comput Syst"},{"issue":"29","key":"1046_CR19","doi-asserted-by":"publisher","first-page":"3621","DOI":"10.1016\/j.vaccine.2017.05.052","volume":"35","author":"GJ Kang","year":"2017","unstructured":"Kang GJ, Ewing-Nelson SR, Mackey L, Schlitt JT, Marathe A, Abbas KM, Swarup S. Semantic network analysis of vaccine sentiment in online social media. Vaccine. 2017; 35(29):3621\u201338.","journal-title":"Vaccine"},{"issue":"2","key":"1046_CR20","doi-asserted-by":"publisher","first-page":"e166","DOI":"10.2196\/publichealth.6586","volume":"2","author":"Timothy R Tangherlini","year":"2016","unstructured":"Tangherlini TR, Roychowdhury V, Glenn B, Crespi CM, Bandari R, Wadia A, Falahi M, Ebrahimzadeh E, Bastani R. \u201cmommy blogs\u201d and the vaccination exemption narrative: results from a machine-learning approach for story aggregation on parenting social media sites. JMIR Publ Health Surveill. 2016; 2(2). https:\/\/doi.org\/10.2196\/publichealth.6586.","journal-title":"JMIR Public Health and Surveillance"},{"key":"1046_CR21","first-page":"993","volume":"3","author":"DM Blei","year":"2003","unstructured":"Blei DM, Ng AY, Jordan MI. Latent dirichlet allocation. J Mach Learn Res. 2003; 3:993\u20131022.","journal-title":"J Mach Learn Res"},{"issue":"8","key":"1046_CR22","doi-asserted-by":"publisher","first-page":"e232","DOI":"10.2196\/jmir.6045","volume":"18","author":"Didi Surian","year":"2016","unstructured":"Surian D, Nguyen DQ, Kennedy G, Johnson M, Coiera E, Dunn AG. Characterizing twitter discussions about hpv vaccines using topic modeling and community detection. J Med Internet Res. 2016; 18(8). https:\/\/doi.org\/10.2196\/jmir.6045.","journal-title":"Journal of Medical Internet Research"},{"key":"1046_CR23","first-page":"121","volume":"3","author":"E Sang","year":"2013","unstructured":"Tjong K, Sang E, van den Bosch A. Dealing with big data: The case of twitter. Comput Linguist Neth J. 2013; 3:121\u201334.","journal-title":"Comput Linguist Neth J"},{"issue":"1","key":"1046_CR24","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1080\/19312450709336664","volume":"1","author":"AF Hayes","year":"2007","unstructured":"Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Measures. 2007; 1(1):77\u201389.","journal-title":"Commun Methods Measures"},{"key":"1046_CR25","volume-title":"Proceedings of Recent Advances in Slavonic Natural Language Processing","author":"V Kov\u00e1r","year":"2014","unstructured":"Kov\u00e1r V, Rychl\u00fd P, Jakub\u00edcek M. Low inter-annotator agreement=an ill-defined problem? In: Proceedings of Recent Advances in Slavonic Natural Language Processing. Brno: NLP Consulting: 2014. p. 57\u201362."},{"key":"1046_CR26","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"K Krippendorff","year":"2004","unstructured":"Krippendorff K. Content Analysis: An Introduction to Its Methodology. Thousand Oaks: SAGE Publications; 2004."},{"issue":"3","key":"1046_CR27","first-page":"385","volume":"69","author":"DJ Hand","year":"2001","unstructured":"Hand DJ, Yu K. Idiot\u2019s bayes\u2014not so stupid after all?Int Stat Rev. 2001; 69(3):385\u201398.","journal-title":"Int Stat Rev"},{"issue":"4","key":"1046_CR28","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/5254.708428","volume":"13","author":"MA Hearst","year":"1998","unstructured":"Hearst MA, Dumais ST, Osuna E, Platt J, Scholkopf B. Support vector machines. IEEE Intell Syst Appl. 1998; 13(4):18\u201328.","journal-title":"IEEE Intell Syst Appl"},{"key":"1046_CR29","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, et al. Scikit-learn: Machine learning in python. J Mach Learn Res. 2011; 12:2825\u201330.","journal-title":"J Mach Learn Res"},{"key":"1046_CR30","first-page":"2063","volume":"13","author":"TD Smedt","year":"2012","unstructured":"Smedt TD, Daelemans W. Pattern for python. J Mach Learn Res. 2012; 13:2063\u20137.","journal-title":"J Mach Learn Res"},{"key":"1046_CR31","first-page":"45","volume":"2","author":"S Tong","year":"2001","unstructured":"Tong S, Koller D. Support vector machine active learning with applications to text classification. J Mach Learn Res. 2001; 2:45\u201366.","journal-title":"J Mach Learn Res"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1046-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12911-020-1046-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1046-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,15]],"date-time":"2022-10-15T22:17:51Z","timestamp":1665872271000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-020-1046-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,18]]},"references-count":31,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["1046"],"URL":"https:\/\/doi.org\/10.1186\/s12911-020-1046-y","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,18]]},"assertion":[{"value":"13 September 2018","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 February 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 February 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This research is based on the textual content of Twitter messages, which were collected according to the Twitter developer policy. In line with this policy, we only share the ID\u2019s of these tweets as a dataset, thereby safeguarding the privacy of the users. No other personal information has been collected as part of this research. The European GDPR legislation applies to any research conducted in the Netherlands. The current study complies with this legislation.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"33"}}