{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T13:19:49Z","timestamp":1771075189387,"version":"3.50.1"},"reference-count":60,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2022,11,16]],"date-time":"2022-11-16T00:00:00Z","timestamp":1668556800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["UL1TR003017"],"award-info":[{"award-number":["UL1TR003017"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["NRF-2017S1A3A2066084"],"award-info":[{"award-number":["NRF-2017S1A3A2066084"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Research Foundation of Korea","award":["UL1TR003017"],"award-info":[{"award-number":["UL1TR003017"]}]},{"name":"National Research Foundation of Korea","award":["NRF-2017S1A3A2066084"],"award-info":[{"award-number":["NRF-2017S1A3A2066084"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Since the start of the COVID-19 pandemic, government authorities have responded by issuing new public health policies, many of which were intended to contain its spread but ended up limiting economic and social activities. The citizen responses to these policies are diverse, ranging from goodwill to fear and anger. It is challenging to determine whether or not these public health policies achieved the intended impact. This requires systematic data collection and scientific studies, which can be very time-consuming. To overcome such challenges, in this paper, we provide an alternative approach to continuously monitor and dynamically make sense of how public health policies impact citizens. Our approach is to continuously collect Twitter posts related to COVID-19 policies and to analyze the public reactions. We have developed a web-based system that collects tweets daily and generates timelines and geographical displays of citizens\u2019 \u201cconcern levels\u201d. Tracking the public reactions towards different policies can help government officials assess the policy impacts in a more dynamic and real-time manner. For this paper, we collected and analyzed over 16 million tweets related to ten policies over a 10-month period. We obtained several findings; for example, the \u201cCOVID-19 (General)\u201d and \u201dVentilators\u201d policies engendered the highest concern levels, while the \u201cFace Coverings\u201d policy caused the lowest. Nine out of ten policies exhibited significant changes in concern levels during the observation period.<\/jats:p>","DOI":"10.3390\/info13110543","type":"journal-article","created":{"date-parts":[[2022,11,16]],"date-time":"2022-11-16T02:33:48Z","timestamp":1668566028000},"page":"543","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Public Health Policy Monitoring through Public Perceptions: A Case of COVID-19 Tweet Analysis"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7983-634X","authenticated-orcid":false,"given":"Chih-yuan","family":"Li","sequence":"first","affiliation":[{"name":"New Jersey Institute of Technology, Newark, NJ 07102, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Renda","sequence":"additional","affiliation":[{"name":"New Jersey Institute of Technology, Newark, NJ 07102, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fatima","family":"Yusuf","sequence":"additional","affiliation":[{"name":"New Jersey Institute of Technology, Newark, NJ 07102, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James","family":"Geller","sequence":"additional","affiliation":[{"name":"New Jersey Institute of Technology, Newark, NJ 07102, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Soon Ae","family":"Chun","sequence":"additional","affiliation":[{"name":"College of Staten Island, City University of New York, Staten Island, NY 10314, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,16]]},"reference":[{"key":"ref_1","unstructured":"The World Health Organization (WHO) (2022, January 03). Characterizes COVID-19 as a Pandemic. Available online: https:\/\/www.who.int\/emergencies\/diseases\/novel-coronavirus-2019\/events-as-they-happen."},{"key":"ref_2","unstructured":"Agence France-Presse (2022, March 30). China Locks down City of 9 Million and Reports 4,000 Cases as Omicron Tests Zero-Covid Strategy. The Guardian, 2022. Available online: https:\/\/www.theguardian.com\/world\/2022\/mar\/22\/china-locks-down-city-of-9-million-and-reports-4000-cases-as-omicron-tests-zero-covid-strategy."},{"key":"ref_3","unstructured":"Tsui, K. (2022, August 20). Shanghai Ikea Lockdown Sparks Scramble as China Enforces \u2018Zero COVID\u2019. Washington Post, 2022. Available online: https:\/\/www.washingtonpost.com\/world\/2022\/08\/15\/shanghai-ikea-lockdown-china-zero-covid\/."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/1758-5899.13125","article-title":"Making global public policy work: A survey of international organization effectiveness","volume":"13","author":"Coen","year":"2022","journal-title":"Glob. Policy"},{"key":"ref_5","unstructured":"(2021, March 08). COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). Available online: https:\/\/coronavirus.jhu.edu\/map.html."},{"key":"ref_6","unstructured":"(2021, March 08). Coronavirus in the U.S.: Latest Map and Case Count. Available online: https:\/\/www.nytimes.com\/interactive\/2020\/us\/coronavirus-us-cases.html?action=click&module=Spotlight&pgtype=Homepage."},{"key":"ref_7","unstructured":"(2021, March 08). New Jersey COVID-19 Information Hub, Available online: https:\/\/covid19.nj.gov\/index.html."},{"key":"ref_8","unstructured":"Sayce, D. (2021, December 15). The Number of Tweets per Day in 2020. Available online: https:\/\/www.dsayce.com\/social-media\/tweets-day\/."},{"key":"ref_9","unstructured":"Stoy, L. (2022, February 02). Sentiment Analysis: A Deep Dive Into The Theory, Methods, and Applications. Available online: https:\/\/lazarinastoy.com\/sentiment-analysis-theory-methods-applications\/."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Asur, S., and Huberman, B. (September, January 31). Predicting the Future with Social Media. Proceedings of the 2010 IEEE\/WIC\/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Toronto, ON, Canada.","DOI":"10.1109\/WI-IAT.2010.63"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"F\u00fcrnkranz, J., Scheffer, T., and Spiliopoulou, M. (2006). Efficient Convolution Kernels for Dependency and Constituent Syntactic Trees. Machine Learning: ECML 2006, Springer. Lecture Notes in Computer Science.","DOI":"10.1007\/11871842"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T., and Qin, B. (2014, January 22\u201327). Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, MD, USA.","DOI":"10.3115\/v1\/P14-1146"},{"key":"ref_13","unstructured":"Singh, L.G., Mitra, A., and Singh, S.R. (2020, January 16\u201320). Sentiment Analysis of Tweets Using Heterogeneous Multi-layer Network Representation and Embedding. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, Online."},{"key":"ref_14","unstructured":"Davisson, M. (2021, April 23). Staff Budget Briefing, FY 2017-18; Department of Public Health and Environment, 2016, Available online: https:\/\/leg.colorado.gov\/sites\/default\/files\/fy2017-18_pubheabrf.pdf."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"101408","DOI":"10.1016\/j.aei.2021.101408","article-title":"An approach to user knowledge acquisition in product design","volume":"50","author":"Tan","year":"2021","journal-title":"Adv. Eng. Inform."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"7653","DOI":"10.1016\/j.eswa.2014.06.009","article-title":"Text mining for market prediction: A systematic review","volume":"41","author":"Nassirtoussi","year":"2014","journal-title":"Expert Syst. Appl."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.knosys.2014.05.008","article-title":"PoliTwi: Early detection of emerging political topics on twitter and the impact on concept-level sentiment analysis","volume":"69","author":"Rill","year":"2014","journal-title":"Knowl.-Based Syst."},{"key":"ref_18","unstructured":"Luo, Y. (2014). Analysis and Visualization of Sentiment and Emotion on Crisis Tweets. Cooperative Design, Visualization, and Engineering, Springer International Publishing."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4065","DOI":"10.1016\/j.eswa.2013.01.001","article-title":"Ontology-based sentiment analysis of twitter posts","volume":"40","author":"Kontopoulos","year":"2013","journal-title":"Expert Syst. Appl."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1016\/j.chb.2016.07.035","article-title":"Looking for the perfect tweet. The use of data mining techniques to find influencers on twitter","volume":"64","year":"2016","journal-title":"Comput. Hum. Behav."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ji, X., Chun, S.A., and Geller, J. (2013). Monitoring Public Health Concerns Using Twitter Sentiment Classifications. IEEE International Conference on Healthcare Informatics, IEEE.","DOI":"10.1109\/ICHI.2013.47"},{"key":"ref_22","unstructured":"OECD (2021). The territorial impact of COVID-19: Managing the crisis and recovery across levels of government. OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing."},{"key":"ref_23","unstructured":"(2021, June 15). Colorado Department of Public Health and Environment, Available online: https:\/\/cdphe.colorado.gov\/."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"e2033484","DOI":"10.1001\/jamanetworkopen.2020.33484","article-title":"Public Concern About Violence, Firearms, and the COVID-19 Pandemic in California","volume":"4","author":"Aubel","year":"2021","journal-title":"JAMA Netw. Open"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.jad.2020.06.046","article-title":"An analysis on the panic during COVID-19 pandemic through an online form","volume":"276","author":"Nicomedesa","year":"2020","journal-title":"J. Affect. Disord."},{"key":"ref_26","unstructured":"Centers for Disease Control and Prevention (2021, April 24). COVID Data Tracker, Available online: https:\/\/covid.cdc.gov\/covid-data-tracker\/#datatracker-home."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1007\/s13278-021-00828-x","article-title":"Identification of affective valence of Twitter generated sentiments during the COVID-19 outbreak","volume":"11","author":"Mittal","year":"2021","journal-title":"Soc. Netw. Anal. Min."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Hutto, C.J., and Gilbert, E. (2014, January 1\u20134). VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Proceedings of the International AAAI Conference on Web and Social Media. 8, Ann Arbor, MI, USA. Available online: https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/14550.","DOI":"10.1609\/icwsm.v8i1.14550"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"e22590","DOI":"10.2196\/22590","article-title":"Social Network Analysis of COVID-19 Sentiments: Application of Artificial Intelligence","volume":"22","author":"Hung","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1145\/3475867","article-title":"A Language-independent Network to Analyze the Impact of COVID-19 on the World via Sentiment Analysis","volume":"22","author":"Yadav","year":"2022","journal-title":"ACM Trans. Internet Technol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"126684","DOI":"10.1109\/ACCESS.2021.3111833","article-title":"Senti-COVID19: An Interactive Visual Analytics System for Detecting Public Sentiment and Insights Regarding COVID-19 From Social Media","volume":"9","author":"Yu","year":"2021","journal-title":"IEEE Access"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"107242","DOI":"10.1016\/j.knosys.2021.107242","article-title":"A novel fusion-based deep learning model for sentiment analysis of COVID-19 tweets","volume":"228","author":"Basiri","year":"2021","journal-title":"Knowl.-Based Syst."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"992","DOI":"10.1109\/TCSS.2020.3042446","article-title":"Sentiment Analysis of Lockdown in India During COVID-19: A Case Study on Twitter","volume":"8","author":"Gupta","year":"2021","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1003","DOI":"10.1109\/TCSS.2021.3051189","article-title":"COVIDSenti: A Large-Scale Benchmark Twitter Data Set for COVID-19 Sentiment Analysis","volume":"8","author":"Naseem","year":"2021","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"key":"ref_35","unstructured":"Chen, E., Lerman, K., and Ferrara, E. (2020). COVID-19: The first public coronavirus twitter dataset. arXiv."},{"key":"ref_36","unstructured":"Lopez, C.E., Vasu, M., and Gallemore, C. (2020). Understanding the perception of COVID-19 policies by mining a multilanguage twitter dataset. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3428090","article-title":"Tweeting During the COVID-19 Pandemic: Sentiment Analysis of Twitter Messages by President Trump","volume":"2","author":"Yaqub","year":"2021","journal-title":"Digit. Gov. Res. Pract."},{"key":"ref_38","unstructured":"National Institutes of Health (2022, November 05). Office of Data Science Strategy, Available online: https:\/\/datascience.nih.gov\/COVID-19-open-access-resources."},{"key":"ref_39","unstructured":"European Centre for Disease Prevention and Control (2022, November 06). Download Historical Data (to 14 December 2020) on the Daily Number of New Reported COVID-19 Cases and Deaths Worldwide. Available online: https:\/\/www.ecdc.europa.eu\/en\/publications-data\/download-todays-data-geographic-distribution-COVID-19-cases-worldwide."},{"key":"ref_40","unstructured":"(2021, June 01). Twitter Developer Platform. Available online: https:\/\/developer.twitter.com\/en\/docs."},{"key":"ref_41","unstructured":"Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., and Potts, C. (2013, January 18\u201321). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA. Available online: https:\/\/aclanthology.org\/D13-1170."},{"key":"ref_42","unstructured":"Loria, S. (2021, December 15). TextBlob Documentation. Release 0.16.0. Available online: https:\/\/buildmedia.readthedocs.org\/media\/pdf\/TextBlob\/latest\/TextBlob.pdf."},{"key":"ref_43","unstructured":"(2022, February 28). COVID 19 Daily Trends\u2014New Cases, Deaths by State and Counties. Available online: https:\/\/www.makeready.com\/covid\/."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1177\/001316447003000105","article-title":"Estimating the Reliability, Systematic Error and Random Error of Interval Data","volume":"30","author":"Krippendorff","year":"1970","journal-title":"Educ. Psychol. Meas."},{"key":"ref_45","unstructured":"(2022, March 01). ReCal for Nominal, Ordinal, Interval, and Ratio-Level Data. Available online: http:\/\/dfreelon.org\/recal\/recal-oir.php."},{"key":"ref_46","unstructured":"Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology, Sage Publications. [2nd ed.]."},{"key":"ref_47","unstructured":"(2022, October 20). Sentiment Analysis without Modeling: TextBlob vs. VADER vs. Flair. Available online: https:\/\/pub.towardsai.net\/sentiment-analysis-without-modeling-TextBlob-vs-vader-vs-flair-657b7af855f4."},{"key":"ref_48","unstructured":"(2021, December 15). Free Statistics Calculators, Calculator: P-Value for Correlation Coefficients. Available online: https:\/\/www.danielsoper.com\/statcalc\/calculator.aspx?id=44."},{"key":"ref_49","unstructured":"IMF (2022, September 01). Policy Tracker. Available online: https:\/\/www.imf.org\/en\/Topics\/imf-and-covid19\/Policy-Responses-to-COVID-19."},{"key":"ref_50","unstructured":"(2022, March 01). COVID-19: Local Action Tracker. Available online: https:\/\/www.nlc.org\/resource\/COVID-19-local-action-tracker\/."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"ref_52","unstructured":"Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P.J. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv."},{"key":"ref_53","unstructured":"Barrios, F., L\u00f3pez, F., Argerich, L., and Wachenchauzer, R. (2016). Variations of the Similarity Function of TextRank for Automated Summarization. arXiv."},{"key":"ref_54","unstructured":"Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2022, June 15). Language Models are Unsupervised Multitask Learners. Available online: https:\/\/openai.com\/blog\/better-language-models\/."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Wu, H., Huang, J., Zhang, C.J.P., He, Z., and Ming, W. (2020). Facemask shortage and the coronavirus disease (COVID-19) outbreak: Reflection on public health measures. medRxiv.","DOI":"10.1101\/2020.02.11.20020735"},{"key":"ref_56","first-page":"100098","article-title":"Recreational and philanthropic sectors are the worst-hit US industries in the COVID-19 aftermath","volume":"3","author":"Roy","year":"2021","journal-title":"Soc. Sci. Humanit. Open"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"353","DOI":"10.3389\/fpsyt.2021.626197","article-title":"Adjustment to a \u201cNew Normal:\u201d Coping Flexibility and Mental Health Issues During the COVID-19 Pandemic","volume":"12","author":"Cheng","year":"2021","journal-title":"Front. Psychiatry"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Chun, S.A., Li, C., Toliyat, A., and Geller, J. (2020, January 15\u201319). Tracking Citizen\u2019s Concerns During COVID-19 Pandemic. Proceedings of the 21st Annual International Conference on Digital Government Research, New York, NY, USA.","DOI":"10.1145\/3396956.3397000"},{"key":"ref_59","unstructured":"(2022, November 06). Black Swan in the Stock Market: What Is It, With Examples and History. Available online: https:\/\/www.investopedia.com\/terms\/b\/blackswan.asp."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"103095","DOI":"10.1016\/j.ipm.2022.103095","article-title":"Investigating the COVID-19 vaccine discussions on Twitter through a multilayer network-based approach","volume":"59","author":"Bonifazi","year":"2022","journal-title":"Inf. Process. Manag."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/11\/543\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:19:08Z","timestamp":1760145548000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/11\/543"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,16]]},"references-count":60,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["info13110543"],"URL":"https:\/\/doi.org\/10.3390\/info13110543","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,16]]}}}