{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T19:21:43Z","timestamp":1757618503380,"version":"3.44.0"},"reference-count":72,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T00:00:00Z","timestamp":1753228800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T00:00:00Z","timestamp":1753228800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001839","name":"University Grants Committee","doi-asserted-by":"publisher","award":["Research Matching Grant Scheme"],"award-info":[{"award-number":["Research Matching Grant Scheme"]}],"id":[{"id":"10.13039\/501100001839","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001747","name":"Hong Kong Baptist University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001747","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Comput Soc Sc"],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Despite the significant advancements of Large Language Models (LLMs) in recent years, tokenization remains a critical step in Natural Language Processing (NLP) for social scientific research. This study presents a simple but effective approach to enhance tokenization accuracy in segmenting user-generated content (UGC) in logographic languages, such as Chinese. Existing tokenization techniques often struggle to effectively handle the complexities of UGC on digital platforms, which include informal language, slang, and newly coined terms. To address this challenge, we developed a dynamic tokenization model that incorporates cumulative logic to recognize and adapt to evolving linguistic patterns in social media content. By analyzing large online discussion datasets from LIHKG, a Reddit-like forum in Hong Kong, the model\u2019s effectiveness is demonstrated through its ability to accurately segment domain-specific terms and novel expressions over time. Our results show that the model outperforms traditional tokenizers in recognizing contextually relevant tokens. This innovative approach offers practical advantages for analyzing large-scale UGC data, and has the potential to improve the performance of downstream NLP tasks.<\/jats:p>","DOI":"10.1007\/s42001-025-00406-7","type":"journal-article","created":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T12:16:45Z","timestamp":1753273005000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Enhancing tokenization accuracy with dynamic patterns: cumulative logic for segmenting user-generated content in logographic languages"],"prefix":"10.1007","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9647-2269","authenticated-orcid":false,"given":"Yin","family":"Zhang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhihuai","family":"Lin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Castiel Chi-chiu","family":"Tong","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sam Wai-yeung","family":"Ho","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,7,23]]},"reference":[{"key":"406_CR1","unstructured":"Barbarestani, B., Maks, I., & Vossen, P. T. (2024). Content moderation in online platforms: A study of annotation methods for inappropriate language. Proceedings of the Fourth Workshop on Threat, Aggression & Cyberbullying@ LREC-COLING-2024, 96\u2013104."},{"issue":"4","key":"406_CR2","doi-asserted-by":"publisher","first-page":"663","DOI":"10.1162\/coli_a_00333","volume":"44","author":"F Benamara","year":"2018","unstructured":"Benamara, F., Inkpen, D., & Taboada, M. (2018). Introduction to the special issue on language in social media: Exploiting discourse and other contextual information. Computational Linguistics, 44(4), 663\u2013681.","journal-title":"Computational Linguistics"},{"key":"406_CR3","unstructured":"Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python: Analyzing text with the natural language toolkit. O\u2019Reilly Media, Inc."},{"issue":"1","key":"406_CR4","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1111\/amet.12112","volume":"42","author":"Y Bonilla","year":"2015","unstructured":"Bonilla, Y., & Rosa, J. (2015). #Ferguson: Digital protest, hashtag ethnography, and the racial politics of social media in the United States. American Ethnologist, 42(1), 4\u201317.","journal-title":"American Ethnologist"},{"issue":"2","key":"406_CR5","first-page":"193","volume":"26","author":"S Boulianne","year":"2009","unstructured":"Boulianne, S. (2009). Does internet use affect engagement? A Meta-Analysis of Research. Political Communication, 26(2), 193\u2013211.","journal-title":"A Meta-Analysis of Research. Political Communication"},{"issue":"2","key":"406_CR6","doi-asserted-by":"publisher","first-page":"497","DOI":"10.1177\/1077699018769906","volume":"95","author":"J Brummette","year":"2018","unstructured":"Brummette, J., DiStaso, M., Vafeiadis, M., & Messner, M. (2018). Read all about it: The Politicization of \u201cFake News\u201d on twitter. Journalism & Mass Communication Quarterly, 95(2), 497\u2013517.","journal-title":"Journalism & Mass Communication Quarterly"},{"key":"406_CR7","unstructured":"Bystrov, V., Naboka-Krell, V., Staszewska-Bystrova, A., & Winker, P. (2023). Analysing the impact of removing infrequent words on topic quality in LDA models (No. arXiv:2311.14505). arXiv."},{"key":"406_CR8","first-page":"755","volume":"2014","author":"L Chen","year":"2014","unstructured":"Chen, L., Tozammel Hossain, K. S. M., Butler, P., Ramakrishnan, N., & Prakash, B. A. (2014). Flu gone viral: Syndromic surveillance of flu on twitter using temporal topic models. IEEE International Conference on Data Mining, 2014, 755\u2013760.","journal-title":"IEEE International Conference on Data Mining"},{"issue":"4","key":"406_CR9","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1006\/csla.1999.0128","volume":"13","author":"SF Chen","year":"1999","unstructured":"Chen, S. F., & Goodman, J. (1999). An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4), 359\u2013394.","journal-title":"Computer Speech & Language"},{"issue":"6","key":"406_CR10","doi-asserted-by":"publisher","first-page":"696","DOI":"10.1177\/0894439314556599","volume":"33","author":"S Choi","year":"2015","unstructured":"Choi, S. (2015). The two-step flow of communication in twitter-based public forums. Social Science Computer Review, 33(6), 696\u2013711.","journal-title":"Social Science Computer Review"},{"issue":"1","key":"406_CR11","doi-asserted-by":"publisher","first-page":"16598","DOI":"10.1038\/s41598-020-73510-5","volume":"10","author":"M Cinelli","year":"2020","unstructured":"Cinelli, M., Quattrociocchi, W., Galeazzi, A., Valensise, C. M., Brugnoli, E., Schmidt, A. L., Zola, P., Zollo, F., & Scala, A. (2020). The COVID-19 social media infodemic. Scientific Reports, 10(1), 16598.","journal-title":"Scientific Reports"},{"issue":"1","key":"406_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","volume":"39","author":"AP Dempster","year":"1977","unstructured":"Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 1\u201322.","journal-title":"Journal of the Royal Statistical Society: Series B (Methodological)"},{"issue":"2","key":"406_CR13","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1017\/pan.2017.44","volume":"26","author":"MJ Denny","year":"2018","unstructured":"Denny, M. J., & Spirling, A. (2018). Text preprocessing for unsupervised learning: Why it matters, when it misleads, and what to do about It. Political Analysis, 26(2), 168\u2013189.","journal-title":"Political Analysis"},{"issue":"2","key":"406_CR14","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1111\/0023-8333.00185","volume":"52","author":"J-M Dewaele","year":"2002","unstructured":"Dewaele, J.-M., & Pavlenko, A. (2002). Emotion vocabulary in interlanguage. Language Learning, 52(2), 263\u2013322.","journal-title":"Language Learning"},{"key":"406_CR15","unstructured":"Domingo, M., Garc\u0131a-Mart\u0131nez, M., Helle, A., Casacuberta, F., & Herranz, M. (2019). How Much Does Tokenization Affect Neural Machine Translation? (arXiv:1812.08621; Version 4). arXiv."},{"key":"406_CR16","doi-asserted-by":"crossref","unstructured":"Ellis, N. C., & Larsen-Freeman, D. (2006). Language emergence: Implications for applied linguistics - Introduction to the special issue. Applied Linguistics, 27(4), 558\u2013589.","DOI":"10.1093\/applin\/aml028"},{"issue":"4","key":"406_CR17","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1177\/17480485221094123","volume":"84","author":"JN Erni","year":"2022","unstructured":"Erni, J. N., & Zhang, Y. (2022). Wild hopes: sourcing the political vocabulary of digital citizenship from the LIHKG forum. International Communication Gazette, 84(4), 349\u2013375.","journal-title":"International Communication Gazette"},{"key":"406_CR18","doi-asserted-by":"crossref","unstructured":"Goldberg, Y. (2017). Neural Network Methods for Natural Language Processing. Springer International Publishing.","DOI":"10.1007\/978-3-031-02165-7"},{"issue":"3","key":"406_CR19","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1093\/pan\/mps028","volume":"21","author":"J Grimmer","year":"2013","unstructured":"Grimmer, J., & Stewart, B. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3), 267\u2013297.","journal-title":"Political Analysis"},{"issue":"1","key":"406_CR20","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1007\/s13278-015-0258-0","volume":"5","author":"A Guille","year":"2015","unstructured":"Guille, A., & Favre, C. (2015). Event detection, tracking, and visualization in twitter: A mention-anomaly-based approach. Social Network Analysis and Mining, 5(1), 18.","journal-title":"Social Network Analysis and Mining"},{"issue":"2","key":"406_CR21","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1177\/0093650218819671","volume":"47","author":"M Hameleers","year":"2020","unstructured":"Hameleers, M., & Van Der Meer, T. G. L. A. (2020). Misinformation and polarization in a high-choice media environment: How effective are political fact-checkers? Communication Research, 47(2), 227\u2013250.","journal-title":"Communication Research"},{"issue":"2","key":"406_CR22","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1080\/10584609.2019.1674979","volume":"37","author":"M Hameleers","year":"2020","unstructured":"Hameleers, M., Powell, T. E., Van Der Meer, T. G. L. A., & Bos, L. (2020). A picture paints a thousand lies? The effects and mechanisms of multimodal disinformation and rebuttals disseminated via social media. Political Communication, 37(2), 281\u2013301.","journal-title":"Political Communication"},{"key":"406_CR23","doi-asserted-by":"crossref","unstructured":"Hansen, D., Shneiderman, B., & Smith, M. A. (2010). Analyzing social media networks with NodeXL: Insights from a connected world. Morgan Kaufmann.","DOI":"10.1016\/B978-0-12-382229-1.00002-3"},{"issue":"7","key":"406_CR24","doi-asserted-by":"publisher","first-page":"1382","DOI":"10.1177\/1461444814555096","volume":"18","author":"I Himelboim","year":"2016","unstructured":"Himelboim, I., Sweetser, K. D., Tinkham, S. F., Cameron, K., Danelo, M., & West, K. (2016). Valence-based homophily on twitter: Network analysis of emotions and political talk in the 2012 presidential election. New Media & Society, 18(7), 1382\u20131400.","journal-title":"New Media & Society"},{"key":"406_CR25","doi-asserted-by":"crossref","unstructured":"Hovy, D., & Spruit, S. L. (2016). The social impact of natural language processing. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 591\u2013598.","DOI":"10.18653\/v1\/P16-2096"},{"key":"406_CR26","doi-asserted-by":"crossref","unstructured":"Huang, C.-R., & Hsieh, S.-K. (2015). Chinese lexical semantics. In The Oxford handbook of Chinese linguistics (pp. 290\u2013305). Oxford Univ. Press.","DOI":"10.1093\/oxfordhb\/9780199856336.013.0018"},{"issue":"20","key":"406_CR27","doi-asserted-by":"publisher","first-page":"18510","DOI":"10.1021\/acsomega.9b02060","volume":"4","author":"L Huang","year":"2019","unstructured":"Huang, L., & Ling, C. (2019). Representing multiword chemical terms through phrase-level preprocessing and word embedding. ACS Omega, 4(20), 18510\u201318519.","journal-title":"ACS Omega"},{"key":"406_CR28","doi-asserted-by":"crossref","unstructured":"Jachna, T. (2021). Public Space, the Public Realm and Digital Technologies. In Wiring the Streets, Surfing the Square (pp. 23\u201337). Springer.","DOI":"10.1007\/978-3-030-66672-9_2"},{"key":"406_CR29","volume-title":"Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition","author":"D Jurafsky","year":"2009","unstructured":"Jurafsky, D., & Martin, J. H. (2009). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall."},{"issue":"1","key":"406_CR30","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1017\/S1351324900000048","volume":"1","author":"JS Justeson","year":"1995","unstructured":"Justeson, J. S., & Katz, S. M. (1995). Technical terminology: Some linguistic properties and an algorithm for identification in text. Natural Language Engineering, 1(1), 9\u201327.","journal-title":"Natural Language Engineering"},{"key":"406_CR31","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1016\/j.cmpb.2016.11.009","volume":"140","author":"T Kang","year":"2017","unstructured":"Kang, T., Zhang, S., Xu, N., Wen, D., Zhang, X., & Lei, J. (2017). Detecting negation and scope in Chinese clinical notes using character and word embedding. Computer Methods and Programs in Biomedicine, 140, 53\u201359.","journal-title":"Computer Methods and Programs in Biomedicine"},{"issue":"1","key":"406_CR32","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1016\/j.bushor.2009.09.003","volume":"53","author":"AM Kaplan","year":"2010","unstructured":"Kaplan, A. M., & Haenlein, M. (2010). Users of the world, unite! The challenges and opportunities of Social Media. Business Horizons, 53(1), 59\u201368.","journal-title":"Business Horizons"},{"key":"406_CR33","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1007\/s10796-017-9810-y","volume":"20","author":"KK Kapoor","year":"2018","unstructured":"Kapoor, K. K., Tamilmani, K., Rana, N. P., Patil, P., Dwivedi, Y. K., & Nerur, S. (2018). Advances in social media research: Past, present and future. Information Systems Frontiers, 20, 531\u2013558.","journal-title":"Information Systems Frontiers"},{"issue":"2","key":"406_CR34","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1177\/1073191113514104","volume":"21","author":"ML Kern","year":"2014","unstructured":"Kern, M. L., Eichstaedt, J. C., Schwartz, H. A., Dziurzynski, L., Ungar, L. H., Stillwell, D. J., Kosinski, M., Ramones, S. M., & Seligman, M. E. (2014). The online social self: An open vocabulary approach to personality. Assessment, 21(2), 158\u2013169.","journal-title":"Assessment"},{"issue":"3","key":"406_CR35","doi-asserted-by":"publisher","first-page":"3713","DOI":"10.1007\/s11042-022-13428-4","volume":"82","author":"D Khurana","year":"2023","unstructured":"Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023). Natural language processing: State of the art, current trends and challenges. Multimedia Tools and Applications, 82(3), 3713\u20133744.","journal-title":"Multimedia Tools and Applications"},{"key":"406_CR36","doi-asserted-by":"crossref","unstructured":"Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. In I. Gurevych & Y. Miyao (Eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 66\u201375). Association for Computational Linguistics.","DOI":"10.18653\/v1\/P18-1007"},{"key":"406_CR37","doi-asserted-by":"crossref","unstructured":"Kudo, T., & Richardson, J. (2018). Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv Preprint arXiv:1808.06226.","DOI":"10.18653\/v1\/D18-2012"},{"issue":"12","key":"406_CR38","doi-asserted-by":"publisher","first-page":"1699","DOI":"10.1080\/1369118X.2021.1877772","volume":"25","author":"FLF Lee","year":"2022","unstructured":"Lee, F. L. F., Liang, H., Cheng, E. W., Tang, G. K. Y., & Yuen, S. (2022). Affordances, movement dynamics, and a centralized digital communication platform in a networked movement. Information, Communication & Society, 25(12), 1699\u20131716.","journal-title":"Information, Communication & Society"},{"issue":"4","key":"406_CR39","first-page":"1","volume":"19","author":"FLF Lee","year":"2019","unstructured":"Lee, F. L. F., Yuen, S., Tang, G., & Cheng, E. W. (2019). Hong Kong\u2019s summer of uprising: From anti-extradition to anti-authoritarian protests. China Review, 19(4), 1\u201332.","journal-title":"China Review"},{"issue":"1","key":"406_CR40","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1109\/TKDE.2018.2823758","volume":"31","author":"B Li","year":"2019","unstructured":"Li, B., Yang, X., Zhou, R., Wang, B., Liu, C., & Zhang, Y. (2019). An efficient method for high quality and cohesive topical phrase mining. IEEE Transactions on Knowledge and Data Engineering, 31(1), 120\u2013137.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"406_CR41","doi-asserted-by":"crossref","unstructured":"Liu, C.-L., spsampsps Hsieh, C.-D. (2006). Exploring phrase-based classification of judicial documents for criminal charges in Chinese. International Symposium on Methodologies for Intelligent Systems, 681\u2013690.","DOI":"10.1007\/11875604_75"},{"key":"406_CR42","unstructured":"Lui, M., & Baldwin, T. (2012). Langid. Py: An off-the-shelf language identification tool. Proceedings of the ACL 2012 System Demonstrations, 25\u201330."},{"issue":"2","key":"406_CR43","doi-asserted-by":"publisher","first-page":"139","DOI":"10.5117\/CCR2020.2.001.MAIE","volume":"2","author":"D Maier","year":"2020","unstructured":"Maier, D., Niekler, A., Wiedemann, G., & Stoltenberg, D. (2020). How document sampling and vocabulary pruning affect the results of topic models. Computational Communication Research, 2(2), 139\u2013152.","journal-title":"Computational Communication Research"},{"key":"406_CR44","volume-title":"Foundations of Statistical Natural Language Processing","author":"CD Manning","year":"1999","unstructured":"Manning, C. D., & Schutze, H. (1999). Foundations of Statistical Natural Language Processing. MIT Press."},{"key":"406_CR45","doi-asserted-by":"crossref","unstructured":"Manning, C. D., Raghavan, P., & Sch\u00fctze, H. (2008). Introduction to Information Retrieval (1st ed.). Cambridge University Press.","DOI":"10.1017\/CBO9780511809071"},{"key":"406_CR46","doi-asserted-by":"crossref","unstructured":"Matthews, S., & Yip, V. (2011). Unbalanced bilingual acquisition as a mechanism of grammatical change. Bilingualism: Language and Cognition, 14(2), 159\u2013161.","DOI":"10.1017\/S1366728910000593"},{"key":"406_CR47","unstructured":"Maynard, D., & Greenwood, M. (2014). Who cares about Sarcastic Tweets? Investigating the Impact of Sarcasm on Sentiment Analysis. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC\u201914) (pp. 4238\u20134243). European Language Resources Association (ELRA)."},{"key":"406_CR48","unstructured":"Mielke, S. J., Alyafeai, Z., Salesky, E., Raffel, C., Dey, M., Gall\u00e9, M., Raja, A., Si, C., Lee, W. Y., Sagot, B., & Tan, S. (2021). Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (arXiv:2112.10508). arXiv."},{"issue":"6","key":"406_CR49","first-page":"16875","volume":"5","author":"AS Nayak","year":"2016","unstructured":"Nayak, A. S., Kanive, A. P., Chandavekar, N., & Balasubramani, R. (2016). Survey on pre-processing techniques for text mining. International Journal of Engineering and Computer Science, 5(6), 16875\u201316879.","journal-title":"International Journal of Engineering and Computer Science"},{"key":"406_CR50","doi-asserted-by":"publisher","DOI":"10.1016\/j.tele.2022.101785","volume":"69","author":"Y-L Ng","year":"2022","unstructured":"Ng, Y.-L., Song, Y., & Huang, Y. (2022). Supportive and uncivil expressions in discussions on out-groups by in-group members in anonymous computer-mediated communication. Telematics and Informatics, 69, Article 101785.","journal-title":"Telematics and Informatics"},{"key":"406_CR51","doi-asserted-by":"publisher","first-page":"2003","DOI":"10.1609\/icwsm.v18i1.31443","volume":"18","author":"N Oh","year":"2024","unstructured":"Oh, N. (2024). Sentibank: A unified resource of sentiment lexicons and dictionaries. Proceedings of the International AAAI Conference on Web and Social Media, 18, 2003\u20132013.","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media"},{"issue":"2","key":"406_CR52","doi-asserted-by":"publisher","first-page":"266","DOI":"10.1111\/j.1460-2466.2012.01630.x","volume":"62","author":"Z Papacharissi","year":"2012","unstructured":"Papacharissi, Z., & de Fatima Oliveira, M. (2012). Affective news and networked publics: The rhythms of news storytelling on #Egypt. Journal of Communication, 62(2), 266\u2013282.","journal-title":"Journal of Communication"},{"key":"406_CR53","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1007\/s10579-009-9101-4","volume":"44","author":"P Pecina","year":"2010","unstructured":"Pecina, P. (2010). Lexical association measures and collocation extraction. Language Resources and Evaluation, 44, 137\u2013158.","journal-title":"Language Resources and Evaluation"},{"key":"406_CR54","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., & Dubourg, V. (2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12, 2825\u20132830.","journal-title":"Journal of Machine Learning Research"},{"issue":"2","key":"406_CR55","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1080\/21670811.2022.2036623","volume":"11","author":"C Pipal","year":"2023","unstructured":"Pipal, C., Song, H., & Boomgaarden, H. G. (2023). If you have choices, Why not choose (and share) all of them? A multiverse approach to understanding news engagement on social media. Digital Journalism, 11(2), 255\u2013275.","journal-title":"Digital Journalism"},{"issue":"2","key":"406_CR56","first-page":"2","volume":"3","author":"R Rehurek","year":"2011","unstructured":"Rehurek, R., & Sojka, P. (2011). Gensim\u2013python framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic, 3(2), 2.","journal-title":"NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic"},{"issue":"5","key":"406_CR57","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1016\/0306-4573(88)90021-0","volume":"24","author":"G Salton","year":"1988","unstructured":"Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513\u2013523.","journal-title":"Information Processing & Management"},{"key":"406_CR58","doi-asserted-by":"crossref","unstructured":"Sennrich, R., Haddow, B., & Birch, A. (2015). Neural machine translation of rare words with subword units. arXiv Preprint arXiv:1508.07909.","DOI":"10.18653\/v1\/P16-1162"},{"key":"406_CR59","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.dss.2018.01.004","volume":"108","author":"M Siering","year":"2018","unstructured":"Siering, M., Muntermann, J., & Rajagopalan, B. (2018). Explaining and predicting online review helpfulness: The role of content and reviewer-related signals. Decision Support Systems, 108, 1\u201312.","journal-title":"Decision Support Systems"},{"key":"406_CR60","doi-asserted-by":"crossref","unstructured":"Song, Y., Schuldt, J. P., Zhang, Y., Wang, Z., Yuan, Y. C., Zou, S., & Li, J. (2025). Bridging ideologies: analyzing the use of moral language and framing in social media discourse on climate change by U.S. congress members through computational approaches. Climatic Change, 178, 56.","DOI":"10.1007\/s10584-025-03898-4"},{"issue":"3","key":"406_CR61","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1145\/1324185.1324190","volume":"36","author":"A Stavrianou","year":"2007","unstructured":"Stavrianou, A., Andritsos, P., & Nicoloyannis, N. (2007). Overview and semantic issues of text mining. ACM SIGMOD Record, 36(3), 23\u201334.","journal-title":"ACM SIGMOD Record"},{"key":"406_CR62","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1007\/978-3-030-32381-3_16","volume-title":"Chinese Computational Linguistics","author":"C Sun","year":"2019","unstructured":"Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019). How to Fine-Tune BERT for Text Classification? In M. Sun, X. Huang, H. Ji, Z. Liu, & Y. Liu (Eds.), Chinese Computational Linguistics (pp. 194\u2013206). Springer International Publishing."},{"issue":"2","key":"406_CR63","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1162\/COLI_a_00049","volume":"37","author":"M Taboada","year":"2011","unstructured":"Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. (2011). Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(2), 267\u2013307.","journal-title":"Computational Linguistics"},{"key":"406_CR64","unstructured":"Tay, Y., Tran, V. Q., Ruder, S., Gupta, J., Chung, H. W., Bahri, D., Qin, Z., Baumgartner, S., Yu, C., & Metzler, D. (2022). Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (No. arXiv:2106.12672). arXiv."},{"issue":"1","key":"406_CR65","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1609\/icwsm.v8i1.14517","volume":"8","author":"Z Tufekci","year":"2014","unstructured":"Tufekci, Z. (2014). Big questions for social media big data: Representativeness, validity and other methodological pitfalls. Proceedings of the International AAAI Conference on Web and Social Media, 8(1), 505\u2013514.","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media"},{"issue":"1","key":"406_CR66","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1613\/jair.2934","volume":"37","author":"PD Turney","year":"2010","unstructured":"Turney, P. D., & Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37(1), 141\u2013188.","journal-title":"Journal of Artificial Intelligence Research"},{"key":"406_CR67","unstructured":"Van Rossum, G., & Drake, F. L. (2009). Python 3 Reference Manual. CreateSpace."},{"key":"406_CR68","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1007\/978-3-319-46681-1_15","volume-title":"Neural Information Processing","author":"M Wang","year":"2016","unstructured":"Wang, M., Zhao, B., & Huang, Y. (2016). PTR: Phrase-Based Topical Ranking for Automatic Keyphrase Extraction in Scientific Publications. In A. Hirose, S. Ozawa, K. Doya, K. Ikeda, M. Lee, & D. Liu (Eds.), Neural Information Processing (pp. 120\u2013128). Springer International Publishing."},{"key":"406_CR69","doi-asserted-by":"crossref","unstructured":"Wang, Z., & Zhang, Y. (2017). A neural model for joint event detection and summarization. Proceedings of the 26th International Joint Conference on Artificial Intelligence, 4158\u20134164.","DOI":"10.24963\/ijcai.2017\/581"},{"key":"406_CR70","doi-asserted-by":"crossref","unstructured":"Webster, J. J., & Kit, C. (1992). Tokenization as the initial phase in NLP. COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics.","DOI":"10.3115\/992424.992434"},{"issue":"4","key":"406_CR71","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1080\/19312458.2017.1387238","volume":"11","author":"K Welbers","year":"2017","unstructured":"Welbers, K., Van Atteveldt, W., & Benoit, K. (2017). Text analysis in R. Communication Methods and Measures, 11(4), 245\u2013265.","journal-title":"Communication Methods and Measures"},{"issue":"1","key":"406_CR72","doi-asserted-by":"publisher","first-page":"401","DOI":"10.1609\/icwsm.v5i1.14102","volume":"5","author":"J Weng","year":"2011","unstructured":"Weng, J., & Lee, B.-S. (2011). Event detection in twitter. Proceedings of the International Aaai Conference on Web and Social Media, 5(1), 401\u2013408.","journal-title":"Proceedings of the International Aaai Conference on Web and Social Media"}],"container-title":["Journal of Computational Social Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42001-025-00406-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42001-025-00406-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42001-025-00406-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,7]],"date-time":"2025-09-07T00:40:00Z","timestamp":1757205600000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42001-025-00406-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,23]]},"references-count":72,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["406"],"URL":"https:\/\/doi.org\/10.1007\/s42001-025-00406-7","relation":{},"ISSN":["2432-2717","2432-2725"],"issn-type":[{"type":"print","value":"2432-2717"},{"type":"electronic","value":"2432-2725"}],"subject":[],"published":{"date-parts":[[2025,7,23]]},"assertion":[{"value":"27 January 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 June 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 July 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that there is no conflict of interest regarding the publication of this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"80"}}