{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,4,16]],"date-time":"2024-04-16T06:29:52Z","timestamp":1713248992546},"reference-count":17,"publisher":"MIT Press","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2015,9]]},"abstract":"<jats:p>In recent years, many studies have been published on data collected from social media, especially microblogs such as Twitter. However, rather few of these studies have considered evaluation methodologies that take into account the statistically dependent nature of such data, which breaks the theoretical conditions for using cross-validation. Despite concerns raised in the past about using cross-validation for data of similar characteristics, such as time series, some of these studies evaluate their work using standard k-fold cross-validation. Through experiments on Twitter data collected during a two-year period that includes disastrous events, we show that by ignoring the statistical dependence of the text messages published in social media, standard cross-validation can result in misleading conclusions in a machine learning task. We explore alternative evaluation methods that explicitly deal with statistical dependence in text. Our work also raises concerns for any other data for which similar conditions might hold.<\/jats:p>","DOI":"10.1162\/coli_a_00230","type":"journal-article","created":{"date-parts":[[2015,7,20]],"date-time":"2015-07-20T18:19:23Z","timestamp":1437416363000},"page":"539-548","source":"Crossref","is-referenced-by-count":5,"title":["Evaluation Methods for Statistically Dependent Text"],"prefix":"10.1162","volume":"41","author":[{"given":"Sarvnaz","family":"Karimi","sequence":"first","affiliation":[{"name":"CSIRO"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jie","family":"Yin","sequence":"additional","affiliation":[{"name":"CSIRO"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiri","family":"Baum","sequence":"additional","affiliation":[{"name":"Sabik Software Solutions"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1214\/09-SS054"},{"key":"R2","unstructured":"Bengio, Y. and Y. Grandvalet. 2004. No unbiased estimator of the variance of k-fold cross-validation. JMLR, 5:1089\u20131105."},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2011.12.028"},{"key":"R4","unstructured":"Chang, C. and C. Lin. 2011. LIBSVM: A library for support vector machines. ACM TIST, 2(3):27:1\u201327:27."},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176348377"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1145\/1964858.1964874"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/61.1.101"},{"key":"R8","unstructured":"Jiang, L., M. Yu, M. Zhou, X. Liu, and T. Zhao. 2011. Target-dependent Twitter sentiment classification. In ACL-HLT, pages 151\u2013160, Portland, OR."},{"key":"R9","doi-asserted-by":"crossref","unstructured":"Kumar, A., M. Jiang, and Y. Fang. 2014. Where not to go? Detecting road hazards using Twitter. In SIGIR, pages 1223\u20131226, Gold Coast.","DOI":"10.1145\/2600428.2609550"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.3115\/1599081.1599141"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1021\/ci400084k"},{"key":"R12","doi-asserted-by":"crossref","unstructured":"Sriram, B., D. Fuhry, E. Demir, H. Ferhatosmanoglu, and M. Demirbas. 2010. Short text classification in Twitter to improve information filtering. In SIGIR, pages 841\u2013842, Geneva.","DOI":"10.1145\/1835449.1835643"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398642"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Uysal, I. and W. B. Croft. 2011. User oriented tweet ranking: A filtering approach to microblogs. In CIKM, pages 2261\u20132264, Glasgow.","DOI":"10.1145\/2063576.2063941"},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Verma, S., S. Vieweg, W. J. Corvey, L. Palen, J. H. Martin, M. Palmer, A. Schram, and K. M. Anderson. 2011. Natural language processing to the rescue? Extracting \u201csituational awareness\u201d Tweets during mass emergency. In ICWSM, pages 385\u2013392, Barcelona.","DOI":"10.1609\/icwsm.v5i1.14119"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398732"},{"key":"R17","unstructured":"Zanzotto, F. M., M. Pennacchiotti, and K. Tsioutsiouliklis. 2011. Linguistic redundancy in Twitter. In EMNLP, pages 659\u2013669, Edinburgh."}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00230","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,12]],"date-time":"2023-08-12T10:24:46Z","timestamp":1691835886000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/41\/3\/539-548\/1518"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,9]]},"references-count":17,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2015,9]]}},"alternative-id":["10.1162\/COLI_a_00230"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00230","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,9]]}}}