{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:26:54Z","timestamp":1750220814852,"version":"3.41.0"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2019,9,30]],"date-time":"2019-09-30T00:00:00Z","timestamp":1569801600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Trans. Soc. Comput."],"published-print":{"date-parts":[[2019,9,30]]},"abstract":"<jats:p>Geolocating Twitter users\u2014the task of identifying their home locations\u2014serves a wide range of community and business applications such as managing natural crises, journalism, and public health. Many approaches have been proposed for automatically geolocating users based on their tweets; at the same time, various evaluation metrics have been proposed to measure the effectiveness of these approaches, making it challenging to understand which of these metrics is the most suitable for this task. In this article, we propose a guide for a standardized evaluation of Twitter user geolocation by analyzing 15 models and two baselines in a controlled experimental setting. Models are evaluated using 10 metrics over four geographic granularities. We use rank correlations to assess the effectiveness of these metrics.<\/jats:p>\n          <jats:p>\n            Our results demonstrate that the choice of effectiveness metric can have a substantial impact on the conclusions drawn from a geolocation system experiment, potentially leading experimenters to contradictory results about relative effectiveness. We show that for general evaluations, a range of performance metrics should be reported, to ensure that a complete picture of system effectiveness is conveyed. Given the global geographic coverage of this task, we specifically recommend evaluation at micro vs. macro levels to measure the impact of the bias in distribution over locations. Although a lot of complex geolocation algorithms have been applied in recent years, a majority class baseline is still competitive at coarse geographic granularity. We propose a suite of statistical analysis tests, based on the employed metric, to ensure that the results are not coincidental.\n            <jats:sup>1<\/jats:sup>\n          <\/jats:p>","DOI":"10.1145\/3352572","type":"journal-article","created":{"date-parts":[[2019,12,9]],"date-time":"2019-12-09T13:35:27Z","timestamp":1575898527000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["A Practical Guide for the Effective Evaluation of Twitter User Geolocation"],"prefix":"10.1145","volume":"2","author":[{"given":"Ahmed","family":"Mourad","sequence":"first","affiliation":[{"name":"School of Computer Science and Information Technology, RMIT University, Melbourne, VIC, Australia"}]},{"given":"Falk","family":"Scholer","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Technology, RMIT University, Melbourne, VIC, Australia"}]},{"given":"Walid","family":"Magdy","sequence":"additional","affiliation":[{"name":"School of Informatics, The University of Edinburgh, Crichton Street, Edinburgh, United Kingdom"}]},{"given":"Mark","family":"Sanderson","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Technology, RMIT University, Melbourne, VIC, Australia"}]}],"member":"320","published-online":{"date-parts":[[2019,12,7]]},"reference":[{"volume-title":"Proceedings of the 22nd International Conference on World Wide Web. 25--36","author":"Ahmed Amr","key":"e_1_2_1_1_1"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772698"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","DOI":"10.1145\/3178112","volume-title":"Twitter geolocation: A hybrid approach. ACM Transactions on Knowledge Discovery from Data (TKDD) 12","author":"Bakerman Jordan","year":"2018"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0083672"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1871437.1871535"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208409"},{"volume-title":"Evaggelia Tsiligianni, Bruno Cornelis, and Nikos Deligiannis.","year":"2017","author":"Do Tien Huu","key":"e_1_2_1_7_1"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the AAAI Workshop on Expanding the Boundaries of Health Informatics Using AI","volume":"23","author":"Dredze Mark","year":"2013"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00074"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1128"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.24011"},{"volume-title":"Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. 1277--1287","author":"Eisenstein Jacob","key":"e_1_2_1_12_1"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2808797.2809327"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/2655713.2655726"},{"volume-title":"Proceedings of the 2nd Workshop on Noisy User-Generated Text. 213--217","year":"2016","author":"Han Bo","key":"e_1_2_1_15_1"},{"volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 237--246","author":"Hecht Brent","key":"e_1_2_1_16_1"},{"volume-title":"Proceedings of the 8th International AAAI Conference on Web and Social Media. 197--205","author":"Brent","key":"e_1_2_1_17_1"},{"volume-title":"Proceedings of the 2nd Workshop on Noisy User-Generated Text. 218--226","year":"2016","author":"Jayasinghe Gaya","key":"e_1_2_1_18_1"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026015"},{"volume-title":"Proceedings of the 7th International AAAI Conference on Web and Social Media. 273--282","year":"2013","author":"Jurgens David","key":"e_1_2_1_20_1"},{"volume-title":"Proceedings of the 9th International AAAI Conference on Web and Social Media. 8--17","year":"2015","author":"Jurgens David","key":"e_1_2_1_21_1"},{"volume-title":"Proceedings of the 9th International AAAI Conference on Web and Social Media. 188--197","year":"2015","author":"Jurgens David","key":"e_1_2_1_22_1"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0117288"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983363"},{"volume-title":"Proceedings of the ACL-2012 System Demonstrations. 25--30","year":"2012","author":"Lui Marco","key":"e_1_2_1_25_1"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1111\/tgis.12212"},{"volume-title":"Proceedings of the 5th International AAAI Conference on Weblogs and Social Media. 554--557","author":"Mislove Alan","key":"e_1_2_1_27_1"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1116"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2407085.2407092"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-56608-5_26"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASONAM.2018.8508701"},{"volume-title":"Confounds and consequences in geotagged Twitter data. Arxiv Preprint Arxiv:1506.02275","year":"2015","author":"Pavalanathan Umashanthi","key":"e_1_2_1_32_1"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078714.3078719"},{"volume-title":"Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work 8 Social Computing. 1523--1536","author":"Priedhorsky Reid","key":"e_1_2_1_35_1"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1016"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-4022"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1187"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2015.05.066"},{"volume-title":"Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 1500--1510","year":"2012","author":"Roller Stephen","key":"e_1_2_1_40_1"},{"volume-title":"Proceedings of the 9th International AAAI Conference on Web and Social Media. 407--415","year":"2015","author":"Schwartz Raz","key":"e_1_2_1_41_1"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505283"},{"volume-title":"A systematic analysis of performance measures for classification tasks. Information Processing 8 Management 45","year":"2009","author":"Sokolova Marina","key":"e_1_2_1_43_1"},{"volume-title":"Proceedings of 9th International Conference on Information Systems for Crisis Response and Management. 1--10","year":"2012","author":"Starbird Kate","key":"e_1_2_1_44_1"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1039"},{"volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 955--964","author":"Benjamin","key":"e_1_2_1_46_1"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009982220290"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/312624.312647"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2018.2807840"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2449396.2449424"}],"container-title":["ACM Transactions on Social Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3352572","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3352572","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:13:02Z","timestamp":1750201982000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3352572"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,30]]},"references-count":50,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,9,30]]}},"alternative-id":["10.1145\/3352572"],"URL":"https:\/\/doi.org\/10.1145\/3352572","relation":{},"ISSN":["2469-7818","2469-7826"],"issn-type":[{"type":"print","value":"2469-7818"},{"type":"electronic","value":"2469-7826"}],"subject":[],"published":{"date-parts":[[2019,9,30]]},"assertion":[{"value":"2018-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-12-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}