{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T15:52:53Z","timestamp":1778860373258,"version":"3.51.4"},"reference-count":40,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T00:00:00Z","timestamp":1632441600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["42050101"],"award-info":[{"award-number":["42050101"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Strategic Priority Research Program of the Chinese Academy of Sciences","award":["XDA23100100"],"award-info":[{"award-number":["XDA23100100"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJGI"],"abstract":"<jats:p>Social media datasets have been widely used in disaster assessment and management. When a disaster occurs, many users post messages in a variety of formats, e.g., image and text, on social media platforms. Useful information could be mined from these multimodal data to enable situational awareness and to support decision making during disasters. However, the multimodal data collected from social media contain a lot of irrelevant and misleading content that needs to be filtered out. Existing work has mostly used unimodal methods to classify disaster messages. In other words, these methods treated the image and textual features separately. While a few methods adopted multimodality to deal with the data, their accuracy cannot be guaranteed. This research seamlessly integrates image and text information by developing a multimodal fusion approach to identify useful disaster images collected from social media platforms. In particular, a deep learning method is used to extract the visual features from social media, and a FastText framework is then used to extract the textual features. Next, a novel data fusion model is developed to combine both visual and textual features to classify relevant disaster images. Experiments on a real-world disaster dataset, CrisisMMD, are performed, and the validation results demonstrate that the method consistently and significantly outperforms the previously published state-of-the-art work by over 3%, with a performance improvement from 84.4% to 87.6%.<\/jats:p>","DOI":"10.3390\/ijgi10100636","type":"journal-article","created":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T20:15:32Z","timestamp":1632514532000},"page":"636","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":52,"title":["Disaster Image Classification by Fusing Multimodal Social Media Data"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2828-8491","authenticated-orcid":false,"given":"Zhiqiang","family":"Zou","sequence":"first","affiliation":[{"name":"College of Computer, Nanjing University of Posts and Telecommunications, Nanjing 210023, China"},{"name":"Jiangsu Key Laboratory of Big Data Security and Intelligent Processing, Nanjing 210023, China"}]},{"given":"Hongyu","family":"Gan","sequence":"additional","affiliation":[{"name":"College of Computer, Nanjing University of Posts and Telecommunications, Nanjing 210023, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3499-7294","authenticated-orcid":false,"given":"Qunying","family":"Huang","sequence":"additional","affiliation":[{"name":"Spatial Computing and Data Mining Lab, Department of Geography, University of Wisconsin-Madison, 550 N. Park St., Science Hall, Madison, WI 53706, USA"}]},{"given":"Tianhui","family":"Cai","sequence":"additional","affiliation":[{"name":"College of Liberal Arts & Sciences, University of Illinois at Urbana Champaign, Champaign, IL 61820, USA"}]},{"given":"Kai","family":"Cao","sequence":"additional","affiliation":[{"name":"School of Geographic Sciences, East China Normal University, Shanghai 200241, China"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1549","DOI":"10.3390\/ijgi4031549","article-title":"Geographic Situational Awareness: Mining Tweets for Disaster Preparedness, Emergency Response, Impact, and Recovery","volume":"4","author":"Huang","year":"2015","journal-title":"ISPRS Int. J. Geo-Inf."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kumar, A., Sangwan, S.R., and Nayyar, A. (2020). Multimedia Social Big Data: Mining, Springer.","DOI":"10.1007\/978-981-13-8759-3_11"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"781","DOI":"10.1080\/17538947.2018.1480670","article-title":"Assessing relevance of tweets for risk communication","volume":"12","author":"Liu","year":"2019","journal-title":"Int. J. Digit. Earth"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zhao, Z., Resnick, P., and Mei, Q. (2015, January 18\u201322). Enquiring Minds: Early Detection of Rumors in Social Media from Enquiry Posts Categories and Subject Descriptors Detection Problems in Social Media. Proceedings of the 24th International Conference on World Wide Web, Florence, Italy.","DOI":"10.1145\/2736277.2741637"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1089\/big.2020.0062","article-title":"FakeNewsNet: A Data Repository with News Content, Social Context, and Spatiotemporal Information for Studying Fake News on Social Media","volume":"8","author":"Shu","year":"2020","journal-title":"Big Data"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zubiaga, A., Liakata, M., and Procter, R. (2016). Learning Reporting Dynamics during Breaking News for Rumour Detection in Social Media. arXiv.","DOI":"10.1007\/978-3-319-67217-5_8"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mendoza, M., Poblete, B., and Castillo, C. (2010, January 25\u201328). Twitter under crisis: Can we trust what we RT?. Proceedings of the First Workshop on Social Media Analytics\u2014SOMA\u201910, Washington, DC, USA.","DOI":"10.1145\/1964858.1964869"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1002\/poi3.85","article-title":"Cyber hate speech on twitter: An application of machine classification and statistical modeling for policy and decision making","volume":"7","author":"Burnap","year":"2015","journal-title":"Policy Internet"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1016\/j.tre.2017.05.008","article-title":"Social media data analytics to improve supply chain management in food industries","volume":"114","author":"Singh","year":"2018","journal-title":"Transp. Res. Part E Logist. Transp. Rev."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/j.eswa.2016.12.036","article-title":"The impact of microblogging data for stock market prediction: Using Twitter to predict returns, volatility, trading volume and survey sentiment indices","volume":"73","author":"Oliveira","year":"2017","journal-title":"Expert Syst. Appl."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1007\/978-981-15-9509-7_43","article-title":"Analyzing Social Media Data for Better Understanding Students\u2019 Learning Experiences","volume":"57","author":"Ganesan","year":"2021","journal-title":"Lect. Notes Data Eng. Commun. Technol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"101003","DOI":"10.1016\/j.jocs.2019.05.009","article-title":"Emotion and sentiment analysis from Twitter text","volume":"36","author":"Sailunaz","year":"2019","journal-title":"J. Comput. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"70701","DOI":"10.1109\/ACCESS.2019.2918354","article-title":"Predicting Cyberbullying on Social Media in the Big Data Era Using Machine Learning Algorithms: Review of Literature and Open Challenges","volume":"7","author":"Hussain","year":"2019","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Chen, Y., Zhou, Y., Zhu, S., and Xu, H. (2012, January 3\u20135). Detecting Offensive Language in Social Media to Protect Adolescent Online Safety. Proceedings of the 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, Amsterdam, The Netherlands.","DOI":"10.1109\/SocialCom-PASSAT.2012.55"},{"key":"ref_15","first-page":"5534","article-title":"Affective image content analysis: A comprehensive survey","volume":"2018","author":"Zhao","year":"2018","journal-title":"IJCAI Int. Jt. Conf. Artif. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1016\/j.neucom.2018.12.053","article-title":"Multi-level region-based Convolutional Neural Network for image emotion classification","volume":"333","author":"Rao","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_17","first-page":"423","article-title":"Multimodal machine learning: A survey and taxonomy","volume":"41","author":"Ahuja","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1109\/35.41402","article-title":"Integration of Acoustic and Visual Speech Signals Using Neural Networks","volume":"27","author":"Yuhas","year":"1989","journal-title":"IEEE Commun. Mag."},{"key":"ref_19","first-page":"4188","article-title":"Framing image description as a ranking task: Data, models and evaluation metrics","volume":"2015","author":"Hodosh","year":"2015","journal-title":"IJCAI Int. Jt. Conf. Artif. Intell."},{"key":"ref_20","first-page":"290","article-title":"Comparison of Automatic Shot Boundary Detection Algorithms","volume":"3656","author":"Lienhart","year":"1999","journal-title":"Event Electron. Imaging"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1553","DOI":"10.1109\/TMM.2013.2267205","article-title":"Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention","volume":"15","author":"Evangelopoulos","year":"2013","journal-title":"IEEE Trans. Multimed."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zou, Z., He, X., and Zhu, A. (2019). An Automatic Annotation Method for Discovering Semantic Information of Geographical Locations from Location-Based Social Networks. ISPRS Int. J. Geo-Inf., 8.","DOI":"10.3390\/ijgi8110487"},{"key":"ref_23","unstructured":"Ofli, F., Alam, F., and Imran, M. (2020). Analysis of Social Media Data Using Multimodal Deep Learning for Disaster Response. arXiv."},{"key":"ref_24","unstructured":"Simonyan, K., and Zisserman, A. (2015, January 7\u20139). Very deep convolutional networks for large-scale image recognition. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015\u2014Conference Track Proceedings, San Diego, CA, USA."},{"key":"ref_25","first-page":"427","article-title":"Bag of Tricks for Efficient Text Classification","volume":"Volume 2","author":"Joulin","year":"2017","journal-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1080\/19475683.2020.1817146","article-title":"Situational awareness extraction: A comprehensive review of social media data classification during natural hazards","volume":"27","author":"Vongkusolkit","year":"2021","journal-title":"Ann. GIS"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Imran, M., Castillo, C., Diaz, F., and Vieweg, S. (2018, January 23\u201327). Processing Social Media Messages in Mass Emergency: Survey Summary. Proceedings of the Companion of the The Web Conference 2018 on The Web Conference 2018\u2014WWW\u201918, Lyon, France.","DOI":"10.1145\/3184558.3186242"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Liu, X., Kar, B., Montiel Ishino, F.A., Zhang, C., and Williams, F. (2020). Assessing the Reliability of Relevant Tweets and Validation Using Manual and Automatic Approaches for Flood Risk Communication. ISPRS Int. J. Geo-Inf., 9.","DOI":"10.3390\/ijgi9090532"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Yang, T., Xie, J., Li, G., Mou, N., Li, Z., Tian, C., and Zhao, J. (2019). Social Media Big Data Mining and Spatio-Temporal Analysis on Public Emotions for Disaster Mitigation. ISPRS Int. J. Geo-Inf., 8.","DOI":"10.3390\/ijgi8010029"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1177","DOI":"10.1007\/s11069-016-2370-5","article-title":"A Weibo-based approach to disaster informatics: Incidents monitor in post-disaster situation via Weibo text negative sentiment analysis","volume":"83","author":"Bai","year":"2016","journal-title":"Nat. Hazards"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.ijinfomgt.2018.05.004","article-title":"Big data analytics for disaster response and recovery through sentiment analysis","volume":"42","author":"Ragini","year":"2018","journal-title":"Int. J. Inf. Manag."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wu, Z., Shen, Y., and Wang, H. (2019). Assessing Urban Areas\u2019 Vulnerability to Flood Disaster Based on Text Data: A Case Study in Zhengzhou City. Sustainability, 11.","DOI":"10.3390\/su11174548"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Alam, F., Imran, M., and Ofli, F. (August, January 31). Image4Act: Online social media image processing for disaster response. Proceedings of the 2017 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, Sydney, Australia.","DOI":"10.1145\/3110025.3110164"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"31267","DOI":"10.1007\/s11042-019-07942-1","article-title":"Natural disasters detection in social media and satellite imagery: A survey","volume":"78","author":"Said","year":"2019","journal-title":"Multimed. Tools Appl."},{"key":"ref_35","unstructured":"Hassan, S.Z., Ahmad, K., Hicks, S., Halvorsen, P., Al-Fuqaha, A., Conci, N., and Riegler, M. (2020). Visual Sentiment Analysis from Disaster Images in Social Media. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Dao, M.-S., Quang Nhat Minh, P., Kasem, A., and Haja Nazmudeen, M.S. (2018, January 11\u201314). A Context-Aware Late-Fusion Approach for Disaster Image Retrieval from Social Media. Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, Yokohama, Japan.","DOI":"10.1145\/3206025.3206047"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Gautam, A.K., Misra, L., Kumar, A., Misra, K., Aggarwal, S., and Shah, R.R. (2019, January 11\u201313). Multimodal Analysis of Disaster Tweets. Proceedings of the 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), Singapore.","DOI":"10.1109\/BigMM.2019.00-38"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Alam, F., Ofli, F., and Imran, M. (2018, January 25\u201328). CrisisMMD: Multimodal twitter datasets from natural disasters. Proceedings of the 12th International AAAI Conference on Web and Social Media, ICWSM 2018, Palo Alto, CA, USA.","DOI":"10.1609\/icwsm.v12i1.14983"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A Survey on Transfer Learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Tilon, S., Nex, F., Kerle, N., and Vosselman, G. (2020). Post-Disaster Building Damage Detection from Earth Observation Imagery Using Unsupervised and Transferable Anomaly Detecting Generative Adversarial Networks. Remote Sens., 12.","DOI":"10.3390\/rs12244193"}],"container-title":["ISPRS International Journal of Geo-Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2220-9964\/10\/10\/636\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:04:15Z","timestamp":1760166255000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2220-9964\/10\/10\/636"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,24]]},"references-count":40,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["ijgi10100636"],"URL":"https:\/\/doi.org\/10.3390\/ijgi10100636","relation":{},"ISSN":["2220-9964"],"issn-type":[{"value":"2220-9964","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,24]]}}}