{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T22:05:54Z","timestamp":1767650754277,"version":"build-2065373602"},"reference-count":64,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2023,4,6]],"date-time":"2023-04-06T00:00:00Z","timestamp":1680739200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"EU\u2019s Horizon 2020 research and innovation programme","award":["H2020-832876 aqua3S","H2020-101004157 WQeMS","H2020-883484 PathoCERT"],"award-info":[{"award-number":["H2020-832876 aqua3S","H2020-101004157 WQeMS","H2020-883484 PathoCERT"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper discusses the importance of detecting breaking events in real time to help emergency response workers, and how social media can be used to process large amounts of data quickly. Most event detection techniques have focused on either images or text, but combining the two can improve performance. The authors present lessons learned from the Flood-related multimedia task in MediaEval2020, provide a dataset for reproducibility, and propose a new multimodal fusion method that uses Graph Neural Networks to combine image, text, and time information. Their method outperforms state-of-the-art approaches and can handle low-sample labelled data.<\/jats:p>","DOI":"10.3390\/s23073767","type":"journal-article","created":{"date-parts":[[2023,4,6]],"date-time":"2023-04-06T05:21:23Z","timestamp":1680758483000},"page":"3767","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Flood-Related Multimedia Benchmark Evaluation: Challenges, Results and a Novel GNN Approach"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5902-482X","authenticated-orcid":false,"given":"Thomas","family":"Papadimos","sequence":"first","affiliation":[{"name":"Information Technologies Institute, Centre for Research and Technology Hellas, 6th km Charilaou-Thermi Rd, 570 01 Thessaloniki, Greece"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5519-1962","authenticated-orcid":false,"given":"Stelios","family":"Andreadis","sequence":"additional","affiliation":[{"name":"Information Technologies Institute, Centre for Research and Technology Hellas, 6th km Charilaou-Thermi Rd, 570 01 Thessaloniki, Greece"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5234-9795","authenticated-orcid":false,"given":"Ilias","family":"Gialampoukidis","sequence":"additional","affiliation":[{"name":"Information Technologies Institute, Centre for Research and Technology Hellas, 6th km Charilaou-Thermi Rd, 570 01 Thessaloniki, Greece"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2505-9178","authenticated-orcid":false,"given":"Stefanos","family":"Vrochidis","sequence":"additional","affiliation":[{"name":"Information Technologies Institute, Centre for Research and Technology Hellas, 6th km Charilaou-Thermi Rd, 570 01 Thessaloniki, Greece"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6447-9020","authenticated-orcid":false,"given":"Ioannis","family":"Kompatsiaris","sequence":"additional","affiliation":[{"name":"Information Technologies Institute, Centre for Research and Technology Hellas, 6th km Charilaou-Thermi Rd, 570 01 Thessaloniki, Greece"}]}],"member":"1968","published-online":{"date-parts":[[2023,4,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1056\/NEJMp1103591","article-title":"Integrating social media into emergency-preparedness efforts","volume":"365","author":"Merchant","year":"2011","journal-title":"N. Engl. J. Med."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1080\/10447318.2018.1427832","article-title":"Social media in crisis management: An evaluation and analysis of crisis informatics research","volume":"34","author":"Reuter","year":"2018","journal-title":"Int. J. Hum.-Comput. Interact."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1016\/j.techfore.2016.06.010","article-title":"Social media in emergency management: Twitter as a tool for communicating risks to the public","volume":"111","author":"Panagiotopoulos","year":"2016","journal-title":"Technol. Forecast. Soc. Chang."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Imran, M., Castillo, C., Diaz, F., and Vieweg, S. (2018, January 23\u201327). Processing social media messages in mass emergency: Survey summary. Proceedings of the The Web Conference 2018, Lyon, France.","DOI":"10.1145\/3184558.3186242"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1177\/0361198118773896","article-title":"Crisis communication patterns in social media during Hurricane Sandy","volume":"2672","author":"Sadri","year":"2018","journal-title":"Transp. Res. Rec."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"101212","DOI":"10.1016\/j.ijdrr.2019.101212","article-title":"Using social media to call for help in Hurricane Harvey: Bonding emotion, culture, and community relationships","volume":"38","author":"Li","year":"2019","journal-title":"Int. J. Disaster Risk Reduct."},{"key":"ref_7","first-page":"44","article-title":"The use of social media during the 2019\u20132020 Australian bushfires","volume":"35","author":"Lim","year":"2020","journal-title":"Aust. J. Emerg. Manag."},{"key":"ref_8","first-page":"18","article-title":"The Lebanese Revolution on Twitter: Framing, sources, and networked publics","volume":"15","author":"Thomas","year":"2021","journal-title":"Int. J. Commun."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Bui, T.D., Ravi, S., and Ramavajjala, V. (2018, January 6\u20138). Neural graph learning: Training neural networks using graphs. Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, Los Angeles, CA, USA.","DOI":"10.1145\/3159652.3159731"},{"key":"ref_10","unstructured":"Awad, G., Butt, A., Liu, J., and Drew, W. (2023, April 01). Disaster Scene Description and Indexing (DSDI). 2020 TREC Video Retrieval Evaluation Notebook Papers and Slides, Available online: https:\/\/tsapps.nist.gov\/publication\/get_pdf.cfm?pub_id=931766."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ionescu, B., M\u00fcller, H., Villegas, M., Arenas, H., Boato, G., Dang-Nguyen, D.T., Cid, Y.D., Eickhoff, C., de Herrera, A.G.S., and Gurrin, C. (2017, January 11\u201314). Overview of ImageCLEF 2017: Information extraction from images. Proceedings of the International Conference of the Cross-Language Evaluation Forum for European Languages, Dublin, Ireland.","DOI":"10.1007\/978-3-319-65813-1_28"},{"key":"ref_12","unstructured":"Bischke, B., Helber, P., Schulze, C., Srinivasan, V., Dengel, A., and Borth, D. (2017, January 13\u201315). The Multimedia Satellite Task at MediaEval 2017. Proceedings of the MediaEval, Dublin, Ireland."},{"key":"ref_13","unstructured":"Bischke, B., Helber, P., Zhao, Z., De Bruijn, J., and Borth, D. (2018, January 29\u201331). The Multimedia Satellite Task at MediaEval 2018: Emergency response for flooding events. Proceedings of the MediaEval, Sophia Antipolies, France."},{"key":"ref_14","unstructured":"Bischke, B., Helber, P., Brugman, S., Basar, E., Zhao, Z., Larson, M., and Pogorelov, K. (2019, January 27\u201330). The Multimedia Satellite Task at MediaEval 2019: Estimation of Flood Severity. Proceedings of the MediaEval 2019 Workshop, Sophia Antipolis, France."},{"key":"ref_15","unstructured":"Andreadis, S., Gialampoukidis, I., Karakostas, A., Vrochidis, S., Kompatsiaris, I., Fiorin, R., Norbiato, D., and Ferri, M. (2020, January 14\u201315). The Flood-related Multimedia Task at MediaEval 2020. Proceedings of the MediaEval 2020, Online."},{"key":"ref_16","unstructured":"Imran, M., Mitra, P., and Castillo, C. (2016, January 23\u201328). Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portoroz, Slovenia."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Imran, M., Elbassuoni, S., Castillo, C., Diaz, F., and Meier, P. (2013, January 13\u201317). Practical extraction of disaster-relevant information from social media. Proceedings of the 22nd international Conference on World Wide Web Companion, Rio de Janeiro, Brazil.","DOI":"10.1145\/2487788.2488109"},{"key":"ref_18","unstructured":"Imran, M., Elbassuoni, S.M., Castillo, C., Diaz, F., and Meier, P. (2013, January 12\u201315). Extracting information nuggets from disaster-related messages in social media. Proceedings of the ISCRAM, Baden-Baden, Germany."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Alam, F., Ofli, F., and Imran, M. (2018, January 25\u201328). CrisisMMD: Multimodal Twitter Datasets from Natural Disasters. Proceedings of the 12th International AAAI Conference on Web and Social Media (ICWSM), Palo Alto, CA, USA.","DOI":"10.1609\/icwsm.v12i1.14983"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Nguyen, D.T., Ofli, F., Imran, M., and Mitra, P. (August, January 31). Damage assessment from social media imagery data during disasters. Proceedings of the 2017 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining, Sydney, NSW, Australia.","DOI":"10.1145\/3110025.3110109"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"102107","DOI":"10.1016\/j.ipm.2019.102107","article-title":"Automatic identification of eyewitness messages on twitter during disasters","volume":"57","author":"Zahra","year":"2020","journal-title":"Inf. Process. Manag."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Alam, F., Joty, S., and Imran, M. (2018). Domain Adaptation with Adversarial Training and Graph Embeddings. arXiv.","DOI":"10.18653\/v1\/P18-1099"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Alam, F., Ofli, F., Imran, M., Alam, T., and Qazi, U. (2020, January 7\u201310). Deep Learning Benchmarks and Datasets for Social Media Image Classification for Disaster Response. Proceedings of the 2020 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), The Hague, The Netherlands.","DOI":"10.1109\/ASONAM49781.2020.9381294"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Alam, F., Sajjad, H., Imran, M., and Ofli, F. (2021, January 7\u201310). CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing. Proceedings of the 15th International Conference on Web and Social Media (ICWSM), Online.","DOI":"10.1609\/icwsm.v15i1.18115"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Madichetty, S., and Sridevi, M. (2019, January 7\u201311). Detecting informative tweets during disaster using deep neural networks. Proceedings of the 2019 11th International Conference on Communication Systems & Networks (COMSNETS), Bangalore, India.","DOI":"10.1109\/COMSNETS.2019.8711095"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Nguyen, D.T., Mannai, K.A.A., Joty, S., Sajjad, H., Imran, M., and Mitra, P. (2016). Rapid classification of crisis-related data on social networks using convolutional neural networks. arXiv.","DOI":"10.1609\/icwsm.v11i1.14950"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"104485","DOI":"10.1016\/j.cageo.2020.104485","article-title":"Improving the classification of flood tweets with contextual hydrological information in a multimodal neural network","volume":"140","author":"Weerts","year":"2020","journal-title":"Comput. Geosci."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2016). Bag of tricks for efficient text classification. arXiv.","DOI":"10.18653\/v1\/E17-2068"},{"key":"ref_29","unstructured":"Conneau, A., Lample, G., Ranzato, M., Denoyer, L., and J\u00e9gou, H. (2017). Word translation without parallel data. arXiv."},{"key":"ref_30","unstructured":"Mouzannar, H., Rizk, Y., and Awad, M. (2018, January 20\u201323). Damage Identification in Social Media Posts using Multimodal Deep Learning. Proceedings of the ISCRAM, Rochester, NY, USA."},{"key":"ref_31","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Gautam, A.K., Misra, L., Kumar, A., Misra, K., Aggarwal, S., and Shah, R.R. (2019, January 11\u201313). Multimodal analysis of disaster tweets. Proceedings of the 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), Singapore.","DOI":"10.1109\/BigMM.2019.00-38"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Suk, H.I., and Shen, D. (2013, January 18\u201322). Deep learning-based feature representation for AD\/MCI classification. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Singapore.","DOI":"10.1007\/978-3-642-40763-5_72"},{"key":"ref_34","unstructured":"Ofli, F., Alam, F., and Imran, M. (2020). Analysis of social media data using multimodal deep learning for disaster response. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Khan, Q., Kalbus, E., Zaki, N., and Mohamed, M.M. (2022). Utilization of social media in floods assessment using data mining techniques. PLoS ONE, 17.","DOI":"10.1371\/journal.pone.0267079"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1007\/s00477-021-02161-3","article-title":"Deep learning-based assessment of flood severity using social media streams","volume":"36","author":"Kanth","year":"2022","journal-title":"Stoch. Environ. Res. Risk Assess."},{"key":"ref_37","first-page":"1","article-title":"Learning early detection of emergencies from word usage patterns on social media","volume":"2022","author":"Bono","year":"2022","journal-title":"Inf. Technol. Disaster Risk Reduct."},{"key":"ref_38","unstructured":"Bischke, B., Bhardwaj, P., Gautam, A., Helber, P., Borth, D., and Dengel, A. (2017, January 13\u201315). Detection of Flooding Events in Social Multimedia and Satellite Imagery using Deep Neural Networks. Proceedings of the MediaEval, Dublin, Ireland."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Abavisani, M., Wu, L., Hu, S., Tetreault, J., and Jaimes, A. (2020, January 13\u201319). Multimodal categorization of crisis events in social media. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01469"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"102783","DOI":"10.1016\/j.ijdrr.2022.102783","article-title":"Social media use in disaster recovery: A systematic literature review","volume":"70","author":"Ogie","year":"2022","journal-title":"Int. J. Disaster Risk Reduct."},{"key":"ref_42","unstructured":"Zhou, J., Cui, G., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., and Sun, M. (2018). Graph neural networks: A review of methods and applications. arXiv."},{"key":"ref_43","unstructured":"Wu, L., Li, S., Hsieh, C.J., and Sharpnack, J. (2019). Stochastic shared embeddings: Data-driven regularization of embedding layers. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Jang, B., Kim, M., Harerimana, G., Kang, S.u., and Kim, J.W. (2020). Bi-LSTM model to increase accuracy in text classification: Combining Word2vec CNN and attention mechanism. Appl. Sci., 10.","DOI":"10.3390\/app10175841"},{"key":"ref_45","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv."},{"key":"ref_46","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Sammut, C., and Webb, G.I. (2010). Encyclopedia of Machine Learning, Springer.","DOI":"10.1007\/978-0-387-30164-8"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 11\u201314). Identity mappings in deep residual networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46493-0_38"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2017, January 4\u20139). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Efron, M., Lin, J., He, J., and De Vries, A. (2014, January 6\u201311). Temporal feedback for tweet search with non-parametric density estimation. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, Gold Coast, Australia.","DOI":"10.1145\/2600428.2609575"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1111\/j.2517-6161.1991.tb01857.x","article-title":"A reliable data-based bandwidth selection method for kernel density estimation","volume":"53","author":"Sheather","year":"1991","journal-title":"J. R. Stat. Soc. Ser. (Methodol.)"},{"key":"ref_53","unstructured":"Nikoletopoulos, T., and Wolff, C. (2020, January 14\u201315). A tweet text binary Artificial Neural Network classifier. Proceedings of the MediaEval 2020, Online."},{"key":"ref_54","unstructured":"Said, N., Ahmad, K., Gul, A., Ahmad, N., and Al-Fuqaha, A. (2020). Floods Detection in Twitter Text and Images. arXiv."},{"key":"ref_55","unstructured":"Jony, R.I., and Woodley, A. (2020, January 14\u201315). Flood Detection in Twitter Using a Novel Learning Method for Neural Networks. Proceedings of the MediaEval 2020, Online."},{"key":"ref_56","unstructured":"Hanif, M., Joozer, H., Tahir, M.A., and Rafi, M. (2020, January 14\u201315). Ensemble based method for the classification of flooding event using social media data. Proceedings of the MediaEval 2020, Online."},{"key":"ref_57","unstructured":"Alam, F., Hassan, Z., Ahmad, K., Gul, A., Reiglar, M., Conci, N., and Al-Fuqaha, A. (2020). Flood Detection via Twitter Streams using Textual and Visual Features. arXiv."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Jacovi, A., Shalom, O.S., and Goldberg, Y. (2018). Understanding convolutional neural networks for text classification. arXiv.","DOI":"10.18653\/v1\/W18-5408"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"51522","DOI":"10.1109\/ACCESS.2019.2909919","article-title":"Sentiment Analysis of Comment Texts Based on BiLSTM","volume":"7","author":"Xu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Bloice, M.D., Stocker, C., and Holzinger, A. (2017). Augmentor: An image augmentation library for machine learning. arXiv.","DOI":"10.21105\/joss.00432"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"54","DOI":"10.3389\/frobt.2016.00054","article-title":"Image classification to support emergency situation awareness","volume":"3","author":"Lagerstrom","year":"2016","journal-title":"Front. Robot. AI"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Prusa, J., Khoshgoftaar, T.M., Dittman, D.J., and Napolitano, A. (2015, January 13\u201315). Using Random Undersampling to Alleviate Class Imbalance on Tweet Sentiment Data. Proceedings of the 2015 IEEE International Conference on Information Reuse and Integration, San Francisco, CA, USA.","DOI":"10.1109\/IRI.2015.39"},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Alam, F., Joty, S., and Imran, M. (2018, January 25\u201328). Graph based semi-supervised learning with convolution neural networks to classify crisis related tweets. Proceedings of the International AAAI Conference on Web and Social Media, Palo Alto, CA, USA.","DOI":"10.1609\/icwsm.v12i1.15047"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/7\/3767\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:11:00Z","timestamp":1760123460000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/7\/3767"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,6]]},"references-count":64,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["s23073767"],"URL":"https:\/\/doi.org\/10.3390\/s23073767","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,4,6]]}}}