{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,3]],"date-time":"2025-12-03T20:38:07Z","timestamp":1764794287163,"version":"build-2065373602"},"reference-count":39,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,5,5]],"date-time":"2022-05-05T00:00:00Z","timestamp":1651708800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"EmoRe"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Users of web or chat social networks typically use emojis (e.g., smilies, memes, hearts) to convey in their textual interactions the emotions underlying the context of the communication, aiming for better interpretability, especially for short polysemous phrases. Semantic-based context recognition tools, employed in any chat or social network, can directly comprehend text-based emoticons (i.e., emojis created from a combination of symbols and characters) and translate them into audio information (e.g., text-to-speech readers for individuals with vision impairment). On the other hand, for a comprehensive understanding of the semantic context, image-based emojis require image-recognition algorithms. This study aims to explore and compare different classification methods for pictograms, applied to emojis collected from Internet sources. Each emoji is labeled according to the basic Ekman model of six emotional states. The first step involves extraction of emoji features through convolutional neural networks, which are then used to train conventional supervised machine learning classifiers for purposes of comparison. The second experimental step broadens the comparison to deep learning networks. The results reveal that both the conventional and deep learning classification approaches accomplish the goal effectively, with deep transfer learning exhibiting a highly satisfactory performance, as expected.<\/jats:p>","DOI":"10.3390\/fi14050142","type":"journal-article","created":{"date-parts":[[2022,5,5]],"date-time":"2022-05-05T13:10:26Z","timestamp":1651756226000},"page":"142","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Tell Me More: Automating Emojis Classification for Better Accessibility and Emotional Context Recognition"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3460-7164","authenticated-orcid":false,"given":"Muhammad","family":"Atif","sequence":"first","affiliation":[{"name":"Department of Mathematics and Computer Science, University of Florence, Viale Morgagni 67\/a, 50134 Florence, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2972-7188","authenticated-orcid":false,"given":"Valentina","family":"Franzoni","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Computer Science, University of Perugia, Via Vanvitelli 1, 06123 Perugia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Read, J. (2005). Using Emoticons to Reduce Dependency in Machine Learning Techniques for Sentiment Classification. Proceedings of the ACL Student Research Workshop, Association for Computational Linguistics.","DOI":"10.3115\/1628960.1628969"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1804","DOI":"10.1108\/IMDS-12-2017-0582","article-title":"Multi-class Twitter sentiment classification with emojis","volume":"118","author":"Li","year":"2018","journal-title":"Ind. Manag. Data Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1007\/s40031-021-00620-7","article-title":"Exploiting Emojis in Sentiment Analysis: A Survey","volume":"103","author":"Grover","year":"2022","journal-title":"J. Inst. Eng. India Ser. B"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Atif, M., Franzoni, V., and Milani, A. (2021, January 17\u201319). Emojis Pictogram Classification for Semantic Recognition of Emotional Context. Proceedings of the 14th International Conference on Brain Informatics, Virtual Event.","DOI":"10.1007\/978-3-030-86993-9_14"},{"key":"ref_5","unstructured":"Chandra, P., and Prasad, U. (2021, January 17\u201319). Classification of Emojis using Artificial Neural Network and Natural Language Processing. Proceedings of the 2021 8th International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Guo, G., Wang, H., Bell, D., Bi, Y., and Greer, K. (2003, January 3\u20137). KNN model-based approach in classification. Proceedings of the OTM Confederated International Conferences CoopIS, DOA, and ODBASE 2003 Catania, Sicily, Italy.","DOI":"10.1007\/978-3-540-39964-3_62"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1109\/72.991427","article-title":"A comparison of methods for multiclass support vector machines","volume":"13","author":"Hsu","year":"2002","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging Predictors","volume":"26","author":"Breiman","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_9","unstructured":"Mitchell, T.M. (1997). Machine Learning, McGraw Hill."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1006\/jcss.1997.1504","article-title":"A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting","volume":"55","author":"Freund","year":"1997","journal-title":"J. Comput. Syst. Sci."},{"key":"ref_11","first-page":"73","article-title":"Discriminant Function Analysis: Concept and Application","volume":"33","year":"2008","journal-title":"Eurasian J. Educ. Res. (EJER)"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"17","DOI":"10.3233\/WEB-190397","article-title":"Automating facial emotion recognition","volume":"17","author":"Gervasi","year":"2019","journal-title":"Web Intell."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Carlini, N., and Wagner, D. (2017, January 22\u201324). Towards Evaluating the Robustness of Neural Networks. Proceedings of the 2017 IEEE Symposium on Security and Privacy, San Jose, CA, USA.","DOI":"10.1109\/SP.2017.49"},{"key":"ref_14","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_16","unstructured":"Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2022, March 15). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Available online: https:\/\/openaccess.thecvf.com\/content_cvpr_2018\/papers\/Sandler_MobileNetV2_Inverted_Residuals_CVPR_2018_paper.pdf.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the Inception Architecture for Computer Vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Biondi, G., Franzoni, V., and Poggioni, V. (2017). A deep learning semantic approach to emotion recognition using the IBM watson bluemix alchemy language. Computational Science and Its Applications\u2014ICCSA 2017, Springer International Publishing. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics).","DOI":"10.1007\/978-3-319-62398-6_51"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Franzoni, V., Milani, A., and Biondi, G. (2017, January 23\u201326). SEMO: A semantic model for emotion recognition in web objects. Proceedings of the 2017 IEEE\/WIC\/ACM International Conference on Web Intelligence, Leipzig, Germany.","DOI":"10.1145\/3106426.3109417"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1007\/s13278-021-00776-6","article-title":"A review on sentiment analysis and emotion detection from text","volume":"11","author":"Nandwani","year":"2021","journal-title":"Soc. Netw. Anal. Min."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"e26","DOI":"10.7717\/peerj-cs.26","article-title":"Quantifying the effect of sentiment on information diffusion in social media","volume":"1","author":"Ferrara","year":"2015","journal-title":"PeerJ Comput. Sci."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1080\/02699939208411068","article-title":"An argument for basic emotions","volume":"6","author":"Ekman","year":"1992","journal-title":"Cogn. Emot."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.patrec.2019.01.008","article-title":"Extended deep neural network for facial emotion recognition","volume":"120","author":"Jain","year":"2019","journal-title":"Pattern Recognit. Lett."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1319","DOI":"10.1109\/TMM.2016.2557721","article-title":"Sparse kernel reduced-rank regression for bimodal emotion recognition from facial expression and speech","volume":"18","author":"Yan","year":"2016","journal-title":"IEEE Trans. Multimed."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Wollmer, M., Metallinou, A., Eyben, F., Schuller, B., and Narayanan, S. (2010, January 26\u201330). Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling. Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH 2010), Chiba, Japan.","DOI":"10.21437\/Interspeech.2010-646"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"He, G., Liu, X., Fan, F., and You, J. (2020, January 14\u201319). Image2audio: Facilitating semi-supervised audio emotion recognition with facial expression image. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00464"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Franzoni, V., and Milani, A. (2015, January 6\u20138). Semantic context extraction from collaborative networks. Proceedings of the 2015 IEEE 19th International Conference on Computer Supported Cooperative Work in Design, CSCWD 2015, Calabria, Italy.","DOI":"10.1109\/CSCWD.2015.7230946"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Sahoo, J., Prakash, S.A., and Patra, S.K. (2019, January 16\u201318). Hand gesture recognition using PCA based deep CNN reduced features and SVM classifier. Proceedings of the 2019 IEEE International Symposium on Smart Electronic Systems (iSES) (Formerly iNiS), Rourkela, India.","DOI":"10.1109\/iSES47678.2019.00056"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Tarawneh, A.S., Hassanat, A.B., Chetverikov, D., Lendak, I., and Verma, C. (2019, January 9\u201311). Invoice classification using deep features and machine learning techniques. Proceedings of the 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), Amman, Jordan.","DOI":"10.1109\/JEEIT.2019.8717504"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"16","DOI":"10.33640\/2405-609X.1675","article-title":"Fruit Recognition using Support Vector Machine based on Deep Features","volume":"6","author":"Behera","year":"2020","journal-title":"Karbala Int. J. Mod. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Mahbod, A., Schaefer, G., Wang, C., Ecker, R., and Ellinge, I. (2019, January 12-17). Skin lesion classification using hybrid deep neural networks. Proceedings of the ICASSP 2019\u20142019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8683352"},{"key":"ref_33","unstructured":"Liu, Y., Yuan, G., and Wotao, Y. (2020, January 6\u201312). An improved analysis of stochastic gradient descent with momentum. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Warmuth, M., Liao, J., and Ratsch, G. (2006, January 25\u201329). Totally corrective boosting algorithms that maximize the margin. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.","DOI":"10.1145\/1143844.1143970"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Franzoni, V., Biondi, G., Perri, D., and Gervasi, O. (2020). Enhancing Mouth-Based Emotion Recognition Using Transfer Learning. Sensors, 20.","DOI":"10.20944\/preprints202007.0379.v1"},{"key":"ref_38","unstructured":"Lei Ba, J., and Kingma, D.P. (2015, January 7\u20139). Adam: A method for stochastic gradient descent. Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Dogo, E.M., Afolabi, O.J., Nwulu, N.I., Twala, B., and Aigbavboa, C.O. (2018, January 21\u201322). A comparative analysis of gradient descent-based optimization algorithms on convolutional neural networks. Proceedings of the 2018 International Conference on Computational Techniques, Electronics and Mechanical Systems (CTEMS), Belgaum, India.","DOI":"10.1109\/CTEMS.2018.8769211"}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/14\/5\/142\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:06:27Z","timestamp":1760137587000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/14\/5\/142"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,5]]},"references-count":39,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["fi14050142"],"URL":"https:\/\/doi.org\/10.3390\/fi14050142","relation":{},"ISSN":["1999-5903"],"issn-type":[{"type":"electronic","value":"1999-5903"}],"subject":[],"published":{"date-parts":[[2022,5,5]]}}}