{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T05:07:39Z","timestamp":1780636059287,"version":"3.54.1"},"reference-count":35,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2019,4,15]],"date-time":"2019-04-15T00:00:00Z","timestamp":1555286400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"published-print":{"date-parts":[[2019,5,14]]},"abstract":"<jats:p>\n                    Natural Language Processing problems has recently been benefited for the advances in Deep Learning. Many of these problems can be addressed as a multi-label classification problem. Usually, the metrics used to evaluate classification models are different from the loss functions used in the learning process. In this paper, we present a strategy to incorporate evaluation metrics in the learning process in order to increase the performance of the classifier according to the measure we are interested to favor. Concretely, we propose soft versions of the Accuracy, micro-\n                    <jats:italic>F<\/jats:italic>\n                    <jats:sub>1<\/jats:sub>\n                    , and macro-\n                    <jats:italic>F<\/jats:italic>\n                    <jats:sub>1<\/jats:sub>\n                    measures that can be used as loss functions in the back-propagation algorithm. In order to experimentally validate our approach, we tested our system in an Emotion Classification task proposed at the International Workshop on Semantic Evaluation, SemEval-2018. Using a Convolutional Neural Network trained with the proposed loss functions we obtained significant improvements both for the English and the Spanish corpora.\n                  <\/jats:p>","DOI":"10.3233\/jifs-179019","type":"journal-article","created":{"date-parts":[[2019,4,16]],"date-time":"2019-04-16T17:16:38Z","timestamp":1555434998000},"page":"4697-4708","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":5,"title":["Choosing the right loss function for multi-label Emotion Classification"],"prefix":"10.1177","volume":"36","author":[{"given":"Llu\u00eds-F.","family":"Hurtado","sequence":"first","affiliation":[{"name":"Departament de Sistemes Inform\u00e0tics i Computaci\u00f3 Universitat Polit\u00e8cnica de Val\u00e8ncia Cam\u00ed de Vera sn, Val\u00e8ncia, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jos\u00e9-\u00c1ngel","family":"Gonz\u00e1lez","sequence":"additional","affiliation":[{"name":"Departament de Sistemes Inform\u00e0tics i Computaci\u00f3 Universitat Polit\u00e8cnica de Val\u00e8ncia Cam\u00ed de Vera sn, Val\u00e8ncia, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ferran","family":"Pla","sequence":"additional","affiliation":[{"name":"Departament de Sistemes Inform\u00e0tics i Computaci\u00f3 Universitat Polit\u00e8cnica de Val\u00e8ncia Cam\u00ed de Vera sn, Val\u00e8ncia, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2019,4,15]]},"reference":[{"key":"e_1_3_3_2_2","article-title":"Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining","author":"Baccianella S.","year":"2010","unstructured":"BaccianellaS., EsuliA. and SebastianiF., Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining, In in Proc of LREC, 2010.","journal-title":"In in Proc of LREC"},{"key":"e_1_3_3_3_2","doi-asserted-by":"crossref","first-page":"4153","DOI":"10.1109\/ICASSP.1997.604861","volume-title":"In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing","volume":"5","author":"Bilmes J.","year":"1997","unstructured":"BilmesJ., AsanovicK., ChinC.-W. and DemmelJ., Using phipac to speed error back-propagation learning, In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 5, 1997, pp. 4153\u20134156."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2014.04.005"},{"key":"e_1_3_3_5_2","first-page":"1130","volume-title":"Proceedings of the 30th International Conference on Machine Learning volume 28 of Proceedings of Machine Learning Research","author":"Dembczynski K.","year":"2013","unstructured":"DembczynskiK., JachnikA., KotlowskiW., WaegemanW. and HuellermeierE., Optimizing the F-Measure in Multi-Label Classification: Plug-in Rule Approach versus Structured Loss Minimization, In DasguptaS. and McAllesterD., editors, Proceedings of the 30th International Conference on Machine Learning volume 28 of Proceedings of Machine Learning Research, Atlanta, Georgia, USA, PMLR, 2013, pp. 1130\u20131138."},{"key":"e_1_3_3_6_2","volume-title":"Twitter sentiment classification using distant supervision","author":"Go A.","year":"2009","unstructured":"GoA., BhayaniR., HuangL., Twitter sentiment classification using distant supervision, Stanford University, Technical report, 2009."},{"key":"e_1_3_3_7_2","first-page":"146","article-title":"Multimedia lab @ ACL W-NUT NER sharedtask: Named entity recognition for Twitter microposts using distributed word representations","volume":"2015","author":"Godin F.","year":"2015","unstructured":"GodinF., VandersmissenB., De NeveW. and Van de WalleR. , Multimedia lab @ ACL W-NUT NER sharedtask: Named entity recognition for Twitter microposts using distributed word representations, ACL-IJCNLP2015 (2015), 146\u2013153.","journal-title":"ACL-IJCNLP"},{"key":"e_1_3_3_8_2","unstructured":"GoodfellowI. BengioY. and CourvilleA. Deep Learning MIT Press http:\/\/www.deeplearningbook.org (2016)."},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014073"},{"key":"e_1_3_3_10_2","article-title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","author":"Ioffe S.","year":"2015","unstructured":"IoffeS. and SzegedyC., Batch normalization: Accelerating deep network training by reducing internal covariate shift, CoRR, abs\/1502.03167 (2015).","journal-title":"CoRR, abs\/1502.03167"},{"key":"e_1_3_3_11_2","article-title":"On loss functions for deep neural networks in classification","author":"Janocha K.","year":"2017","unstructured":"JanochaK. and CzarneckiW.M., On loss functions for deep neural networks in classification, CoRR, abs\/1702.05659 (2017).","journal-title":"CoRR, abs\/1702.05659"},{"key":"e_1_3_3_12_2","volume-title":"In Proc of AAAI Conference on Weblogs and Social","author":"Krieger M.","year":"2010","unstructured":"KriegerM. and AhnD., Tweetmotif: Exploratory search and topic summarization for twitter, In Proc of AAAI Conference on Weblogs and Social, 2010."},{"key":"e_1_3_3_13_2","unstructured":"LiuB. Sentiment Analysis and Opinion Mining A Comprehensive Introduction and Survey. Morgan & Claypool Publishers 2012."},{"key":"e_1_3_3_14_2","article-title":"Distributed representations of words and phrases and their compositionality","author":"Mikolov T.","year":"2013","unstructured":"MikolovT., SutskeverI., ChenK., CorradoG. and DeanJ., Distributed representations of words and phrases and their compositionality, CoRR, abs\/1310.4546 (2013a).","journal-title":"CoRR, abs\/1310.4546"},{"key":"e_1_3_3_15_2","unstructured":"MikolovT. ChenK. CorradoG. and DeanJ. Efficient estimation of word representations in vector space CoRR abs\/1301.3781 2013b."},{"key":"e_1_3_3_16_2","first-page":"246","volume-title":"*SEM 2012: The First Joint Conference on Lexical and Computational Semantics \u2013 Volume 1: Proceedings of the Main Conference and the Shared Task and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)","author":"Mohammad S.","year":"2012","unstructured":"MohammadS., #emotional tweets, In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics \u2013 Volume 1: Proceedings of the Main Conference and the Shared Task and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), Montr\u00e9al, Canada. Association for Computational Linguistics, 2012, pp. 246\u2013255."},{"key":"e_1_3_3_17_2","first-page":"31","volume-title":"Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)","author":"Mohammad S.","year":"2016","unstructured":"MohammadS., KiritchenkoS., SobhaniP., ZhuX. and CherryC., Semeval-task 6: Detecting stance in tweets, In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016, pp. 31\u201341."},{"key":"e_1_3_3_18_2","doi-asserted-by":"crossref","unstructured":"MohammadS.M. and Bravo-MarquezF. WASSA-shared task on emotion intensity CoRR abs\/1708.03700 2017.","DOI":"10.18653\/v1\/W17-5205"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8640.2012.00460.x"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3003433"},{"key":"e_1_3_3_21_2","volume-title":"Proceedings of International Workshop on Semantic Evaluation (SemEval-2018)","author":"Mohammad S.M.","year":"2018","unstructured":"MohammadS.M., Bravo-MarquezF., SalamehM. and KiritchenkoS., Semeval-2018 Task 1: Affect in tweets, In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA, 2018."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2013.06.076"},{"key":"e_1_3_3_23_2","volume-title":"The Practice of Business Statistics Companion Chapter 18: Bootstrap Methods and Permutation Tests","author":"Moore D.","year":"2003","unstructured":"MooreD., McCabeG., DuckworthW., ScloveS., The Practice of Business Statistics Companion Chapter 18: Bootstrap Methods and Permutation Tests, W. H. Freeman, 2003."},{"key":"e_1_3_3_24_2","first-page":"807","volume-title":"Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML\u201910","author":"Nair V.","year":"2010","unstructured":"NairV. and HintonG.E., Rectified linear units improve restricted boltzmann machines, In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML\u201910, USA, 2010, pp. 807\u2013814. Omnipress."},{"key":"e_1_3_3_25_2","unstructured":"NielsenF.\u00c5. AFINN 2011."},{"key":"e_1_3_3_26_2","volume-title":"IWANN Proceedings","author":"Pastor-Pellicer J.","year":"2013","unstructured":"Pastor-PellicerJ., Zamora-Mart\u00ednezF., Espa\u00f1a BoqueraS. and Castro BledaM.J. , F-Measure as the Error Function to Train Neural Networks, In IWANN Proceedings, 2013."},{"key":"e_1_3_3_27_2","unstructured":"PennebakerJ. ChungC. IrelandM. GonzalesA. and BoothR. The development and psychological properties of liwc2007 2014."},{"issue":"3","key":"e_1_3_3_28_2","doi-asserted-by":"crossref","first-page":"965","DOI":"10.1007\/s10115-016-0997-x","article-title":"Language identification of multilingual posts from twitter: A case study","volume":"51","author":"Pla F.","year":"2017","unstructured":"PlaF. and HurtadoL.-F., Language identification of multilingual posts from twitter: A case study, Knowledge and Information Systems51(3) (2017), 965\u2013989.","journal-title":"Knowledge and Information Systems"},{"key":"e_1_3_3_29_2","volume-title":"Proceedings of the 11th International Workshop on Semantic Evaluation","author":"Rosenthal S.","year":"2017","unstructured":"RosenthalS., FarraN. and NakovP., SemEval-2017 task 4: Sentiment analysis in Twitter, In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval \u201917, Vancouver, Canada, Association for Computational Linguistics, 2017."},{"key":"e_1_3_3_30_2","first-page":"143","volume-title":"XXIX Congreso de la Sociedad Espaola de Procesamiento de Lenguaje Natural, Workshop on Sentiment Analysis at SEPLN (TASS2013)","author":"Saralegi X.","year":"2013","unstructured":"SaralegiX. and SanI., Vicente, Elhuyar at tass 2013, In XXIX Congreso de la Sociedad Espaola de Procesamiento de Lenguaje Natural, Workshop on Sentiment Analysis at SEPLN (TASS2013), 2013, pp. 143\u2013150."},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505283"},{"key":"e_1_3_3_32_2","first-page":"2017","volume-title":"In Notebook Papers of 2nd SEPLN Workshop on Evaluation of Human Language Technologies for Iberian Languages (IBEREVAL)","author":"Taul\u00e9 M.","year":"2017","unstructured":"Taul\u00e9M., Mart\u00edM., RangelF., RossoP., BoscoC. and PattiV., Overview of the task of Stance and Gender Detection in Tweets on Catalan Independence at IBEREVAL 2017, In Notebook Papers of 2nd SEPLN Workshop on Evaluation of Human Language Technologies for Iberian Languages (IBEREVAL), Murcia (Spain). CEUR Workshop Proceedings. CEUR-WS.org, 2017, 2017."},{"key":"e_1_3_3_33_2","first-page":"512","article-title":"Humans require context to infer ironic intent (so computers probably do, too)","author":"Wallace B.C.","year":"2014","unstructured":"WallaceB.C., ChoeD.K., KertzL. and CharniakE., Humans require context to infer ironic intent (so computers probably do, too), In ACL (2), The Association for Computer Linguistics, 2014, pp. 512\u2013516.","journal-title":"In ACL (2)"},{"issue":"2","key":"e_1_3_3_34_2","article-title":"Annotating expressions of opinions and emotions in language","volume":"1","author":"Wiebe J.","year":"2005","unstructured":"WiebeJ., WilsonT. and CardieC., Annotating expressions of opinions and emotions in language, Language Resources and Evaluation1(2) (2005).","journal-title":"Language Resources and Evaluation"},{"key":"e_1_3_3_35_2","doi-asserted-by":"crossref","first-page":"347","DOI":"10.3115\/1220575.1220619","volume-title":"Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT \u201905","author":"Wilson T.","year":"2005","unstructured":"WilsonT., WiebeJ. and HoffmannP., Recognizing contextual polarity in phrase-level sentiment analysis, In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT \u201905, Stroudsburg, PA, USA, 2005, pp. 347\u2013354. Association for Computational Linguistics."},{"key":"e_1_3_3_36_2","first-page":"253","volume-title":"Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Zhang Y.","year":"2017","unstructured":"ZhangY. and WallaceB., A sensitivity analysis of (and practitioners\u2019 guide to) convolutional neural networks for sentence classification, In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2017, pp. 253\u2013263. Asian Federation of Natural Language Processing."}],"container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-179019","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/JIFS-179019","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-179019","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:37:38Z","timestamp":1777455458000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/JIFS-179019"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4,15]]},"references-count":35,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2019,5,14]]}},"alternative-id":["10.3233\/JIFS-179019"],"URL":"https:\/\/doi.org\/10.3233\/jifs-179019","relation":{},"ISSN":["1064-1246","1875-8967"],"issn-type":[{"value":"1064-1246","type":"print"},{"value":"1875-8967","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,4,15]]}}}