{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T18:43:43Z","timestamp":1774550623317,"version":"3.50.1"},"reference-count":50,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2020,6,19]],"date-time":"2020-06-19T00:00:00Z","timestamp":1592524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,6,19]],"date-time":"2020-06-19T00:00:00Z","timestamp":1592524800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Stiftelsen f\u00f6r Kunskapsoch Kompetensutveckling","award":["20140032"],"award-info":[{"award-number":["20140032"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2021,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Classifying e-mails into distinct labels can have a great impact on customer support. By using machine learning to label e-mails, the system can set up queues containing e-mails of a specific category. This enables support personnel to handle request quicker and more easily by selecting a queue that match their expertise. This study aims to improve a manually defined rule-based algorithm, currently implemented at a large telecom company, by using machine learning. The proposed model should have higher<jats:inline-formula><jats:alternatives><jats:tex-math>$$F_1$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:msub><mml:mi>F<\/mml:mi><mml:mn>1<\/mml:mn><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>-score and classification rate. Integrating or migrating from a manually defined rule-based model to a machine learning model should also reduce the administrative and maintenance work. It should also make the model more flexible. By using the frameworks, TensorFlow, Scikit-learn and Gensim, the authors conduct a number of experiments to test the performance of several common machine learning algorithms, text-representations, word embeddings to investigate how they work together. A long short-term memory network showed best classification performance with an<jats:inline-formula><jats:alternatives><jats:tex-math>$$F_1$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:msub><mml:mi>F<\/mml:mi><mml:mn>1<\/mml:mn><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>-score of 0.91. The authors conclude that long short-term memory networks outperform other non-sequential models such as support vector machines and AdaBoost when predicting labels for e-mails. Further, the study also presents a Web-based interface that were implemented around the LSTM network, which can classify e-mails into 33 different labels.<\/jats:p>","DOI":"10.1007\/s00521-020-05058-4","type":"journal-article","created":{"date-parts":[[2020,6,19]],"date-time":"2020-06-19T06:02:36Z","timestamp":1592546556000},"page":"1881-1902","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["E-mail classification with machine learning and word embeddings for improved customer support"],"prefix":"10.1007","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8929-7220","authenticated-orcid":false,"given":"Anton","family":"Borg","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Martin","family":"Boldt","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Oliver","family":"Rosander","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jim","family":"Ahlstrand","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,6,19]]},"reference":[{"key":"5058_CR1","unstructured":"Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Man\u00e9 D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Vi\u00e9gas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org"},{"issue":"2","key":"5058_CR2","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1016\/S0271-5309(98)00005-6","volume":"18","author":"NS Baron","year":"1998","unstructured":"Baron NS (1998) Letters by phone or speech by other means: the linguistics of email. Lang Commun 18(2):133\u2013170","journal-title":"Lang Commun"},{"key":"5058_CR3","doi-asserted-by":"crossref","unstructured":"Borg A, Boldt M, Svensson J (2019) Using conformal prediction for multi-label document classification in e-mail support systems. In: International conference on industrial, engineering and other applications of applied intelligent systems. Springer, pp 308\u2013322","DOI":"10.1007\/978-3-030-22999-3_28"},{"key":"5058_CR4","doi-asserted-by":"crossref","unstructured":"Bottou L (2012) Stochastic gradient descent tricks. In: Neural networks: tricks of the trade, 2nd edn. Springer, Berlin, pp 421\u2013436","DOI":"10.1007\/978-3-642-35289-8_25"},{"issue":"4","key":"5058_CR5","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1177\/0092070303254412","volume":"31","author":"R Bougie","year":"2003","unstructured":"Bougie R, Pieters R, Zeelenberg M (2003) Angry customers don\u2019t come back, they get back: the experience and behavioral implications of anger and dissatisfaction in services. J Acad Mark Sci 31(4):377\u2013393","journal-title":"J Acad Mark Sci"},{"key":"5058_CR6","unstructured":"Buitinck L, Louppe G, Blondel M, Pedregosa F, Mueller A, Grisel O, Niculae V, Prettenhofer P, Gramfort A, Grobler J, Layton R, VanderPlas J, Joly A, Holt B, Varoquaux G (2013) API design for machine learning software: experiences from the scikit-learn project. In: ECML PKDD workshop: languages for data mining and machine learning, pp 108\u2013122"},{"key":"5058_CR7","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1109\/MCI.2014.2307227","volume":"9","author":"E Cambria","year":"2014","unstructured":"Cambria E, White B (2014) Jumping NLP curves: a review of natural language processing research. IEEE Comput Intell Mag 9:48\u201357","journal-title":"IEEE Comput Intell Mag"},{"issue":"1","key":"5058_CR8","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1002\/aris.1440370103","volume":"37","author":"GG Chowdhury","year":"2003","unstructured":"Chowdhury GG (2003) Natural language processing. Annu Rev Inf Sci Technol 37(1):51\u201389. https:\/\/doi.org\/10.1002\/aris.1440370103","journal-title":"Annu Rev Inf Sci Technol"},{"key":"5058_CR9","first-page":"177","volume":"151","author":"DM Christopher","year":"2008","unstructured":"Christopher DM, Prabhakar R, Hinrich S (2008) Introduction to information retrieval. Introd Inform Retriev 151:177","journal-title":"Introd Inform Retriev"},{"issue":"6","key":"5058_CR10","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1016\/j.jbi.2005.02.009","volume":"38","author":"AR Coden","year":"2005","unstructured":"Coden AR, Pakhomov SV, Ando RK, Duffy PH, Chute CG (2005) Domain-specific language models and lexicons for tagging. J Biomed Inform 38(6):422\u2013430","journal-title":"J Biomed Inform"},{"issue":"3","key":"5058_CR11","first-page":"273","volume":"20","author":"C Cortes","year":"1995","unstructured":"Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273\u2013297","journal-title":"Mach Learn"},{"issue":"4","key":"5058_CR12","doi-asserted-by":"publisher","first-page":"870","DOI":"10.1016\/j.dss.2007.10.010","volume":"44","author":"K Coussement","year":"2008","unstructured":"Coussement K, den Poel DV (2008) Improving customer complaint management by automatic email classification using linguistic style features as predictors. Decis Support Syst 44(4):870\u2013882","journal-title":"Decis Support Syst"},{"key":"5058_CR13","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1016\/j.patrec.2016.06.012","volume":"80","author":"C De Boom","year":"2016","unstructured":"De Boom C, Van Canneyt S, Demeester T, Dhoedt B (2016) Representation learning for very short texts using weighted word embedding aggregation. Pattern Recogn Lett 80:150\u2013156","journal-title":"Pattern Recogn Lett"},{"key":"5058_CR14","unstructured":"Domingos P, Pazzani M (1996) Beyond independence: conditions for the optimality of the simple bayesian classifer. In: Proceedings of 13th international conference machine learning, pp 105\u2013112"},{"key":"5058_CR15","unstructured":"Eide SR, Tahmasebi N, Borin L (2016) The Swedish culturomics gigaword corpus: a one billion word Swedish reference dataset for NLP"},{"key":"5058_CR16","unstructured":"Fallgren P, Segeblad J, Kuhlmann M (2016) Towards a standard dataset of Swedish word vectors. In: Sixth Swedish Language Technology Conference (SLTC)"},{"key":"5058_CR17","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511973000","volume-title":"Machine learning: the art and science of algorithms that make sense of data","author":"P Flach","year":"2012","unstructured":"Flach P (2012) Machine learning: the art and science of algorithms that make sense of data. Cambridge University Press, Cambridge"},{"key":"5058_CR18","doi-asserted-by":"crossref","unstructured":"Freund Y, Schapire RE (1995) A desicion-theoretic generalization of on-line learning and an application to boosting. In: European conference on computational learning theory. Springer, pp 23\u201337","DOI":"10.1007\/3-540-59119-2_166"},{"key":"5058_CR19","doi-asserted-by":"crossref","unstructured":"Gabrilovich E, Markovitch S (2004) Text categorization with many redundant features: using aggressive feature selection to make SVMS competitive with c4. 5. In: Proceedings of the twenty-first international conference on Machine learning. ACM, p\u00a041","DOI":"10.1145\/1015330.1015388"},{"key":"5058_CR20","doi-asserted-by":"publisher","unstructured":"Graves A, Mohamed A, Hinton G (2013) Speech recognition with deep recurrent neural networks. In: 2013 IEEE International conference on acoustics, speech and signal processing, pp 6645\u20136649. https:\/\/doi.org\/10.1109\/ICASSP.2013.6638947","DOI":"10.1109\/ICASSP.2013.6638947"},{"issue":"5","key":"5058_CR21","doi-asserted-by":"publisher","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","volume":"18","author":"A Graves","year":"2005","unstructured":"Graves A, Schmidhuber J (2005) Framewise phoneme classification with bidirectional lSTM and other neural network architectures. Neural Netw 18(5):602\u2013610 IJCNN 2005","journal-title":"Neural Netw"},{"key":"5058_CR22","unstructured":"Halpin N (2016) The customer service report: Why great customer service matters even more in the age of e-commerce and the channels that perform best. http:\/\/www.businessinsider.com\/customer-service-experiences-are-more-important-than-ever-in-the-age-of-e-commerce-2016-3?r=US&IR=T&IR=T"},{"issue":"8","key":"5058_CR23","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735\u20131780. https:\/\/doi.org\/10.1162\/neco.1997.9.8.1735","journal-title":"Neural Comput"},{"key":"5058_CR24","volume-title":"Data mining\u2014practical machine learning tools and techniques","author":"IH Witten","year":"2011","unstructured":"Witten IH, Frank E, Hall M (2011) Data mining\u2014practical machine learning tools and techniques, 3rd edn. Elsevier, Amsterdam","edition":"3"},{"key":"5058_CR25","volume-title":"Taming text: how to find, organize, and manipulate it","author":"GS Ingersoll","year":"2013","unstructured":"Ingersoll GS, Morton TSM, Farris D (2013) Taming text: how to find, organize, and manipulate it, 1st edn. Manning Publications, Shelter Island","edition":"1"},{"key":"5058_CR26","doi-asserted-by":"publisher","unstructured":"Kar P, Li S, Narasimhan H, Chawla S, Sebastiani F (2016) Online optimization methods for the quantification problem, pp 1625\u20131634. https:\/\/doi.org\/10.1145\/2939672.2939832","DOI":"10.1145\/2939672.2939832"},{"key":"5058_CR27","unstructured":"Kusner M, Sun Y, Kolkin N, Weinberger K (2015) From word embeddings to document distances. In: International conference on machine learning, pp 957\u2013966"},{"key":"5058_CR28","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1162\/tacl_a_00134","volume":"3","author":"O Levy","year":"2015","unstructured":"Levy O, Goldberg Y, Dagan I (2015) Improving distributional similarity with lessons learned from word embeddings. Trans Assoc Comput Linguist 3:211\u2013225","journal-title":"Trans Assoc Comput Linguist"},{"key":"5058_CR29","doi-asserted-by":"crossref","unstructured":"Lewis DD (1998) Naive (bayes) at forty: the independence assumption in information retrieval. In: European conference on machine learning. Springer, pp 4\u201315","DOI":"10.1007\/BFb0026666"},{"key":"5058_CR30","unstructured":"Loshchilov I, Hutter F (2016) SQDR: stochastic gradient descent with restarts. arXiv preprint arXiv:1608.03983"},{"key":"5058_CR31","unstructured":"McCormick C (2017) Word2vec tutorial\u2014the skip-gram model. http:\/\/mccormickml.com\/2016\/04\/19\/word2vec-tutorial-the-skip-gram-model\/"},{"key":"5058_CR32","unstructured":"Meta: Data dumps \u2014 meta, discussion about wikimedia projects (2017) https:\/\/meta.wikimedia.org\/w\/index.php?title=Data_dumps&oldid=17422082. Online; Accessed 18 December 2017"},{"key":"5058_CR33","unstructured":"Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781"},{"key":"5058_CR34","first-page":"3111","volume":"26","author":"T Mikolov","year":"2013","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. Adv Neural Inform Process Syst 26:3111\u20133119","journal-title":"Adv Neural Inform Process Syst"},{"key":"5058_CR35","doi-asserted-by":"crossref","unstructured":"Nowak J, Taspinar A, Scherer R (2017) LSTM recurrent neural networks for short text and sentiment classification. Springer, Cham, pp 553\u2013562","DOI":"10.1007\/978-3-319-59060-8_50"},{"key":"5058_CR36","volume-title":"Advanced data mining techniques","author":"DL Olson","year":"2008","unstructured":"Olson DL, Delen D (2008) Advanced data mining techniques. Springer, Berlin"},{"key":"5058_CR37","doi-asserted-by":"crossref","unstructured":"Pennington J, Socher R, Manning C (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532\u20131543","DOI":"10.3115\/v1\/D14-1162"},{"issue":"4","key":"5058_CR38","doi-asserted-by":"publisher","first-page":"761","DOI":"10.1016\/S0893-6080(98)00010-0","volume":"11","author":"L Prechelt","year":"1998","unstructured":"Prechelt L (1998) Automatic early stopping using cross validation: quantifying the criteria. Neural Netw 11(4):761\u2013767","journal-title":"Neural Netw"},{"key":"5058_CR39","unstructured":"\u0158eh\u016f\u0159ek R, Sojka P (2010) Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 workshop on new challenges for NLP frameworks, pp 45\u201350. ELRA, Valletta, Malta. http:\/\/is.muni.cz\/publication\/884893\/en"},{"issue":"1","key":"5058_CR40","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1177\/002224298304700107","volume":"47","author":"ML Richins","year":"1983","unstructured":"Richins ML (1983) Negative word-of-mouth by dissatisfied consumers: a pilot study. J Mark 47(1):68\u201378","journal-title":"J Mark"},{"issue":"1","key":"5058_CR41","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/505282.505283","volume":"34","author":"F Sebastiani","year":"2002","unstructured":"Sebastiani F (2002) Machine learning in automated text categorization. ACM Comput Surv (CSUR) 34(1):1\u201347","journal-title":"ACM Comput Surv (CSUR)"},{"issue":"1","key":"5058_CR42","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1145\/584091.584093","volume":"5","author":"CE Shannon","year":"2001","unstructured":"Shannon CE (2001) A mathematical theory of communication. ACM SIGMOBILE Mob Comput Commun Rev 5(1):3\u201355","journal-title":"ACM SIGMOBILE Mob Comput Commun Rev"},{"issue":"1","key":"5058_CR43","first-page":"1929","volume":"15","author":"N Srivastava","year":"2014","unstructured":"Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929\u20131958","journal-title":"J Mach Learn Res"},{"key":"5058_CR44","doi-asserted-by":"crossref","unstructured":"Sundermeyer M, Schl\u00fcter R, Ney H (2012) LSTM neural networks for language modeling. In: Thirteenth annual conference of the international speech communication association","DOI":"10.21437\/Interspeech.2012-65"},{"key":"5058_CR45","doi-asserted-by":"crossref","unstructured":"Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1\u20139","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"5058_CR46","doi-asserted-by":"crossref","unstructured":"Tsvetkov Y, Faruqui M, Dyer C (2016) Correlation-based intrinsic evaluation of word vector representations. arXiv preprint arXiv:1606.06710","DOI":"10.18653\/v1\/W16-2520"},{"key":"5058_CR47","unstructured":"Vorontsov E, Trabelsi C, Kadoury S, Pal C (2017) On orthogonality and learning recurrent networks with long term dependencies. arXiv preprint arXiv:1702.00071"},{"key":"5058_CR48","doi-asserted-by":"publisher","DOI":"10.1007\/s11063-017-9636-0","author":"Y Yan","year":"2017","unstructured":"Yan Y, Wang Y, Gao WC, Zhang BW, Yang C, Yin XC (2017) Lstm2: multi-label ranking for document classification. Neural Process Lett. https:\/\/doi.org\/10.1007\/s11063-017-9636-0","journal-title":"Neural Process Lett"},{"issue":"1","key":"5058_CR49","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1023\/A:1009982220290","volume":"1","author":"Y Yang","year":"1999","unstructured":"Yang Y (1999) An evaluation of statistical approaches to text categorization. Inform Retriev 1(1):69\u201390","journal-title":"Inform Retriev"},{"issue":"2","key":"5058_CR50","first-page":"3","volume":"1","author":"H Zhang","year":"2004","unstructured":"Zhang H (2004) The optimality of Naive Bayes. Assoc Adv Artif Intell 1(2):3","journal-title":"Assoc Adv Artif Intell"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-020-05058-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-020-05058-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-020-05058-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,29]],"date-time":"2022-10-29T11:36:38Z","timestamp":1667043398000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-020-05058-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,19]]},"references-count":50,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,3]]}},"alternative-id":["5058"],"URL":"https:\/\/doi.org\/10.1007\/s00521-020-05058-4","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,19]]},"assertion":[{"value":"16 August 2018","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 June 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 June 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}