{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T04:08:59Z","timestamp":1778126939596,"version":"3.51.4"},"reference-count":67,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,1,19]],"date-time":"2020-01-19T00:00:00Z","timestamp":1579392000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,1,19]],"date-time":"2020-01-19T00:00:00Z","timestamp":1579392000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["K\u00fcnstl Intell"],"published-print":{"date-parts":[[2020,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In the following article, we introduce a novel workflow, which we subsume under the term \u201cexplainable cooperative machine learning\u201d and show its practical application in a data annotation and model training tool called<jats:sc>NOVA<\/jats:sc>. The main idea of our approach is to interactively incorporate the \u2018human in the loop\u2019 when training classification models from annotated data. In particular, NOVA offers a collaborative annotation backend where multiple annotators join their workforce. A main aspect is the possibility of applying semi-supervised active learning techniques already during the annotation process by giving the possibility to pre-label data automatically, resulting in a drastic acceleration of the annotation process. Furthermore, the user-interface implements recent eXplainable AI techniques to provide users with both, a confidence value of the automatically predicted annotations, as well as visual explanation. We show in an use-case evaluation that our workflow is able to speed up the annotation process, and further argue that by providing additional visual explanations annotators get to understand the decision making process as well as the trustworthiness of their trained machine learning models.<\/jats:p>","DOI":"10.1007\/s13218-020-00632-3","type":"journal-article","created":{"date-parts":[[2020,1,19]],"date-time":"2020-01-19T06:02:20Z","timestamp":1579413740000},"page":"143-164","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["eXplainable Cooperative Machine Learning with NOVA"],"prefix":"10.1007","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2797-605X","authenticated-orcid":false,"given":"Tobias","family":"Baur","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander","family":"Heimerl","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Florian","family":"Lingenfelser","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Johannes","family":"Wagner","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michel F.","family":"Valstar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bj\u00f6rn","family":"Schuller","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elisabeth","family":"Andr\u00e9","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,1,19]]},"reference":[{"key":"632_CR1","unstructured":"Alber M, Lapuschkin S, Seegerer P, H\u00e4gele M, Sch\u00fctt KT, Montavon G, Samek W, M\u00fcller K, D\u00e4hne S, Kindermans P (2018) Investigate neural networks! CoRR. arXiv:abs\/1808.04260"},{"issue":"4","key":"632_CR2","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1609\/aimag.v35i4.2513","volume":"35","author":"S Amershi","year":"2014","unstructured":"Amershi S, Cakmak M, Knox WB, Kulesza T (2014) Power to the people: the role of humans in interactive machine learning. AI Mag 35(4):105\u2013120","journal-title":"AI Mag"},{"key":"632_CR3","doi-asserted-by":"crossref","unstructured":"Amershi S, Chickering M, Drucker SM, Lee B, Simard P, Suh J (2015) Modeltracker: redesigning performance analysis tools for machine learning. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, pp 337\u2013346","DOI":"10.1145\/2702123.2702509"},{"key":"632_CR4","doi-asserted-by":"crossref","unstructured":"Amershi S, Fogarty J, Kapoor A, Tan DS (2009) Overview based example selection in end user interactive concept learning. In: Proceedings of the 22nd annual ACM symposium on user interface software and technology, Victoria, October 4\u20137, 2009, pp 247\u2013256","DOI":"10.1145\/1622176.1622222"},{"key":"632_CR5","doi-asserted-by":"crossref","unstructured":"Baltru\u0161aitis T, Robinson P, Morency LP (2016) Openface: an open source facial behavior analysis toolkit. In: 2016 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 1\u201310","DOI":"10.1109\/WACV.2016.7477553"},{"issue":"2","key":"632_CR6","first-page":"11","volume":"5","author":"T Baur","year":"2015","unstructured":"Baur T, Mehlmann G, Damian I, Lingenfelser F, Wagner J, Lugrin B, Andr\u00e9 E, Gebhard P (2015) Context-aware automated analysis and annotation of social human\u2013agent interactions. ACM Trans Interact Intell Syst (TiiS) 5(2):11","journal-title":"ACM Trans Interact Intell Syst (TiiS)"},{"key":"632_CR7","doi-asserted-by":"crossref","unstructured":"Beritelli F, Casale S, Russo A, Serrano S, Ettorre D (2006) Speech emotion recognition using MFCCs extracted from a mobile terminal based on ETSI front end. In: International conference on signal processing, vol.\u00a02","DOI":"10.1109\/ICOSP.2006.345670"},{"key":"632_CR8","doi-asserted-by":"crossref","unstructured":"Cafaro A, Wagner J, Baur T, Dermouche S, Torres\u00a0Torres M, Pelachaud C, Andr\u00e9 E, Valstar MF (2017) The noxi database: multimodal recordings of mediated novice\u2013expert interactions. In: Proceedings of the 19th international conference on multimodal interaction. ACM (in press)","DOI":"10.1145\/3136755.3136780"},{"key":"632_CR9","unstructured":"Chen NC, Kocielnik R, Drouhard M, Pe\u00f1a-Araya V, Suh J, Cen K, Zheng X, Aragon CR (2016) Challenges of applying machine learning to qualitative coding. In: CHI 2016 workshop on human centred machine learning"},{"key":"632_CR10","doi-asserted-by":"crossref","unstructured":"Cheng J, Bernstein MS (2015) Flock: hybrid crowd-machine learning classifiers. In: Proceedings of the 18th ACM conference on computer supported cooperative work and social computing. ACM, pp 600\u2013611","DOI":"10.1145\/2675133.2675214"},{"issue":"1","key":"632_CR11","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37\u201346","journal-title":"Educ Psychol Meas"},{"key":"632_CR12","unstructured":"Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M, Schr\u00f6der M (2000) \u2019feeltrace\u2019: an instrument for recording perceived emotion in real time. In: ISCA tutorial and research workshop (ITRW) on speech and emotion"},{"issue":"1","key":"632_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.4018\/jse.2012010101","volume":"3","author":"R Cowie","year":"2012","unstructured":"Cowie R, McKeown G, Douglas-Cowie E (2012) Tracing emotion: an overview. Int J Synth Emot (IJSE) 3(1):1\u201317","journal-title":"Int J Synth Emot (IJSE)"},{"issue":"3","key":"632_CR14","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1007\/BF02310555","volume":"16","author":"LJ Cronbach","year":"1951","unstructured":"Cronbach LJ (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16(3):297\u2013334","journal-title":"Psychometrika"},{"key":"632_CR15","unstructured":"Dong M, Sun Z (2003) On human machine cooperative learning control. In: Proceedings of the 2003 IEEE international symposium on intelligent control, pp 81\u201386"},{"issue":"c","key":"632_CR16","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/S0167-6393(02)00070-5","volume":"40","author":"E Douglas-Cowie","year":"2003","unstructured":"Douglas-Cowie E, Campbell N, Cowie R, Roach P (2003) Emotional speech: towards a new generation of databases. Speech Commun 40(c):33\u201360","journal-title":"Speech Commun"},{"key":"632_CR17","doi-asserted-by":"crossref","unstructured":"Eyben F, Weninger F, Gross F, Schuller B (2013) Recent developments in opensmile, the Munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM international conference on multimedia, MM \u201913. ACM, New York, pp 835\u2013838","DOI":"10.1145\/2502081.2502224"},{"key":"632_CR18","doi-asserted-by":"crossref","unstructured":"Fails JA, Olsen Jr, DR (2003) Interactive machine learning. In: Proceedings of the 8th international conference on intelligent user interfaces, IUI \u201903. ACM, New York, pp 39\u201345","DOI":"10.1145\/604045.604056"},{"key":"632_CR19","first-page":"1871","volume":"9","author":"RE Fan","year":"2008","unstructured":"Fan RE, Chang KW, Hsieh CJ, Wang XR, Lin CJ (2008) Liblinear: a library for large linear classification. J Mach Learn Res 9:1871\u20131874","journal-title":"J Mach Learn Res"},{"key":"632_CR20","unstructured":"Ganchev T, Fakotakis N, Kokkinakis G (2005) Comparative evaluation of various MFCC implementations on the speaker verification task. In: Proceedings of the SPECOM-2005, pp 191\u2013194"},{"issue":"1","key":"632_CR21","first-page":"e5","volume":"2","author":"JM Girard","year":"2014","unstructured":"Girard JM (2014) Carma: software for continuous affect rating and media annotation. J Open Res Softw 2(1):e5","journal-title":"J Open Res Softw"},{"key":"632_CR22","doi-asserted-by":"crossref","unstructured":"Girard JM, Wright AGC (2016) DARMA: dual axis rating and media annotation (submitted)","DOI":"10.31219\/osf.io\/xhmu6"},{"key":"632_CR23","doi-asserted-by":"crossref","unstructured":"Hantke S, Eyben F, Appel T, Schuller B (2015) iHEARu-PLAY: introducing a game for crowdsourced data collection for affective computing. In: 2015 International conference on affective computing and intelligent interaction (ACII). IEEE, pp 891\u2013897","DOI":"10.1109\/ACII.2015.7344680"},{"issue":"2","key":"632_CR24","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s40708-016-0042-6","volume":"3","author":"A Holzinger","year":"2016","unstructured":"Holzinger A (2016) Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Inform 3(2):119\u2013131","journal-title":"Brain Inform"},{"key":"632_CR25","doi-asserted-by":"crossref","unstructured":"Holzinger A (2018) From machine learning to explainable AI. In: 2018 World symposium on digital intelligence for systems and machines (DISA). IEEE, pp 55\u201366","DOI":"10.1109\/DISA.2018.8490530"},{"key":"632_CR26","doi-asserted-by":"crossref","unstructured":"Holzinger A, Plass M, Holzinger K, Cri\u015fan GC, Pintea CM, Palade V (2016) Towards interactive machine learning (IML): applying ant colony algorithms to solve the traveling salesman problem with the human-in-the-loop approach. In: International conference on availability, reliability, and security. Springer, pp 81\u201395","DOI":"10.1007\/978-3-319-45507-5_6"},{"key":"632_CR27","unstructured":"Kamar E, Hacker S, Horvitz E (2012) Combining human and machine intelligence in large-scale crowdsourcing. In: International conference on autonomous agents and multiagent systems, AAMAS 2012, Valencia, June 4\u20138, 2012 (3 volumes), pp 467\u2013474"},{"key":"632_CR28","unstructured":"Kennedy L, Ellis DPW (2004) Laughter detection in meetings. In: Proceedings of NIST meeting recognition workshop, Montreal, pp 118\u2013121"},{"key":"632_CR29","doi-asserted-by":"crossref","unstructured":"Kim B, Pardo B (2017) I-SED: an interactive sound event detector. In: Proceedings of the 22nd international conference on intelligent user interfaces, IUI \u201917. ACM, New York, pp 553\u2013557","DOI":"10.1145\/3025171.3025231"},{"key":"632_CR30","doi-asserted-by":"crossref","unstructured":"Kipp M (2013) Anvil: the video annotation research tool. In: Handbook of corpus phonology. Oxford University Press, Oxford","DOI":"10.1093\/oxfordhb\/9780199571932.013.024"},{"key":"632_CR31","unstructured":"Kishore KK, Satish KP (2013) Emotion recognition in speech using MFCC and wavelet features. In: International conference on advance computing conference (IACC), pp 842\u2013847"},{"key":"632_CR32","unstructured":"Knox MT, Mirghafori N (2007) Automatic laughter detection using neural networks. In: INTERSPEECH 2007, 8th annual conference of the International Speech Communication Association, Antwerp, August 27\u201331, 2007, pp 2973\u20132976"},{"key":"632_CR33","doi-asserted-by":"crossref","unstructured":"Lee CM, Yildirim S, Bulut M, Kazemzadeh A, Busso C, Deng Z, Lee S, Narayanan S (2004) Emotion recognition based on phoneme classes. In: International conference on spoken language processing (ICSLP), pp 889\u2013892","DOI":"10.21437\/Interspeech.2004-322"},{"key":"632_CR34","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1145\/2070481.2070487","volume-title":"International conference on multimodal interfaces (ICMI), ICMI \u201911","author":"F Lingenfelser","year":"2011","unstructured":"Lingenfelser F, Wagner J, Andr\u00e9 E (2011) A systematic discussion of fusion techniques for multi-modal affect recognition tasks. International conference on multimodal interfaces (ICMI), ICMI \u201911. ACM, New York, pp 19\u201326"},{"key":"632_CR35","doi-asserted-by":"crossref","unstructured":"Lingenfelser F, Wagner J, Andr\u00e9 E, McKeown G, Curran W (2014) An event driven fusion approach for enjoyment recognition in real-time. In: International conference on multimedia (MM), MM \u201914. ACM, New York, pp 377\u2013386","DOI":"10.1145\/2647868.2654924"},{"issue":"4","key":"632_CR36","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1109\/TAFFC.2017.2736999","volume":"10","author":"R Lotfian","year":"2017","unstructured":"Lotfian R, Busso C (2017) Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Trans Affect Comput 10(4):471\u2013483","journal-title":"IEEE Trans Affect Comput"},{"key":"632_CR37","doi-asserted-by":"crossref","unstructured":"Mayor O, Llimona Q, Marchini M, Papiotis P, Maestre E (2013) repoVizz: a framework for remote storage, browsing, annotation, and exchange of multi-modal data. In: Proceedings of the 21st ACM international conference on multimedia, MM \u201913. ACM, New York, pp 415\u2013416","DOI":"10.1145\/2502081.2502247"},{"key":"632_CR38","doi-asserted-by":"crossref","unstructured":"Neiberg D, Elenius K, Laskowski K (2006) Emotion recognition in spontaneous speech using GMMs. In: Conference of the International Speech Communication Association (INTERSPEECH)","DOI":"10.21437\/Interspeech.2006-277"},{"key":"632_CR39","unstructured":"Poignant J, Budnik M, Bredin H, Barras C, Stefas M, Bruneau P, Adda G, Besacier L, Ekenel HK, Francopoulo G, Hernando J, Mariani J, Morros R, Qu\u00e9not G, Rosset S, Tamisier T (2016) The CAMOMILE collaborative annotation platform for multi-modal, multi-lingual and multi-media documents. In: Proceedings of the tenth international conference on language resources and evaluation LREC 2016, Portoro\u017e, May 23\u201328, 2016"},{"key":"632_CR40","volume-title":"Fundamentals of speech recognition","author":"L Rabiner","year":"1993","unstructured":"Rabiner L, Juang BH (1993) Fundamentals of speech recognition. Prentice-Hall, Upper Saddle River"},{"key":"632_CR41","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) \u201cWhy should I trust you?\u201d: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco, August 13\u201317, 2016, pp 1135\u20131144","DOI":"10.1145\/2939672.2939778"},{"key":"632_CR42","doi-asserted-by":"crossref","unstructured":"Rosenthal S, Dey AK (2010) Towards maximizing the accuracy of human-labeled sensor data. In: Proceedings of the 2010 international conference on intelligent user interfaces, February 7\u201310, 2010, Hong Kong, pp 259\u2013268","DOI":"10.1145\/1719970.1720006"},{"key":"632_CR43","unstructured":"Schmidt T (2004) Transcribing and annotating spoken language with EXMARaLDA. In: Proceedings of the international conference on language resources and evaluation: workshop on XML based richly annotated corpora, Lisbon 2004. ELRA, Paris, pp 879\u2013896"},{"key":"632_CR44","doi-asserted-by":"crossref","unstructured":"Schuller B, Batliner A, Seppi D, Steidl S, Vogt T, Wagner J, Devillers L, Vidrascu L, Amir N, Kessous L, Aharonson V (2007) The relevance of feature type for the automatic classification of emotional user states: low level descriptors and functionals. In: INTERSPEECH. ISCA, pp 2253\u20132256","DOI":"10.21437\/Interspeech.2007-612"},{"key":"632_CR45","doi-asserted-by":"crossref","unstructured":"Schuller B, Steidl S, Batliner A, Vinciarelli A, Scherer KR, Ringeval F, Chetouani M, Weninger F, Eyben F, Marchi E, Mortillaro M, Salamin H, Polychroniou A, Valente F, Kim S (2013) The INTERSPEECH 2013 computational paralinguistics challenge: social signals, conflict, emotion, autism. In: INTERSPEECH 2013, 14th annual conference of the international Speech Communication Association, Lyon, August 25\u201329, 2013, pp 148\u2013152","DOI":"10.21437\/Interspeech.2013-56"},{"key":"632_CR46","unstructured":"Settles B (2010) Active learning literature survey. University of Wisconsin-Madison Department of Computer Sciences, vol 52, pp 55\u201366"},{"key":"632_CR47","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-031-01560-1","volume-title":"Active learning: synthesis lectures on artificial intelligence and machine learning","author":"B Settles","year":"2012","unstructured":"Settles B (2012) Active learning: synthesis lectures on artificial intelligence and machine learning. Morgan and Claypool, San Rafael"},{"issue":"3","key":"632_CR48","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1016\/0025-5564(75)90047-4","volume":"23","author":"EH Shortliffe","year":"1975","unstructured":"Shortliffe EH, Buchanan BG (1975) A model of inexact reasoning in medicine. Math Biosci 23(3):351\u2013379","journal-title":"Math Biosci"},{"key":"632_CR49","doi-asserted-by":"crossref","unstructured":"Stikic M, Laerhoven KV, Schiele B (2008) Exploring semi-supervised and active learning for activity recognition. In: 12th IEEE international symposium on wearable computers (ISWC 2008), September 28\u2013October 1, 2008, Pittsburgh, pp 81\u201388","DOI":"10.1109\/ISWC.2008.4911590"},{"key":"632_CR50","first-page":"45","volume":"2","author":"S Tong","year":"2002","unstructured":"Tong S, Koller D (2002) Support vector machine active learning with applications to text classification. J Mach Learn Res 2:45\u201366","journal-title":"J Mach Learn Res"},{"issue":"1","key":"632_CR51","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1007\/s12193-010-0053-1","volume":"4","author":"J Urbain","year":"2010","unstructured":"Urbain J, Niewiadomski R, Bevacqua E, Dutoit T, Moinet A, Pelachaud C, Picart B, Tilmanne J, Wagner J (2010) Avlaughtercycle. J Multimodal User Interfaces 4(1):47\u201358","journal-title":"J Multimodal User Interfaces"},{"key":"632_CR52","doi-asserted-by":"crossref","unstructured":"Valstar MF, Baur T, Cafaro A, Ghitulescu A, Potard B, Wagner J, Andr\u00e9 E, Durieu L, Aylett M, Dermouche S, Pelachaud C, Coutinho E, Schuller B, Zhang Y, Heylen D, Theune M, van Waterschoot J (2016) Ask Alice: an artificial retrieval of information agent. In: Proceedings of the 18th ACM international conference on multimodal interaction. ACM, pp 419\u2013420","DOI":"10.1145\/2993148.2998535"},{"key":"632_CR53","doi-asserted-by":"crossref","unstructured":"Valstar MF, Gunes H, Pantic M (2007) How to distinguish posed from spontaneous smiles using geometric features. In: Proceedings of the 9th international conference on multimodal interfaces. ACM, pp 38\u201345","DOI":"10.1145\/1322192.1322202"},{"issue":"12","key":"632_CR54","doi-asserted-by":"publisher","first-page":"1743","DOI":"10.1016\/j.imavis.2008.11.007","volume":"27","author":"A Vinciarelli","year":"2009","unstructured":"Vinciarelli A, Pantic M, Bourlard H (2009) Social signal processing: survey of an emerging domain. Image Vis Comput 27(12):1743\u20131759","journal-title":"Image Vis Comput"},{"key":"632_CR55","doi-asserted-by":"crossref","unstructured":"Vinciarelli A, Pantic M, Bourlard H, Pentland A (2008) Social signal processing: state-of-the-art and future perspectives of an emerging domain. In: International conference on multimedia (MM), Vancouver, pp 1061\u20131070","DOI":"10.1145\/1459359.1459573"},{"key":"632_CR56","doi-asserted-by":"crossref","unstructured":"Vogt T, Andr\u00e9 E (2005) Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In: International conference on multimedia and expo (ICME), pp 474\u2013477","DOI":"10.1109\/ICME.2005.1521463"},{"key":"632_CR57","unstructured":"Wagner J, Andr\u00e9 E, Kugler M, Leberle D (2010) SSI\/ModelUI\u2014a tool for the acquisition and annotation of human generated signals. In: DAGA 2010. TU Berlin, Berlin"},{"issue":"4","key":"632_CR58","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1109\/T-AFFC.2011.12","volume":"2","author":"J Wagner","year":"2011","unstructured":"Wagner J, Lingenfelser F, Andr\u00e9 E, Kim J, Vogt T (2011) Exploring fusion methods for multimodal emotion recognition with missing data. Affect Comput 2(4):206\u2013218","journal-title":"Affect Comput"},{"key":"632_CR59","doi-asserted-by":"crossref","unstructured":"Wagner J, Lingenfelser F, Andr\u00e9 E, Mazzei D, Tognetti A, Lanat\u00e0 A, Rossi DD, Betella A, Zucca R, Omedas P, Verschure PF (2013) A sensing architecture for empathetic data systems. In: Augmented human international conference (AH). ACM, Stuttgart, pp 96\u201399","DOI":"10.1145\/2459236.2459253"},{"key":"632_CR60","doi-asserted-by":"crossref","unstructured":"Wagner J, Lingenfelser F, Baur T, Damian I, Kistler F, Andr\u00e9 E (2013) The social signal interpretation (ssi) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM international conference on Multimedia, MM \u201913. ACM, New York, pp 831\u2013834","DOI":"10.1145\/2502081.2502223"},{"key":"632_CR61","doi-asserted-by":"crossref","unstructured":"Wagner J, Seiderer A, Lingenfelser F, Andr\u00e9 E (2015) Combining hierarchical classification with frequency weighting for the recognition of eating conditions. In: INTERSPEECH 2015, 16th annual conference of the International Speech Communication Association, Dresden, September 6\u201310, 2015, pp 889\u2013893","DOI":"10.21437\/Interspeech.2015-189"},{"issue":"2","key":"632_CR62","doi-asserted-by":"publisher","first-page":"10:1","DOI":"10.1145\/1899412.1899414","volume":"2","author":"M Wang","year":"2011","unstructured":"Wang M, Hua XS (2011) Active learning in multimedia annotation and retrieval: a survey. ACM Trans Intell Syst Technol 2(2):10:1\u201310:21","journal-title":"ACM Trans Intell Syst Technol"},{"key":"632_CR63","unstructured":"Wittenburg P, Brugman H, Russel A, Klassmann A, Sloetjes H (2006) Elan: a professional framework for multimodality research. In: Proceedings of the fifth international conference on language resources and evaluation (LREC), pp 879\u2013896"},{"key":"632_CR64","doi-asserted-by":"crossref","unstructured":"Zhang Y, Coutinho E, Schuller B, Zhang Z, Adam M (2015) On rater reliability and agreement based dynamic active learning. In: International conference on affective computing and intelligent interaction, ACII. Xi\u2019an, pp 70\u201376","DOI":"10.1109\/ACII.2015.7344553"},{"key":"632_CR65","doi-asserted-by":"crossref","unstructured":"Zhang Y, Coutinho E, Zhang Z, Quan C, Schuller B (2015) Dynamic active learning based on agreement and applied to emotion recognition in spoken interactions. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, ICMI \u201915. ACM, New York, pp 275\u2013278","DOI":"10.1145\/2818346.2820774"},{"issue":"1","key":"632_CR66","first-page":"115","volume":"23","author":"Z Zhang","year":"2015","unstructured":"Zhang Z, Coutinho E, Deng J, Schuller B (2015) Cooperative learning and its application to emotion recognition from speech. IEEE\/ACM Trans Audio Speech Lang Process 23(1):115\u2013126","journal-title":"IEEE\/ACM Trans Audio Speech Lang Process"},{"key":"632_CR67","unstructured":"Zhu X (2005) Semi-supervised learning literature survey. Tech. rep., Computer Sciences, University of Wisconsin-Madison"}],"container-title":["KI - K\u00fcnstliche Intelligenz"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-020-00632-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s13218-020-00632-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-020-00632-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,11]],"date-time":"2022-10-11T22:45:13Z","timestamp":1665528313000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s13218-020-00632-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,19]]},"references-count":67,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6]]}},"alternative-id":["632"],"URL":"https:\/\/doi.org\/10.1007\/s13218-020-00632-3","relation":{},"ISSN":["0933-1875","1610-1987"],"issn-type":[{"value":"0933-1875","type":"print"},{"value":"1610-1987","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,19]]},"assertion":[{"value":"30 September 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 January 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 January 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}