{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:46:16Z","timestamp":1760240776579,"version":"build-2065373602"},"reference-count":90,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2019,9,2]],"date-time":"2019-09-02T00:00:00Z","timestamp":1567382400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>Many systems rely on the expertise from human operators, who have acquired their knowledge through practical experience over the course of many years. For the detection of anomalies in industrial settings, sensor units have been introduced to predict and classify such anomalous events, but these critically rely on annotated data for training. Lengthy data collection campaigns are needed, which tend to be combined with domain expert annotations of the data afterwards, resulting in costly and slow process. This work presents an alternative by studying live annotation of rare anomalous events in sensor streams in a real-world manufacturing setting by experienced human operators that can also observe the machinery itself. A prototype for visualization and in situ annotation of sensor signals is developed with embedded unsupervised anomaly detection algorithms to propose signals for annotation and which allows the operators to give feedback on the detection and classify anomalous events. This prototype allowed assembling a corpus of several weeks of sensor data measured in a real manufacturing surrounding and was annotated by domain experts as an evaluation basis for this study. The evaluation of live annotations reveals high user motivation after getting accustomed to the labeling prototype. After this initial period, clear anomalies with characteristic signal patterns are detected reliably in visualized envelope signals. More subtle signal deviations were less likely to be confirmed an anomaly due to either an insufficient visibility in envelope signals or the absence of characteristic signal patterns.<\/jats:p>","DOI":"10.3390\/informatics6030038","type":"journal-article","created":{"date-parts":[[2019,9,3]],"date-time":"2019-09-03T03:06:14Z","timestamp":1567479974000},"page":"38","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Collecting Labels for Rare Anomalies via Direct Human Feedback\u2014An Industrial Application Study"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-0449","authenticated-orcid":false,"given":"Christian","family":"Reich","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of Siegen, 57068 Siegen, Germany"},{"name":"Corporate Research, Robert Bosch GmbH, 71272 Renningen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmad","family":"Mansour","sequence":"additional","affiliation":[{"name":"Corporate Research, Robert Bosch GmbH, 71272 Renningen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5296-5347","authenticated-orcid":false,"given":"Kristof","family":"Van Laerhoven","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of Siegen, 57068 Siegen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,9,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1016\/j.jmsy.2018.06.004","article-title":"Smart Optimization of a Friction-Drilling Process Based on Boosting Ensembles","volume":"48","author":"Bustillo","year":"2018","journal-title":"J. Manuf. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"15:1","DOI":"10.1145\/1541880.1541882","article-title":"Anomaly Detection: A Survey","volume":"41","author":"Chandola","year":"2009","journal-title":"Acm Comput. Surv."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"137","DOI":"10.3182\/20130522-3-RO-4035.00010","article-title":"Comparison between envelope detection methods for bearing defects diagnose","volume":"46","author":"Betea","year":"2013","journal-title":"Ifac Proc. Vol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/S0888-3270(03)00046-3","article-title":"Constructing a wavelet-based envelope function for vibration signal analysis","volume":"18","author":"Sheen","year":"2004","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Liao, Z., Song, L., Chen, P., Guan, Z., Fang, Z., and Li, K. (2018). An Effective Singular Value Selection and Bearing Fault Signal Filtering Diagnosis Method Based on False Nearest Neighbors and Statistical Information Criteria. Sensors, 18.","DOI":"10.3390\/s18072235"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1145\/2949741.2949758","article-title":"k-Shape: Efficient and Accurate Clustering of Time Series","volume":"45","author":"Paparrizos","year":"2016","journal-title":"SIGMOD Rec."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1016\/j.cirp.2010.05.010","article-title":"Advanced Monitoring of Machining Operations","volume":"59","author":"Teti","year":"2010","journal-title":"Cirp-Ann.-Manuf. Technol."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Benkedjouh, T., Zerhouni, N., and Rechak, S. (2017, January 29\u201331). Tool condition monitoring based on mel-frequency cepstral coefficients and support vector regression. Proceedings of the 5th International Conference on Electrical Engineering, Boumerdes, Algeria.","DOI":"10.1109\/ICEE-B.2017.8192024"},{"key":"ref_9","unstructured":"Kingma, D.P., and Welling, M. (2014, January 14\u201316). Auto-Encoding Variational Bayes. Proceedings of the 2nd International Conference on Learning Representations, Banff, AB, Canada."},{"key":"ref_10","unstructured":"Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., and Bengio, Y. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, 8\u201313 December 2014, MIT Press."},{"key":"ref_11","unstructured":"Oord, A.V., Kalchbrenner, N., and Kavukcuoglu, K. (2016, January 19\u201324). Pixel Recurrent Neural Networks. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_12","unstructured":"Dinh, L., Krueger, D., and Bengio, Y. (2014). NICE: Non-linear independent components estimation. arXiv."},{"key":"ref_13","unstructured":"Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2016). Density Estimation Using Real NVP. arXiv."},{"key":"ref_14","unstructured":"Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (2018). Glow: Generative Flow with Invertible 1x1 Convolutions. Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Curran Associates, Inc."},{"key":"ref_15","first-page":"3371","article-title":"Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion","volume":"11","author":"Vincent","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_16","unstructured":"Rifai, S., Vincent, P., Muller, X., Glorot, X., and Bengio, Y. (July, January 28). Contractive Auto-encoders: Explicit Invariance During Feature Extraction. Proceedings of the 28th International Conference on International Conference on Machine Learning, Bellevue, WA, USA."},{"key":"ref_17","unstructured":"Zong, B., Song, Q., Min, M.R., Cheng, W., Lumezanu, C., Cho, D., and Chen, H. (May, January 30). Deep autoencoding gaussian mixture model for unsupervised anomaly detection. Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_18","unstructured":"Berndt, D.J., and Clifford, J. (1994). Using Dynamic Time Warping to Find Patterns in Time Series. AAAIWS\u201994, Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining, Seattle, WA, USA, 31 July\u20131 August 1994, AAAI Press."},{"key":"ref_19","unstructured":"Cuturi, M., and Blondel, M. (2017, January 6\u201311). Soft-DTW: a Differentiable Loss Function for Time-Series. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"678","DOI":"10.1016\/j.patcog.2010.09.013","article-title":"A Global Averaging Method for Dynamic Time Warping, with Applications to Clustering","volume":"44","author":"Petitjean","year":"2011","journal-title":"Pattern Recogn."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Xi, X., Keogh, E., Shelton, C., Wei, L., and Ratanamahatana, C.A. (2006, January 25\u201329). Fast Time Series Classification Using Numerosity Reduction. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.","DOI":"10.1145\/1143844.1143974"},{"key":"ref_22","unstructured":"Bagnall, A., and Lines, J. (2014). An Experimental Evaluation of Nearest Neighbour Time Series Classification. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1007\/s10618-012-0250-5","article-title":"Experimental Comparison of Representation Methods and Distance Measures for Time Series Data","volume":"26","author":"Wang","year":"2013","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10115-015-0878-8","article-title":"Faster and More Accurate Classification of Time Series by Exploiting a Novel Dynamic Time Warping Averaging Algorithm","volume":"47","author":"Petitjean","year":"2016","journal-title":"Knowl. Inf. Syst."},{"key":"ref_25","unstructured":"Dodge, Y. (1987). Clustering by means of medoids. Statistical Data Analysis Based on the L1 Norm and Related Methods, Elsevier."},{"key":"ref_26","unstructured":"Ester, M., Kriegel, H.P., Sander, J., and Xu, X. (1996). A Density-based Algorithm for Discovering Clusters a Density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, AAAI Press."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Ankerst, M., Breunig, M.M., Kriegel, H.P., and Sander, J. (1999). OPTICS: Ordering Points to Identify the Clustering Structure. Proceedings of the 1999 ACM SIGMOD International Conference on Management of Data, ACM.","DOI":"10.1145\/304182.304187"},{"key":"ref_28","unstructured":"Pei, J., Tseng, V.S., Cao, L., Motoda, H., and Xu, G. (2013). Density-Based Clustering Based on Hierarchical Density Estimates. Advances in Knowledge Discovery and Data Mining, Springer."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Elhamifar, E., and Vidal, R. (2009, January 20\u201325). Sparse subspace clustering. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPRW.2009.5206547"},{"key":"ref_30","unstructured":"Bahadori, M.T., Kale, D., Fan, Y., and Liu, Y. (2015, January 6\u201311). Functional Subspace Clustering with Application to Time Series. Proceedings of the 32nd International Conference on International Conference on Machine Learning, Lille, France."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1857","DOI":"10.1016\/j.patcog.2005.01.025","article-title":"Clustering of Time Series Data-a Survey","volume":"38","author":"Liao","year":"2005","journal-title":"Pattern Recogn."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.is.2015.04.007","article-title":"Time-series Clustering\u2014A Decade Review","volume":"53","author":"Aghabozorgi","year":"2015","journal-title":"Inf. Syst."},{"key":"ref_33","unstructured":"Malhotra, P., Vig, L., Shroff, G., and Agarwal, P. (2015, January 22\u201324). Long Short Term Memory Networks for Anomaly Detection in Time Series. Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zhang, C., Song, D., Chen, Y., Feng, X., Lumezanu, C., Cheng, W., Ni, J., Zong, B., Chen, H., and Chawla, N.V. (2018). A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data. arXiv.","DOI":"10.1609\/aaai.v33i01.33011409"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Xu, H., Chen, W., Zhao, N., Li, Z., Bu, J., Li, Z., Liu, Y., Zhao, Y., Pei, D., and Feng, Y. (2018, January 23\u201327). Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications. Proceedings of the 2018 World Wide Web Conference on World Wide Web, Lyon, France.","DOI":"10.1145\/3178876.3185996"},{"key":"ref_36","unstructured":"Bai, S., Kolter, J.Z., and Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv."},{"key":"ref_37","unstructured":"Aggarwal, C.C. (2016). Outlier Analysis, Springer International Publishing. [2nd ed.]."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s00778-004-0125-5","article-title":"An Effective and Efficient Algorithm for High-dimensional Outlier Detection","volume":"14","author":"Aggarwal","year":"2005","journal-title":"VLDB J."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1080\/00401706.1999.10485670","article-title":"A fast algorithm for the minimum covariance determinant estimator","volume":"41","author":"Rousseeuw","year":"1999","journal-title":"Technometrics"},{"key":"ref_40","first-page":"1","article-title":"Outlier detection techniques","volume":"10","author":"Kriegel","year":"2010","journal-title":"Tutorial at KDD"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1443","DOI":"10.1162\/089976601750264965","article-title":"Estimating the Support of a High-Dimensional Distribution","volume":"13","author":"Platt","year":"2001","journal-title":"Neural Comput."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1023\/B:MACH.0000008084.60811.49","article-title":"Support Vector Data Description","volume":"54","author":"Tax","year":"2004","journal-title":"Mach. Learn."},{"key":"ref_43","unstructured":"Ruff, L., Vandermeulen, R., G\u00f6rnitz, N., Deecke, L., Siddiqui, S.A., Binder, A., M\u00fcller, E., and Kloft, M. (2018, January 10\u201315). Deep One-Class Classification. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Hautamaki, V., Karkkainen, I., and Franti, P. (2004, January 26). Outlier Detection Using k-Nearest Neighbour Graph. Proceedings of the Pattern Recognition, 17th International Conference on (ICPR\u201904), Cambridge, UK.","DOI":"10.1109\/ICPR.2004.1334558"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Breunig, M.M., Kriegel, H.P., Ng, R.T., and Sander, J. LOF: Identifying Density-based Local Outliers. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, ACM.","DOI":"10.1145\/342009.335388"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1007\/s10618-012-0300-z","article-title":"Local outlier detection reconsidered: a generalized view on locality with applications to spatial, video, and network outlier detection","volume":"28","author":"Schubert","year":"2014","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Tang, J., Chen, Z., Fu, A.W.C., and Cheung, D.W.L. (2002). Enhancing Effectiveness of Outlier Detections for Low Density Patterns. Proceedings of the 6th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, Springer.","DOI":"10.1007\/3-540-47887-6_53"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1641","DOI":"10.1016\/S0167-8655(03)00003-5","article-title":"Discovering Cluster-based Local Outliers","volume":"24","author":"He","year":"2003","journal-title":"Pattern Recogn. Lett."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Jin, W., Tung, A.K.H., Han, J., and Wang, W. (2006). Ranking Outliers Using Symmetric Neighborhood Relationship. Proceedings of the 10th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, Springer.","DOI":"10.1007\/11731139_68"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Kriegel, H.P., Kr\u00f6ger, P., Schubert, E., and Zimek, A. (2009). LoOP: Local Outlier Probabilities. Proceedings of the 18th ACM Conference on Information and Knowledge Management, ACM.","DOI":"10.1145\/1645953.1646195"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Zhang, K., Hutter, M., and Jin, H. (2009). A New Local Distance-Based Outlier Detection Approach for Scattered Real-World Data. Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, Springer.","DOI":"10.1007\/978-3-642-01307-2_84"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Latecki, L.J., Lazarevic, A., and Pokrajac, D. (2007). Outlier Detection with Kernel Density Functions. Proceedings of the 5th International Conference on Machine Learning and Data Mining in Pattern Recognition, Springer.","DOI":"10.1007\/978-3-540-73499-4_6"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Schubert, E., Zimek, A., and Kriegel, H.P. (2014, January 24\u201326). Generalized outlier detection with flexible kernel density estimates. Proceedings of the 2014 SIAM International Conference on Data Mining, Philadelphia, PA, USA.","DOI":"10.1137\/1.9781611973440.63"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1145\/2481244.2481252","article-title":"Outlier Ensembles: Position Paper","volume":"14","author":"Aggarwal","year":"2013","journal-title":"SIGKDD Explor. Newsl."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1145\/2594473.2594476","article-title":"Ensembles for Unsupervised Outlier Detection: Challenges and Research Questions a Position Paper","volume":"15","author":"Zimek","year":"2014","journal-title":"SIGKDD Explor. Newsl."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Liu, F.T., Ting, K.M., and Zhou, Z.H. (2008). Isolation Forest. Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, IEEE Computer Society.","DOI":"10.1109\/ICDM.2008.17"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1145\/2830544.2830549","article-title":"Theoretical Foundations and Algorithms for Outlier Ensembles","volume":"17","author":"Aggarwal","year":"2015","journal-title":"SIGKDD Explor. Newsl."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Manzoor, E., Lamba, H., and Akoglu, L. (2018, January 19\u201323). xStream: Outlier Detection in Feature-Evolving Data Streams. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.","DOI":"10.1145\/3219819.3220107"},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Zimek, A., Gaudet, M., Campello, R.J., and Sander, J. (2013). Subsampling for Efficient and Effective Unsupervised Outlier Detection Ensembles. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM.","DOI":"10.1145\/2487575.2487676"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Wu, K., Zhang, K., Fan, W., Edwards, A., and Yu, P.S. (2014). RS-Forest: A Rapid Density Estimator for Streaming Anomaly Detection. Proceedings of the 2014 IEEE International Conference on Data Mining, IEEE Computer Society.","DOI":"10.1109\/ICDM.2014.45"},{"key":"ref_61","unstructured":"Tan, S.C., Ting, K.M., and Liu, T.F. (2011). Fast Anomaly Detection for Streaming Data. Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence\u2014Volume Two, AAAI Press."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"42:1","DOI":"10.1145\/2890508","article-title":"Less is More: Building Selective Anomaly Ensembles","volume":"10","author":"Rayana","year":"2016","journal-title":"ACM Trans. Knowl. Discov. Data"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Rayana, S., Zhong, W., and Akoglu, L. (2016). Sequential Ensemble Learning for Outlier Detection: A Bias-Variance Perspective. 2016 IEEE 16th International Conference on Data Mining (ICDM), IEEE.","DOI":"10.1109\/ICDM.2016.0154"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1007\/s10994-015-5521-0","article-title":"Loda: Lightweight On-line Detector of Anomalies","volume":"102","author":"Pevny","year":"2016","journal-title":"Mach. Learn."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Sathe, S., and Aggarwal, C.C. (2016). Subspace Outlier Detection in Linear Time with Randomized Hashing. 2016 IEEE 16th International Conference on Data Mining (ICDM), IEEE.","DOI":"10.1109\/ICDM.2016.0057"},{"key":"ref_66","first-page":"xxx","article-title":"Tutorial i: Outlier detection in high-dimensional data","volume":"Volume 10","author":"Zimek","year":"2012","journal-title":"2012 IEEE 12th International Conference on Data Mining (ICDM)"},{"key":"ref_67","first-page":"157","article-title":"Learning Time Series Detection Models from Temporally Imprecise Labels","volume":"54","author":"Adams","year":"2017","journal-title":"Proc. Mach. Learn. Res."},{"key":"ref_68","unstructured":"Adams, R., and Marlin, B. (2018, January 6\u201310). Learning Time Series Segmentation Models from Temporally Imprecise Labels. Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence (UAI), Monterey, CA, USA."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/s12193-013-0129-9","article-title":"Inter-rater reliability for emotion annotation in human\u2013computer interaction: Comparison and methodological improvements","volume":"8","author":"Siegert","year":"2014","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_70","first-page":"28","article-title":"A Review of Human Activity Recognition Methods","volume":"2","author":"Vrigkas","year":"2015","journal-title":"Front. Robot."},{"key":"ref_71","unstructured":"Minor, B.D. (2015). Toward Learning and Mining from Uncertain Time-Series Data for Activity Prediction. SIGKDD Workshop on Mining and Learning from Time Series (MiLeTS\u20192015), ACM."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"15861","DOI":"10.3390\/s140915861","article-title":"Evaluation of Prompted Annotation of Activity Data Recorded from a Smart Phone","volume":"14","author":"Cleland","year":"2014","journal-title":"Sensors"},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Zenonos, A., Khan, A., Kalogridis, G., Vatsikas, S., Lewis, T., and Sooriyabandara, M. (2016). HealthyOffice: Mood recognition at work using smartphones and wearable sensors. IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops), IEEE.","DOI":"10.1109\/PERCOMW.2016.7457166"},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"2203","DOI":"10.3390\/s18072203","article-title":"Automatic Annotation for Human Activity Recognition in Free Living Using a Smartphone","volume":"18","author":"Cruciani","year":"2018","journal-title":"Sensors"},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Vaizman, Y., Ellis, K., Lanckriet, G., and Weibel, N. (2018). ExtraSensory App: Data Collection In-the-Wild with Rich User Interface to Self-Report Behavior. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ACM.","DOI":"10.1145\/3173574.3174128"},{"key":"ref_76","doi-asserted-by":"crossref","unstructured":"Lewis, D.D., and Gale, W.A. (1994). A Sequential Algorithm for Training Text Classifiers. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Springer.","DOI":"10.1007\/978-1-4471-2099-5_1"},{"key":"ref_77","unstructured":"Gwet, K.L. (2014). Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Raters, Advanced Analytics, LLC. [4th ed.]."},{"key":"ref_78","first-page":"2349","article-title":"Estimating Labels from Label Proportions","volume":"10","author":"Quadrianto","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_79","first-page":"1607","article-title":"Bayesian Nonparametric Crowdsourcing","volume":"16","author":"Moreno","year":"2015","journal-title":"J. Mach. Learn. Res."},{"key":"ref_80","doi-asserted-by":"crossref","unstructured":"Yang, Y., Zhang, M., Chen, W., Zhang, W., Wang, H., and Zhang, M. (2018). Adversarial Learning for Chinese NER From Crowd Annotations. 32nd AAAI Conference on Artificial Intelligence (AAAI18), AAAI Press.","DOI":"10.1609\/aaai.v32i1.11507"},{"key":"ref_81","doi-asserted-by":"crossref","unstructured":"Snow, R., O\u2019Connor, B., Jurafsky, D., and Ng, A. (2008, January 25\u201327). Cheap and fast\u2014but is it good? Evaluating non-expert annotations for natural language tasks. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Honolulu, HI, USA.","DOI":"10.3115\/1613715.1613751"},{"key":"ref_82","unstructured":"Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., and Hovy, E. (2013, January 9\u201314). Learning Whom to Trust with MACE. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics (ACL), Atlanta, GA, USA."},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1162\/tacl_a_00185","article-title":"The Benefits of a Model of Annotation","volume":"2","author":"Passonneau","year":"2014","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_84","doi-asserted-by":"crossref","unstructured":"Pl\u00f6tz, T., Chen, C., Hammerla, N.Y., and Abowd, G.D. (2012). Automatic Synchronization of Wearable Sensors and Video-Cameras for Ground Truth Annotation\u2014A Practical Approach. Proceedings of the 2012 16th Annual International Symposium on Wearable Computers (ISWC), IEEE Computer Society.","DOI":"10.1109\/ISWC.2012.15"},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Miu, T., Missier, P., and Pl\u00f6tz, T. (2015, January 26\u201328). Bootstrapping Personalised Human Activity Recognition Models Using Online Active Learning. Proceedings of the 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing, Liverpool, UK.","DOI":"10.1109\/CIT\/IUCC\/DASC\/PICOM.2015.170"},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Schr\u00f6der, M., Yordanova, K., Bader, S., and Kirste, T. (2016). Tool Support for the Online Annotation of Sensor Data. Proceedings of the 3rd International Workshop on Sensor-based Activity Recognition and Interaction, ACM.","DOI":"10.1145\/2948963.2948972"},{"key":"ref_87","doi-asserted-by":"crossref","unstructured":"Miu, T., Pl\u00f6tz, T., Missier, P., and Roggen, D. (2014). On Strategies for Budget-based Online Annotation in Human Activity Recognition. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, ACM.","DOI":"10.1145\/2638728.2641300"},{"key":"ref_88","doi-asserted-by":"crossref","unstructured":"Gjoreski, H., and Roggen, D. (2017). Unsupervised Online Activity Discovery Using Temporal Behaviour Assumption. Proceedings of the 2017 ACM International Symposium on Wearable Computers, ACM.","DOI":"10.1145\/3123021.3123044"},{"key":"ref_89","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_90","unstructured":"Zhao, Y., Nasrullah, Z., and Li, Z. (2019). PyOD: A Python Toolbox for Scalable Outlier Detection. arXiv."}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/6\/3\/38\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:16:13Z","timestamp":1760188573000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/6\/3\/38"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,2]]},"references-count":90,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2019,9]]}},"alternative-id":["informatics6030038"],"URL":"https:\/\/doi.org\/10.3390\/informatics6030038","relation":{},"ISSN":["2227-9709"],"issn-type":[{"type":"electronic","value":"2227-9709"}],"subject":[],"published":{"date-parts":[[2019,9,2]]}}}