{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T03:14:34Z","timestamp":1761621274522,"version":"build-2065373602"},"reference-count":55,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2016,8,10]],"date-time":"2016-08-10T00:00:00Z","timestamp":1470787200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004083","name":"Ministry of Science ICT and Future Planning","doi-asserted-by":"publisher","award":["IITP-2015-R6812-15-0001","IITP-2015-H8501-15-1015"],"award-info":[{"award-number":["IITP-2015-R6812-15-0001","IITP-2015-H8501-15-1015"]}],"id":[{"id":"10.13039\/501100004083","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>There is sufficient evidence proving the impact that negative lifestyle choices have on people\u2019s health and wellness. Changing unhealthy behaviours requires raising people\u2019s self-awareness and also providing healthcare experts with a thorough and continuous description of the user\u2019s conduct. Several monitoring techniques have been proposed in the past to track users\u2019 behaviour; however, these approaches are either subjective and prone to misreporting, such as questionnaires, or only focus on a specific component of context, such as activity counters. This work presents an innovative multimodal context mining framework to inspect and infer human behaviour in a more holistic fashion. The proposed approach extends beyond the state-of-the-art, since it not only explores a sole type of context, but also combines diverse levels of context in an integral manner. Namely, low-level contexts, including activities, emotions and locations, are identified from heterogeneous sensory data through machine learning techniques. Low-level contexts are combined using ontological mechanisms to derive a more abstract representation of the user\u2019s context, here referred to as high-level context. An initial implementation of the proposed framework supporting real-time context identification is also presented. The developed system is evaluated for various realistic scenarios making use of a novel multimodal context open dataset and data on-the-go, demonstrating prominent context-aware capabilities at both low and high levels.<\/jats:p>","DOI":"10.3390\/s16081264","type":"journal-article","created":{"date-parts":[[2016,8,10]],"date-time":"2016-08-10T10:14:10Z","timestamp":1470824050000},"page":"1264","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["Human Behavior Analysis by Means of Multimodal Context Mining"],"prefix":"10.3390","volume":"16","author":[{"given":"Oresti","family":"Banos","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"},{"name":"Telemedicine Group, Center for Telematics and Information Technology, University of Twente, Enschede 7500AE, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5434-4253","authenticated-orcid":false,"given":"Claudia","family":"Villalonga","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"},{"name":"Department of Computer Architecture and Computer Technology, Research Center on Information and Communications Technology, University of Granada, Granada E18071, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jaehun","family":"Bang","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taeho","family":"Hur","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3675-2258","authenticated-orcid":false,"given":"Donguk","family":"Kang","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7458-3745","authenticated-orcid":false,"given":"Sangbeom","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thien","family":"Huynh-The","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vui","family":"Le-Ba","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Muhammad","family":"Amin","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Muhammad","family":"Razzaq","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wahajat","family":"Khan","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Choong","family":"Hong","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0061-8834","authenticated-orcid":false,"given":"Sungyoung","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kyung Hee University, Yongin-si 446-701, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2016,8,10]]},"reference":[{"key":"ref_1","unstructured":"WHO (2014). Global Status Report on Noncommunicable Diseases 2014, World Health Organization. Technical Report."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"815","DOI":"10.1177\/1049732309335395","article-title":"Experiences of Self-Monitoring: Successes and Struggles During Treatment for Weight Loss","volume":"19","author":"Burke","year":"2009","journal-title":"Qual. Health Res."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1136\/bjsm.37.3.197","article-title":"Limits to the measurement of habitual physical activity by questionnaires","volume":"37","author":"Shephard","year":"2003","journal-title":"Br. J. Sports Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"93","DOI":"10.3390\/jpm2030093","article-title":"Health 2050: The realization of personalized medicine through crowdsourcing, the Quantified Self, and the participatory biocitizen","volume":"2","author":"Swan","year":"2012","journal-title":"J. Pers. Med."},{"key":"ref_5","unstructured":"Fitbit Surge. Available online: https:\/\/www.fitbit.com\/surge."},{"key":"ref_6","unstructured":"Jawbone Up. Available online: https:\/\/jawbone.com\/up."},{"key":"ref_7","unstructured":"Misfit Shine. Available online: http:\/\/www.misfitwearables.com\/products\/shine."},{"key":"ref_8","unstructured":"Garmin Vivoactive. Available online: http:\/\/sites.garmin.com\/en-US\/vivo\/vivofit\/."},{"key":"ref_9","unstructured":"Empatica Embrace. Available online: https:\/\/www.empatica.com\/product-embrace."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"734","DOI":"10.1109\/TITB.2010.2047865","article-title":"A Wearable Smartphone-Based Platform for Real-Time Cardiovascular Disease Detection Via Electrocardiogram Processing","volume":"14","author":"Oresko","year":"2010","journal-title":"IEEE Trans. Inf. Technol. Biomed."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2014\/490824","article-title":"PhysioDroid: Combining Wearable Health Sensors and Mobile Devices for a Ubiquitous, Continuous, and Personal Monitoring","volume":"2014","author":"Banos","year":"2014","journal-title":"Sci. World J."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Patel, S., Mancinelli, C., Healey, J., Moy, M., and Bonato, P. (2009, January 3\u20135). Using Wearable Sensors to Monitor Physical Activities of Patients with COPD: A Comparison of Classifier Performance. Proceedings of the 6th International Workshop on Wearable and Implantable Body Sensor Networks, Berkeley, CA, USA.","DOI":"10.1109\/BSN.2009.53"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Pierce, W.D., and Cheney, C.D. (2013). Behavior Analysis and Learning, Psychology Press.","DOI":"10.4324\/9780203441817"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"510","DOI":"10.1109\/JSAC.2009.090513","article-title":"Context-aware middleware for pervasive elderly homecare","volume":"27","author":"Pung","year":"2009","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1109\/JBHI.2014.2385694","article-title":"Personalized, Multi-Layer Daily Life Profiling through Context Enabled Activity Classification and Motion Reconstruction: An Integrated Systems Approach","volume":"20","author":"Xu","year":"2015","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1007\/s00779-011-0465-2","article-title":"A Mobile Data Collection Platform for Mental Health Research","volume":"17","author":"Gaggioli","year":"2013","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_17","unstructured":"Apple HealthKit. Available online: https:\/\/developer.apple.com\/healthkit\/."},{"key":"ref_18","unstructured":"Google Fit. Available online: https:\/\/developers.google.com\/fit\/overview."},{"key":"ref_19","unstructured":"SAMI, S. Available online: https:\/\/developer.samsungsami.io\/sami\/sami-documentation\/."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Banos, O., Amin, M., Khan, W., Ali, T., Afzal, M., Kang, B., and Lee, S. (2015, January 20\u201323). Mining Minds: An innovative framework for personalized health and wellness support. Proceedings of the 2015 9th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), Istanbul, Turkey.","DOI":"10.4108\/icst.pervasivehealth.2015.259083"},{"key":"ref_21","first-page":"164","article-title":"High-Level Context Inference for Human Behavior Identification","volume":"Volume 9455","author":"Cleland","year":"2015","journal-title":"International Work-Conference on Ambient Assisted Living an Active Ageing (IWAAL 2015)"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1186\/s12938-016-0179-9","article-title":"The Mining Minds Digital Health and Wellness Framework","volume":"15","author":"Banos","year":"2016","journal-title":"BioMed. Eng. OnLine"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1109\/MC.2012.393","article-title":"Opportunistic Human Activity and Context Recognition","volume":"46","author":"Roggen","year":"2013","journal-title":"Computer"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Amin, M.B., Banos, O., Khan, W.A., Muhammad Bilal, H.S., Gong, J., Bui, D.M., Cho, S.H., Hussain, S., Ali, T., and Akhtar, U. (2016). On Curating Multimodal Sensory Data for Health and Wellness Platforms. Sensors, 16.","DOI":"10.3390\/s16070980"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"6474","DOI":"10.3390\/s140406474","article-title":"Window Size Impact in Human Activity Recognition","volume":"14","author":"Banos","year":"2014","journal-title":"Sensors"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"871","DOI":"10.1109\/TBME.2008.2006190","article-title":"A Comparison of Feature Extraction Methods for the Classification of Dynamic Activities From Accelerometer Data","volume":"56","author":"Preece","year":"2009","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1109\/TIT.1967.1053964","article-title":"Nearest neighbor pattern classification","volume":"13","author":"Cover","year":"1967","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"8013","DOI":"10.1016\/j.eswa.2012.01.164","article-title":"Daily living activity recognition based on statistical feature quality group selection","volume":"39","author":"Banos","year":"2012","journal-title":"Expert Syst. Appl."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"8039","DOI":"10.3390\/s120608039","article-title":"On the Use of Sensor Fusion to Reduce the Impact of Rotational and Additive Noise in Human Activity Recognition","volume":"12","author":"Banos","year":"2012","journal-title":"Sensors"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Banos, O., Calatroni, A., Damas, M., Pomares, H., Rojas, I., Sagha, H., del R Millan, J., Troster, G., Chavarriaga, R., and Roggen, D. (2012, January 18\u201322). Kinect=IMU? Learning MIMO Signal Mappings to Automatically Translate Activity Recognition Systems across Sensor Modalities. Proceeding of the 2012 16th International Symposium on Wearable Computers (ISWC), Newcastle, Australia.","DOI":"10.1109\/ISWC.2012.17"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"12588","DOI":"10.3390\/s120912588","article-title":"Comprehensive Context Recognizer Based on Multimodal Sensors in a Smartphone","volume":"12","author":"Han","year":"2012","journal-title":"Sensors"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"786","DOI":"10.1016\/j.eswa.2013.08.009","article-title":"Evolutionary joint selection to improve human action recognition with RGB-D devices","volume":"41","author":"Chaaraoui","year":"2014","journal-title":"Expert Syst. Appl."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1318","DOI":"10.1109\/TCYB.2013.2265378","article-title":"Enhanced computer vision with microsoft kinect sensor: A review","volume":"43","author":"Han","year":"2013","journal-title":"IEEE Trans. Cybern."},{"key":"ref_34","unstructured":"Soomro, K., and Zamir, A.R. (2014). Computer Vision in Sports, Springer."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1109\/TCSVT.2014.2376139","article-title":"CoDe4D: Color-depth local spatio-temporal features for human activity recognition from RGB-D videos","volume":"26","author":"Zhang","year":"2016","journal-title":"IEEE Trans. Circ. Syst. Video Technol."},{"key":"ref_36","unstructured":"Gupta, S., and Mooney, R.J. Using Closed Captions as Supervision for Video Activity Recognition. Available online: http:\/\/www.cs.utexas.edu\/ml\/papers\/gupta.aaai10.pdf."},{"key":"ref_37","first-page":"208","article-title":"Call Speech Emotion Recognition for Emotion based Services","volume":"41","author":"Bang","year":"2014","journal-title":"J. KIISE Softw. Appl."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Le, B.V., Bang, J., and Lee, S. (2013, January 5). Hierarchical Emotion Classification using Genetic Algorithms. Proceedings of the Fourth Symposium on Information and Communication Technology, New York, NY, USA.","DOI":"10.1145\/2542050.2542075"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Cristianini, N., and Shawe-Taylor, J. (2000). An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods, Cambridge University Press.","DOI":"10.1017\/CBO9780511801389"},{"key":"ref_40","unstructured":"Naver Maps API. Available online: http:\/\/developer.naver.com\/wiki\/pages\/MapAPI."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1007\/s00500-012-0896-3","article-title":"Human activity recognition based on a sensor weighting hierarchical classifier","volume":"17","author":"Banos","year":"2013","journal-title":"Soft Comput."},{"key":"ref_42","first-page":"290","article-title":"Multiwindow Fusion for Wearable Activity Recognition","volume":"Volume 9095","author":"Rojas","year":"2015","journal-title":"International Work-Conference on Artificial Neural Networks"},{"key":"ref_43","unstructured":"W3C OWL Working Group OWL 2 Web Ontology Language: Document Overview (Second Edition); W3C Recommendation. Available online: http:\/\/www.w3.org\/TR\/owl2-overview\/."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"473","DOI":"10.2307\/2274395","article-title":"Systematization of finite many-valued logics through the method of tableaux","volume":"52","author":"Carnielli","year":"1987","journal-title":"J. Symb. Log."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1145\/1656274.1656278","article-title":"The WEKA data mining software: an update","volume":"11","author":"Hall","year":"2009","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_46","unstructured":"Apache Jena. Available online: https:\/\/jena.apache.org\/."},{"key":"ref_47","unstructured":"Brickley, D., and Guha, R.V. RDF Schema 1.1; W3C Recommendation. Available online: https:\/\/www.w3.org\/TR\/rdf-schema\/."},{"key":"ref_48","unstructured":"Harris, S., and Seaborne, A. SPARQL 1.1 (SPARQL Query Language for RDF); W3C Recommendation. Available online: http:\/\/www.w3.org\/TR\/sparql11-query\/."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.websem.2007.03.004","article-title":"Pellet: A practical OWL-DL reasoner","volume":"5","author":"Sirin","year":"2007","journal-title":"J. Web Semant."},{"key":"ref_50","unstructured":"Microsoft Azure. Available online: http:\/\/azure.microsoft.com."},{"key":"ref_51","unstructured":"Richardson, L., and Ruby, S. (2008). RESTful Web Services, O\u2019Reilly Media, Inc."},{"key":"ref_52","unstructured":"Kinect for Windows v2. Available online: https:\/\/support.xbox.com\/en-US\/xbox-on-windows\/accessories\/kinect-for-windows-v2-info."},{"key":"ref_53","unstructured":"Samsung Galaxy S5. Available online: http:\/\/www.samsung.com\/us\/search\/searchMain?Dy=1Nty=1Ntt=samsung+galaxy+s53Acqcsrftoken=undefined."},{"key":"ref_54","unstructured":"LG G Watch R. Available online: http:\/\/www.lg.com\/us\/smart-watches\/lg-W110-g-watch-r."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2499621","article-title":"A Tutorial on Human Activity Recognition Using Body-worn Inertial Sensors","volume":"46","author":"Bulling","year":"2014","journal-title":"ACM Comput. Surv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/8\/1264\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:28:06Z","timestamp":1760210886000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/8\/1264"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,8,10]]},"references-count":55,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2016,8]]}},"alternative-id":["s16081264"],"URL":"https:\/\/doi.org\/10.3390\/s16081264","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2016,8,10]]}}}