{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T03:02:22Z","timestamp":1775790142855,"version":"3.50.1"},"reference-count":55,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2020,3,30]],"date-time":"2020-03-30T00:00:00Z","timestamp":1585526400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1553528"],"award-info":[{"award-number":["IIS-1553528"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Mobile health monitoring plays a central role in the future of cyber physical systems (CPS) for healthcare applications. Such monitoring systems need to process user data accurately. Unlike in other human-centered CPS, in healthcare CPS, the user functions in multiple roles all at the same time: as an operator, an actuator, the physical environment and, most importantly, the target that needs to be monitored in the process. Therefore, mobile health CPS devices face highly dynamic settings generally, and accuracy of the machine learning models the devices employ may drop dramatically every time a change in setting happens. Novel learning architecture that specifically address challenges associated with dynamic environments are therefore needed. Using active learning and transfer learning as organizing principles, we propose a collaborative multiple-expert architecture and accompanying algorithms for the design of machine learning models that autonomously adapt to a new configuration, context, or user need. Specifically, our architecture and its constituent algorithms are designed to manage heterogeneous knowledge sources or experts with varying levels of confidence and type while minimizing adaptation cost. Additionally, our framework incorporates a mechanism for collaboration among experts to enrich their knowledge, which in turn decreases both cost and uncertainty of data labeling in future steps. We evaluate the efficacy of the architecture using two publicly available human activity datasets. We attain activity recognition accuracy of over     85 %     (for the first dataset) and     92 %     (for the second dataset) by labeling only     15 %     of unlabeled data.<\/jats:p>","DOI":"10.3390\/s20071932","type":"journal-article","created":{"date-parts":[[2020,4,1]],"date-time":"2020-04-01T03:44:13Z","timestamp":1585712653000},"page":"1932","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Collaborative Multi-Expert Active Learning for Mobile Health Monitoring: Architecture, Algorithms, and Evaluation"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5687-3420","authenticated-orcid":false,"given":"Ramyar","family":"Saeedi","sequence":"first","affiliation":[{"name":"School of Electrical Engineering and Computer Science, Washington State University, Pullman, WA 991642250, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Keyvan","family":"Sasani","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer Science, Washington State University, Pullman, WA 991642250, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Assefaw H.","family":"Gebremedhin","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer Science, Washington State University, Pullman, WA 991642250, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,3,30]]},"reference":[{"key":"ref_1","first-page":"66","article-title":"mHealth: New horizons for health through mobile technologies","volume":"3","author":"Kay","year":"2011","journal-title":"World Health Organ."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Mosa, A.S.M., Yoo, I., and Sheets, L. (2012). A systematic review of healthcare applications for smartphones. BMC Med. Inf. Decis. Mak., 12.","DOI":"10.1186\/1472-6947-12-67"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jpsychires.2016.10.019","article-title":"Mobile devices for the remote acquisition of physiological and behavioral biomarkers in psychiatric clinical research","volume":"85","author":"Adams","year":"2017","journal-title":"J. Psychiatr. Res."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1109\/JSYST.2015.2460747","article-title":"Health-CPS: Healthcare cyber-physical system assisted by cloud and big data","volume":"11","author":"Zhang","year":"2017","journal-title":"IEEE Syst. J."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1589\/jpts.29.613","article-title":"Interrelationship between the general characteristics of Korean stroke patients and the variables of the sexual functions: random forest and boosting algorithm","volume":"29","author":"Yoon","year":"2017","journal-title":"J. Phys. Ther. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Free, C., Phillips, G., Watson, L., Galli, L., Felix, L., Edwards, P., Patel, V., and Haines, A. (2013). The effectiveness of mobile-health technologies to improve health care service delivery processes: A systematic review and meta-analysis. PLoS Med., 10.","DOI":"10.1371\/journal.pmed.1001363"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hayes, D.F., Markus, H.S., Leslie, R.D., and Topol, E.J. (2014). Personalized medicine: risk prediction, targeted therapies and mobile health technology. BMC Med., 12.","DOI":"10.1186\/1741-7015-12-37"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1089\/tmj.2014.0103","article-title":"Older people with access to hand-held devices: Who are they?","volume":"21","author":"Shahrokni","year":"2015","journal-title":"Telemed. e-Health"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1097\/MAT.0000000000000446","article-title":"Multi-Sense CardioPatch: A Wearable Patch for Remote Monitoring of Electro-Mechanical Cardiac Activity","volume":"63","author":"Marcelli","year":"2017","journal-title":"Asaio J."},{"key":"ref_10","unstructured":"Eggers, P.E., Eggers, E.A., and Bailey, B.Z. (2016). Wearable Apparatus, System and Method for Detection of Cardiac Arrest and Alerting Emergency Response. (20,160,174,857), US Patent."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/j.compbiomed.2017.01.011","article-title":"Automated seizure detection using limited-channel EEG and non-linear dimension reduction","volume":"82","author":"Birjandtalab","year":"2017","journal-title":"Comput. Biol. Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.tips.2014.11.002","article-title":"Data collection as a barrier to personalized medicine","volume":"36","author":"Zakim","year":"2015","journal-title":"Trends Pharmacol. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Fallahzadeh, R., Pedram, M., Saeedi, R., Sadeghi, B., Ong, M., and Ghasemzadeh, H. (2015, January 23\u201327). Smart-cuff: A wearable bio-sensing platform with activity-sensitive information quality assessment for monitoring ankle edema. Proceedings of the 2015 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops), St. Louis, MI, USA.","DOI":"10.1109\/PERCOMW.2015.7133994"},{"key":"ref_14","first-page":"500","article-title":"Situation awareness and complexity: The role of wearable technologies in sports science","volume":"10","year":"2015","journal-title":"J. Human Sport Exerc."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Lomasky, R., Brodley, C.E., Aernecke, M., Walt, D., and Friedl, M. (2007). Active class selection. European Conference on Machine Learning, Springer.","DOI":"10.1007\/978-3-540-74958-5_63"},{"key":"ref_16","unstructured":"Settles, B., Craven, M., and Friedland, L. (2008, January 13). Active learning with real annotation costs. Proceedings of the NIPS Workshop on Cost-Sensitive Learning, Vancouver, BC, Canada."},{"key":"ref_17","unstructured":"Sagha, H., Mill\u00e1n, J.d.R., and Chavarriaga, R. (2010, January 26\u201329). A probabilistic approach to handle missing data for multi-sensory activity recognition. Proceedings of theWorkshop on Context Awareness and Information Processing in Opportunistic Ubiquitous Systems at 12th ACM International Conference on Ubiquitous Computing, Copenhagen, Denmark. number EPFL-CONF-151974."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Fallahzadeh, R., and Ghasemzadeh, H. (2017, January 18\u201321). Personalization without User Interruption: Boosting Activity Recognition in New Subjects Using Unlabeled Data (An Uninformed Cross-Subject Transfer Learning Algorithm). Proceedings of the International Conference on Cyber-Physical Systems (ICCPS), Pittsburgh, PA, USA.","DOI":"10.1145\/3055004.3055015"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zheng, V.W., Hu, D.H., and Yang, Q. (2009, January 12\u201316). Cross-domain activity recognition. Proceedings of the 11th International Conference on Ubiquitous Computing, Orlando, FL, USA.","DOI":"10.1145\/1620545.1620554"},{"key":"ref_20","unstructured":"Settles, B. (2010). Active Learning Literature Survey, University of Wisconsin. Computer Science Technical Report 1648."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Chu, W., Zinkevich, M., Li, L., Thomas, A., and Tseng, B. (2011, January 22\u201327). Unbiased online active learning in data streams. Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA.","DOI":"10.1145\/2020408.2020444"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1016\/j.patcog.2015.10.006","article-title":"Dynamic label propagation for semi-supervised multi-class multi-label classification","volume":"52","author":"Wang","year":"2016","journal-title":"Pattern Recognit."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Doukas, C., and Maglogiannis, I. (2012, January 4\u20136). Bringing IoT and cloud computing towards pervasive healthcare. Proceedings of the 2012 Sixth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), Palermo, Italy.","DOI":"10.1109\/IMIS.2012.26"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1109\/MC.2012.392","article-title":"Mobile health: Revolutionizing healthcare through transdisciplinary research","volume":"46","author":"Kumar","year":"2013","journal-title":"Computer"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"3810","DOI":"10.1109\/JIOT.2018.2849014","article-title":"Internet of medical things: A review of recent contributions dealing with cyber-physical systems in medicine","volume":"5","author":"Gatouillat","year":"2018","journal-title":"IEEE Int. Things J."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Saeedi, R., Sasani, K., and Gebremedhin, A.H. (2017, January 20\u201323). Co-MEAL: Cost-Optimal Multi-Expert Active Learning Architecture for Mobile Health Monitoring. Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Boston, MA, USA.","DOI":"10.1145\/3107411.3107430"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Settles, B., and Craven, M. (2008, January 25\u201327). An analysis of active learning strategies for sequence labeling tasks. Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, HI, USA.","DOI":"10.3115\/1613715.1613855"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Gadde, A., Anis, A., and Ortega, A. (2014, January 24\u201327). Active semi-supervised learning using sampling theory for graph signals. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA.","DOI":"10.1145\/2623330.2623760"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1007\/s11263-014-0781-x","article-title":"Multi-class active learning by uncertainty sampling with diversity maximization","volume":"113","author":"Yang","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Longstaff, B., Reddy, S., and Estrin, D. (2010, January 22\u201325). Improving activity classification for health applications on mobile devices using active and semi-supervised learning. Proceedings of the 2010 4th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), Munchen, Germany.","DOI":"10.4108\/ICST.PERVASIVEHEALTH2010.8851"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1145\/3130959","article-title":"Label Propagation: An Unsupervised Similarity Based Method for Integrating New Sensors in Activity Recognition Systems","volume":"1","author":"Rey","year":"2017","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Bagaveyev, S., and Cook, D.J. (2014, January 13\u201317). Designing and evaluating active learning methods for activity recognition. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, Seattle, WA, USA.","DOI":"10.1145\/2638728.2641674"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Hoque, E., and Stankovic, J. (2012, January 21\u201324). AALO: Activity recognition in smart homes using Active Learning in the presence of Overlapped activities. Proceedings of the 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), San Diego, CA, USA.","DOI":"10.4108\/icst.pervasivehealth.2012.248600"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Donmez, P., and Carbonell, J.G. (2008, January 26\u201330). Proactive learning: cost-sensitive active learning with multiple imperfect oracles. Proceedings of the 17th ACM Conference on Information and Knowledge Management, Napa Valley, CA, USA.","DOI":"10.1145\/1458082.1458165"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Arora, S., Nyberg, E., and Ros\u00e9, C.P. (2009, January 5). Estimating annotation cost for active learning in a multi-annotator environment. Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing. Association for Computational Linguistics, Boulder, CO, USA.","DOI":"10.3115\/1564131.1564136"},{"key":"ref_36","unstructured":"Gorriz, M., Carlier, A., Faure, E., and Giro-i Nieto, X. (2017). Cost-Effective Active Learning for Melanoma Segmentation. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s40708-016-0042-6","article-title":"Interactive machine learning for health informatics: when do we need the human-in-the-loop?","volume":"3","author":"Holzinger","year":"2016","journal-title":"Brain Inform."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Saeedi, R., Ghasemzadeh, H., and Gebremedhin, A.H. (2016, January 5\u20138). Transfer learning algorithms for autonomous reconfiguration of wearable systems. Proceedings of the 2016 IEEE International Conference on Big Data, Washington DC, USA.","DOI":"10.1109\/BigData.2016.7840648"},{"key":"ref_39","unstructured":"Saeedi, R., and Gebremedhin, A. (2018). A Signal-Level Transfer Learning Framework for Autonomous Reconfiguration of Wearable Systems. IEEE Trans. Mob. Comput."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Rokni, S.A., and Ghasemzadeh, H. (2016, January 5\u20139). Plug-n-learn: Automatic learning of computational algorithms in human-centered internet-of-things applications. Proceedings of the 53rd Annual Design Automation Conference, Austin, TX, USA.","DOI":"10.1145\/2897937.2898066"},{"key":"ref_41","unstructured":"Jiang, J. (2020, March 30). A Literature Survey on Domain Adaptation of Statistical Classifiers. Available online: http:\/\/sifaka.cs.uiuc.edu\/jiang4\/domainadaptation\/survey."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_43","first-page":"242","article-title":"Transfer learning","volume":"1","author":"Torrey","year":"2009","journal-title":"Handb. Res. Mach. Learn. Appl. Trends: Algorithm. Methods Tech."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1007\/s10115-013-0665-3","article-title":"Transfer learning for activity recognition: A survey","volume":"36","author":"Cook","year":"2013","journal-title":"Knowl. inf. syst."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1108\/IJPCC-03-2014-0020","article-title":"Heterogeneous transfer learning for activity recognition using heuristic search techniques","volume":"10","author":"Cook","year":"2014","journal-title":"Int. J. Pervasive Comput. Commun."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Rokni, S.A., and Ghasemzadeh, H. (2017). Synchronous dynamic view learning: A framework for autonomous training of activity recognition models using wearable sensors. Proceedings of the 16th ACM\/IEEE International Conference on Information Processing in Sensor Networks, Association for Computing Machinery.","DOI":"10.1145\/3055031.3055087"},{"key":"ref_47","unstructured":"Calma, A., Leimeister, J.M., Lukowicz, P., Oeste-Rei\u00df, S., Reitmaier, T., Schmidt, A., Sick, B., Stumme, G., and Zweig, K.A. (2016, January 4\u20137). From active learning to dedicated collaborative interactive learning. Proceedings of the ARCS 2016, Nuremberg, Germany."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"4340","DOI":"10.1109\/JSEN.2014.2331463","article-title":"Improved use of foot force sensors and mobile phone GPS for mobility activity recognition","volume":"14","author":"Zhang","year":"2014","journal-title":"IEEE Sens. J."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Saeedi, R., Amini, N., and Ghasemzadeh, H. (2014, January 2\u20135). Patient-centric on-body sensor localization in smart health systems. Proceedings of the 2014 48th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA.","DOI":"10.1109\/ACSSC.2014.7094840"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Sztyler, T., and Stuckenschmidt, H. (2016, January 14\u201318). On-body localization of wearable devices: an investigation of position-aware activity recognition. Proceedings of the 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom), Sydney, Australia.","DOI":"10.1109\/PERCOM.2016.7456521"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"3095","DOI":"10.1109\/ACCESS.2017.2676168","article-title":"Performance Analysis of Smartphone-Sensor Behavior for Human Activity Recognition","volume":"5","author":"Chen","year":"2017","journal-title":"IEEE Access"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1649","DOI":"10.1093\/comjnl\/bxt075","article-title":"Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units","volume":"57","author":"Barshan","year":"2013","journal-title":"Comput. J."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1109\/JSAC.2009.090107","article-title":"Energy-Efficient Information-Driven Coverage for Physical Movement Monitoring in Body Sensor Networks","volume":"27","author":"Ghasemzadeh","year":"2009","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/JSEN.2012.2222572","article-title":"Wireless medical-embedded systems: A review of signal-processing techniques for classification","volume":"13","author":"Ghasemzadeh","year":"2013","journal-title":"IEEE Sens. J."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Zappi, P., Stiefmeier, T., Farella, E., Roggen, D., Benini, L., and Troster, G. (2007, January 3\u20136). Activity recognition from on-body sensors by classifier fusion: sensor scalability and robustness. Proceedings of the 3rd International Conference on Intelligent Sensors, Sensor Networks and Information, Melbourne, Australia.","DOI":"10.1109\/ISSNIP.2007.4496857"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/7\/1932\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:13:35Z","timestamp":1760174015000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/7\/1932"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,30]]},"references-count":55,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2020,4]]}},"alternative-id":["s20071932"],"URL":"https:\/\/doi.org\/10.3390\/s20071932","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,30]]}}}