{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T18:02:46Z","timestamp":1773511366440,"version":"3.50.1"},"reference-count":30,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2019,8,9]],"date-time":"2019-08-09T00:00:00Z","timestamp":1565308800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004073","name":"Killam Trusts","doi-asserted-by":"publisher","award":["The Izaak Walton Killam Memorial Scholarships"],"award-info":[{"award-number":["The Izaak Walton Killam Memorial Scholarships"]}],"id":[{"id":"10.13039\/501100004073","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Natural Sciences and Engineering Research Council of Canada (NSERC)","award":["Discovery Grant"],"award-info":[{"award-number":["Discovery Grant"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Accuracy evaluation in machine learning is based on the split of data into a training set and a test set. This critical step is applied to develop machine learning models including models based on sensor data. For sensor-based problems, comparing the accuracy of machine learning models using the train\/test split provides only a baseline comparison in ideal situations. Such comparisons won\u2019t consider practical production problems that can impact the inference accuracy such as the sensors\u2019 thermal noise, performance with lower inference quantization, and tolerance to sensor failure. Therefore, this paper proposes a set of practical tests that can be applied when comparing the accuracy of machine learning models for sensor-based problems. First, the impact of the sensors\u2019 thermal noise on the models\u2019 inference accuracy was simulated. Machine learning algorithms have different levels of error resilience to thermal noise, as will be presented. Second, the models\u2019 accuracy using lower inference quantization was compared. Lowering inference quantization leads to lowering the analog-to-digital converter (ADC) resolution which is cost-effective in embedded designs. Moreover, in custom designs, analog-to-digital converters\u2019 (ADCs) effective number of bits (ENOB) is usually lower than the ideal number of bits due to various design factors. Therefore, it is practical to compare models\u2019 accuracy using lower inference quantization. Third, the models\u2019 accuracy tolerance to sensor failure was evaluated and compared. For this study, University of California Irvine (UCI) \u2018Daily and Sports Activities\u2019 dataset was used to present these practical tests and their impact on model selection.<\/jats:p>","DOI":"10.3390\/s19163491","type":"journal-article","created":{"date-parts":[[2019,8,9]],"date-time":"2019-08-09T11:11:31Z","timestamp":1565349091000},"page":"3491","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1895-3438","authenticated-orcid":false,"given":"Issam","family":"Hammad","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Dalhousie University, Halifax, NS B3H 4R2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kamal","family":"El-Sankary","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Dalhousie University, Halifax, NS B3H 4R2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,8,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"521","DOI":"10.3390\/make1010032","article-title":"Evaluation of Regression Models: Model Assessment, Model Selection and Generalization Error","volume":"1","author":"Matthias","year":"2019","journal-title":"Mach. Learn. Knowl. Extr."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Emmert-Streib, F., Salisou, M., and Matthias, D. (2019). A comprehensive survey of error measures for evaluating binary decision making in data science. Wiley Interdiscip. Rev. Data Min. Knowl. Discov., e1303.","DOI":"10.1002\/widm.1303"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1050","DOI":"10.1109\/72.870038","article-title":"On overfitting, generalization, and randomly expanded training sets","volume":"11","author":"Karystinos","year":"2000","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3605","DOI":"10.1016\/j.patcog.2010.04.019","article-title":"Comparative study on classifying human activities with miniature inertial and magnetic sensors","volume":"43","author":"Altun","year":"2010","journal-title":"Pattern Recognit."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1649","DOI":"10.1093\/comjnl\/bxt075","article-title":"Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units","volume":"57","author":"Barshan","year":"2014","journal-title":"Comput. J."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Chung, S. (2019). Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning. Sensors, 19.","DOI":"10.3390\/s19071716"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1426","DOI":"10.1109\/JSEN.2018.2880434","article-title":"A Wearable Sensor for Arterial Stiffness Monitoring Based on Machine Learning Algorithms","volume":"19","author":"Miao","year":"2018","journal-title":"IEEE Sens. J."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Yeh, C. (2019). Machine Learning for Long Cycle Maintenance Prediction of Wind Turbine. Sensors, 19.","DOI":"10.3390\/s19071671"},{"key":"ref_9","unstructured":"(2019, March 01). University of Califonia Irvine Machine Learning Repository, Daily and Sports Activities Data Set. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/daily+and+sports+activities."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"6261","DOI":"10.1109\/JSEN.2016.2582262","article-title":"Signal quality and compactness of a dual-accelerometer system for gyro-free human motion analysis","volume":"16","author":"Villeneuve","year":"2016","journal-title":"IEEE Sens. J."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"518","DOI":"10.1016\/j.mechatronics.2013.04.003","article-title":"Measuring motion with kinematically redundant accelerometer arrays: Theory, simulation and implementation","volume":"23","author":"Madgwick","year":"2013","journal-title":"Mechatronics"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1109\/TCSII.2002.801191","article-title":"Speed-power-accuracy tradeoff in high-speed CMOS ADCs","volume":"49","author":"Uyttenhove","year":"2002","journal-title":"IEEE Trans. Circuits Syst. Analog Digit. Signal Proc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"3183","DOI":"10.1109\/TIM.2015.2450296","article-title":"ADC standard IEC 60748-4-3: Precision measurement of alternative ENOB without a sine wave","volume":"64","author":"Belcher","year":"2015","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_14","unstructured":"Technologies, X., and MTi, B.V. (2019, April 02). MTx User Manual. Available online: http:\/\/www.xsens.com."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"31314","DOI":"10.3390\/s151229858","article-title":"Physical human activity recognition using wearable sensors","volume":"15","author":"Attal","year":"2015","journal-title":"Sensors"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1109\/TITB.2012.2196440","article-title":"An incremental learning method based on probabilistic neural networks and adjustable fuzzy clustering for human activity recognition by using wearable sensors","volume":"16","author":"Wang","year":"2012","journal-title":"IEEE Trans. Inf. Technol. Biomed."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"829","DOI":"10.1109\/TASE.2013.2256349","article-title":"An unsupervised approach for automatic activity recognition based on hidden Markov model regression","volume":"10","author":"Trabelsi","year":"2013","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_18","unstructured":"Chollet, F. (2018, October 12). 2015 Keras. Available online: https:\/\/github.com\/fchollet\/keras."},{"key":"ref_19","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1016\/j.sigpro.2013.10.002","article-title":"Simplified noise model parameter estimation for signal-dependent noise","volume":"96","author":"Jeong","year":"2014","journal-title":"Signal Process."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1109\/JSEN.2005.844538","article-title":"Mechanical-thermal noise in MEMS gyroscopes","volume":"5","author":"Robert","year":"2005","journal-title":"IEEE Sens. J."},{"key":"ref_23","unstructured":"Lent, B. (2019, May 01). Practical Considerations of Accelerometers Noise. Available online: https:\/\/www.endevco.com\/news\/archivednews\/2009\/2009_12\/TP324.pdf."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"6454","DOI":"10.1109\/JSEN.2015.2461432","article-title":"A generalized study of coil-core-aspect ratio optimization for noise reduction and SNR enhancement in search coil magnetometers at low frequencies","volume":"15","author":"Abolghasem","year":"2015","journal-title":"IEEE Sens. J."},{"key":"ref_25","unstructured":"(2016). How to Increase the Analog-to-Digital Converter Accuracy in an Application, Freescale Semiconductor, Inc."},{"key":"ref_26","unstructured":"Large, D., and James, F. (2009). The HFC Plant. Broadband Cable Access Networks, Elsevier Inc."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"434","DOI":"10.1109\/TCSII.2009.2020929","article-title":"Spurious-free dynamic range of a uniform quantizer","volume":"56","author":"Alink","year":"2009","journal-title":"IEEE Trans. Circuits Syst. Express Briefs"},{"key":"ref_28","unstructured":"Johns, D.A., and Ken, M. (2008). Analog Integrated Circuit Design, John Wiley & Sons."},{"key":"ref_29","unstructured":"Murmann, B. (2019, June 08). ADC Performance Survey 1997\u20132019. Available online: http:\/\/web.stanford.edu\/~murmann\/adcsurvey.html."},{"key":"ref_30","unstructured":"Timothy, H.W. (2011, January 1\u20134). The hitchhiker\u2019s guide to successful residential sensing deployments. Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems, Seattle, WA, USA."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/16\/3491\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:09:52Z","timestamp":1760188192000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/16\/3491"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8,9]]},"references-count":30,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2019,8]]}},"alternative-id":["s19163491"],"URL":"https:\/\/doi.org\/10.3390\/s19163491","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,8,9]]}}}