{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T02:43:23Z","timestamp":1777085003363,"version":"3.51.4"},"reference-count":37,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2021,3,27]],"date-time":"2021-03-27T00:00:00Z","timestamp":1616803200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010665","name":"H2020 Marie Sk\u0142odowska-Curie Actions","doi-asserted-by":"publisher","award":["676201"],"award-info":[{"award-number":["676201"]}],"id":[{"id":"10.13039\/100010665","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"publisher","award":["12\/RC\/2289_P2"],"award-info":[{"award-number":["12\/RC\/2289_P2"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Machine learning models are being utilized to provide wearable sensor-based exercise biofeedback to patients undertaking physical therapy. However, most systems are validated at a technical level using lab-based cross validation approaches. These results do not necessarily reflect the performance levels that patients and clinicians can expect in the real-world environment. This study aimed to conduct a thorough evaluation of an example wearable exercise biofeedback system from laboratory testing through to clinical validation in the target setting, illustrating the importance of context when validating such systems. Each of the various components of the system were evaluated independently, and then in combination as the system is designed to be deployed. The results show a reduction in overall system accuracy between lab-based cross validation (&gt;94%), testing on healthy participants (n = 10) in the target setting (&gt;75%), through to test data collected from the clinical cohort (n = 11) (&gt;59%). This study illustrates that the reliance on lab-based validation approaches may be misleading key stakeholders in the inertial sensor-based exercise biofeedback sector, makes recommendations for clinicians, developers and researchers, and discusses factors that may influence system performance at each stage of evaluation.<\/jats:p>","DOI":"10.3390\/s21072346","type":"journal-article","created":{"date-parts":[[2021,3,28]],"date-time":"2021-03-28T23:27:25Z","timestamp":1616974045000},"page":"2346","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["The Importance of Real-World Validation of Machine Learning Systems in Wearable Exercise Biofeedback Platforms: A Case Study"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3522-9409","authenticated-orcid":false,"given":"Rob","family":"Argent","sequence":"first","affiliation":[{"name":"Insight Centre for Data Analytics, University College Dublin, D04 V1W8 Dublin, Ireland"},{"name":"School of Public Health, Physiotherapy and Sport Science, University College Dublin, D04 V1W8 Dublin, Ireland"}]},{"given":"Antonio","family":"Bevilacqua","sequence":"additional","affiliation":[{"name":"Insight Centre for Data Analytics, University College Dublin, D04 V1W8 Dublin, Ireland"},{"name":"School of Computer Science, University College Dublin, D04 V1W8 Dublin, Ireland"}]},{"given":"Alison","family":"Keogh","sequence":"additional","affiliation":[{"name":"Insight Centre for Data Analytics, University College Dublin, D04 V1W8 Dublin, Ireland"},{"name":"School of Public Health, Physiotherapy and Sport Science, University College Dublin, D04 V1W8 Dublin, Ireland"}]},{"given":"Ailish","family":"Daly","sequence":"additional","affiliation":[{"name":"Beacon Hospital, D18 AK68 Dublin, Ireland"}]},{"given":"Brian","family":"Caulfield","sequence":"additional","affiliation":[{"name":"Insight Centre for Data Analytics, University College Dublin, D04 V1W8 Dublin, Ireland"},{"name":"School of Public Health, Physiotherapy and Sport Science, University College Dublin, D04 V1W8 Dublin, Ireland"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,27]]},"reference":[{"key":"ref_1","first-page":"231","article-title":"Telerehabilitation:Enabling the Remote Delivery of Healthcare, Rehabilitation, and Self Management","volume":"145","author":"Brennan","year":"2009","journal-title":"Stud. Heal. Technol. Inf."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1080\/09638280802062553","article-title":"A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation","volume":"31","author":"Kairy","year":"2009","journal-title":"Disabil. Rehabil."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1743-0003-10-60","article-title":"Biofeedback in rehabilitation","volume":"10","author":"Giggins","year":"2013","journal-title":"J. Neuroeng. Rehabil."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"e4","DOI":"10.2196\/rehab.7258","article-title":"Translating Comprehensive Conservative Care for Chronic Knee Pain Into a Digital Care Pathway: 12-Week and 6-Month Outcomes for the Hinge Health Program","volume":"4","author":"Smittenaar","year":"2017","journal-title":"JMIR Rehabil. Assist. Technol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-018-29668-0","article-title":"Home-based Rehabilitation With A Novel Digital Biofeedback System versus Conventional In-person Rehabilitation after Total Knee Replacement: a feasibility study","volume":"8","author":"Correia","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Argent, R., Slevin, P., Bevilacqua, A., Neligan, M., Daly, A., and Caulfield, B. (2019). Wearable Sensor-Based Exercise Biofeedback for Orthopaedic Rehabilitation: A Mixed Methods User Evaluation of a Prototype System. Sensors, 19.","DOI":"10.3390\/s19020432"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-018-0076-7","article-title":"Randomized controlled trial of a 12-week digital care program in improving low back pain","volume":"2","author":"Shebib","year":"2019","journal-title":"NPJ Digit Med."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1221","DOI":"10.1007\/s40279-018-0878-4","article-title":"Wearable Inertial Sensor Systems for Lower Limb Exercise Detection and Evaluation: A Systematic Review","volume":"48","author":"Caulfield","year":"2018","journal-title":"Sport Med."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1016\/j.msksp.2018.11.012","article-title":"Artificial intelligence and machine learning | applications in musculoskeletal physiotherapy","volume":"39","author":"Tack","year":"2019","journal-title":"Musculoskelet Sci. Pract."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Taylor, P.E., Almeida, G.J.M., Hodgins, J.K., and Kanade, T. (September, January 8). Multi-label classification for the analysis of human motion quality. Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, USA.","DOI":"10.1109\/EMBC.2012.6346402"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bevilacqua, A., Huang, B., Argent, R., Caulfield, B., and Kechadi, T. (2018, January 4\u20137). Automatic Classification of Knee Rehabilitation Exercises Using a Single Inertial Sensor: A Case Study. Proceedings of the 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN), Las Vegas, NV, USA.","DOI":"10.1109\/BSN.2018.8329649"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jbiomech.2018.09.009","article-title":"Machine learning in human movement biomechanics: Best practices, common pitfalls, and new opportunities","volume":"81","author":"Halilaj","year":"2018","journal-title":"J. Biomech."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1080\/14763141.2017.1314544","article-title":"Classification of lunge biomechanics with multiple and individual inertial measurement units","volume":"16","author":"Whelan","year":"2017","journal-title":"Sport Biomech."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Bevilacqua, A., Ciampi, G., Argent, R., Caulfield, B., and Kechadi, T. (2020, January 7\u201312). Combining Real-Time Segmentation and Classification of Rehabilitation Exercises with LSTM Networks and Pointwise Boosting. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i08.7028"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Huang, B., Giggins, O., Kechadi, T., and Caulfield, B. (2016, January 16\u201320). The Limb Movement Analysis of Rehabilitation Exercises using Wearable Inertial Sensors. Proceedings of the IEEE 38th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.","DOI":"10.1109\/EMBC.2016.7591773"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Whelan, D., O\u2019Reilly, M., Huang, B., Giggins, O., Kechadi, T., and Caulfield, B. (2016, January 16\u201320). Leveraging IMU data for accurate exercise performance classification and musculoskeletal injury risk screening. Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.","DOI":"10.1109\/EMBC.2016.7590788"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1049\/htl.2015.0019","article-title":"Review: Are we stumbling in our quest to find the best predictor? Over-optimism in sensor-based models for predicting falls in older adults","volume":"2","author":"Shany","year":"2015","journal-title":"Healthc. Technol. Lett."},{"key":"ref_18","unstructured":"WHO (2016). Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment, World Health Organization."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"e9","DOI":"10.2196\/rehab.7259","article-title":"Mobile App to Streamline the Development of Wearable Sensor-Based Exercise Biofeedback Systems: System Development and Evaluation","volume":"4","author":"Duffin","year":"2017","journal-title":"JMIR Rehabil. Assist. Technol."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Jarm, T., Cvetkoska, A., Mahni\u010d-Kalamiza, S., and Miklavcic, D. A Novel Validation Framework to Assess Segmentation Accuracy of Inertial Sensor Data for Rehabilitation Exercises. Proceedings of the 8th European Medical and Biological Engineering Conference, Portoro\u017e, Slovenia, 29 November\u20133 December 2020.","DOI":"10.1007\/978-3-030-64610-3"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1186\/1743-0003-11-158","article-title":"Rehabilitation exercise assessment using inertial sensors: a cross-sectional analytical study","volume":"11","author":"Giggins","year":"2014","journal-title":"J. Neuroeng. Rehabil."},{"key":"ref_22","unstructured":"American Academy of Orthopaedic Surgeons (2018, May 24). Total Knee Replacement Exercise Guide [Internet]. Available online: http:\/\/orthoinfo.aaos.org\/topic.cfm?topic=a00301."},{"key":"ref_23","unstructured":"Arthritis Research UK (2018, May 24). Knee Replacement Surgery [Internet]. Available online: http:\/\/www.arthritisresearchuk.org\/arthritis-information\/surgery\/knee-replacement.aspx."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1007\/s00779-010-0293-9","article-title":"Preprocessing techniques for context recognition from accelerometer data","volume":"14","author":"Figo","year":"2010","journal-title":"Pers Ubiquitous Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"R1","DOI":"10.1088\/0967-3334\/30\/4\/R01","article-title":"Activity identification using body-mounted sensors\u2014A review of classification techniques","volume":"30","author":"Preece","year":"2009","journal-title":"Physiol. Meas."},{"key":"ref_26","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1527","DOI":"10.1109\/JSEN.2010.2045498","article-title":"SHIMMERTM\u2013A Wireless Sensor Platform for Noninvasive Biomedical Research","volume":"10","author":"Burns","year":"2010","journal-title":"IEEE Sens. J."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1386","DOI":"10.2106\/JBJS.N.01141","article-title":"Prevalence of total hip and knee replacement in the United States","volume":"97","author":"Kremers","year":"2015","journal-title":"J. Bone Jt. Surg. Am."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1109\/THMS.2015.2493536","article-title":"Movement primitive segmentation for human motion modeling: A framework for analysis","volume":"46","author":"Lin","year":"2016","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1109\/TNSRE.2013.2259640","article-title":"Online segmentation of human motion for automated rehabilitation exercise analysis","volume":"22","author":"Lin","year":"2013","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1158","DOI":"10.1109\/TRO.2009.2026508","article-title":"Online segmentation and clustering from continuous observation of whole body motions","volume":"25","author":"Kulic","year":"2009","journal-title":"IEEE Trans. Robot."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1109\/89.917689","article-title":"Audio content analysis for online audiovisual data segmentation and classification","volume":"9","author":"Zhang","year":"2001","journal-title":"IEEE Trans. Speech Audio Process."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2303","DOI":"10.1519\/JSC.0000000000001957","article-title":"Technology in Strength and Conditioning: Assessing Bodyweight Squat Technique with Wearable Sensors","volume":"31","author":"Whelan","year":"2017","journal-title":"J. Strength Cond. Res."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"4193","DOI":"10.3390\/s150204193","article-title":"Wearable sensor-based rehabilitation exercise assessment for knee osteoarthritis","volume":"15","author":"Chen","year":"2015","journal-title":"Sensors"},{"key":"ref_35","first-page":"1","article-title":"Classification-based Segmentation for Rehabilitation Exercise Monitoring","volume":"5","author":"Lin","year":"2018","journal-title":"J. Rehabil. Assist. Technol. Eng."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1145\/1007730.1007735","article-title":"A study of the behavior of several methods for balancing machine learning training data","volume":"6","author":"Batista","year":"2004","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"478","DOI":"10.1093\/ptj\/pzz004","article-title":"Determining Interrater and Intrarater Levels of Agreement in Students and Clinicians When Visually Evaluating Movement Proficiency During Screening Assessments","volume":"99","author":"Whelan","year":"2019","journal-title":"Phys. Ther."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/7\/2346\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:32:59Z","timestamp":1760362379000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/7\/2346"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,27]]},"references-count":37,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["s21072346"],"URL":"https:\/\/doi.org\/10.3390\/s21072346","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,27]]}}}