{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,5]],"date-time":"2026-05-05T20:02:09Z","timestamp":1778011329018,"version":"3.51.4"},"reference-count":57,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,9,24]],"date-time":"2024-09-24T00:00:00Z","timestamp":1727136000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Human activity recognition (HAR) and brain-machine interface (BMI) are two emerging technologies that can enhance human-robot collaboration (HRC) in domains such as industry or healthcare. HAR uses sensors or cameras to capture and analyze the movements and actions of humans, while BMI uses human brain signals to decode action intentions. Both technologies face challenges impacting accuracy, reliability, and usability. In this article, we review the state-of-the-art techniques and methods for HAR and BMI and highlight their strengths and limitations. We then propose a hybrid framework that fuses HAR and BMI data, which can integrate the complementary information from the brain and body motion signals and improve the performance of human state decoding. We also discuss our hybrid method\u2019s potential benefits and implications for HRC.<\/jats:p>","DOI":"10.3389\/fnbot.2024.1383089","type":"journal-article","created":{"date-parts":[[2024,9,24]],"date-time":"2024-09-24T05:10:25Z","timestamp":1727154625000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Human in the collaborative loop: a strategy for integrating human activity recognition and non-invasive brain-machine interfaces to control collaborative robots"],"prefix":"10.3389","volume":"18","author":[{"given":"Artur","family":"Pilacinski","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lukas","family":"Christ","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marius","family":"Boshoff","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ioannis","family":"Iossifidis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Patrick","family":"Adler","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Miro","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bernd","family":"Kuhlenk\u00f6tter","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christian","family":"Klaes","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,9,24]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"906","DOI":"10.1126\/science.aaa5417","article-title":"Decoding motor imagery from the posterior parietal cortex of a tetraplegic human","volume":"348","author":"Aflalo","year":"2015","journal-title":"Science"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1922649.1922653","article-title":"Human activity analysis","volume":"43","author":"Aggarwal","year":"2011","journal-title":"ACM Comput. Surv."},{"key":"ref4","first-page":"134","article-title":"Challenges in sensor-based human activity recognition and a comparative analysis of benchmark datasets: a review","author":"Antar","year":"2019"},{"key":"ref5","volume-title":"Leichtbauroboter in der manuellen Montage\u2013einfach einfach anfangen","author":"Bauer","year":"2016"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"1546","DOI":"10.1167\/iovs.02-0468","article-title":"Latency of the pupil light reflex: sample rate, stimulus intensity, and variation in normal subjects","volume":"44","author":"Bergamin","year":"2003","journal-title":"Invest. Ophthalmol. Vis. Sci."},{"key":"ref7","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1038\/35086023","article-title":"From the perception of action to the understanding of intention","volume":"2","author":"Blakemore","year":"2001","journal-title":"Nat. Rev. Neurosci."},{"key":"ref9001","first-page":"336","article-title":"Alpha power is influenced by performance errors","author":"Carp","year":"2009"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"208","DOI":"10.3389\/fnins.2014.00208","article-title":"Errare machinale Est: the use of error-related potentials in brain-machine interfaces","volume":"8","author":"Chavarriaga","year":"2014","journal-title":"Front. Neurosci."},{"key":"ref9","doi-asserted-by":"publisher","first-page":"345","DOI":"10.3390\/automation4040020","article-title":"Implementation of digital twin and real production system to address actual and future challenges in assembly technology","volume":"4","author":"Christ","year":"2023","journal-title":"Automation"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1016\/j.rcim.2010.07.005","article-title":"Safe human\u2013robot interaction based on dynamic sphere-swept line bounding volumes","volume":"27","author":"Corrales","year":"2011","journal-title":"Robot. Comput. Integr. Manuf."},{"key":"ref11","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1109\/TNSRE.2017.2755018","article-title":"Double ErrP detection for automatic error correction in an ERP-based BMI speller","volume":"26","author":"Cruz","year":"2018","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref12","doi-asserted-by":"publisher","first-page":"806517","DOI":"10.3389\/fnhum.2022.806517","article-title":"Error-related potentials in reinforcement learning-based brain-machine interfaces","volume":"16","author":"Fid\u00eancio","year":"2022","journal-title":"Front. Hum. Neurosci."},{"key":"ref13","doi-asserted-by":"publisher","first-page":"1309","DOI":"10.1007\/s10846-020-01309-7","article-title":"Robot speed adaption in multiple trajectory planning and integration in a simulation tool for human-robot interaction","volume":"102","author":"Glogowski","year":"2021","journal-title":"J Intell Robot Syst"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1515\/RNS.2011.047","article-title":"Cognition in action: imaging brain\/body dynamics in mobile humans","volume":"22","author":"Gramann","year":"2011","journal-title":"Rev. Neurosci."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"4755","DOI":"10.1007\/s10462-021-10116-x","article-title":"Human activity recognition in artificial intelligence framework: a narrative review","volume":"55","author":"Gupta","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"ref16","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1111\/j.1469-8986.2005.00270.x","article-title":"On the ERN and the significance of errors","volume":"42","author":"Hajcak","year":"2005","journal-title":"Psychophysiology"},{"key":"ref17","first-page":"6282","article-title":"Human-robot trust assessment using motion tracking & galvanic skin response","author":"Hald","year":"2020"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1016\/j.procir.2018.03.043","article-title":"Review of vision-based safety systems for human-robot collaboration","volume":"72","author":"Halme","year":"2018","journal-title":"Proc. CIRP"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1038\/nature04970","article-title":"Neuronal ensemble control of prosthetic devices by a human with tetraplegia","volume":"442","author":"Hochberg","year":"2006","journal-title":"Nature"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"7452","DOI":"10.3390\/s23177452","article-title":"An explainable EEG-based human activity recognition model using machine-learning approach and LIME","volume":"23","author":"Hussain","year":"2023","journal-title":"Sensors"},{"key":"ref21","volume-title":"Demistyfying Collaborative Industrial Robots. International Federation of Robotics, Frankfurt, Germany","year":"2020"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"503","DOI":"10.1016\/j.bbe.2021.03.009","article-title":"Artifact removal from EEG signals recorded in non-restricted environment","volume":"41","author":"Jamil","year":"2021","journal-title":"Biocybernet. Biomed. Eng."},{"key":"ref23","first-page":"200901","article-title":"Vision-based human action recognition: an overview and real world challenges","volume":"32","author":"Jegham","year":"2020","journal-title":"For. Sci. Int. Digit. Investig."},{"key":"ref24","doi-asserted-by":"crossref","DOI":"10.1109\/IWW-BCI.2017.7858156","article-title":"Single-trial analysis of readiness potentials for lower limb exoskeleton control","author":"Jeong","year":"2017"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1016\/j.neuroimage.2018.09.012","article-title":"Systematic comparison between a wireless EEG system with dry electrodes and a wired EEG system with wet electrodes","volume":"184","author":"Kam","year":"2019","journal-title":"NeuroImage"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"767","DOI":"10.1016\/j.neuroimage.2007.12.018","article-title":"Motor-cortical beta oscillations are modulated by correctness of observed action","volume":"40","author":"Koelewijn","year":"2008","journal-title":"NeuroImage"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"1366","DOI":"10.1007\/s11263-022-01594-9","article-title":"Human action recognition and prediction: a survey","volume":"130","author":"Kong","year":"2022","journal-title":"Int. J. Comput. Vis."},{"key":"ref9002","first-page":"289","article-title":"Aging and error processing: Time-frequency analysis of error-related potentials","volume-title":"Journal of Psychophysiology","author":"Kolev","year":"2005"},{"key":"ref28","first-page":"433","article-title":"Meyendtris: a hands-free, multimodal tetris clone using eye tracking and passive BCI for intuitive neuroadaptive gaming","author":"Krol","year":"2017"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"14","DOI":"10.3389\/fnhum.2018.00014","article-title":"EEG-based brain\u2013computer interfaces for communication and rehabilitation of people with motor impairment: a novel approach of the 21st century","volume":"12","author":"Lazarou","year":"2018","journal-title":"Front. Hum. Neurosci."},{"key":"ref30","doi-asserted-by":"crossref","first-page":"99","DOI":"10.2478\/s13380-014-0212-z","article-title":"Brain-machine interfaces: an overview","volume":"5","author":"Lebedev","year":"2014","journal-title":"Transl. Neurosci."},{"key":"ref31","doi-asserted-by":"publisher","first-page":"025011","DOI":"10.1088\/1741-2560\/8\/2\/025011","article-title":"A hybrid brain\u2013computer interface based on the fusion of electroencephalographic and electromyographic activities","volume":"8","author":"Leeb","year":"2011","journal-title":"J. Neural Eng."},{"key":"ref32","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-662-56714-2_1","article-title":"Task-based potential analysis for human-robot collaboration within assembly systems","volume-title":"Tagungsband des 3. Kongresses Montage Handhabung Industrieroboter","author":"Linsinger","year":"2018"},{"key":"ref33","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1016\/j.biopsycho.2018.12.013","article-title":"Error-related pupil dilation is sensitive to the evaluation of different error types","volume":"141","author":"Maier","year":"2019","journal-title":"Biol. Psychol."},{"key":"ref34","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1016\/j.cobme.2017.11.004","article-title":"EEG-based brain-computer interfaces","volume":"4","author":"McFarland","year":"2017","journal-title":"Curr. Opin. Biomed. Eng."},{"key":"ref35","doi-asserted-by":"publisher","first-page":"107561","DOI":"10.1016\/j.patcog.2020.107561","article-title":"Sensor-based and vision-based human activity recognition: a comprehensive survey","volume":"108","author":"Minh Dang","year":"2020","journal-title":"Pattern Recogn."},{"key":"ref36","article-title":"Simulation technology and application of safe collaborative operations in human-robot interaction","author":"Miro","year":"2022"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1101\/402149","article-title":"Frontal and occipito-temporal Theta activity as marker of error monitoring in human-avatar joint performance","author":"Moreau","year":"2018","journal-title":"BioRxiv"},{"key":"ref38","doi-asserted-by":"crossref","first-page":"1817","DOI":"10.1007\/978-981-16-5540-1_52","article-title":"Brain-machine interfaces for neurorobotics","volume-title":"Handbook of Neuroengineering","author":"Ortiz","year":"2023"},{"key":"ref39","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1016\/j.brainresrev.2005.04.005","article-title":"The functional significance of mu rhythms: translating \u201cseeing\u201d and \u201chearing\u201d into \u201cdoing\u201d","volume":"50","author":"Pineda","year":"2005","journal-title":"Brain Res. Rev."},{"key":"ref40","doi-asserted-by":"publisher","first-page":"18172","DOI":"10.3390\/s141018172","article-title":"Evaluating classifiers to detect arm movement intention from EEG signals","volume":"14","author":"Planelles","year":"2014","journal-title":"Sensors"},{"key":"ref41","doi-asserted-by":"publisher","first-page":"1154480","DOI":"10.3389\/fnins.2023.1154480","article-title":"Brain-machine interface based on transfer-learning for detecting the appearance of obstacles during exoskeleton-assisted walking","volume":"17","author":"Quiles","year":"2023","journal-title":"Front. Neurosci."},{"key":"ref42","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1016\/j.inffus.2021.12.003","article-title":"Multimodal co-learning: challenges, applications with datasets, recent advances and future directions","volume":"81","author":"Rahate","year":"2022","journal-title":"Inform. Fusion"},{"key":"ref43","doi-asserted-by":"publisher","first-page":"25","DOI":"10.3389\/fnbot.2020.00025","article-title":"Current status, challenges, and possible solutions of EEG-based brain-computer Interface: a comprehensive review","volume":"14","author":"Rashid","year":"2020","journal-title":"Front. Neurorobot."},{"key":"ref44","doi-asserted-by":"crossref","DOI":"10.1109\/ICRA.2017.7989777","article-title":"Correcting robot mistakes in real time using EEG signals","author":"Salazar-Gomez","year":"2017"},{"key":"ref45","doi-asserted-by":"publisher","first-page":"102094","DOI":"10.1016\/j.bspc.2020.102094","article-title":"Human activity recognition using deep electroencephalography learning","volume":"62","author":"Salehzadeh","year":"2020","journal-title":"Biomed. Sign. Process. Control"},{"key":"ref46","doi-asserted-by":"publisher","first-page":"558","DOI":"10.1016\/j.tics.2021.04.001","article-title":"What is the readiness potential?","volume":"25","author":"Schurger","year":"2021","journal-title":"Trends Cogn. Sci."},{"key":"ref48","first-page":"1","article-title":"Prediction error negativity in physical human-robot collaboration","author":"Singh","year":"2020"},{"key":"ref49","doi-asserted-by":"publisher","first-page":"155","DOI":"10.3389\/fnhum.2015.00155","article-title":"Error-related potentials during continuous feedback: using EEG to detect errors of different type and severity","volume":"9","author":"Sp\u00fcler","year":"2015","journal-title":"Front. Hum. Neurosci."},{"key":"ref51","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1109\/MRA.2019.2940413","article-title":"Planetary exploration with robot teams: implementing higher autonomy with swarm intelligence","volume":"27","author":"St-Onge","year":"2019","journal-title":"IEEE Robot. Automat. Magaz."},{"key":"ref9004","doi-asserted-by":"crossref","DOI":"10.1016\/S0301-0511(03)00097-8","article-title":"Error negativity on correct trials: a reexamination of available data","author":"Vidal","year":"2003"},{"key":"ref52","first-page":"647","article-title":"EEGlass: an EEG-eyeware prototype for ubiquitous brain-computer interaction","author":"Vourvopoulos","year":"2019"},{"key":"ref53","doi-asserted-by":"publisher","first-page":"903","DOI":"10.17973\/Mmsj.2016_06_201611","article-title":"Human\u2013robot collaboration in industry","volume":"2016","author":"Vysocky","year":"2016","journal-title":"MM SJ"},{"key":"ref54","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1109\/TCDS.2022.3151860","article-title":"Implicit robot control using error-related potential-based brain\u2013computer interface","volume":"15","author":"Wang","year":"2022","journal-title":"IEEE Trans. Cogn. Dev. Syst."},{"key":"ref55","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1708.01465","article-title":"Brain responses during robot-error observation","author":"Welke","year":"2017","journal-title":"arXiv"},{"key":"ref9003","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1093\/brain\/awh042","article-title":"Sensorimotor slowing with ageing is mediated by a functional dysregulation of motor\u2010generation processes: evidence from high\u2010resolution event\u2010related potentials","volume":"127","author":"Yordanova","year":"2004","journal-title":"Brain"},{"key":"ref56","doi-asserted-by":"publisher","first-page":"1425","DOI":"10.1109\/TBME.2014.2312397","article-title":"Brain-computer interfaces using sensorimotor rhythms: current state and future perspectives","volume":"61","author":"Yuan","year":"2014","journal-title":"I.E.E.E. Trans. Biomed. Eng."}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2024.1383089\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,24]],"date-time":"2024-09-24T05:10:30Z","timestamp":1727154630000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2024.1383089\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,24]]},"references-count":57,"alternative-id":["10.3389\/fnbot.2024.1383089"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2024.1383089","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,24]]},"article-number":"1383089"}}