{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T10:55:01Z","timestamp":1781780101850,"version":"3.54.5"},"reference-count":47,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T00:00:00Z","timestamp":1781481600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Proactive robotic assistance in human\u2013robot collaboration (HRC) requires systems that can perceive evolving task contexts, anticipate user needs, and intervene appropriately without disrupting human workflow. We present the Agentic Unified Robotic Assistance (AURA) Framework, which couples Large Language Model (LLM) reasoning grounded by Standard Operating Procedures (SOPs) with a modular layer of specialized Intent, Motion, Perception, Sound, Affordance, and Performance Monitors that supply structured context to a central decision-making module, making the framework reconfigurable and auditable without retraining or re-prompting. We introduce a human-in-the-loop teleoperation data collection methodology and an offline evaluation scheme with an Appropriateness Score (A-Score) tailored to proactive intervention timing, and release a benchmark dataset of annotated multimodal HRC episodes containing workspace and robot wrist camera videos, robot joint states, and labeled intervention events. Across three tasks of varying complexity, we observe progressive gains in intent prediction and decision-making as the modules are supplied with richer grounded context (prior-state memory and tracked object locations), with Combined F1 rising by over 20 points between context-poor and context-rich conditions. The structured grounding allows lightweight multimodal backbones such as Gemini 3.1 Flash Lite to perform on par with heavier reasoning-tier models at roughly one-fifth the inference latency. Together, these contributions establish a scalable framework, benchmark, and evaluation methodology for advancing proactive robotic assistance in collaborative environments.<\/jats:p>","DOI":"10.3390\/s26123810","type":"journal-article","created":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T15:00:48Z","timestamp":1781535648000},"page":"3810","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Modular Framework for Responsive and Explainable Robotic Assistance with Intention Prediction Using Human-Centric Digital Twins"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0640-8026","authenticated-orcid":false,"given":"Usman","family":"Asad","sequence":"first","affiliation":[{"name":"Department of Mechanical Engineering, Capital University of Science and Technology, Islamabad 45750, Pakistan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5270-6599","authenticated-orcid":false,"given":"Azfar","family":"Khalid","sequence":"additional","affiliation":[{"name":"Digital Innovation Research Group, Department of Engineering, School of Science & Technology, Nottingham Trent University, Nottingham NG11 8NS, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2989-1559","authenticated-orcid":false,"given":"Waqas Akbar","family":"Lughmani","sequence":"additional","affiliation":[{"name":"Department of Engineering, Birmingham City University, Birmingham B4 7XG, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4381-7435","authenticated-orcid":false,"given":"Shummaila","family":"Rasheed","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Capital University of Science and Technology, Islamabad 45750, Pakistan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7974-4472","authenticated-orcid":false,"given":"Muhammad Mahabat","family":"Khan","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Capital University of Science and Technology, Islamabad 45750, Pakistan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,6,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1016\/j.procir.2020.08.005","article-title":"On modelling and handling of flexible materials: A review on digital twins and planning systems","volume":"97","author":"Andronas","year":"2021","journal-title":"Procedia CIRP"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"5238","DOI":"10.1080\/00207543.2020.1714091","article-title":"Digital twin of composite assembly manufacturing process","volume":"58","author":"Polini","year":"2020","journal-title":"Int. J. Prod. Res."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Asad, U., Khan, M., Khalid, A., and Lughmani, W.A. (2023). Human-centric digital twins in industry: A comprehensive review of enabling technologies and implementation strategies. Sensors, 23.","DOI":"10.3390\/s23083938"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/j.jmsy.2023.07.013","article-title":"Deploying OWL ontologies for semantic mediation of mixed-reality interactions for human\u2013robot collaborative assembly","volume":"70","author":"David","year":"2023","journal-title":"J. Manuf. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"eaay7120","DOI":"10.1126\/scirobotics.aay7120","article-title":"XAI\u2014Explainable artificial intelligence","volume":"4","author":"Gunning","year":"2019","journal-title":"Sci. Robot."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Asad, U., Khalid, A., Lughmani, W.A., Rasheed, S., and Khan, M.M. (2025). A Human Intention and Motion Prediction Framework for Applications in Human-Centric Digital Twins. Biomimetics, 10.","DOI":"10.3390\/biomimetics10100656"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Umbrico, A., Orlandini, A., Cesta, A., Faroni, M., Beschi, M., Pedrocchi, N., Scala, A., Tavormina, P., Koukas, S., and Zalonis, A. (2022). Design of Advanced Human\u2013Robot Collaborative Cells for Personalized Human\u2013Robot Collaborations. Appl. Sci., 12.","DOI":"10.3390\/app12146839"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1109\/TRO.2007.907483","article-title":"Cost-Based Anticipatory Action Selection for Human\u2013Robot Fluency","volume":"23","author":"Hoffman","year":"2007","journal-title":"IEEE Trans. Robot."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1016\/j.cirp.2023.05.003","article-title":"Automated assembly of non-rigid objects","volume":"72","author":"Makris","year":"2023","journal-title":"CIRP Ann."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1901","DOI":"10.3390\/smartcities6040088","article-title":"Towards the cognitive factory in industry 5.0: From concept to implementation","volume":"6","author":"Cotta","year":"2023","journal-title":"Smart Cities"},{"key":"ref_12","first-page":"2220","article-title":"MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking","volume":"Volume 270","author":"Agrawal","year":"2025","journal-title":"Proceedings of the 8th Conference on Robot Learning"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"102727","DOI":"10.1016\/j.rcim.2024.102727","article-title":"Reactive human-robot collaborative manipulation of deformable linear objects using a new topological latent control model","volume":"88","author":"Zhou","year":"2024","journal-title":"Robot. Comput.-Integr. Manuf."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"104725","DOI":"10.1016\/j.robot.2024.104725","article-title":"Enhancing human-robot collaborative transportation through obstacle-aware vibrotactile warning and virtual fixtures","volume":"178","author":"Sirintuna","year":"2024","journal-title":"Robot. Auton. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1016\/j.ifacol.2020.11.034","article-title":"Development of an augmented reality equipped composites bonded assembly and repair for aerospace applications","volume":"53","author":"Nezhad","year":"2020","journal-title":"IFAC-PapersOnLine"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"De Lazzari, D., Terreran, M., Giacomuzzo, G., Jain, S., Falco, P., Carli, R., and Romeres, D. (2025). PACE: Proactive Assistance in Human-Robot Collaboration Through Action-Completion Estimation. Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA55743.2025.11127399"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1141","DOI":"10.1007\/s10845-023-02294-y","article-title":"Embodied intelligence in manufacturing: Leveraging large language models for autonomous industrial robotics","volume":"36","author":"Fan","year":"2024","journal-title":"J. Intell. Manuf."},{"key":"ref_18","unstructured":"Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. (2022). Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. Proceedings of the International Conference on Machine Learning, PMLR."},{"key":"ref_19","unstructured":"Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., and Julian, R. (2023). Do as i can, not as i say: Grounding language in robotic affordances. Proceedings of the Conference on Robot Learning, PMLR."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"999","DOI":"10.1007\/s10514-023-10135-3","article-title":"Progprompt: Program generation for situated robot task planning using large language models","volume":"47","author":"Singh","year":"2023","journal-title":"Auton. Robot."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Gao, J., Sarkar, B., Xia, F., Xiao, T., Wu, J., Ichter, B., Majumdar, A., and Sadigh, D. (2024). Physically Grounded Vision-Language Models for Robotic Manipulation. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA57147.2024.10610090"},{"key":"ref_22","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_23","first-page":"2165","article-title":"RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control","volume":"Volume 229","author":"Tan","year":"2023","journal-title":"Proceedings of the 7th Conference on Robot Learning"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1684","DOI":"10.1177\/02783649241273668","article-title":"Diffusion policy: Visuomotor policy learning via action diffusion","volume":"44","author":"Chi","year":"2025","journal-title":"Int. J. Robot. Res."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., and Ichter, B. (2024). \u03c00: A Vision-Language-Action Flow Model for General Robot Control. arXiv.","DOI":"10.15607\/RSS.2025.XXI.010"},{"key":"ref_26","first-page":"2679","article-title":"OpenVLA: An Open-Source Vision-Language-Action Model","volume":"Volume 270","author":"Agrawal","year":"2025","journal-title":"Proceedings of the 8th Conference on Robot Learning"},{"key":"ref_27","unstructured":"Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., and Mart\u00edn-Mart\u00edn, R. (2022). What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. Proceedings of the Conference on Robot Learning, PMLR."},{"key":"ref_28","unstructured":"Huang, Z., Pohovey, J., Yammanuru, A., and Driggs-Campbell, K. (2024). LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration\u2013A Robot Sous-Chef Application. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Sun, N., Mao, B., Li, Y., Guo, D., and Liu, H. (2025). AssistantX: An LLM-Powered Proactive Assistant in Collaborative Human-Populated Environments. Proceedings of the 2025 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.","DOI":"10.1109\/IROS60139.2025.11246901"},{"key":"ref_30","unstructured":"Zhang, Y., Bai, S., Zhou, W., Zhang, H., Zhang, Q., Luan, Z., and Chen, B. (2026). Assistance Without Interruption: A Benchmark and LLM-based Framework for Non-Intrusive Human-Robot Assistance. arXiv."},{"key":"ref_31","unstructured":"Wang, S., Fu, J., Liu, F., He, X., Wu, H., Shi, J., Huang, K., Fei, Z., Gong, J., and Wu, Z. (2025). RoboOmni: Proactive Robot Manipulation in Omni-modal Context. arXiv."},{"key":"ref_32","unstructured":"Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Coll-Vinent, D.S., Ryali, C., Alwala, K.V., Khedr, H., and Huang, A. (2026, January 23\u201327). SAM 3: Segment Anything with Concepts. Proceedings of the Fourteenth International Conference on Learning Representations, Rio de Janeiro, Brazil."},{"key":"ref_33","first-page":"1","article-title":"Face-Pedestrian Joint Feature Modeling with Cross-Category Dynamic Matching for Occlusion-Robust Multi-Object Tracking","volume":"86","year":"2026","journal-title":"Comput. Mater. Contin."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Lee, T., Wen, B., Kang, M., Kang, G., Kweon, I.S., and Yoon, K.J. (2025). Any6D: Model-free 6D Pose Estimation of Novel Objects. Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), IEEE.","DOI":"10.1109\/CVPR52734.2025.01086"},{"key":"ref_35","unstructured":"Lin, H., Chen, S., Liew, J.H., Chen, D.Y., Li, Z., Zhao, Y., Peng, S., Guo, H., Zhou, X., and Shi, G. (2026, January 23\u201327). Depth Anything 3: Recovering the Visual Space from Any Views. Proceedings of the Fourteenth International Conference on Learning Representations, Rio de Janeiro, Brazil."},{"key":"ref_36","unstructured":"Chen, X., Chu, F.J., Gleize, P., Liang, K.J., Sax, A., Tang, H., Wang, W., Guo, M., Hardin, T., and Li, X. (2026). Sam 3d: 3dfy anything in images. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, IEEE."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3414685.3417861","article-title":"Modular Primitives for High-Performance Differentiable Rendering","volume":"39","author":"Laine","year":"2020","journal-title":"ACM Trans. Graph."},{"key":"ref_38","unstructured":"Yang, T., He, S., Jing, H., Yang, J., Liu, Z., Zou, C., and Wang, Y. (2026). Fast SAM 3D Body: Accelerating SAM 3D Body for Real-Time Full-Body Human Mesh Recovery. arXiv."},{"key":"ref_39","first-page":"1769","article-title":"Inner Monologue: Embodied Reasoning through Planning with Language Models","volume":"Volume 205","author":"Liu","year":"2023","journal-title":"Proceedings of the 6th Conference on Robot Learning"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"3964","DOI":"10.1109\/LRA.2023.3264766","article-title":"Robotic manipulation of deformable rope-like objects using differentiable compliant position-based dynamics","volume":"8","author":"Liu","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. (2023). Code as Policies: Language Model Programs for Embodied Control. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA48891.2023.10160591"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M.K., Chen, L.Y., and Ellis, K. (2024). DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. arXiv.","DOI":"10.15607\/RSS.2024.XX.120"},{"key":"ref_43","unstructured":"Cadene, R., Alibert, S., Soare, A., Gallouedec, Q., Zouitine, A., Palma, S., Kooijmans, P., Aractingi, M., Shukor, M., and Aubakirova, D. (2026, June 12). LeRobot: State-of-the-Art Machine Learning for Real-World Robotics in Pytorch. Available online: https:\/\/github.com\/huggingface\/lerobot."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"030903","DOI":"10.1115\/1.4056295","article-title":"Physics informed synthetic image generation for deep learning-based detection of wrinkles and folds","volume":"23","author":"Manyar","year":"2023","journal-title":"J. Comput. Inf. Sci. Eng."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., and Boote, B. (2024). Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives. Proceedings of the 2024 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE.","DOI":"10.1109\/CVPR52733.2024.01834"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Bo, V., Garrell, A., and Sanfeliu, A. (2026). Fast or Accurate? How Intention-Recognition Models Shape Human Perception of a Mobile Robot. Proceedings of the Companion Proceedings of the 21st ACM\/IEEE International Conference on Human-Robot Interaction, IEEE.","DOI":"10.1145\/3776734.3794445"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"9877","DOI":"10.1609\/aaai.v36i9.21224","article-title":"Solving disjunctive temporal networks with uncertainty under restricted time-based controllability using tree search and graph neural networks","volume":"Volume 36","author":"Osanlou","year":"2022","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/12\/3810\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T10:28:56Z","timestamp":1781778536000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/12\/3810"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,15]]},"references-count":47,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2026,6]]}},"alternative-id":["s26123810"],"URL":"https:\/\/doi.org\/10.3390\/s26123810","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,15]]}}}