{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T11:02:53Z","timestamp":1777633373810,"version":"3.51.4"},"reference-count":53,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T00:00:00Z","timestamp":1777420800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Mobile manipulators show incredible promise as domestic service robots \u2013 interacting with a wide variety of objects using increasingly affordable hardware. But although perception, manipulation, and mobility have advanced, there remain fundamental challenges in making robots more useful. How can a robot proactively identify tasks that it can complete while supporting individual human preferences for how a home should be configured? We propose using foundation models to first detect what has changed and then select appropriate tasks for the service robot. Change affords action. Only those objects that have been interacted with need to be considered for tasking. Other objects, even if located in non-standard positions in the house, can be ignored. Open-vocabulary based object detection and neural radiance field models are used to identify changes corresponding to fixed phrases. Large language models then validate which tasks should be completed by the robot. Experiments are conducted on data collected by both mobile phone and Stretch 2 Mobile Manipulator, demonstrating general applicability to a wide range of applications in the home.<\/jats:p>","DOI":"10.3389\/frobt.2026.1772005","type":"journal-article","created":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T04:35:15Z","timestamp":1777437315000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Integrating foundation models with change detection to identify tasking for service robots"],"prefix":"10.3389","volume":"13","author":[{"given":"Eric","family":"Martinson","sequence":"first","affiliation":[{"name":"College of Arts and Sciences, Lawrence Technological University","place":["Southfield, MI, United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Igri","family":"Fishta","sequence":"additional","affiliation":[{"name":"College of Arts and Sciences, Lawrence Technological University","place":["Southfield, MI, United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Devson","family":"Butani","sequence":"additional","affiliation":[{"name":"College of Arts and Sciences, Lawrence Technological University","place":["Southfield, MI, United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2026,4,29]]},"reference":[{"key":"B1","volume-title":"Module 1: detecting change in a resident's condition","year":"2014"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1301","DOI":"10.1007\/s10514-018-9734-5","article-title":"Street-view change detection with deconvolutional networks","volume":"42","author":"Alcantarilla","year":"2018","journal-title":"Aut. Robots"},{"key":"B3","volume-title":"Dementia and hiding, hoarding or losing things","year":"2017"},{"key":"B4","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1016\/j.jbi.2018.03.009","article-title":"Automatic assessment of functional health decline in older adults based on smart home data","volume":"81","author":"Aramendi","year":"2018","journal-title":"J. Biomed. Inf."},{"key":"B5","volume-title":"SAM3: segment anything with concepts","author":"Carion","year":"2025"},{"key":"B6","volume-title":"A density-based algorithm for discovering clusters in large spatial databases with noise","author":"Ester","year":"1996"},{"key":"B7","volume-title":"Amcl","author":"Gerkey","year":"2020"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.3390\/s22176342","article-title":"Object-based change detection algorithm with a spatial AI Stereo Camera","volume":"22","author":"G\u00f6ncz","year":"2022","journal-title":"Sensors"},{"key":"B9","doi-asserted-by":"publisher","first-page":"59","DOI":"10.14358\/pers.69.1.59","article-title":"Block adjustment of high-resolution satellite images described by rational polynomials","volume":"69","author":"Grodecki","year":"2003","journal-title":"Photogrammetric Eng. and Remote Sens."},{"key":"B10","volume-title":"Open-vocabulary object detection via vision and Language knowledge distillation","author":"Gu","year":"2022"},{"key":"B11","article-title":"\u201cScaling up visual and vision-language representation learning,\u201d in","volume-title":"Int conf on machine learning","author":"Jia","year":"2021"},{"key":"B12","doi-asserted-by":"publisher","first-page":"1232","DOI":"10.1080\/01691864.2024.2408593","article-title":"Real-world robot applications of foundation models: a review","volume":"38","author":"Kawaharazuka","year":"2024","journal-title":"Adv. Robot."},{"key":"B13","doi-asserted-by":"crossref","DOI":"10.1145\/3592433","article-title":"3D gaussian splatting for real-time radiance field rendering","volume":"42","author":"Kerbl","year":"2023","journal-title":"ACM Trans. Graph."},{"key":"B14","doi-asserted-by":"crossref","DOI":"10.1109\/ICCV51070.2023.01807","article-title":"\u201cLERF: Language Embedded Radiance Fields,\u201d in","volume-title":"Int conf on computer vision","author":"Kerr","year":"2023"},{"key":"B15","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52733.2024.00915","article-title":"\"Lisa: Reasoning Segmentation via Large Language Model,\" in","volume-title":"Computer vision and pattern recognition","author":"Lai","year":"2024"},{"key":"B16","doi-asserted-by":"publisher","first-page":"112308","DOI":"10.1016\/j.rse.2021.112308","article-title":"Change detection using deep learning approach with object-based image analysis","volume":"256","author":"Liu","year":"2021","journal-title":"Remote Sens. Environ."},{"key":"B17","doi-asserted-by":"publisher","first-page":"2365","DOI":"10.1080\/0143116031000139863","article-title":"Change detection techniques","volume":"25","author":"Lu","year":"2004","journal-title":"Int J. Remote Sens."},{"key":"B18","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52688.2022.00695","volume-title":"Image segmentation using text and image prompts","author":"L\u00fcddecke","year":"2022"},{"key":"B19","doi-asserted-by":"publisher","first-page":"6101","DOI":"10.1109\/tits.2021.3077883","article-title":"An empirical review of deep learning frameworks for change detection: model design, experimental frameworks, challenges and research needs","volume":"23","author":"Mandal","year":"2022","journal-title":"IEEE Trans Intelligent Transp. Syst."},{"key":"B20","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1186\/s12889-020-08587-8","article-title":"The relationship between household chaos and child, parent, and family outcomes: a systematic scoping review","volume":"20","author":"Marsh","year":"2020","journal-title":"BMC Public Health"},{"key":"B21","doi-asserted-by":"crossref","DOI":"10.1109\/UR61395.2024.10597526","volume-title":"Interactive, privacy-aware semantic mapping for homes","author":"Martinson","year":"2024"},{"key":"B22","doi-asserted-by":"crossref","DOI":"10.1109\/CASE58245.2025.11164053","volume-title":"Augmenting open vocabulary object detection with large Language models for home service robots","author":"Martinson","year":"2025"},{"key":"B23","doi-asserted-by":"crossref","DOI":"10.1109\/UR61395.2024.10597504","volume-title":"Meaningful change detection in indoor environments using CLIP models and NeRF-based image synthesis","author":"Martinson","year":"2024"},{"key":"B24","doi-asserted-by":"crossref","DOI":"10.1109\/ROMAN.2016.7745252","volume-title":"Personalizing object handover with an electronic health record","author":"Martinson","year":"2016"},{"key":"B25","doi-asserted-by":"publisher","first-page":"1635","DOI":"10.1017\/S104161021800220X","article-title":"Characterizing the symptom of misplacing objects in people with dementia: findings from an online tracking tool","volume":"31","author":"McGarrigle","year":"2019","journal-title":"Int. Psychogeriatrics"},{"key":"B26","volume-title":"The Llama 4 herd: the beginning of a new era of natively multimodal AI innovation","year":"2025"},{"key":"B27","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1145\/3503250","article-title":"Nerf: representing scenes as neural radiance fields","volume":"65","author":"Mildenhall","year":"2021","journal-title":"Commun. ACM"},{"key":"B28","article-title":"\u201cSimple open-vocabulary object detection with vision transformers,\u201d in","volume-title":"European conf on computer vision","author":"Minderer","year":"2022"},{"key":"B29","first-page":"3228","article-title":"\u201cSemantic Scene Difference Detection in Daily Life Patroling by Mobile Robots using Pre-Trained Large-Scale Vision-Language Model,\u201d in","volume-title":"Intelligent robotics and systems","author":"Obinata","year":"2023"},{"key":"B30","volume-title":"The link between messy rooms and depression","author":"Pattemore","year":"2022"},{"key":"B31","first-page":"26","volume-title":"Kosmos-2: grounding multimodal large Language models to the world","author":"Peng","year":"2022"},{"key":"B32","doi-asserted-by":"crossref","DOI":"10.1109\/ICCV.2019.00937","volume-title":"Deep hough voting for 3D object detection in point clouds","author":"Qi","year":"2019"},{"key":"B33","article-title":"\u201cLearning transferable visual models from natural language supervision,\u201d in","volume-title":"Int. conference on machine learning","author":"Radford","year":"2021"},{"key":"B34","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52733.2024.01233","article-title":"\u201cLearning to Localize Objects Improves Spatial Reasoning in Visual-LLMs,\u201d in","volume-title":"Computer vision and pattern recognition (CVPR)","author":"Ranasinghe","year":"2024"},{"key":"B35","doi-asserted-by":"publisher","first-page":"9913","DOI":"10.1109\/lra.2022.3191215","article-title":"Combining navigation and manipulation costs for time-efficient robot placement in Mobile manipulation tasks","volume":"7","author":"Reister","year":"2022","journal-title":"IEEE Robotics Automation Lett."},{"key":"B36","doi-asserted-by":"crossref","DOI":"10.5244\/C.29.61","article-title":"Change detection from a street image pair using CNN features and superpixel segmentation","author":"Sakurada","year":"2015"},{"key":"B37","volume-title":"Colmap FAQ","author":"Schoenberger","year":"2016"},{"key":"B38","volume-title":"LAION-5B: an open large-scale dataset for training next generation image-text models","author":"Schuhmann","year":"2022"},{"key":"B39","doi-asserted-by":"crossref","DOI":"10.1109\/WACV51458.2022.00036","article-title":"\u201cHybVIO: pushing the limits of real-time visual-inertial odometry,\u201d in","volume-title":"IEEE\/CVF winter conference on applications of computer vision","author":"Seiskari","year":"2022"},{"key":"B40","volume-title":"Gaussian Splatting and NeRFs","author":"Spectacular","year":"2025"},{"key":"B41","first-page":"1","article-title":"Nerfstudio: a modular framework for neural radiance field development","author":"Tancik","year":"2023"},{"key":"B53","volume-title":"YOLOv8","year":"2023"},{"key":"B42","volume-title":"ChangeNet: a deep learning Architecture for visual","author":"Varghese","year":"2018"},{"key":"B43","volume-title":"SegLLM: Multi-Round reasoning segmentation","author":"Wang","year":"2024"},{"key":"B44","doi-asserted-by":"publisher","first-page":"109384","DOI":"10.1016\/j.patcog.2023.109384","article-title":"How to reduce change detection to semantic segmentation","volume":"138","author":"Wanga","year":"2023","journal-title":"Pattern Recognit."},{"key":"B45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3448671","article-title":"Early detection of health changes in the elderly using in-home multi-sensor data streams","volume":"2","author":"Wu","year":"2021","journal-title":"ACM Trans. Comput. Healthc."},{"key":"B46","doi-asserted-by":"publisher","first-page":"1087","DOI":"10.1007\/s10514-023-10139-z","article-title":"TidyBot: personalized robot assistance with large Language models","volume":"47","author":"Wu","year":"","journal-title":"Aut. Robots"},{"key":"B47","article-title":"\u201cMultimodal large language models: a survey,\u201d in","author":"Wu","year":"","journal-title":"Int conf on big data"},{"key":"B48","volume-title":"Lisa++: an improved baseline for reasoning segmentation with a large Language model","author":"Yang","year":"2023"},{"key":"B49","doi-asserted-by":"publisher","first-page":"825","DOI":"10.1007\/s11263-024-02214-4","article-title":"Contextual object detection with multimodal large language models","volume":"133","author":"Zang","year":"2025","journal-title":"Int J Comput. Vis"},{"key":"B50","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52688.2022.01759","article-title":"\u201cLiT: Zero-shot transfer with locked-image text tuning,\u201d in","volume-title":"Computer vision and pattern recognition","author":"Zhai","year":"2022"},{"key":"B51","first-page":"36067","article-title":"\u201cGlipv2: unifying localization and vision-language understanding,\u201d in","volume-title":"Advances in neural information processing systems (NeurIPS)","author":"Zhang","year":"2022"},{"key":"B52","volume-title":"VLPrompt: Vision-Language prompting for panoptic scene graph generation","author":"Zhou","year":"2023"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2026.1772005\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T04:36:38Z","timestamp":1777437398000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2026.1772005\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,29]]},"references-count":53,"alternative-id":["10.3389\/frobt.2026.1772005"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2026.1772005","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,29]]},"article-number":"1772005"}}