{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T15:12:27Z","timestamp":1793286747531,"version":"4.5.0"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"8","license":[{"start":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T00:00:00Z","timestamp":1695427200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T00:00:00Z","timestamp":1695427200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100006919","name":"Massachusetts Institute of Technology","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100006919","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Auton Robot"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Mechanical search is a robotic problem where a robot needs to retrieve a target item that is partially or fully-occluded from its camera. State-of-the-art approaches for mechanical search either require an expensive search process to find the target item, or they require the item to be tagged with a radio frequency identification tag (e.g., RFID), making their approach beneficial only to tagged items in the environment. We present FuseBot, the first robotic system for RF-Visual mechanical search that enables efficient retrieval of both RF-tagged and untagged items in a pile. Rather than requiring all target items in a pile to be RF-tagged, FuseBot leverages the mere existence of an RF-tagged item in the pile to benefit both tagged and untagged items. Our design introduces two key innovations. The first is\n                    <jats:italic>RF-Visual Mapping<\/jats:italic>\n                    , a technique that identifies and locates RF-tagged items in a pile and uses this information to construct an RF-Visual occupancy distribution map. The second is\n                    <jats:italic>RF-Visual Extraction<\/jats:italic>\n                    , a policy formulated as an optimization problem that minimizes the number of actions required to extract the target object by accounting for the probabilistic occupancy distribution, the expected grasp quality, and the expected information gain from future actions. We built a real-time end-to-end prototype of our system on a UR5e robotic arm with in-hand vision and RF perception modules. We conducted over 200 real-world experimental trials to evaluate FuseBot and compare its performance to a state-of-the-art vision-based system named X-Ray (Danielczuk et al., in: 2020 IEEE\/RSJ international conference on intelligent robots and systems (IROS), IEEE, 2020). Our experimental results demonstrate that FuseBot outperforms X-Ray\u2019s efficiency by more than 40% in terms of the number of actions required for successful mechanical search. Furthermore, in comparison to X-Ray\u2019s success rate of 84%, FuseBot achieves a success rate of 95% in retrieving untagged items, demonstrating for the first time that the benefits of RF perception extend beyond tagged objects in the mechanical search problem.\n                  <\/jats:p>","DOI":"10.1007\/s10514-023-10137-1","type":"journal-article","created":{"date-parts":[[2023,9,22]],"date-time":"2023-09-22T23:21:25Z","timestamp":1695424885000},"page":"1137-1154","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["FuseBot: mechanical search of rigid and deformable objects via multi-modal perception"],"prefix":"10.1007","volume":"47","author":[{"given":"Tara","family":"Boroushaki","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura","family":"Dodds","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nazish","family":"Naeem","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fadel","family":"Adib","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,9,23]]},"reference":[{"key":"10137_CR1","doi-asserted-by":"crossref","unstructured":"Avigal, Y., Satish, V., Tam, Z., Huang, H., Zhang, H., Danielczuk, M., Ichnowski, J., & Goldberg, K. (2021). AVPLUG: Approach vector planning for unicontact grasping amid clutter. In 2021 IEEE 17th international conference on automation science and engineering (CASE) (pp. 1140\u20131147). IEEE.","DOI":"10.1109\/CASE49439.2021.9551652"},{"key":"10137_CR2","doi-asserted-by":"crossref","unstructured":"Aydemir, A., Sj\u00f6\u00f6, K., Folkesson, J., Pronobis, A., & Jensfelt, P. (2011). Search in the real world: Active visual object search based on spatial relations. In 2011 IEEE international conference on robotics and automation (pp. 2818\u20132824). IEEE.","DOI":"10.1109\/ICRA.2011.5980495"},{"issue":"8","key":"10137_CR3","doi-asserted-by":"publisher","first-page":"966","DOI":"10.1109\/5.5968","volume":"76","author":"R Bajcsy","year":"1988","unstructured":"Bajcsy, R. (1988). Active perception. Proceedings of the IEEE, 76(8), 966\u20131005.","journal-title":"Proceedings of the IEEE"},{"issue":"6","key":"10137_CR4","doi-asserted-by":"publisher","first-page":"1273","DOI":"10.1109\/TRO.2017.2721939","volume":"33","author":"J Bohg","year":"2017","unstructured":"Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., & Sukhatme, G. S. (2017). Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6), 1273\u20131291.","journal-title":"IEEE Transactions on Robotics"},{"key":"10137_CR5","doi-asserted-by":"crossref","unstructured":"Boroushaki, T., Leng, J., Clester, I., Rodriguez, A., & Adib, F. (2021a). Robotic grasping of fully-occluded objects using rf perception. In 2021 international conference on robotics and automation (ICRA). IEEE.","DOI":"10.1109\/ICRA48506.2021.9560956"},{"key":"10137_CR6","doi-asserted-by":"crossref","unstructured":"Boroushaki, T., Perper, I., Nachin, M., Rodriguez, A., & Adib, F. (2021b). RFusion: Robotic grasping via RF-visual sensing and learning. In Proceedings of the 19th ACM conference on embedded networked sensor systems. SenSys \u201921 (pp. 192\u2013205). Association for Computing Machinery.","DOI":"10.1145\/3485730.3485944"},{"key":"10137_CR7","first-page":"5621","volume":"33","author":"Y Chen","year":"2020","unstructured":"Chen, Y., Zhang, Z., Cao, Y., Wang, L., Lin, S., & Hu, H. (2020). RepPoints V2: Verification meets regression for object detection. Advances in Neural Information Processing Systems, 33, 5621\u20135631.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"10137_CR8","unstructured":"Curlander, J. C., & McDonough, R. N. (1991). Synthetic aperture radar (Vol. 11)."},{"key":"10137_CR9","doi-asserted-by":"crossref","unstructured":"Danielczuk, M., Angelova, A., Vanhoucke, V., & Goldberg, K. (2020). X-ray: Mechanical search for an occluded object by minimizing support of learned occupancy distributions. In 2020 IEEE\/RSJ international conference on intelligent robots and systems (IROS) (pp. 9577\u20139584). IEEE.","DOI":"10.1109\/IROS45743.2020.9340984"},{"key":"10137_CR10","doi-asserted-by":"crossref","unstructured":"Danielczuk, M., Angelova, A., Vanhoucke, V., & Goldberg, K. (2020). X-ray: Mechanical search for an occluded object by minimizing support of learned occupancy distributions. In 2020 IEEE\/RSJ international conference on intelligent robots and systems (IROS) (pp. 9577\u20139584). IEEE.","DOI":"10.1109\/IROS45743.2020.9340984"},{"key":"10137_CR11","unstructured":"Danielczuk, M., Angelova, A., Vanhoucke, V., & Goldberg, K. (2021). X-ray code. https:\/\/github.com\/BerkeleyAutomation\/xray"},{"key":"10137_CR12","doi-asserted-by":"crossref","unstructured":"Danielczuk, M., Kurenkov, A., Balakrishna, A., Matl, M., Wang, D., Mart\u00edn-Mart\u00edn, R., Garg, A., Savarese, S., & Goldberg, K. (2019). Mechanical search: Multi-step retrieval of a target object occluded by clutter. In 2019 international conference on robotics and automation (ICRA) (pp. 1614\u20131621). IEEE.","DOI":"10.1109\/ICRA.2019.8794143"},{"key":"10137_CR13","doi-asserted-by":"crossref","unstructured":"Danielczuk, M., Matl, M., Gupta, S., Li, A., Lee, A., Mahler, J., & Goldberg, K. (2019). Segmenting unknown 3D objects from real depth images using mask R-CNN trained on synthetic data. In 2019 international conference on robotics and automation (ICRA) (pp. 7283\u20137290). IEEE.","DOI":"10.1109\/ICRA.2019.8793744"},{"key":"10137_CR14","doi-asserted-by":"crossref","unstructured":"Dogar, M., Hsiao, K., Ciocarlie, M., & Srinivasa, S. (2012). Physics-based grasp planning through clutter. In Proceedings of robotics: Science and systems (RSS \u201912).","DOI":"10.15607\/RSS.2012.VIII.008"},{"key":"10137_CR15","unstructured":"EPC UHF Gen2 air interface protocol. http:\/\/www.gs1.org\/epcrfid\/epc-rfid-uhf-air-interface-protocol\/2-0-1"},{"key":"10137_CR16","unstructured":"Ettus Research, CDA-2990. https:\/\/moveit.ros.org\/"},{"key":"10137_CR17","doi-asserted-by":"crossref","unstructured":"Hornung, A., Wurm, K. M., Bennewitz, M., Stachniss, C., & Burgard, W. (2013). OctoMap: An efficient probabilistic 3D mapping framework based on octrees. Autonomous Robots.","DOI":"10.1007\/s10514-012-9321-0"},{"key":"10137_CR18","doi-asserted-by":"crossref","unstructured":"Huang, H., Dominguez-Kuhne, M., Satish, V., Danielczuk, M., Sanders, K., Ichnowski, J., Lee, A., Angelova, A., Vanhoucke, V., & Goldberg, K. (2020). Mechanical search on shelves using lateral access X-ray. In 2021 IEEE\/RSJ international conference on intelligent robots and systems (IROS) (pp. 2045\u20132052). IEEE.","DOI":"10.1109\/IROS51168.2021.9636629"},{"key":"10137_CR19","doi-asserted-by":"crossref","unstructured":"Huang, X., Walker, I., & Birchfield, S. (2012). Occlusion-aware reconstruction and manipulation of 3D articulated objects. In 2012 IEEE international conference on robotics and automation (pp. 1365\u20131371). IEEE.","DOI":"10.1109\/ICRA.2012.6224911"},{"key":"10137_CR20","unstructured":"Inlay, S. S. (2021). www.smartrac-group.com"},{"key":"10137_CR21","unstructured":"Intel RealSense. (2019). https:\/\/www.intelrealsense.com"},{"key":"10137_CR22","unstructured":"Kent Electronics. (2021). http:\/\/www.wa5vjb.com"},{"key":"10137_CR23","unstructured":"Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In Advances in neural information processing systems."},{"key":"10137_CR24","doi-asserted-by":"crossref","unstructured":"Liu, S., & Deng, W. (2015). Very deep convolutional neural network based image classification using small training sample size. In 2015 3rd IAPR Asian conference on pattern recognition (ACPR) (pp. 730\u2013734).","DOI":"10.1109\/ACPR.2015.7486599"},{"key":"10137_CR25","unstructured":"Luo, Z., Zhang, Q., Ma, Y., Singh, M., & Adib, F. (2019). 3D backscatter localization for fine-grained robotics. In 16th USENIX symposium on networked systems design and implementation (NSDI 19) (pp. 765\u2013782)."},{"key":"10137_CR26","doi-asserted-by":"crossref","unstructured":"Ma, Y., Selby, N., & Adib, F. (2017). Minding the billions: Ultra-wideband localization for deployed RFID tags. In Proceedings of the 23rd annual international conference on mobile computing and networking (MobiCom) (pp. 248\u2013260).","DOI":"10.1145\/3117811.3117833"},{"key":"10137_CR27","doi-asserted-by":"crossref","unstructured":"Morrison, D., Corke, P., & Leitner, J. (2018a). Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach. In Robotics: Science and systems XIV (RSS).","DOI":"10.15607\/RSS.2018.XIV.021"},{"key":"10137_CR28","unstructured":"Morrison, D., Corke, P., & Leitner, J. (2018b). GG-CNN code. https:\/\/github.com\/dougsm\/ggcnn"},{"key":"10137_CR29","unstructured":"Nuand. (2021). BladeRF 2.0 Micro. https:\/\/www.nuand.com\/bladerf-2-0-micro\/"},{"key":"10137_CR30","unstructured":"Optitrack. (2017). http:\/\/www.optitrack.com."},{"key":"10137_CR31","unstructured":"Price, A., Jin, L., & Berenson, D. (2019). Inferring occluded geometry improves performance when retrieving an object from dense clutter. arXiv:1907.08770"},{"key":"10137_CR32","unstructured":"Robotiq. (2019). https:\/\/robotiq.com\/products\/2f85-140-adaptive-robot-gripper"},{"key":"10137_CR33","doi-asserted-by":"crossref","unstructured":"Tse, D., & Viswanath, P. (2005). Fundamentals of wireless communication.","DOI":"10.1017\/CBO9780511807213"},{"key":"10137_CR34","unstructured":"Universal Robots ROS Driver. (2020). https:\/\/github.com\/UniversalRobots\/Universal_Robots_ROS_Driver"},{"key":"10137_CR35","unstructured":"Universal Robots. (2021). UR5e. https:\/\/www.universal-robots.com\/products\/ur5-robot\/"},{"key":"10137_CR36","doi-asserted-by":"crossref","unstructured":"Wang, J., & Katabi, D. (2013). Dude, where\u2019s my card? RFID positioning that works with multipath and non-line of sight. In Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM (pp. 51\u201362).","DOI":"10.1145\/2486001.2486029"},{"key":"10137_CR37","doi-asserted-by":"crossref","unstructured":"Wang, J., Adib, F., Knepper, R., Katabi, D., & Rus, D. (2013). RF-compass: Robot object manipulation using RFIDs. In Proceedings of the 19th annual international conference on mobile computing & networking (MobiCom) (pp. 3\u201314).","DOI":"10.1145\/2500423.2500451"}],"container-title":["Autonomous Robots"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10514-023-10137-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10514-023-10137-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10514-023-10137-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T13:12:53Z","timestamp":1701177173000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10514-023-10137-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,23]]},"references-count":37,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["10137"],"URL":"https:\/\/doi.org\/10.1007\/s10514-023-10137-1","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-2580051\/v1","asserted-by":"object"}]},"ISSN":["0929-5593","1573-7527"],"issn-type":[{"value":"0929-5593","type":"print"},{"value":"1573-7527","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,23]]},"assertion":[{"value":"13 February 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 September 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"F.A. is founder of Cartesian Systems. The remaining authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"The authors declare that this manuscript is an extended version of a conference paper presented at RSS 2022, titled \u201cFuseBot: RF-Visual Mechanical Search.\u201d This manuscript includes new techniques (Sect.\u00a0\n                      \n                      ) and additional real-world performance evaluations in Sect.\u00a0\n                      \n                      and Sect.\u00a0\n                      \n                      with 57 new real-world trials, which have not been previously published or submitted elsewhere. The authors affirm that the research reported in this manuscript is original, and that the data and findings presented are accurate and reliable. The authors have disclosed any potential conflicts of interest, financial or otherwise.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical statement"}}]}}