{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T08:35:57Z","timestamp":1781598957948,"version":"3.54.5"},"reference-count":61,"publisher":"SAGE Publications","issue":"1-3","license":[{"start":{"date-parts":[[2015,10,6]],"date-time":"2015-10-06T00:00:00Z","timestamp":1444089600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2016,1]]},"abstract":"<jats:p>Most approaches to robot localization rely on low-level geometric features such as points, lines, and planes. In this paper, we use object recognition to obtain semantic information from the robot\u2019s sensors and consider the task of localizing the robot within a prior map of landmarks, which are annotated with semantic labels. As object recognition algorithms miss detections and produce false alarms, correct data association between the detections and the landmarks on the map is central to the semantic localization problem. Instead of the traditional vector-based representation, we propose a sensor model, which encodes the semantic observations via random finite sets and enables a unified treatment of missed detections, false alarms, and data association. Our second contribution is to reduce the problem of computing the likelihood of a set-valued observation to the problem of computing a matrix permanent. It is this crucial transformation that allows us to solve the semantic localization problem with a polynomial-time approximation to the set-based Bayes filter. Finally, we address the active semantic localization problem, in which the observer\u2019s trajectory is planned in order to improve the accuracy and efficiency of the localization process. The performance of our approach is demonstrated in simulation and in real environments using deformable-part-model-based object detectors. Robust global localization from semantic observations is demonstrated for a mobile robot, for the Project Tango phone, and on the KITTI visual odometry dataset. Comparisons are made with the traditional lidar-based geometric Monte Carlo localization.<\/jats:p>","DOI":"10.1177\/0278364915596589","type":"journal-article","created":{"date-parts":[[2015,10,7]],"date-time":"2015-10-07T21:27:51Z","timestamp":1444253271000},"page":"73-99","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":59,"title":["Localization from semantic observations via the matrix           permanent"],"prefix":"10.1177","volume":"35","author":[{"given":"Nikolay","family":"Atanasov","sequence":"first","affiliation":[{"name":"GRASP Laboratory, University of Pennsylvania, Philadelphia,\r          PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Menglong","family":"Zhu","sequence":"additional","affiliation":[{"name":"GRASP Laboratory, University of Pennsylvania, Philadelphia,\r          PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kostas","family":"Daniilidis","sequence":"additional","affiliation":[{"name":"GRASP Laboratory, University of Pennsylvania, Philadelphia,\r          PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"George J.","family":"Pappas","sequence":"additional","affiliation":[{"name":"GRASP Laboratory, University of Pennsylvania, Philadelphia,\r          PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2015,10,6]]},"reference":[{"key":"bibr1-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2012.6225216"},{"key":"bibr2-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2009.5152501"},{"key":"bibr3-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139863"},{"key":"bibr4-0278364915596589","doi-asserted-by":"crossref","unstructured":"Atanasov N, Zhu M, Daniilidis K, Pappas G (2014) Semantic localization via the matrix             permanent. In: Robotics: science and systems             (RSS), pp. 1\u201310. Available at: http:\/\/www.roboticsproceedings.org\/rss10\/p43.html","DOI":"10.15607\/RSS.2014.X.043"},{"key":"bibr5-0278364915596589","volume-title":"Mobile Robot Localisation and Mapping in Extensive Outdoor Environments","author":"Bailey T","year":"2002"},{"key":"bibr6-0278364915596589","doi-asserted-by":"crossref","unstructured":"Bao S, Savarese S (2011) Semantic structure from             motion. In: Computer vision and pattern recognition             (CVPR).","DOI":"10.1109\/CVPR.2011.5995462"},{"key":"bibr7-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/MCS.2009.934469"},{"key":"bibr8-0278364915596589","first-page":"900","volume-title":"ACM\u2013SIAM symposium on discrete algorithms","author":"Bez\u00e1kov\u00e1 I","year":"2006"},{"key":"bibr9-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/ICIF.2010.5711873"},{"key":"bibr10-0278364915596589","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2013.IX.053"},{"key":"bibr11-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2011.6094648"},{"key":"bibr12-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/7.256316"},{"key":"bibr13-0278364915596589","volume-title":"Elements of Information Theory","author":"Cover T","year":"2006","edition":"2"},{"key":"bibr14-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1177\/0278364908090961"},{"key":"bibr15-0278364915596589","doi-asserted-by":"crossref","unstructured":"Dame A, Prisacariu V, Ren C, Reid I (2013) Dense reconstruction using 3D object             shape priors. In: Computer vision and pattern recognition             (CVPR).","DOI":"10.1109\/CVPR.2013.170"},{"key":"bibr16-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/MRA.2013.2295947"},{"key":"bibr17-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.1999.772544"},{"key":"bibr18-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2009.167"},{"key":"bibr19-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1016\/S0921-8890(98)00049-9"},{"key":"bibr20-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2005.1545511"},{"key":"bibr21-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1177\/0278364913491297"},{"key":"bibr22-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2011.5940405"},{"key":"bibr23-0278364915596589","unstructured":"Google ATAP group (2014) Project\n            Tango. https:\/\/www.google.com\/atap\/projecttango."},{"key":"bibr24-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36279-8_34"},{"key":"bibr25-0278364915596589","unstructured":"Howard A, Gerkey B (2002) Adaptive Monte-Carlo Localization             (AMCL) package. Robot Operating System (ROS), available at: http:\/\/wiki.ros.org\/amcl."},{"key":"bibr26-0278364915596589","first-page":"398","volume-title":"International conference on artificial intelligence and statistics (AISTATS)","author":"Jadbabaie A","year":"2015"},{"key":"bibr27-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1145\/1008731.1008738"},{"key":"bibr28-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1177\/0278364911430419"},{"key":"bibr29-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2008.2006706"},{"key":"bibr30-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1177\/0278364909349948"},{"key":"bibr31-0278364915596589","doi-asserted-by":"crossref","unstructured":"Karasev V, Chiuso A, Soatto S (2012) Controlled recognition bounds for             visual learning and exploration. In: Advances in Neural             Information Processing Systems (NIPS).","DOI":"10.1109\/ITA.2013.6502995"},{"key":"bibr32-0278364915596589","first-page":"2630","volume-title":"IEEE\/RSJ international conference on intelligent robots and systems (IROS)","author":"Ko D","year":"2013"},{"key":"bibr33-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2004.1308033"},{"key":"bibr34-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2013.07.008"},{"key":"bibr35-0278364915596589","first-page":"3607","volume-title":"IEEE international conference on robotics and automation (ICRA)","author":"Kummerle R","year":"2011"},{"key":"bibr36-0278364915596589","volume-title":"Approximately Counting Perfect and General Matchings in Bipartite and General Graphs","author":"Law W","year":"2009"},{"key":"bibr37-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/JSTSP.2013.2251606"},{"key":"bibr38-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1201\/9781420053098"},{"key":"bibr39-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/TSP.2006.877658"},{"key":"bibr40-0278364915596589","doi-asserted-by":"crossref","unstructured":"Mahler R (2007) Statistical Multisource-Multitarget             Information Fusion. Artech House.","DOI":"10.1201\/9781420053098.ch16"},{"key":"bibr41-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2011.5980340"},{"key":"bibr42-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2003.1241885"},{"key":"bibr43-0278364915596589","first-page":"292","volume-title":"International conference on information fusion","author":"Morelande M","year":"2009"},{"key":"bibr44-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21390-8"},{"key":"bibr45-0278364915596589","volume-title":"Combinatorial Algorithms","author":"Nijenhuis A","year":"1978"},{"key":"bibr46-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2008.08.001"},{"key":"bibr47-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2009.2012975"},{"key":"bibr48-0278364915596589","first-page":"1160","volume-title":"Proceedings of the international joint conference on artificial intelligence (IJCAI)","volume":"2","author":"Pasula H","year":"1999"},{"key":"bibr49-0278364915596589","unstructured":"Pronobis A (2011) Semantic Mapping with Mobile Robots.           PhD Dissertation, KTH Royal Institute of Technology.           Available at: http:\/\/www.pronobis.pro\/phd."},{"key":"bibr50-0278364915596589","doi-asserted-by":"publisher","DOI":"10.5948\/UPO9781614440147"},{"key":"bibr51-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2004.839228"},{"key":"bibr52-0278364915596589","doi-asserted-by":"crossref","unstructured":"Sidenbladh H, Wirkander SL (2003) Tracking random sets of vehicles in             terrain. In: Computer vision and pattern recognition             workshop, vol. 9, p. 98","DOI":"10.1109\/CVPRW.2003.10097"},{"key":"bibr53-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2011.6094921"},{"key":"bibr54-0278364915596589","volume-title":"Probabilistic Robotics","author":"Thrun S","year":"2005"},{"key":"bibr55-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1016\/0304-3975(79)90044-6"},{"key":"bibr56-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2005.859085"},{"key":"bibr57-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2004.835453"},{"key":"bibr58-0278364915596589","unstructured":"Wong L, Kaelbling L, Lozano-P\u00e9rez T (2013) Data assocation for semantic world             modeling from partial views. In: International symposium on             robotics research (ISRR). Available at: http:\/\/people.csail.mit.edu\/lsw\/research.html"},{"key":"bibr59-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/ICSMC.2009.5346258"},{"key":"bibr60-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6385532"},{"key":"bibr61-0278364915596589","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10584-0_19"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364915596589","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0278364915596589","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364915596589","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T10:18:55Z","timestamp":1777457935000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0278364915596589"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,10,6]]},"references-count":61,"journal-issue":{"issue":"1-3","published-print":{"date-parts":[[2016,1]]}},"alternative-id":["10.1177\/0278364915596589"],"URL":"https:\/\/doi.org\/10.1177\/0278364915596589","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,10,6]]}}}