{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T12:02:56Z","timestamp":1775476976793,"version":"3.50.1"},"reference-count":69,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"78","content-domain":{"domain":["www.science.org"],"crossmark-restriction":true},"short-container-title":["Sci. Robot."],"published-print":{"date-parts":[[2023,5,31]]},"abstract":"<jats:p>Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recognition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both conventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural networks of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronously using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting robustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi\u2013neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot processor Jetson Xavier NX.<\/jats:p>","DOI":"10.1126\/scirobotics.abm6996","type":"journal-article","created":{"date-parts":[[2023,5,10]],"date-time":"2023-05-10T17:58:38Z","timestamp":1683741518000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark","source":"Crossref","is-referenced-by-count":72,"title":["Brain-inspired multimodal hybrid neural network for robot place recognition"],"prefix":"10.1126","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7031-4185","authenticated-orcid":true,"given":"Fangwen","family":"Yu","sequence":"first","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3738-4311","authenticated-orcid":true,"given":"Yujie","family":"Wu","sequence":"additional","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."},{"name":"Institute of Theoretical Computer Science, Graz University of Technology, Graz, Austria."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7913-046X","authenticated-orcid":true,"given":"Songchen","family":"Ma","sequence":"additional","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4329-8735","authenticated-orcid":true,"given":"Mingkun","family":"Xu","sequence":"additional","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7495-9930","authenticated-orcid":true,"given":"Hongyi","family":"Li","sequence":"additional","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0389-0032","authenticated-orcid":true,"given":"Huanyu","family":"Qu","sequence":"additional","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5371-6155","authenticated-orcid":true,"given":"Chenhang","family":"Song","sequence":"additional","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1878-5451","authenticated-orcid":true,"given":"Taoyi","family":"Wang","sequence":"additional","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2320-0326","authenticated-orcid":true,"given":"Rong","family":"Zhao","sequence":"additional","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."},{"name":"IDG\/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9829-2202","authenticated-orcid":true,"given":"Luping","family":"Shi","sequence":"additional","affiliation":[{"name":"Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China."},{"name":"IDG\/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China."},{"name":"THU-CET HIK Joint Research Center for Brain-Inspired Computing, Tsinghua University, Beijing 100084, China."}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2015.2496823"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","unstructured":"T. Fischer S.\u00a0Garg M.\u00a0Milford Where is your place visual place recognition in International Joint Conference on Artificial Intelligence (IJCAI) 19 to 26 August 2021 pp.\u00a04416\u20134425.","DOI":"10.24963\/ijcai.2021\/603"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2021.3123907"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2021.3096745"},{"key":"e_1_3_2_6_2","unstructured":"P. Yin S.\u00a0Zhao I.\u00a0Cisneros A.\u00a0Abuduweili G.\u00a0Huang M.\u00a0Milford C.\u00a0Liu H.\u00a0Choset S.\u00a0Scherer General place recognition survey: Towards the real-world autonomy age. arXiv:2209.04497 (2022)."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364909340592"},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","unstructured":"M. Milford G.\u00a0Wyeth SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights in IEEE International Conference on Robotics and Automation (ICRA) Saint Paul MN USA 14May to 18 May 2012 pp.\u00a01643\u20131649.","DOI":"10.1109\/ICRA.2012.6224623"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00422-019-00806-9"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2967324"},{"key":"e_1_3_2_11_2","unstructured":"M. Chanc\u00e1n M.\u00a0Milford DeepSeqSLAM: A trainable CNN+RNN for joint global description and sequence-based place recognition in the Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS) Workshop on Machine Learning for Autonomous Driving (ML4AD) 6 to 12 December 2020 pp.\u00a01-9."},{"key":"e_1_3_2_12_2","unstructured":"M. Chanc\u00e1n M.\u00a0Milford Sequential place learning: Heuristic-free high-performance long-term place recognition. arXiv:2103.02074 [cs.CV] (2021)."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2022.3150505"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.3008413"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aaz9712"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11633-022-1370-z"},{"key":"e_1_3_2_17_2","unstructured":"M. Milford H.\u00a0Kim M.\u00a0Mangan S.\u00a0Leutenegger T.\u00a0Stone B.\u00a0Webb A.\u00a0Davison Place recognition with event-based cameras and a neural implementation of SeqSLAM arXiv:1505.04548 [cs.RO] (2015)."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00422-017-0745-7"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.3025505"},{"key":"e_1_3_2_20_2","doi-asserted-by":"crossref","unstructured":"A.\u00a0J.\u00a0Lee A.\u00a0Kim EventVLAD: Visual place recognition with reconstructed edges from event cameras in 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) Prague Czech Republic 27 September to 1 October 2021 pp.\u00a02247\u20132252.","DOI":"10.1109\/IROS51168.2021.9635907"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIM.2022.3168892"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2022.3216226"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2022.3149030"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2012.01.010"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature11587"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2021.06.079"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abg1278"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1093\/cercor\/9.3.197"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2005.08.006"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1038\/nrn1828"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.aaw6503"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-04335-z"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2019.03.034"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2004642117"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-04242-3"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/0006-8993(71)90358-1"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1235338"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.10-02-00420.1990"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14031"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature03721"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-03783-x"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-018-0459-6"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-neuro-101419-011117"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41593-022-01226-y"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.43140"},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","unstructured":"S. Inayat B.\u00a0B.\u00a0McAllister B.\u00a0L.\u00a0McNaughton I.\u00a0Q.\u00a0Whishaw M.\u00a0H.\u00a0Mohajerani Conjunctive and complementary CA1 hippocampal cell populations relate sensory events to immobility and locomotion. bioRxiv 2022.07.06.498996 (2022).","DOI":"10.1101\/2022.07.06.498996"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1424-8"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSSC.2020.2970709"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2021.615279"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2021.3067593"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abl8419"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abk2948"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-28487-2"},{"key":"e_1_3_2_54_2","doi-asserted-by":"crossref","unstructured":"R. Kreiser A.\u00a0Renner Y.\u00a0Sandamirskaya P.\u00a0Pienroj Pose estimation and map formation with spiking neural networks: Towards neuromorphic SLAM in 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) Madrid Spain 1 to 5 October 2018 pp.\u00a02159\u20132166.","DOI":"10.1109\/IROS.2018.8594228"},{"key":"e_1_3_2_55_2","doi-asserted-by":"crossref","unstructured":"S. Baumgartner A.\u00a0Renner R.\u00a0Kreiser D.\u00a0Liang G.\u00a0Indiveri Y.\u00a0Sandamirskaya Visual pattern recognition with on on-chip learning: Towards a fully neuromorphic approach in 2020 IEEE International Symposium on Circuits and Systems (ISCAS) Seville Spain 12 to 14 October 2020 pp.\u00a01\u20135.","DOI":"10.1109\/ISCAS45731.2020.9180628"},{"key":"e_1_3_2_56_2","doi-asserted-by":"crossref","unstructured":"E.\u00a0P.\u00a0Frady G.\u00a0Orchard D.\u00a0Florey N.\u00a0Imam R.\u00a0Liu J.\u00a0Mishra J.\u00a0Tse A.\u00a0Wild F.\u00a0T.\u00a0Sommer M.\u00a0Davies Neuromorphic nearest neighbor search using Intel\u2019s Pohoiki Springs in Proceedings of the Neuro-inspired Computational Elements Workshop (NICE\u201920) Heidelberg Germany 17 to 20 March 2020 23 pp.\u00a01\u201310.","DOI":"10.1145\/3381755.3398695"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-0159-4"},{"key":"e_1_3_2_58_2","doi-asserted-by":"crossref","unstructured":"G. Tang N.\u00a0Kumar K.\u00a0P.\u00a0Michmizos Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware in IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) Las Vegas NV USA 2020 pp.\u00a06090\u20136097.","DOI":"10.1109\/IROS45743.2020.9340948"},{"key":"e_1_3_2_59_2","doi-asserted-by":"crossref","unstructured":"A. Vitale A.\u00a0Renner C.\u00a0Nauer D.\u00a0Scaramuzza Y.\u00a0Sandamirskaya Event-driven vision and control for UAVs on a neuromorphic chip in IEEE International Conference on Robotics and Automation Xi\u2019an China 30 May to 5 June 2021 pp.\u00a0103\u2013109.","DOI":"10.1109\/ICRA48506.2021.9560881"},{"key":"e_1_3_2_60_2","doi-asserted-by":"crossref","unstructured":"M. Quigley K.\u00a0Conley B.\u00a0Gerkey J.\u00a0Faust T.\u00a0Foote J.\u00a0Leibs R.\u00a0Wheeler A.\u00a0Ng ROS: An open-source Robot Operating System in IEEE International Conference on Robotics and Automation (ICRA) Workshop on Open Source Software Kobe Japan 12 to 17 May 2009 3 (3.2) pp.\u00a01\u20135.","DOI":"10.1109\/MRA.2010.936956"},{"key":"e_1_3_2_61_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11633-022-1390-8","article-title":"Brain-inspired intelligent robotics: Theoretical analysis and systematic application","volume":"20","author":"Qiao H.","year":"2023","unstructured":"H.\u00a0Qiao, Y.\u00a0Wu, S.\u00a0Zhong, P.\u00a0Yin, J.\u00a0Chen, Brain-inspired intelligent robotics: Theoretical analysis and systematic application. Machine Intelligence Research 20(1), 1\u201318 (2023).","journal-title":"Machine Intelligence Research"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2018.00331"},{"key":"e_1_3_2_63_2","doi-asserted-by":"crossref","unstructured":"Y. Wu L.\u00a0Deng G.\u00a0Li J.\u00a0Zhu Y.\u00a0Xie L.\u00a0Shi Direct training for spiking neural networks: Faster larger better in Proceedings of the AAAI Conference on Artificial Intelligence Honolulu Hawaii USA 27 January to 1 February 2019 33(01) pp.\u00a01311\u20131318.","DOI":"10.1609\/aaai.v33i01.33011311"},{"key":"e_1_3_2_64_2","doi-asserted-by":"crossref","unstructured":"S. Wu K.\u00a0Y.\u00a0M. Wong C.\u00a0C.\u00a0A. Fung Y.\u00a0Mi W.\u00a0Zhang Continuous attractor neural networks: Candidate of a canonical model for neural information representation. F1000Research 5 (2016).","DOI":"10.12688\/f1000research.7387.1"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-017-02717-4"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.3389\/fncom.2015.00099"},{"key":"e_1_3_2_67_2","unstructured":"R.\u00a0K.\u00a0Srivastava J.\u00a0Masci S.\u00a0Kazerounian F.\u00a0Gomez J.\u00a0Schmidhuber Compete to compute in NIPS\u201913: Proceedings of the 26th International Conference on Neural Information Processing Systems Lake Tahoe Nevada 5 to 10 December 2013 2 pp.\u00a02310\u20132318."},{"key":"e_1_3_2_68_2","unstructured":"D.\u00a0P.\u00a0Kingma J.\u00a0Ba Adam: A method for stochastic optimization arXiv:1412.6980 (2014)."},{"key":"e_1_3_2_69_2","doi-asserted-by":"crossref","unstructured":"H. Li S.\u00a0Ma T.\u00a0Wang W.\u00a0Zhang G.\u00a0Wang C.\u00a0Song HASP: Hierarchical asynchronous parallelism for multi-NN tasks. TechRxiv (2023); https:\/\/doi.org\/10.36227\/techrxiv.22338757.v1.","DOI":"10.36227\/techrxiv.22338757"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-021-01469-5"}],"container-title":["Science Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.abm6996","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,20]],"date-time":"2024-10-20T04:19:46Z","timestamp":1729397986000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/scirobotics.abm6996"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,31]]},"references-count":69,"journal-issue":{"issue":"78","published-print":{"date-parts":[[2023,5,31]]}},"alternative-id":["10.1126\/scirobotics.abm6996"],"URL":"https:\/\/doi.org\/10.1126\/scirobotics.abm6996","relation":{},"ISSN":["2470-9476"],"issn-type":[{"value":"2470-9476","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,31]]},"assertion":[{"value":"2021-10-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-14","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-05-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"eabm6996"}}