{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T16:34:13Z","timestamp":1785429253543,"version":"3.56.0"},"reference-count":38,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2026,6,14]],"date-time":"2026-06-14T00:00:00Z","timestamp":1781395200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ricks and the Department of Electrical and Computer Engineering, The University of Alabama"},{"id":[{"id":"https:\/\/ror.org\/03xrrjk67","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Autonomous navigation in unstructured off-road environments presents fundamental challenges due to terrain heterogeneity, the absence of structured road markings, and the necessity for real-time traversability reasoning from raw sensory observations. We present TerrainFormer, a hierarchical framework that integrates a world model for terrain dynamics prediction with a temporal decision transformer for action selection. Our methodology employs a two-phase training paradigm: (1) self-supervised world model pretraining on LiDAR point clouds to learn terrain representations encompassing traversability, elevation, and semantic segmentation; (2) behavioral cloning of the decision transformer conditioned on frozen world model features with temporally derived goal directions. The world model processes raw 3D LiDAR point clouds through a PointPillars encoder for real-time bird\u2019s-eye-view (BEV) projection, followed by a Vision Transformer backbone that produces latent terrain representations. A principal contribution is our cross-dataset generalization paradigm: the world model is trained on separate datasets while the decision transformer is trained on separate sequences, ensuring zero data overlap between training phases. We introduce automatic goal direction computation from vehicle pose trajectories, enabling the model to learn directionally conditioned navigation policies. To address the class imbalance inherent in off-road driving data, we employ focal loss with inverse-frequency class weighting and action-chunk supervision. Experimental evaluation on the RELLIS-3D dataset achieves 87.31% test accuracy with 0.7948 macro F1 across all 12 action classes. The world model\u2019s predicted future frames produce only a 0.79% accuracy drop versus ground-truth observations, with 98.82% action agreement, demonstrating effective cross-dataset generalization for real-time off-road navigation.<\/jats:p>","DOI":"10.3390\/s26123795","type":"journal-article","created":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T05:59:47Z","timestamp":1781503187000},"page":"3795","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["TerrainFormer: World Model-Guided Decision Transformer for Autonomous Off-Road Navigation"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-1535-5572","authenticated-orcid":false,"given":"Yongzhi","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, The University of Alabama, Tuscaloosa, AL 35487, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1740-631X","authenticated-orcid":false,"given":"Kenneth","family":"Ricks","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, The University of Alabama, Tuscaloosa, AL 35487, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,6,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1509","DOI":"10.1109\/LRA.2019.2895390","article-title":"Where should I walk? Predicting terrain properties from images via self-supervised learning","volume":"4","author":"Wellhausen","year":"2019","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_2","unstructured":"Xiao, X., Zhang, T., Choromanski, K., Lee, E., Francis, A., Varley, J., Tu, S., Singh, S., Xu, P., and Xia, F. (2022). Learning model predictive controllers with real-time attention for real-world navigation. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1373","DOI":"10.1016\/j.engappai.2013.01.006","article-title":"Terrain traversability analysis methods for unmanned ground vehicles: A survey","volume":"26","author":"Papadakis","year":"2013","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Maturana, D., Chou, P., Uenoyama, M., and Scherer, S. (2018). Real-time semantic mapping for autonomous off-road navigation. Proceedings of the International Conference on Field and Service Robotics, Springer.","DOI":"10.1007\/978-3-319-67361-5_22"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wigness, M., Edelstein, S., Gathright, A., Rogers, J., and Schwarz, P. (2019). A RUGD dataset for autonomous navigation and visual perception in unstructured outdoor environments. 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.","DOI":"10.1109\/IROS40897.2019.8968283"},{"key":"ref_6","unstructured":"Ha, D., and Schmidhuber, J. (2018). World models. arXiv."},{"key":"ref_7","unstructured":"Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2020). Dream to control: Learning behaviors by latent imagination. Proceedings of the International Conference on Learning Representations (ICLR), Curran Associates, Inc."},{"key":"ref_8","unstructured":"Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. (2023). Mastering diverse domains through world models. arXiv."},{"key":"ref_9","unstructured":"Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. (2021). Decision transformer: Reinforcement learning via sequence modeling. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_10","unstructured":"Zheng, Q., Zhang, A., and Grover, A. (2022). Online decision transformer. Proceedings of the International Conference on Machine Learning (ICML), JMLR."},{"key":"ref_11","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhao, T.Z., Kumar, V., Levine, S., and Finn, C. (2023). Learning fine-grained bimanual manipulation with low-cost hardware. Proceedings of the Robotics: Science and Systems (RSS), RSS Foundation.","DOI":"10.15607\/RSS.2023.XIX.016"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wei, C., Wu, Q., Zuo, S., Xu, J., Zhao, B., Yang, Z., Xie, G., and Wang, S. (2025). LiDARDustX: A LiDAR dataset for dusty unstructured road environments. Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA55743.2025.11127917"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mortimer, P., Hagmanns, R., Granero, M., Luettel, T., Petereit, J., and Wuensche, H.-J. (2024). The GOOSE Dataset for Perception in Unstructured Environments. Proceedings of the 2024 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS 2024), IEEE.","DOI":"10.1109\/ICRA57147.2024.10611298"},{"key":"ref_15","unstructured":"Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017). PointNet: Deep learning on point sets for 3D classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE."},{"key":"ref_16","unstructured":"Qi, C.R., Yi, L., Su, H., and Guibas, L.J. (2017). PointNet++: Deep hierarchical feature learning on point sets in a metric space. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zhou, Y., and Tuzel, O. (2018). VoxelNet: End-to-end learning for point cloud based 3D object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE.","DOI":"10.1109\/CVPR.2018.00472"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. (2019). PointPillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE.","DOI":"10.1109\/CVPR.2019.01298"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Jiang, P., Osteen, P., Wigness, M., and Saripalli, S. (2021). RELLIS-3D dataset: Data, benchmarks and analysis for off-road robotics. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA48506.2021.9561251"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Sock, J., Kim, J., Min, J., and Kwak, K. (2016). Probabilistic traversability map generation using 3D-LIDAR and camera. Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA.2016.7487782"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Frey, J., Mattamala, M., Chebrolu, N., Cadena, C., Fallon, M., and Hutter, M. (2023). Fast traversability estimation for wild visual navigation. Proceedings of the Robotics: Science and Systems (RSS), RSS Foundation.","DOI":"10.15607\/RSS.2023.XIX.054"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Castro, M.G., Triest, S., Wang, W., Gregory, J.M., Dellaert, F., and Scherer, S. (2023). How does it feel? Self-supervised costmap learning for off-road vehicle traversability. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA48891.2023.10160856"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Frey, J., Mattamala, M., and Hutter, M. (2024). Roadrunner\u2013Learning traversability estimation for autonomous off-road driving. arXiv.","DOI":"10.1109\/TFR.2024.3464369"},{"key":"ref_24","unstructured":"Hu, A., Corrado, G., Griffiths, N., Murez, Z., Gurau, C., Yeo, H., Kendall, A., Cipolla, R., and Shotton, J. (2022). Model-based imitation learning for urban driving. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_25","unstructured":"Yang, X., Chen, J., and Scherer, S. (2023). Point cloud world models for autonomous driving. Proceedings of the 2023 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE."},{"key":"ref_26","unstructured":"Janner, M., Li, Q., and Levine, S. (2021). Offline reinforcement learning as one big sequence modeling problem. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_27","unstructured":"Xu, M., Shen, Y., Zhang, S., Lu, Y., Zhao, D., Tenenbaum, J., and Gan, C. (2022). Prompting decision transformer for few-shot policy generalization. Proceedings of the International Conference on Machine Learning (ICML), JMLR."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Nayakanti, N., Al-Rfou, R., Zhou, A., Goel, K., Refaat, K.S., and Sapp, B. (2023). Wayformer: Motion forecasting via simple & efficient attention networks. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA48891.2023.10160609"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., and Wang, W. (2023). Planning-oriented autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE.","DOI":"10.1109\/CVPR52729.2023.01712"},{"key":"ref_30","unstructured":"Pomerleau, D.A. (1989). ALVINN: An autonomous land vehicle in a neural network. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_31","unstructured":"Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., and Zhang, J. (2016). End to end learning for self-driving cars. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Codevilla, F., M\u00fcller, M., L\u00f3pez, A., Koltun, V., and Dosovitskiy, A. (2018). End-to-end driving via conditional imitation learning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA.2018.8460487"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1312","DOI":"10.1109\/LRA.2021.3057023","article-title":"BADGR: An autonomous self-supervised learning-based navigation system","volume":"6","author":"Kahn","year":"2021","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Triest, S., Sivaprakasam, M., Wang, S.J., Wang, W., Johnson, A.M., and Scherer, S. (2023). TartanDrive: A large-scale dataset for learning off-road dynamics models. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA46639.2022.9811648"},{"key":"ref_35","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021). An image is worth 16 \u00d7 16 words: Transformers for image recognition at scale. Proceedings of the International Conference on Learning Representations (ICLR), Curran Associates, Inc."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Doll\u00e1r, P. (2017). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, IEEE.","DOI":"10.1109\/ICCV.2017.324"},{"key":"ref_37","unstructured":"Loshchilov, I., and Hutter, F. (2019). Decoupled weight decay regularization. Proceedings of the International Conference on Learning Representations (ICLR), Curran Associates, Inc."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Xu, Z., Yi, G., Zhang, X., Xie, Y., and Kaynak, O. (IEEE\/ASME Trans. Mechatron., 2026). Enhancing Attitude Availability in Star-Depleted Cases: An Inertial\/Star Sensor Fusion Method, IEEE\/ASME Trans. Mechatron., early access.","DOI":"10.1109\/TMECH.2026.3679689"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/12\/3795\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T06:06:31Z","timestamp":1781503591000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/12\/3795"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,14]]},"references-count":38,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2026,6]]}},"alternative-id":["s26123795"],"URL":"https:\/\/doi.org\/10.3390\/s26123795","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,14]]}}}