{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T06:37:47Z","timestamp":1774507067788,"version":"3.50.1"},"reference-count":65,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"81","content-domain":{"domain":["www.science.org"],"crossmark-restriction":true},"short-container-title":["Sci. Robot."],"published-print":{"date-parts":[[2023,8,30]]},"abstract":"<jats:p>Robots are active agents that operate in dynamic scenarios with noisy sensors. Predictions based on these noisy sensor measurements often lead to errors and can be unreliable. To this end, roboticists have used fusion methods using multiple observations. Lately, neural networks have dominated the accuracy charts for perception-driven predictions for robotic decision-making and often lack uncertainty metrics associated with the predictions. Here, we present a mathematical formulation to obtain the heteroscedastic aleatoric uncertainty of any arbitrary distribution without prior knowledge about the data. The approach has no prior assumptions about the prediction labels and is agnostic to network architecture. Furthermore, our class of networks, Ajna, adds minimal computation and requires only a small change to the loss function while training neural networks to obtain uncertainty of predictions, enabling real-time operation even on resource-constrained robots. In addition, we study the informational cues present in the uncertainties of predicted values and their utility in the unification of common robotics problems. In particular, we present an approach to dodge dynamic obstacles, navigate through a cluttered scene, fly through unknown gaps, and segment an object pile, without computing depth but rather using the uncertainties of optical flow obtained from a monocular camera with onboard sensing and computation. We successfully evaluate and demonstrate the proposed Ajna network on four aforementioned common robotics and computer vision tasks and show comparable results to methods directly using depth. Our work demonstrates a generalized deep uncertainty method and demonstrates its utilization in robotics applications.<\/jats:p>","DOI":"10.1126\/scirobotics.add5139","type":"journal-article","created":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T17:58:14Z","timestamp":1692208694000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark","source":"Crossref","is-referenced-by-count":12,"title":["Ajna: Generalized deep uncertainty for minimal perception on parsimonious robots"],"prefix":"10.1126","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9681-7602","authenticated-orcid":true,"given":"Nitin J.","family":"Sanket","sequence":"first","affiliation":[{"name":"Perception and Robotics Group (PRG), University of Maryland, College Park, MD, USA."},{"name":"Perception and Autonomous Robotics (PeAR) Group, Worcester Polytechnic Institute, Worcester, MA, USA."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9972-5559","authenticated-orcid":true,"given":"Chahat Deep","family":"Singh","sequence":"additional","affiliation":[{"name":"Perception and Robotics Group (PRG), University of Maryland, College Park, MD, USA."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2044-2386","authenticated-orcid":true,"given":"Cornelia","family":"Ferm\u00fcller","sequence":"additional","affiliation":[{"name":"Perception and Robotics Group (PRG), University of Maryland, College Park, MD, USA."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8152-4281","authenticated-orcid":true,"given":"Yiannis","family":"Aloimonos","sequence":"additional","affiliation":[{"name":"Perception and Robotics Group (PRG), University of Maryland, College Park, MD, USA."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"221","reference":[{"key":"e_1_3_1_2_2","unstructured":"A. Grant Think Again: The Power of Knowing What You Don\u2019t Know (Penguin 2021)."},{"key":"e_1_3_1_3_2","first-page":"853","article-title":"Transforming neural-net output levels to probability distributions","volume":"3","author":"Denker J.","year":"1990","unstructured":"J. Denker, Y. LeCun, Transforming neural-net output levels to probability distributions. Adv. Neural Inf. Process. Syst. 3, 853\u2013859 (1990).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1992.4.3.448"},{"key":"e_1_3_1_5_2","unstructured":"R. M. Neal. Bayesian Learning for Neural Networks (Springer Science & Business Media 2012) vol. 118."},{"key":"e_1_3_1_6_2","unstructured":"Y. Gal Z. Ghahramani Bayesian convolutional neural networks with Bernoulli approximate variational inference in 4th International Conference on Learning Representations ( ICLR ) Workshop Track (2016) https:\/\/openreview.net\/pdf?id=3QxqXoJEyfp7y9wltP11."},{"key":"e_1_3_1_7_2","unstructured":"Y. Gal \u201cUncertainty in deep learning \u201d thesis University of Cambridge Cambridge UK (2016)."},{"key":"e_1_3_1_8_2","unstructured":"A. Kendall and Y. Gal. What uncertainties do we need in Bayesian deep learning for computer vision? in Proceedings of the 31st International Conference on Neural Information Processing Systems NIPS\u201917 (Curran Associates Inc. 2017) pp. 5580\u20135590."},{"key":"e_1_3_1_9_2","doi-asserted-by":"crossref","unstructured":"J. Gast S. Roth Lightweight probabilistic deep networks in Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (IEEE 2018) pp. 3369\u20133378.","DOI":"10.1109\/CVPR.2018.00355"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2974682"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2021.05.008"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.3007421"},{"key":"e_1_3_1_13_2","doi-asserted-by":"crossref","unstructured":"L. Zhou Z. Luo T. Shen J. Zhang M. Zhen Y. Yao T. Fang L. Quan Kfnet: Learning temporal camera relocalization using kalman filtering in Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (IEEE 2020) pp. 4919\u20134928.","DOI":"10.1109\/CVPR42600.2020.00497"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1115\/1.3662552"},{"key":"e_1_3_1_15_2","doi-asserted-by":"crossref","unstructured":"S. Rabiee J. Biswas IVOA: Introspective vision for obstacle avoidance in Proceedings of the 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems ( IROS ) (IEEE 2019) pp. 1230\u20131235.","DOI":"10.1109\/IROS40897.2019.8968176"},{"key":"#cr-split#-e_1_3_1_16_2.1","doi-asserted-by":"crossref","unstructured":"D. Feng L. Rosenbaum F. Timm K. Dietmayer Leveraging heteroscedastic aleatoric uncertainties for robust real-time LiDAR 3D object detection in Proceedings of the 2019 IEEE Intelligent Vehicles Symposium","DOI":"10.1109\/IVS.2019.8814046"},{"key":"#cr-split#-e_1_3_1_16_2.2","unstructured":"(IV) (IEEE 2019) pp. 1280-1287."},{"key":"e_1_3_1_17_2","doi-asserted-by":"crossref","unstructured":"S. Lee V. Capuano A. Harvard S.-J. Chung Fast uncertainty estimation for deep learning based optical flow in 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems ( IROS ) (IEEE 2020) pp. 10138\u201310144.","DOI":"10.1109\/IROS45743.2020.9340963"},{"key":"e_1_3_1_18_2","doi-asserted-by":"crossref","unstructured":"J.-G. Kang S.-D. Roh K.-S. Chung FlowNetU: Accurate uncertainty estimation of optical flow for video object detection in Proceedings of the 2021 4th International Conference on Artificial Intelligence and Pattern Recognition (Association for Computing Machinery 2021) pp. 36\u201341.","DOI":"10.1145\/3488933.3489027"},{"key":"e_1_3_1_19_2","doi-asserted-by":"crossref","unstructured":"E. Ilg O. Cicek S. Galesso A. Klein O. Makansi F. Hutter T. Brox Uncertainty estimates and multi-hypotheses networks for optical flow in Proceedings of the European Conference on Computer Vision ( ECCV ) (2018) pp. 652\u2013667.","DOI":"10.1007\/978-3-030-01234-2_40"},{"key":"e_1_3_1_20_2","doi-asserted-by":"crossref","unstructured":"S. Li X. Wu Y. Cao H. Zha Generalizing to the open world: Deep visual odometry with online adaptation in Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (IEEE 2021) pp. 13184\u201313193.","DOI":"10.1109\/CVPR46437.2021.01298"},{"key":"e_1_3_1_21_2","doi-asserted-by":"crossref","unstructured":"W. Yuan Y. Zhang B. Wu S. Zhu P. Tan M. Y. Wang Q. Chen Stereo matching by self-supervision of multiscopic vision in Proceedings of the 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE 2021) pp. 5702\u20135709.","DOI":"10.1109\/IROS51168.2021.9636616"},{"key":"e_1_3_1_22_2","doi-asserted-by":"crossref","unstructured":"G. Bae I. Budvytis R. Cipolla Estimating and exploiting the aleatoric uncertainty in surface normal estimation in Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision ( ICCV ) (IEEE 2021) pp. 13137\u201313146.","DOI":"10.1109\/ICCV48922.2021.01289"},{"key":"e_1_3_1_23_2","doi-asserted-by":"crossref","unstructured":"B. Roessle J. T. Barron B. Mildenhall P. P. Srinivasan M. Nie\u00dfner Dense depth priors for neural radiance fields from sparse input views. arXiv:2112.03288 [cs.CV] (6 December 2021).","DOI":"10.1109\/CVPR52688.2022.01255"},{"key":"e_1_3_1_24_2","unstructured":"D. Bhatt K. Mani D. Bansal K. Murthy H. Lee L. Paull f -Cal: Calibrated aleatoric uncertainty estimation from neural networks for robot perception. arXiv:2109.13913 [cs.CV] (28 September 2021)."},{"key":"e_1_3_1_25_2","doi-asserted-by":"crossref","unstructured":"R. Martin-Brualla N. Radwan M. S. M. Sajjadi J. T. Barron A. Dosovitskiy D. Duckworth Nerf in the wild: Neural radiance fields for unconstrained photo collections in Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (IEEE 2021) pp. 7210\u20137219.","DOI":"10.1109\/CVPR46437.2021.00713"},{"key":"e_1_3_1_26_2","doi-asserted-by":"crossref","unstructured":"A. Eldesokey M. Felsberg K. Holmquist M. Persson Uncertainty-aware CNNs for depth completion: Uncertainty from beginning to end in Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (IEEE 2020) pp. 12014\u201312023.","DOI":"10.1109\/CVPR42600.2020.01203"},{"key":"e_1_3_1_27_2","doi-asserted-by":"crossref","unstructured":"M. Poggi F. Aleotti F. Tosi S. Mattoccia On the uncertainty of self-supervised monocular depth estimation in Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (IEEE 2020) pp. 3227\u20133237.","DOI":"10.1109\/CVPR42600.2020.00329"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2020.3001674"},{"key":"e_1_3_1_29_2","doi-asserted-by":"crossref","unstructured":"T. Kawashima Q. Yu A. Asai D. Ikami K. Aizawa. The aleatoric uncertainty estimation using a separate formulation with virtual residuals in 2020 25th International Conference on Pattern Recognition ( ICPR ) (IEEE 2021) pp. 1438\u20131445.","DOI":"10.1109\/ICPR48806.2021.9412324"},{"key":"e_1_3_1_30_2","doi-asserted-by":"crossref","unstructured":"T. Zhou M. Brown N. Snavely D. G. Lowe. Unsupervised learning of depth and ego-motion from video in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (IEEE 2017) pp. 1851\u20131858.","DOI":"10.1109\/CVPR.2017.700"},{"key":"e_1_3_1_31_2","unstructured":"N. J. Sanket C. D. Singh C. Ferm\u00fcller Y. Aloimonos PRGFlyt: AI based indoor quadrotor autonomy framework for navigation and interaction tasks (2019); https:\/\/github.com\/prgumd\/PRGFlyt\/wiki."},{"key":"e_1_3_1_32_2","unstructured":"NVIDIA Jetson TX2; https:\/\/developer.nvidia.com\/embedded\/jetson-tx2."},{"key":"e_1_3_1_33_2","doi-asserted-by":"crossref","unstructured":"N. J. Sanket C. D. Singh C. M. Parameshwara C. Ferm\u00fcller G. C. H. E. de Croon Y. Aloimonos EVPropNet: Detecting drones by finding propellers for mid-air landing and following. arXiv:2106.15045 [cs.CV] (29 June 2021).","DOI":"10.15607\/RSS.2021.XVII.074"},{"key":"e_1_3_1_34_2","doi-asserted-by":"crossref","unstructured":"A. Dosovitskiy P. Fischer E. Ilg P. H\u00e4usser C. Hazirbas V. Golkov P. van der Smagt D. Cremers T. Brox. Flownet: Learning optical flow with convolutional networks in Proceedings of the 2015 IEEE International Conference on Computer Vision ( ICCV ) (IEEE 2015).","DOI":"10.1109\/ICCV.2015.316"},{"key":"e_1_3_1_35_2","doi-asserted-by":"crossref","unstructured":"E. Ilg T. Saikia M. Keuper T. Brox. Occlusions motion and depth boundaries with a generic network for disparity optical flow or scene flow estimation in European Conference on Computer Vision (ECCV) V. Ferrari M. Hebert C. Sminchisescu Y. Weiss Eds. (Springer 2018).","DOI":"10.1007\/978-3-030-01258-8_38"},{"key":"e_1_3_1_36_2","doi-asserted-by":"crossref","unstructured":"N. Mayer E. Ilg P. H\u00e4usser P. Fischer D. Cremers A. Dosovitskiy T. Brox A large dataset to train convolutional networks for disparity optical flow and scene flow estimation. arXiv:1512.02134 [cs.CV] (7 December 2016).","DOI":"10.1109\/CVPR.2016.438"},{"key":"e_1_3_1_37_2","doi-asserted-by":"crossref","unstructured":"C. D. Singh N. J. Sanket C. M. Parameshwara C. Ferm\u00fcller Y. Aloimonos. Nudgeseg: Zero-shot object segmentation by repeated physical interaction in 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems ( IROS ) (IEEE 2021) pp. 2714\u20132712.","DOI":"10.1109\/IROS51168.2021.9635832"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abm5954"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abl6334"},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","unstructured":"N. J. Sanket C. M. Parameshwara C. D. Singh A. V. Kuruttukulam C. Ferm\u00fcller D. Scaramuzza Y. Aloimonos. EVDodgeNet: Deep dynamic obstacle dodging with event cameras in Proceedings of the 2020 IEEE International Conference on Robotics and Automation ( ICRA ) (IEEE 2020) pp. 10651\u201310657.","DOI":"10.1109\/ICRA40945.2020.9196877"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.3019967"},{"key":"e_1_3_1_42_2","doi-asserted-by":"crossref","unstructured":"R. Ranftl A. Bochkovskiy V. Koltun Vision transformers for dense prediction. arXiv:2103.13413 [cs.CV] (24 March 2021).","DOI":"10.1109\/ICCV48922.2021.01196"},{"key":"e_1_3_1_43_2","doi-asserted-by":"crossref","unstructured":"S. Zhao Y. Sheng Y. Dong E. I.-C. Chang Y. Xu Maskflownet: Asymmetric feature matching with learnable occlusion mask in Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE 2020) pp. 6278\u20136287.","DOI":"10.1109\/CVPR42600.2020.00631"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aaz9712"},{"key":"e_1_3_1_45_2","doi-asserted-by":"crossref","unstructured":"N. J. Sanket C. D. Singh V. Asthana C. Ferm\u00fcller Y. Aloimonos MorphEyes: Variable baseline stereo for quadrotor navigation in Proceedings of the 2021 IEEE International Conference on Robotics and Automation ( ICRA ) (IEEE 2021) pp. 413\u2013419.","DOI":"10.1109\/ICRA48506.2021.9561116"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2018.2843445"},{"key":"e_1_3_1_47_2","doi-asserted-by":"crossref","unstructured":"A. Bircher M. Kamel K. Alexis H. Oleynikova R. Siegwart Receding horizon \u201cnext-best-view\u201d planner for 3D exploration in Proceedings of the 2016 IEEE International Conference on Robotics and Automation ( ICRA ) (IEEE 2016) pp. 1462\u2013146.","DOI":"10.1109\/ICRA.2016.7487281"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-017-9615-3"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00128132"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2018.2789841"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-00279-7"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00129684"},{"key":"e_1_3_1_53_2","doi-asserted-by":"crossref","unstructured":"K. He G. Gkioxari P. Doll\u00e1r R. Girshick Mask R-CNN in Proceedings of the 2017 IEEE International Conference on Computer Vision ( ICCV ) (IEEE 2017) pp. 2961\u20132969.","DOI":"10.1109\/ICCV.2017.322"},{"key":"e_1_3_1_54_2","doi-asserted-by":"crossref","unstructured":"A. Kirillov Y. Wu K. He R. Girshick Pointrend: Image segmentation as rendering in Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (IEEE 2020) pp. 9799\u20139808.","DOI":"10.1109\/CVPR42600.2020.00982"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1049\/ell2.12274"},{"key":"e_1_3_1_56_2","doi-asserted-by":"crossref","unstructured":"J.-L. Stevens R. Mahony Vision based forward sensitive reactive control for a quadrotor VTOL in Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems ( IROS ) (IEEE 2018) pp. 5232\u20135238.","DOI":"10.1109\/IROS.2018.8593606"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abg5810"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aaw9710"},{"key":"e_1_3_1_59_2","doi-asserted-by":"crossref","unstructured":"A. Ranjan J. Janai A. Geiger M. J. Black. Attacking optical flow in Proceedings of the IEEE\/CVF International Conference on Computer Vision (IEEE 2019) pp. 2404\u2013413.","DOI":"10.1109\/ICCV.2019.00249"},{"key":"e_1_3_1_60_2","doi-asserted-by":"crossref","unstructured":"T. van Dijk G. de Croon How do neural networks see depth in single images? in Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV) (IEEE 2019) pp. 2183\u20132191.","DOI":"10.1109\/ICCV.2019.00227"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2019.2900980"},{"key":"e_1_3_1_62_2","unstructured":"Intel RealSense Depth Camera D435i. https:\/\/intelrealsense.com\/depth-camera-d435i\/."},{"key":"e_1_3_1_63_2","doi-asserted-by":"crossref","unstructured":"D. Sun X. Yang M.-Y. Liu J. Kautz PWC-Net: CNNs for optical flow using pyramid warping and cost volume in Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (IEEE 2018) pp. 8934\u20138943.","DOI":"10.1109\/CVPR.2018.00931"},{"key":"e_1_3_1_64_2","doi-asserted-by":"crossref","unstructured":"E. Ilg N. Mayer T. Saikia M. Keuper A. Dosovitskiy T. Brox FlowNet 2.0: Evolution of optical flow estimation with deep networks in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) (IEEE 2017) pp. 2462\u20132470.","DOI":"10.1109\/CVPR.2017.179"},{"key":"e_1_3_1_65_2","doi-asserted-by":"crossref","unstructured":"A. Ranjan M. J. Black Optical flow estimation using a spatial pyramid network in Proceedings of the IEEE 2017 Conference on Computer Vision and Pattern Recognition ( CVPR ) (IEEE 2017) pp. 4161\u20134170.","DOI":"10.1109\/CVPR.2017.291"}],"container-title":["Science Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.add5139","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,16]],"date-time":"2024-01-16T12:33:08Z","timestamp":1705408388000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/scirobotics.add5139"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,30]]},"references-count":65,"journal-issue":{"issue":"81","published-print":{"date-parts":[[2023,8,30]]}},"alternative-id":["10.1126\/scirobotics.add5139"],"URL":"https:\/\/doi.org\/10.1126\/scirobotics.add5139","relation":{},"ISSN":["2470-9476"],"issn-type":[{"value":"2470-9476","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,30]]},"assertion":[{"value":"2022-06-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"eadd5139"}}