{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T11:27:07Z","timestamp":1775042827326,"version":"3.50.1"},"reference-count":39,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,4,16]],"date-time":"2021-04-16T00:00:00Z","timestamp":1618531200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Object detection plays a critical role in autonomous driving, but current state-of-the-art object detectors will inevitably fail in many driving scenes, which is unacceptable for safety-critical automated vehicles. Given the complexity of the real traffic scenarios, it is impractical to guarantee zero detection failure; thus, online failure prediction is of crucial importance to mitigate the risk of traffic accidents. Of all the failure cases, False Negative (FN) objects are most likely to cause catastrophic consequences, but little attention has been paid to the online FN prediction. In this paper, we propose a general introspection framework that can make online prediction of FN objects for black-box object detectors. In contrast to existing methods which rely on empirical assumptions or handcrafted features, we facilitate the FN feature extraction by an introspective FN predictor we designed in this framework. For this purpose, we extend the original concept of introspection to object-wise FN predictions, and propose a multi-branch cooperation mechanism to address the distinct foreground-background imbalance problem of FN objects. The effectiveness of the proposed framework is verified through extensive experiments and analysis, and the results show that our method successfully predicts the FN objects with 81.95% precision for 88.10% recall on the challenging KITTI Benchmark, and effectively improves object detection performance by taking FN predictions into consideration.<\/jats:p>","DOI":"10.3390\/s21082819","type":"journal-article","created":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T06:35:53Z","timestamp":1618814153000},"page":"2819","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Introspective False Negative Prediction for Black-Box Object Detectors in Autonomous Driving"],"prefix":"10.3390","volume":"21","author":[{"given":"Qinghua","family":"Yang","sequence":"first","affiliation":[{"name":"School of Automotive Studies, Tongji University, Shanghai 201804, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hui","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Automotive Studies, Tongji University, Shanghai 201804, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhe","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Automotive Studies, Tongji University, Shanghai 201804, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junzhe","family":"Su","sequence":"additional","affiliation":[{"name":"School of Automotive Studies, Tongji University, Shanghai 201804, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,4,16]]},"reference":[{"key":"ref_1","first-page":"3873","article-title":"Uncertainty Estimation for Deep Neural Object Detectors in Safety-Critical Applications","volume":"2018-Novem","author":"Le","year":"2018","journal-title":"IEEE Conf. Intell. Transp. Syst. Proc. ITSC"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3860","DOI":"10.1109\/LRA.2018.2857402","article-title":"Failing to Learn: Autonomously Identifying Perception Failures for Self-Driving Cars","volume":"3","author":"Ramanagopal","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Ponn, T., Kr\u00f6ger, T., and Diermeyer, F. (2020). Identification and explanation of challenging conditions for camera-based object detection of automated vehicles. Sensors (Switzerland), 20.","DOI":"10.3390\/s20133699"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Rahman, Q.M., and Niko, S. (2020). Online Monitoring of Object Detection Performance Post-Deployment. arXiv.","DOI":"10.1109\/IROS51168.2021.9635940"},{"key":"ref_5","unstructured":"Liang, S., Srikant, R., and Li, Y. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv."},{"key":"ref_6","unstructured":"Hendrycks, D., and Gimpel, K. (2016). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Gupta, A., and Carlone, L. (2020, January 20\u201323). Online Monitoring for Neural Network Based Monocular Pedestrian Pose Estimation. Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece.","DOI":"10.1109\/ITSC45102.2020.9294609"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Miller, D., Nicholson, L., Dayoub, F., and Sunderhauf, N. (2018, January 21\u201325). Dropout Sampling for Robust Object Detection in Open-Set Conditions. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8460700"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Harakeh, A., Smart, M., and Waslander, S.L. (August, January 31). BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.","DOI":"10.1109\/ICRA40945.2020.9196544"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Kraus, F., and Dietmayer, K. (2019, January 34\u201330). Uncertainty Estimation in One-Stage Object Detection. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019, Auckland, New Zealand.","DOI":"10.1109\/ITSC.2019.8917494"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Rahman, Q.M., S\u00a8underhauf, N., and Dayoub, F. (2021, January 5\u20139). Per-frame mAP Prediction for Continuous Performance Monitoring of Object Detection During Deployment. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Hilton Waikoloa Village, HI, USA.","DOI":"10.1109\/WACVW52041.2021.00021"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Rahman, Q.M., S\u00a8underhauf, N., and Dayoub, F. (2019). Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors. arXiv.","DOI":"10.1109\/IROS40897.2019.8968525"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Rabiee, S., and Biswas, J. (2019, January 4\u20138). IVOA: Introspective Vision for Obstacle Avoidance. Proceedings of the 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.","DOI":"10.1109\/IROS40897.2019.8968176"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16\u201321). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.","DOI":"10.1109\/CVPR.2012.6248074"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/978-3-319-46448-0_2","article-title":"SSD: Single shot multibox detector","volume":"9905 LNCS","author":"Liu","year":"2016","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_16","first-page":"110","article-title":"Robotic introspection for exploration and mapping of subterranean environments","volume":"3304517","author":"Morris","year":"2007","journal-title":"ProQuest Diss. Theses"},{"key":"ref_17","first-page":"1743","article-title":"Introspective perception: Learning to predict failures in vision systems","volume":"2016","author":"Daftry","year":"2016","journal-title":"IEEE Int. Conf. Intell. Robot. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Oksuz, K., Cam, B.C., Kalkan, S., and Akbas, E. (2020). Imbalance Problems in Object Detection: A Review. IEEE Trans. Pattern Anal. Mach. Intell.","DOI":"10.1109\/TPAMI.2020.2981890"},{"key":"ref_19","first-page":"6568","article-title":"CenterNet: Keypoint triplets for object detection","volume":"2019","author":"Duan","year":"2019","journal-title":"Proc. IEEE Int. Conf. Comput. Vis."},{"key":"ref_20","unstructured":"Liu, W., Hasan, I., and Liao, S. (2019). Center and Scale Prediction: A Box-free Approach for Pedestrian and Face Detection. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Grimmett, H., Paul, R., Triebel, R., and Posner, I. (2013). Knowing when we don\u2019t know: Introspective classification for mission-critical decision making. Proc. IEEE Int. Conf. Robot. Autom., 4531\u20134538.","DOI":"10.1109\/ICRA.2013.6631221"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1007\/978-3-319-28872-7_26","article-title":"Driven learning for driving: How introspection improves semantic mapping","volume":"114","author":"Triebel","year":"2016","journal-title":"Springer Tracts Adv. Robot."},{"key":"ref_23","first-page":"33","article-title":"Introspective evaluation of perception performance for parameter tuning without ground truth","volume":"13","author":"Hu","year":"2017","journal-title":"Robot. Sci. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Kuhn, C.B., Hofbauer, M., Petrovic, G., and Steinbach, E. (November, January 19). Introspective Black Box Failure Prediction for Autonomous Driving. Proceedings of the 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA.","DOI":"10.1109\/IV47402.2020.9304844"},{"key":"ref_25","unstructured":"Kuhn, C.B., Hofbauer, M., Lee, S., Petrovic, G., and Steinbach, E. (2020, January 20\u201323). Deep Residual Learning for Image Recognition. Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Schubert, M., Kahl, K., Rottmann, M., and Sciences, N. (2020). MetaDetect: Uncertainty Quantification and Prediction Quality Estimates for Object Detection. arXiv.","DOI":"10.1109\/IJCNN52387.2021.9534289"},{"key":"ref_27","first-page":"1022","article-title":"Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning","volume":"28","author":"Gal","year":"2016","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Lyu, Z., Gutierrez, N., Rajguru, A., and Beksi, W.J. (2020). Probabilistic Object Detection via Deep Ensembles, Springer International Publishing.","DOI":"10.1007\/978-3-030-65414-6_7"},{"key":"ref_29","unstructured":"(2020, December 30). False Positive and False Negatives. Available online: https:\/\/en.wikipedia.org\/wiki\/False_positives_and_false_negatives."},{"key":"ref_30","unstructured":"Simonyan, K., and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_32","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Chen, J., Wu, Q., Liu, D., and Xu, T. (2020, January 6\u20138). Foreground-Background Imbalance Problem in Deep Object Detectors: A Review. Proceedings of the 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), Shenzhen, China.","DOI":"10.1109\/MIPR49039.2020.00066"},{"key":"ref_34","unstructured":"Zhou, X., Wang, D., and Kr\u00e4henb\u00fchl, P. (2019). Objects as Points. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Rezatofighi, H., Tsoi, N., Gwak, J.Y., Sadeghian, A., Reid, I., and Savarese, S. (2019, January 15\u201320). Generalized intersection over union: A metric and a loss for bounding box regression. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00075"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/s11263-009-0275-4","article-title":"The pascal visual object classes (VOC) challenge","volume":"88","author":"Everingham","year":"2010","journal-title":"Int. J. Comput. Vis."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13\u201319). nuScenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Lin, Z., Wu, Q., Fu, S., Wang, S., Zhang, Z., and Kong, Y. (2019). Dual-NMS: A method for autonomously removing false detection boxes from aerial image object detection results. Sensors, 19.","DOI":"10.3390\/s19214691"},{"key":"ref_39","first-page":"6452","article-title":"1-CVPR-Adaptive NMS: Refining pedestrian detection in a crowd","volume":"2019","author":"Liu","year":"2019","journal-title":"Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/8\/2819\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:49:00Z","timestamp":1760161740000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/8\/2819"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,16]]},"references-count":39,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["s21082819"],"URL":"https:\/\/doi.org\/10.3390\/s21082819","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,16]]}}}