{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,21]],"date-time":"2025-12-21T07:11:47Z","timestamp":1766301107100,"version":"build-2065373602"},"reference-count":40,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,4,8]],"date-time":"2021-04-08T00:00:00Z","timestamp":1617840000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["HR0011-17-2-0045"],"award-info":[{"award-number":["HR0011-17-2-0045"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Deep Neural Network (DNN) systems tend to produce overconfident or uncalibrated outputs. This poses problems for active sensor systems that have a DNN module as the main feedback controller. In this paper, we study a closed-loop feedback smart camera from the lens of uncertainty estimation. The uncertainty of the task output is used to characterize and facilitate the feedback operation. The DNN uncertainty in the feedback system is estimated and characterized using both sampling and non-sampling based methods. In addition, we propose a closed-loop control that incorporates uncertainty information when providing feedback. We show two modes of control, one that prioritizes false positives and one that prioritizes false negatives, and a hybrid approach combining the two. We apply the uncertainty-driven control to the tasks of object detection, object tracking, and action detection. The hybrid system improves object detection and tracking accuracy on the CAMEL dataset by 1.1% each respectively. For the action detection task, the hybrid approach improves accuracy by 1.4%.<\/jats:p>","DOI":"10.3390\/s21082610","type":"journal-article","created":{"date-parts":[[2021,4,8]],"date-time":"2021-04-08T21:27:44Z","timestamp":1617917264000},"page":"2610","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["A Task-Driven Feedback Imager with Uncertainty Driven Hybrid Control"],"prefix":"10.3390","volume":"21","author":[{"given":"Burhan A.","family":"Mudassar","sequence":"first","affiliation":[{"name":"School of ECE, Georgia Institute of Technology, Atlanta, GA 30332, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Priyabrata","family":"Saha","sequence":"additional","affiliation":[{"name":"School of ECE, Georgia Institute of Technology, Atlanta, GA 30332, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marilyn","family":"Wolf","sequence":"additional","affiliation":[{"name":"Deparment of CSE, University of Nebraska-Lincoln, Lincoln, NE 68588, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Saibal","family":"Mukhopadhyay","sequence":"additional","affiliation":[{"name":"School of ECE, Georgia Institute of Technology, Atlanta, GA 30332, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,4,8]]},"reference":[{"unstructured":"Chalimbaud, P., and Berry, F. (2004, January 6\u20138). Design of an imaging system based on FPGA technology and CMOS imager. Proceedings of the 2004 IEEE International Conference on Field-Programmable Technology (IEEE Cat. No. 04EX921), Brisbane, QLD, Australia.","key":"ref_1"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1565","DOI":"10.1109\/JPROC.2008.928742","article-title":"An introduction to distributed smart cameras","volume":"96","author":"Rinner","year":"2008","journal-title":"Proc. IEEE"},{"doi-asserted-by":"crossref","unstructured":"Saha, P., Mudassar, B.A., and Mukhopadhyay, S. (2018, January 27\u201330). Adaptive control of camera modality with deep neural network-based feedback for efficient object tracking. Proceedings of the 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Auckland, New Zealand.","key":"ref_3","DOI":"10.1109\/AVSS.2018.8639423"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1109\/JETCAS.2019.2935207","article-title":"CAMEL: An Adaptive Camera With Embedded Machine Learning-Based Sensor Parameter Control","volume":"9","author":"Mudassar","year":"2019","journal-title":"IEEE J. Emerg. Sel. Top. Circuits Syst."},{"doi-asserted-by":"crossref","unstructured":"Feng, D., Rosenbaum, L., Timm, F., and Dietmayer, K. (2018). Leveraging heteroscedastic aleatoric uncertainties for robust real-time lidar 3d object detection. arXiv.","key":"ref_5","DOI":"10.1109\/IVS.2019.8814046"},{"doi-asserted-by":"crossref","unstructured":"Le, M.T., Diehl, F., Brunner, T., and Knol, A. (2018, January 4\u20137). Uncertainty Estimation for Deep Neural Object Detectors in Safety-Critical Applications. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.","key":"ref_6","DOI":"10.1109\/ITSC.2018.8569637"},{"unstructured":"Henaff, M., Canziani, A., and LeCun, Y. (2019). Model-predictive policy learning with uncertainty regularization for driving in dense traffic. arXiv.","key":"ref_7"},{"unstructured":"Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017). On calibration of modern neural networks. arXiv.","key":"ref_8"},{"unstructured":"Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. (2015). Weight uncertainty in neural networks. arXiv.","key":"ref_9"},{"unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 19\u201324). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the International Conference on Machine Learning, New York, NY, USA.","key":"ref_10"},{"unstructured":"Kendall, A., and Gal, Y. (2017, January 21\u201326). What uncertainties do we need in bayesian deep learning for computer vision?. Proceedings of the 2017 IEEE Conference on Advances in Neural Information Processing Systems, Honolulu, HI, USA.","key":"ref_11"},{"doi-asserted-by":"crossref","unstructured":"Mudassar, B.A., Saha, P., and Mukhopadhyay, S. (2020, January 25\u201328). Uncertainty Characterization in Active Sensor Systems with DNN-Based Feedback Control. Proceedings of the IEEE SENSORS, Rotterdam, The Netherlands.","key":"ref_12","DOI":"10.1109\/SENSORS47125.2020.9278702"},{"doi-asserted-by":"crossref","unstructured":"Gebhardt, E., and Wolf, M. (2018, January 27\u201330). CAMEL Dataset for Visual and Thermal Infrared Multiple Object Detection and Tracking. Proceedings of the 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Auckland, New Zealand.","key":"ref_13","DOI":"10.1109\/AVSS.2018.8639094"},{"unstructured":"Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv.","key":"ref_14"},{"doi-asserted-by":"crossref","unstructured":"Wells, J.W., Natarajan, J., Chatterjee, A., and Barlas, I. (2012, January 7\u201311). Real-Time, Content Aware Camera\u2013Algorithm\u2013Hardware Co-Adaptation for Minimal Power Video Encoding. Proceedings of the 2012 25th International Conference on VLSI Design, Hyderabad, India.","key":"ref_15","DOI":"10.1109\/VLSID.2012.78"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"578","DOI":"10.1109\/JETCAS.2018.2859218","article-title":"Content-aware low-complexity object detection for tracking using adaptive compressed sensing","volume":"8","author":"Wells","year":"2018","journal-title":"IEEE J. Emerg. Sel. Top. Circuits Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1077","DOI":"10.1109\/TCSVT.2016.2527303","article-title":"Error-Resilient Video Encoding Using Parallel Independent Signature Processing","volume":"27","author":"Wells","year":"2016","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"doi-asserted-by":"crossref","unstructured":"Ko, J.H., Na, T., and Mukhopadhyay, S. (2016, January 23\u201326). An energy-efficient wireless video sensor node with a region-of-interest based multi-parameter rate controller for moving object surveillance. Proceedings of the 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Colorado Springs, CO, USA.","key":"ref_18","DOI":"10.1109\/AVSS.2016.7738054"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4333","DOI":"10.1109\/JSEN.2019.2962834","article-title":"Multispectral Information Fusion With Reinforcement Learning for Object Tracking in IoT Edge Devices","volume":"20","author":"Saha","year":"2019","journal-title":"IEEE Sens. J."},{"doi-asserted-by":"crossref","unstructured":"Mukherjee, M., Mudassar, B.A., Lee, M., and Mukhopadhyay, S. (2020, January 25\u201328). Algorithm-Circuit Cross-layer Control for DigitalPixel Image Sensors. Proceedings of the 2020 IEEE SENSORS, Rotterdam, The Netherlands.","key":"ref_20","DOI":"10.1109\/SENSORS47125.2020.9278588"},{"unstructured":"Miller, D., S\u00fcnderhauf, N., Zhang, H., Hall, D., and Dayoub, F. (2019, January 16\u201320). Benchmarking Sampling-based Probabilistic Object Detectors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.","key":"ref_21"},{"unstructured":"Hall, D., Dayoub, F., Skinner, J., Corke, P., Carneiro, G., and S\u00fcnderhauf, N. (2018). Probability-based detection quality (pdq): A probabilistic approach to detection evaluation. arXiv.","key":"ref_22"},{"doi-asserted-by":"crossref","unstructured":"Miller, D., Nicholson, L., Dayoub, F., and S\u00fcnderhauf, N. (2018, January 21\u201325). Dropout sampling for robust object detection in open-set conditions. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.","key":"ref_23","DOI":"10.1109\/ICRA.2018.8460700"},{"doi-asserted-by":"crossref","unstructured":"Miller, D., Dayoub, F., Milford, M., and S\u00fcnderhauf, N. (2018). Evaluating merging strategies for sampling-based uncertainty techniques in object detection. arXiv.","key":"ref_24","DOI":"10.1109\/ICRA.2019.8793821"},{"doi-asserted-by":"crossref","unstructured":"Harakeh, A., Smart, M., and Waslander, S.L. (2019). BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors. arXiv.","key":"ref_25","DOI":"10.1109\/ICRA40945.2020.9196544"},{"doi-asserted-by":"crossref","unstructured":"He, Y., Zhu, C., Wang, J., Savvides, M., and Zhang, X. (2019, January 15\u201320). Bounding box regression with uncertainty for accurate object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","key":"ref_26","DOI":"10.1109\/CVPR.2019.00300"},{"doi-asserted-by":"crossref","unstructured":"Wirges, S., Reith-Braun, M., Lauer, M., and Stiller, C. (2019). Capturing object detection uncertainty in multi-layer grid maps. arXiv.","key":"ref_27","DOI":"10.1109\/IVS.2019.8814073"},{"unstructured":"Corbi\u00e8re, C., Thome, N., Bar-Hen, A., Cord, M., and P\u00e9rez, P. (2019, January 8\u201314). Addressing Failure Prediction by Learning Model Confidence. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.","key":"ref_28"},{"doi-asserted-by":"crossref","unstructured":"Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 8\u201316). Ssd: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","key":"ref_29","DOI":"10.1007\/978-3-319-46448-0_2"},{"unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.","key":"ref_30"},{"unstructured":"Mudassar, B.A., and Mukhopadhyay, S. (2019, January 9\u201312). Rethinking Convolutional Feature Extraction for Small Object Detection. Proceedings of the British Machine Vision Conference (BMVC), Cardiff, UK.","key":"ref_31"},{"doi-asserted-by":"crossref","unstructured":"Bewley, A., Ge, Z., Ott, L., Ramos, F., and Upcroft, B. (2016, January 25\u201328). Simple online and realtime tracking. Proceedings of the 2016 IEEE International Conference on Image Processing, ICIP 2016, Phoenix, AZ, USA.","key":"ref_32","DOI":"10.1109\/ICIP.2016.7533003"},{"doi-asserted-by":"crossref","unstructured":"Kalogeiton, V., Weinzaepfel, P., Ferrari, V., and Schmid, C. (2017, January 22\u201329). Action tubelet detector for spatio-temporal action localization. Proceedings of the International Conference on Computer Vision (ICCV), Venice, Italy.","key":"ref_33","DOI":"10.1109\/ICCV.2017.472"},{"doi-asserted-by":"crossref","unstructured":"Amir, M., and Mukhopadhyay, S. (2016, January 10\u201313). 3D Stacked High Throughput Pixel Parallel Image Sensor with Integrated ReRAM Based Neural Accelerator. Proceedings of the IEEE SOI-3D-Subthreshold Microelectronics Technology Unified Conference (S3S), Burlingame, CA, USA.","key":"ref_34","DOI":"10.1109\/S3S.2016.7804406"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1038\/nphoton.2017.75","article-title":"Broadband image sensor array based on graphene\u2013CMOS integration","volume":"11","author":"Goossens","year":"2017","journal-title":"Nat. Photonics"},{"unstructured":"Torreli, P., and Bangale, M. (2021, April 05). Measuring Inference Performance of Machine-Learning Frameworks on Edge-Class Devices with the Mlmark Benchmark. Available online: https:\/\/www.eembc.org\/techlit\/articles\/MLMARK-WHITEPAPER-FINAL-1.pdf.","key":"ref_36"},{"unstructured":"Gurau, C., Bewley, A., and Posner, I. (2018). Dropout distillation for efficiently estimating model confidence. arXiv.","key":"ref_37"},{"unstructured":"Mi, L., Wang, H., Tian, Y., and Shavit, N. (2019). Training-free uncertainty estimation for neural networks. arXiv.","key":"ref_38"},{"doi-asserted-by":"crossref","unstructured":"Postels, J., Ferroni, F., Coskun, H., Navab, N., and Tombari, F. (2019, January 27\u201328). Sampling-free epistemic uncertainty estimation using approximated variance propagation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.","key":"ref_39","DOI":"10.1109\/ICCV.2019.00302"},{"unstructured":"Jiang, H., Kim, B., Guan, M., and Gupta, M. (2018, January 3\u20138). To trust or not to trust a classifier. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.","key":"ref_40"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/8\/2610\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:12:04Z","timestamp":1760364724000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/8\/2610"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,8]]},"references-count":40,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["s21082610"],"URL":"https:\/\/doi.org\/10.3390\/s21082610","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,4,8]]}}}