{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T19:56:47Z","timestamp":1768420607190,"version":"3.49.0"},"reference-count":20,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,6,28]],"date-time":"2022-06-28T00:00:00Z","timestamp":1656374400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Florida Fish &amp; Wildlife Conservation Commission","award":["17008"],"award-info":[{"award-number":["17008"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>Recent advances in deep learning, including the development of AlexNet, Residual Network (ResNet), and transfer learning, offer unprecedented classification accuracy in the field of machine vision. A developing application of deep learning is the automated identification and management of aquatic invasive plants. Classification of submersed aquatic vegetation (SAV) presents a unique challenge, namely, the lack of a single source of sensor data that can produce robust, interpretable images across a variable range of depth, turbidity, and lighting conditions. This paper focuses on the development of a multi-sensor (RGB and hydroacoustic) classification system for SAV that is robust to environmental conditions and combines the strengths of each sensing modality. The detection of invasive Hydrilla verticillata (hydrilla) is the primary goal. Over 5000 aerial RGB and hydroacoustic images were generated from two Florida lakes via an unmanned aerial vehicle and boat-mounted sonar unit, and tagged for neural network training and evaluation. Classes included \u201cHYDR\u201d, containing hydrilla; \u201cNONE\u201d, lacking SAV, and \u201cOTHER\u201d, containing SAV other than hydrilla. Using a transfer learning approach, deep neural networks with the ResNet architecture were individually trained on the RGB and hydroacoustic datasets. Multiple data fusion methodologies were evaluated to ensemble the outputs of these neural networks for optimal classification accuracy. A method incorporating logic and a Monte Carlo dropout approach yielded the best overall classification accuracy (84%), with recall and precision of 84.5% and 77.5%, respectively, for the hydrilla class. The training and ensembling approaches were repeated for a DenseNet model with identical training and testing datasets. The overall classification accuracy was similar between the ResNet and DenseNet models when averaged across all approaches (1.9% higher accuracy for the ResNet vs. the DenseNet).<\/jats:p>","DOI":"10.3390\/robotics11040068","type":"journal-article","created":{"date-parts":[[2022,6,29]],"date-time":"2022-06-29T01:48:38Z","timestamp":1656467318000},"page":"68","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Sensor Fusion with Deep Learning for Autonomous Classification and Management of Aquatic Invasive Plant Species"],"prefix":"10.3390","volume":"11","author":[{"given":"Jackson E.","family":"Perrin","sequence":"first","affiliation":[{"name":"Mechanical and Aerospace Engineering, North Carolina State University, Raleigh, NC 27695, USA"}]},{"given":"Shaphan R.","family":"Jernigan","sequence":"additional","affiliation":[{"name":"Mechanical and Aerospace Engineering, North Carolina State University, Raleigh, NC 27695, USA"}]},{"given":"Jacob D.","family":"Thayer","sequence":"additional","affiliation":[{"name":"Center for Aquatic and Invasive Plants, University of Florida, Gainesville, FL 32653, USA"}]},{"given":"Andrew W.","family":"Howell","sequence":"additional","affiliation":[{"name":"Crop and Soil Sciences, North Carolina State University, Raleigh, NC 27695, USA"}]},{"given":"James K.","family":"Leary","sequence":"additional","affiliation":[{"name":"Center for Aquatic and Invasive Plants, University of Florida, Gainesville, FL 32653, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6601-4814","authenticated-orcid":false,"given":"Gregory D.","family":"Buckner","sequence":"additional","affiliation":[{"name":"Mechanical and Aerospace Engineering, North Carolina State University, Raleigh, NC 27695, USA"}]}],"member":"1968","published-online":{"date-parts":[[2022,6,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/BF02551274","article-title":"Approximation by superpositions of a sigmoidal function","volume":"2","author":"Cybenko","year":"1989","journal-title":"Math. Control Signals Syst."},{"key":"ref_2","first-page":"1106","article-title":"Imagenet classification with deep convolutional neural networks","volume":"5","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_4","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1109\/JPROC.2020.3004555","article-title":"A comprehensive survey on transfer learning","volume":"109","author":"Zhuang","year":"2020","journal-title":"Proc. IEEE"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2058","DOI":"10.1038\/s41598-018-38343-3","article-title":"DeepWeeds: A multiclass weed species image dataset for deep learning","volume":"9","author":"Olsen","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Rostami, M., Kolouri, S., Pilly, P., and McClelland, J. (2020, January 7\u201312). Generative continual concept learning. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i04.6006"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ins.2021.04.062","article-title":"Continual learning in sensor-based human activity recognition: An empirical benchmark analysis","volume":"575","author":"Jha","year":"2021","journal-title":"Inf. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ashfahani, A., and Pratama, M. (2019, January 2\u20134). Autonomous deep learning: Continual learning approach for dynamic environments. Proceedings of the 2019 SIAM International Conference on Data Mining, Calgary, AB, Canada.","DOI":"10.1137\/1.9781611975673.75"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"106423","DOI":"10.1016\/j.asoc.2020.106423","article-title":"Continual learning classification method with new labeled data based on the artificial immune system","volume":"94","author":"Li","year":"2020","journal-title":"Appl. Soft Comput."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Patel, M., Jernigan, S., Richardson, R., Ferguson, S., and Buckner, G. (2019). Autonomous robotics for identification and management of invasive aquatic plant species. Appl. Sci., 9.","DOI":"10.3390\/app9122410"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1132","DOI":"10.1111\/j.1523-1739.2004.00391.x","article-title":"Effect of invasive plant species on temperate wetland plant diversity","volume":"18","author":"Houlahan","year":"2004","journal-title":"Conserv. Biol."},{"key":"ref_13","unstructured":"Gettys, L.A., Haller, W.T., and Petty, D.G. (2014). Biology and control of aquatic plants. A Best Management Practices Handbook: Third Edition, Aquatic Ecosystem Restoration Foundation."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1016\/j.ecolecon.2004.10.002","article-title":"Update on the environmental and economic costs associated with alien-invasive species in the United States","volume":"52","author":"Pimentel","year":"2005","journal-title":"Ecol. Econ."},{"key":"ref_15","unstructured":"Madsen, J.D. (2022, April 27). Point intercept and line intercept methods for aquatic plant management. APCRP Technical Notes Collection (TN APCRP-M1-02). U.S. Army Engineer Research and Development Center, Vicksburg, MS. Available online: https:\/\/apps.dtic.mil\/sti\/citations\/ADA361270."},{"key":"ref_16","unstructured":"Hauxwell, J., Knight, S., Wagner, K., Mikulyuk, A., Nault, M., Porzky, M., and Chase, S. (2010). Recommended Baseline Monitoring of Aquatic Plants in Wisconsin: Sampling Design, Field and Laboratory Procedures, Data Entry and Analysis, and Applications, Wisconsin Department of Natural Resources. PUB SS-1068."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Goyal, P., Girshick, R., He, K., and Doll\u00e1r, P. (2017, January 22\u201329). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.324"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ganaie, M.A., and Hu, M. (2021). Ensemble deep learning: A review. arXiv.","DOI":"10.1016\/j.engappai.2022.105151"},{"key":"ref_19","unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 19\u201324). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the International Conference on Machine Learning, New York City, NY, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Laakom, F., Raitoharju, J., Iosifidis, A., Nikkanen, J., and Gabbouj, M. (2021, January 18\u201322). Monte Carlo Dropout Ensembles for Robust Illumination Estimation. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.","DOI":"10.1109\/IJCNN52387.2021.9534314"}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/11\/4\/68\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:39:49Z","timestamp":1760139589000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/11\/4\/68"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,28]]},"references-count":20,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["robotics11040068"],"URL":"https:\/\/doi.org\/10.3390\/robotics11040068","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,28]]}}}