{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T16:52:52Z","timestamp":1780591972006,"version":"3.54.1"},"reference-count":49,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,1,13]],"date-time":"2020-01-13T00:00:00Z","timestamp":1578873600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["DP150104251"],"award-info":[{"award-number":["DP150104251"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["DE120102960"],"award-info":[{"award-number":["DE120102960"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Across the globe, remote image data is rapidly being collected for the assessment of benthic communities from shallow to extremely deep waters on continental slopes to the abyssal seas. Exploiting this data is presently limited by the time it takes for experts to identify organisms found in these images. With this limitation in mind, a large effort has been made globally to introduce automation and machine learning algorithms to accelerate both classification and assessment of marine benthic biota. One major issue lies with organisms that move with swell and currents, such as kelps. This paper presents an automatic hierarchical classification method local binary classification as opposed to the conventional flat classification to classify kelps in images collected by autonomous underwater vehicles. The proposed kelp classification approach exploits learned feature representations extracted from deep residual networks. We show that these generic features outperform the traditional off-the-shelf CNN features and the conventional hand-crafted features. Experiments also demonstrate that the hierarchical classification method outperforms the traditional parallel multi-class classifications by a significant margin (90.0% vs. 57.6% and 77.2% vs. 59.0%) on Benthoz15 and Rottnest datasets respectively. Furthermore, we compare different hierarchical classification approaches and experimentally show that the sibling hierarchical training approach outperforms the inclusive hierarchical approach by a significant margin. We also report an application of our proposed method to study the change in kelp cover over time for annually repeated AUV surveys.<\/jats:p>","DOI":"10.3390\/s20020447","type":"journal-article","created":{"date-parts":[[2020,1,15]],"date-time":"2020-01-15T03:20:22Z","timestamp":1579058422000},"page":"447","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":53,"title":["Automatic Hierarchical Classification of Kelps Using Deep Residual Features"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8553-3935","authenticated-orcid":false,"given":"Ammar","family":"Mahmood","sequence":"first","affiliation":[{"name":"Computer Science and Software Engineering, The University of Western Australia, Crawley, WA 6009, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ana Giraldo","family":"Ospina","sequence":"additional","affiliation":[{"name":"School of Biological Sciences and Oceans Institute, The University of Western Australia, Crawley, WA 6009, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammed","family":"Bennamoun","sequence":"additional","affiliation":[{"name":"Computer Science and Software Engineering, The University of Western Australia, Crawley, WA 6009, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Senjian","family":"An","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University, Bentley, WA 6845, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1557-4907","authenticated-orcid":false,"given":"Ferdous","family":"Sohel","sequence":"additional","affiliation":[{"name":"College of Science, Health, Engineering and Education Murdoch University, Murdoch, WA 6150, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Farid","family":"Boussaid","sequence":"additional","affiliation":[{"name":"Electrical, Electronic and Computer Engineering, The University of Western Australia, Crawley, WA 6009, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Renae","family":"Hovey","sequence":"additional","affiliation":[{"name":"School of Biological Sciences and Oceans Institute, The University of Western Australia, Crawley, WA 6009, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6860-9371","authenticated-orcid":false,"given":"Robert B.","family":"Fisher","sequence":"additional","affiliation":[{"name":"School of Informatics, University of Edinburgh, Edinburgh EH8 9YL, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0276-6064","authenticated-orcid":false,"given":"Gary A.","family":"Kendrick","sequence":"additional","affiliation":[{"name":"School of Biological Sciences and Oceans Institute, The University of Western Australia, Crawley, WA 6009, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1146\/annurev-marine-041911-111611","article-title":"Climate change impacts on marine ecosystems","volume":"4","author":"Doney","year":"2012","journal-title":"Mar. Sci."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1080\/17451000.2011.637561","article-title":"Large-scale shift from sugar kelp (Saccharina latissima) to ephemeral algae along the south and west coast of Norway","volume":"8","author":"Moy","year":"2012","journal-title":"Mar. Biol. Res."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1080\/09670262.2011.617840","article-title":"The retreat of large brown seaweeds on the north coast of Spain: The case of Saccorhiza polyschides","volume":"46","year":"2011","journal-title":"Eur. J. Phycol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.jembe.2011.02.032","article-title":"Climate change cascades: Shifts in oceanography, species\u2019 ranges and subtidal marine community dynamics in eastern Tasmania","volume":"400","author":"Johnson","year":"2011","journal-title":"J. Exp. Mar. Biol. Ecol."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Ridgway, K. (2007). Long-term trend and decadal variability of the southward penetration of the East Australian Current. Geophys. Res. Lett., 34.","DOI":"10.1029\/2007GL030393"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1126\/science.aad8745","article-title":"Climate-driven regime shift of a temperate marine ecosystem","volume":"353","author":"Wernberg","year":"2016","journal-title":"Science"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1071\/MF15232","article-title":"The \u2018Great Southern Reef\u2019: Social, ecological and economic value of Australia\u2019s neglected kelp forests","volume":"67","author":"Bennett","year":"2016","journal-title":"Mar. Freshw. Res."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1109\/MRA.2011.2181772","article-title":"Monitoring of benthic reference sites: Using an autonomous underwater vehicle","volume":"19","author":"Williams","year":"2012","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1108","DOI":"10.1093\/icesjms\/fss082","article-title":"Regional-scale benthic monitoring for ecosystem-based fisheries management (EBFM) using an autonomous underwater vehicle (AUV)","volume":"69","author":"Smale","year":"2012","journal-title":"ICES J. Mar. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Barrett, N., Seiler, J., Anderson, T., Williams, S., Nichol, S., and Hill, S.N. (2010, January 24\u201327). Autonomous Underwater Vehicle (AUV) for mapping marine biodiversity in coastal and shelf waters: Implications for marine management. Proceedings of the OCEANS 2010 IEEE-Sydney, Sydney, Australia.","DOI":"10.1109\/OCEANSSYD.2010.5603860"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bridge, T.C., Ferrari, R., Bryson, M., Hovey, R., Figueira, W.F., Williams, S.B., Pizarro, O., Harborne, A.R., and Byrne, M. (2014). Variable responses of benthic communities to anomalously warm sea temperatures on a high-latitude coral reef. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0113079"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1754","DOI":"10.1016\/j.dsr2.2009.05.020","article-title":"Deep-sea benthic boundary layer communities and food supply: A long-term monitoring strategy","volume":"56","author":"Sherman","year":"2009","journal-title":"Deep Sea Res. Part II Top. Stud. Oceanogr."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1126\/science.1195223","article-title":"Tracking hydrocarbon plume transport and biodegradation at Deepwater Horizon","volume":"330","author":"Camilli","year":"2010","journal-title":"Science"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Marzinelli, E.M., Williams, S.B., Babcock, R.C., Barrett, N.S., Johnson, C.R., Jordan, A., Kendrick, G.A., Pizarro, O.R., Smale, D.A., and Steinberg, P.D. (2015). Large-scale geographic variation in distribution and abundance of Australian deep-water kelp forests. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0118390"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"8766","DOI":"10.1364\/OPEX.13.008766","article-title":"Classification of coral reef images from underwater video using neural networks","volume":"13","author":"Marcos","year":"2005","journal-title":"Opt. Express"},{"key":"ref_16","unstructured":"Denuelle, A., and Dunbabin, M. (2010, January 1\u20133). Kelp detection in highly dynamic environments using texture recognition. Proceedings of the Australasian Conference on Robotics & Automation (ACRA), Brisbane, Australia."},{"key":"ref_17","unstructured":"Bewley, M., Douillard, B., Nourani-Vatani, N., Friedman, A., Pizarro, O., and Williams, S. (2012, January 3\u20135). Automated species detection: An experimental approach to kelp detection from sea-floor AUV images. Proceedings of the Australasian Conference on Robotics and Automation, Wellington, New Zealand."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Beijbom, O., Edmunds, P.J., Kline, D., Mitchell, B.G., and Kriegman, D. (2012, January 16\u201321). Automated annotation of coral reef survey images. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.","DOI":"10.1109\/CVPR.2012.6247798"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mahmood, A., Bennamoun, M., An, S., Sohel, F., Boussaid, F., Hovey, R., Kendrick, G., and Fisher, R. (2016, January 25\u201328). Coral classification with hybrid feature representations. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.","DOI":"10.1109\/ICIP.2016.7532411"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Razavian, A.S., Azizpour, H., Sullivan, J., and Carlsson, S. (2014, January 23\u201328). CNN features off-the-shelf: an astounding baseline for recognition. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Columbus, OH, USA.","DOI":"10.1109\/CVPRW.2014.131"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"157","DOI":"10.4319\/lom.2009.7.157","article-title":"Automated processing of coral reef benthic images","volume":"7","author":"Stokes","year":"2009","journal-title":"Limnol. Oceanogr. Methods"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Pizarro, O., Rigby, P., Johnson-Roberson, M., Williams, S.B., and Colquhoun, J. (2008, January 15\u201318). Towards image-based marine habitat classification. Proceedings of the OCEANS 2008, Quebec, QC, Canada.","DOI":"10.1109\/OCEANS.2008.5152075"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"23166","DOI":"10.1038\/srep23166","article-title":"Improving Automated Annotation of Benthic Survey Images Using Wide-band Fluorescence","volume":"6","author":"Beijbom","year":"2016","journal-title":"Sci. Rep."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Mahmood, A., Bennamoun, M., An, S., Sohel, F., Boussaid, F., Hovey, R., Kendrick, G., and Fisher, R. (2016, January 19\u201323). Automatic annotation of coral reefs using deep learning. Proceedings of the OCEANS 2016 MTS\/IEEE Monterey, Monterey, CA, USA.","DOI":"10.1109\/OCEANS.2016.7761105"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1109\/JOE.2017.2786878","article-title":"Deep image representations for coral image classification","volume":"44","author":"Mahmood","year":"2018","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_27","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Bewley, M., Nourani-Vatani, N., Rao, D., Douillard, B., Pizarro, O., and Williams, S.B. (2015). Hierarchical classification in AUV imagery. Field and Service Robotics, Springer.","DOI":"10.1007\/978-3-319-07488-7_1"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support vector machine","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach. Learn."},{"key":"ref_30","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2020, January 12). Imagenet Classification with Deep Convolutional Neural Networks. Available online: http:\/\/papers.nips.cc\/paper\/4824-imagenet-classification-with-deep-convolutional-neural-networ."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"3573","DOI":"10.1109\/TNNLS.2017.2732482","article-title":"Cost Sensitive Learning of Deep Feature Representations from Imbalanced Data","volume":"29","author":"Khan","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 11\u201314). Identity mappings in deep residual networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46493-0_38"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Mahmood, A., Bennamoun, M., An, S., and Sohel, F. (2017, January 17\u201320). Resfeats: Residual network based features for image classification. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.","DOI":"10.1109\/ICIP.2017.8296551"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"150057","DOI":"10.1038\/sdata.2015.57","article-title":"Australian sea-floor survey data, with images and expert annotations","volume":"2","author":"Bewley","year":"2015","journal-title":"Sci. Data"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1259","DOI":"10.1016\/j.cageo.2005.11.009","article-title":"Coral Point Count with Excel extensions (CPCe): A Visual Basic program for the determination of coral and substrate coverage using random point count methodology","volume":"32","author":"Kohler","year":"2006","journal-title":"Comput. Geosci."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2014, January 6\u201312). Visualizing and understanding convolutional networks. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1007\/s10618-010-0175-9","article-title":"A survey of hierarchical classification across different application domains","volume":"22","author":"Silla","year":"2011","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Vedaldi, A., and Lenc, K. (2015, January 26\u201330). Matconvnet: Convolutional neural networks for matlab. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia.","DOI":"10.1145\/2733373.2807412"},{"key":"ref_39","first-page":"1871","article-title":"LIBLINEAR: A library for large linear classification","volume":"9","author":"Fan","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Azizpour, H., Sharif Razavian, A., Sullivan, J., Maki, A., and Carlsson, S. (2015, January 7\u201312). From generic to specific deep representations for visual recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Boston, MA, USA.","DOI":"10.1109\/CVPRW.2015.7301270"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1528","DOI":"10.1111\/1365-2745.12300","article-title":"Extreme climate events lower resilience of foundation seagrass at edge of biogeographical range","volume":"102","author":"Fraser","year":"2014","journal-title":"J. Ecol."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Giraldo Ospina, A., Kendrick, G.A., and Renae, H. (2019). Depth moderates loss of marine foundation species after and extreme marine heatwave: Will deep temperate reefs act as climate refugia?. under review.","DOI":"10.1098\/rspb.2020.0709"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"16576","DOI":"10.1073\/pnas.0704778104","article-title":"Deep-water kelp refugia as potential hotspots of tropical marine diversity and productivity","volume":"104","author":"Graham","year":"2007","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jembe.2009.05.009","article-title":"Ecology of mesophotic coral reefs","volume":"1","author":"Lesser","year":"2009","journal-title":"J. Exp. Mar. Biol. Ecol."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1007\/s00338-010-0593-6","article-title":"Community ecology of mesophotic coral reef ecosystems","volume":"29","author":"Kahng","year":"2010","journal-title":"Coral Reefs"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Beijbom, O., Edmunds, P.J., Roelfsema, C., Smith, J., Kline, D.I., Neal, B.P., Dunlap, M.J., Moriarty, V., Fan, T.Y., and Tan, C.J. (2015). Towards automated annotation of benthic survey images: Variability of human experts and operational modes of automation. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0130312"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"7122","DOI":"10.1038\/s41598-017-07337-y","article-title":"A diver-operated hyperspectral imaging and topographic surveying system for automated mapping of benthic habitats","volume":"7","author":"Chennu","year":"2017","journal-title":"Sci. Rep."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1016\/j.mio.2016.09.008","article-title":"Automated measurements of fish within a trawl using stereo images from a Camera-Trawl device (CamTrawl)","volume":"17","author":"Williams","year":"2016","journal-title":"Methods Oceanogr."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"4","DOI":"10.4031\/MTSJ.50.1.1","article-title":"Progress in the automated identification, measurement, and counting of fish in underwater image sequences","volume":"50","author":"Shortis","year":"2016","journal-title":"Mar. Technol. Soc. J."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/2\/447\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:33:08Z","timestamp":1760365988000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/2\/447"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,13]]},"references-count":49,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,1]]}},"alternative-id":["s20020447"],"URL":"https:\/\/doi.org\/10.3390\/s20020447","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,13]]}}}