Coverage estimation of benthic habitat features by semantic segmentation of underwater imagery from South-eastern Baltic reefs using deep learning models

• Autorzy: Šiaulys Andrius , Vaičiukynas Evaldas , Medelytė Saulė , Buškus Kazimieras

Identyfikatory

DOI

10.1016/j.oceano.2023.12.004

• Języki publikacji: EN

Abstrakty

EN

Underwater imagery (UI) is an important and sometimes the only tool for mapping hard-bottom habitats. With the development of new camera systems, from hand-held or simple “drop-down” cameras to ROV/AUV-mounted video systems, video data collection has increased considerably. However, the processing and analysing of vast amounts of imagery can become very labour-intensive, thus making it ineffective both time-wise and financially. This task could be simplified if the processes or their intermediate steps could be done automatically. Luckily, the rise of AI applications for automatic image analysis tasks in the last decade has empowered researchers with robust and effective tools. In this study, two ways to make UI analysis more efficient were tested with eight dominant visual features of the Southeastern Baltic reefs: 1) the simplification of video processing and expert annotation efforts by skipping the video mosaicking step and reducing the number of frames analysed; 2) the application of semantic segmentation of UI using deep learning models. The results showed that the annotation of individual frames provides similar results compared to 2D mosaics; moreover, the reduction of frames by 2–3 times resulted in only minor differences from the baseline. Semantic segmentation using the PSPNet model as the deep learning architecture was extensively evaluated, applying three variants of validation. The accuracy of segmentation, as measured by the intersection-over-union, was mediocre; however, estimates of visual coverage percentages were fair: the difference between the expert annotations and model-predicted segmentation was less than 6–8%, which could be considered an encouraging result.

Słowa kluczowe

EN

underwater video; epibenthos; mosaicking; segmentation; computer vision; PSPNet; ResNet; Baltic Sea

• Wydawca: Polish Academy of Sciences, Institute of Oceanology ; Elsevier
• Czasopismo: Oceanologia
• Rocznik: 2024
• Tom: Vol. 66 (2)
• Strony: 286--298
• Opis fizyczny: Bibliogr. 38 poz., fot., map., rys., tab., wykr.

Twórcy

autor

Šiaulys Andrius

• Marine Research Institute, Klaipeda University, Klaipeda, Lithuania

autor

Vaičiukynas Evaldas

• Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania

autor

Medelytė Saulė

• Marine Research Institute, Klaipeda University, Klaipeda, Lithuania

autor

Buškus Kazimieras

• Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania

Bibliografia

• 1. Alonso, I., Yuval, M., Eyal, G., Treibitz, T., Murillo, A.C., 2019. CoralSeg: Learning coral segmentation from sparse annotations. J. Field Robot. 36 (8), 1456-1477. https://doi.org/10.1002/rob.21915
• 2. Beijbom, O., Edmunds, P.J., Roelfsema, C., Smith, J., Kline, D.I., Neal, B.P., Dunlap, M.J., Moriarty, V., Fan, T.Y., Tan, C.J., Chan, S., 2015. Towards automated annotation of benthic survey images: Variability of human experts and operational modes of automation. PLOS One 10 (7), e0130312. https://doi.org/10.1371/journal.pone.0130312
• 3. Bučas, M., Daunys, D., Olenin, S., 2007. Overgrowth patterns of the red algae Furcellaria lumbricalis at an exposed Baltic Sea coast: the results of a remote underwater video data analysis. Estuar. Coast. Shelf S. 75 (3), 308-316. https://doi.org/10.1016/j.ecss.2007.04.038
• 4. Buhl-Mortensen, L., Buhl-Mortensen, P., Dolan, M.F., Holte, B., 2015. The MAREANO programme—A full coverage mapping of the Norwegian off-shore benthic environment and fauna. Mar. Biol. Res. 11 (1), 4-17. https://doi.org/10.1080/17451000.2014.952312
• 5. Buškus, K., Vaiˇciukynas, E., Verikas, A., Medelyt˙e, S., Šiaulys, A., Šaškov, A., 2021. Automated quantification of brittle stars in seabed imagery using computer vision techniques. Sensors 21 (22), 7598. https://doi.org/10.3390/s21227598
• 6. Casoli, E., Ventura, D., Mancini, G., Pace, D.S., Belluscio, A., Ardizzone, G., 2021. High spatial resolution photo mosaicking for the monitoring of coralligenous reefs. Coral Reefs 40 (4), 1267-1280. https://doi.org/10.1007/s00338- 021- 02136- 4
• 7. Chao, L., Wang, M., 2010. Removal of water scattering. In: International conference on computer engineering and technology IEEE, 2, 35. https://doi.org/10.1109/ICCET.2010.5485339
• 8. Dabuleviciene, T., Kozlov, I.E., Vaiciute, D., Dailidiene, I., 2018. Remote sensing of coastal upwelling in the south-eastern Baltic Sea: Statistical properties and implications for the coastal environment. Remote Sens. 10 (11), 1752. https://doi.org/10.3390/rs10111752
• 9. Eelbode, T., Bertels, J., Berman, M., Vandermeulen, D., Maes, F., Bisschops, R., Blaschko, M.B., 2020. Optimization for medical image segmentation: theory and practice when evaluating with dice score or Jaccard index. IEEE T. Med. Imaging 39 (11), 3679-3690. https://doi.org/10.1109/TMI.2020.3002417
• 10. Elhamifar, E., Sapiro, G., Vidal, R., 2012. See all by looking at a few: Sparse modeling for finding representative objects. In: Proc. CVPR IEEE, 1600-1607. https://doi.org/10.1109/CVPR.2012.6247852
• 11. Ferreira, T., Rasband, W., 2012. ImageJ user guide — IJ 1.46. URL: https://imagej.net/docs/guide/ (accessed on 2 August 2023).
• 12. Gracias, N., Garcia, R., Campos, R., Hurtos, N., Prados, R., Shihavuddin, A.S.M., Nicosevici, T., Elibol, A., Neumann, L., Escartin, J., 2017. Application challenges of underwater vision. In: López, A.M., Imiya, A., Pajdla, T., Álvarez, J.M. (Eds.), Computer Vision in Vehicle Technology: Land, Sea & Air. Wiley, Chichester, 133-160. https://doi.org/10.1002/9781118868065.ch7
• 13. He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep Residual Learning for Image Recognition. Proc. CVPR IEEE, 770-778. https://doi.org/10.1109/CVPR.2016.90
• 14. Islam, M.J., Edge, C., Xiao, Y., Luo, P., Mehtaz, M., Morse, C., Enan, S.S., Sattar, J., 2020. Semantic segmentation of underwater imagery: Dataset and benchmark. IEEE Int. C. Int. Robot. 1769-1776. https://doi.org/10.1109/iros45743.2020.9340821
• 15. Kohler, K.E., Gill, S.M., 2006. Coral Point Count with Excel extensions (CPCe): A Visual Basic program for the determination of coral and substrate coverage using random point count methodology. Comput. Geosci. 32 (9), 1259-1269. https://doi.org/10.1016/j.cageo.2005.11.009
• 16. Labelbox. URL: https://labelbox.com (accessed 2 August 2023).
• 17. Langenkämper, D., Zurowietz, M., Schoening, T., Nattkemper, T.W., 2017. Biigle 2.0-browsing and annotating large marine image collections. Front. Mar. Sci. 4, 83. https://doi.org/10.3389/fmars.2017.00083
• 18. Li, Y., Randall, C.J., van Woesik, R., Ribeiro, E., 2019. Underwater video mosaicing using topology and superpixel-based pairwise stitching. Expert Syst. Appl. 119, 171-183. https://doi.org/10.1016/j.eswa.2018.10.041
• 19. Liu, F., Fang, M., 2020. Semantic segmentation of underwater images based on improved Deeplab. J. Mar. Sci. Eng. 8 (3), 188. https://doi.org/10.3390/jmse8030188
• 20. Martin-Abadal, M., Guerrero-Font, E., Bonin-Font, F., Gonzalez-Cid, Y., 2018. Deep semantic segmentation in an AUV for on-line Posidonia oceanica meadows identification. IEEE Access 6, 60956-60967. https://doi.org/10.1109/ACCESS.2018.2875412
• 21. Medelyte, S., Šiaulys, A., Daunys, D., Włodarska-Kowalczuk, M., W˛esławski, J.M., Olenin, S., 2022a. Application of underwater imagery for the description of upper sublittoral benthic communities in glaciated and ice-free Arctic fjords. Polar Biol. 45 (12), 1655-1671. https://doi.org/10.1007/s00300- 022- 03096- 3
• 22. Medelyte, S., Šiaulys, A., Vaiciukynas, E., Buškus, K., Šaškov, A., Olenin, S., 2022b. A fully-annotated imagery dataset of sublittoral benthic species in South Eastern Baltic Sea reefs. Mendeley Data, V1. https://doi.org/10.17632/wsd42v8mk5.1
• 23. Moghaddam, S.H.A., Mokhtarzade, M., Beirami, B.A., 2020. A feature extraction method based on spectral segmentation and integration of hyperspectral images. Int. J. Appl. Earth Obs. 89, 102097. https://doi.org/10.1016/j.jag.2020.102097
• 24. Piechaud, N., Hunt, C., Culverhouse, P.F., Foster, N.L., Howell, K.L., 2019. Automated identification of benthic epifauna with computer vision. Mar. Ecol. Prog. Ser. 615, 15-30. https://doi.org/10.3354/meps12925
• 25. Reeves, B.R., Dowty, P.R., Wyllie-Echeverria, S., Berry, H.D., 2007. Classifying the seagrass Zostera marina L. from underwater video: an assessment of sampling variation. J. Mar. Environ. Eng. 9 (1), 1-15.
• 26. Rimavičius, T., Gelžinis, A., Verikas, A., Vaiˇciukynas, E., Bačauskien˙e, M., Šaškov, A., 2018. Automatic benthic imagery recognition using a hierarchical two-stage approach. Signal Image Video P 12 (6), 1107-1114. https://doi.org/10.1007/s11760- 018- 1262- 4
• 27. Rzhanov, Y., Mayer, L., 2004. Deep-sea image processing, 647-652, Oceans ’04 MTS/IEEE Techno-Ocean ’04 (IEEE Cat. No.04CH37600). https://doi.org/10.1109/OCEANS.2004.1405498
• 28. Šaškov, A., Dahlgren, T.G., Rzhanov, Y., Schläppy, M.L., 2015. Comparison of manual and semi-automatic underwater imagery analyses for monitoring of benthic hard-bottom organisms at offshore renewable energy installations. Hydrobiologia 756, 139-153. https://doi.org/10.1007/s10750- 014- 2072- 5
• 29. Šiaulys, A., Vaiˇciukynas, E., Medelyt˙e, S., Olenin, S., Šaškov, A., Buškus, K., Verikas, A., 2021. A fully-annotated imagery dataset of sublittoral benthic species in Svalbard. Arctic. Data Br. 35, 106823. https://doi.org/10.1016/j.dib.2021.106823
• 30. Smith Menandro, P., Cardoso Bastos, A., 2020. Seabed mapping: A brief history from meaningful words. Geosciences 10 (7), 273. https://doi.org/10.3390/geosciences10070273
• 31. Tan, M., Le, Q., 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, 6105-6114. https://doi.org/10.48550/arXiv.1905.11946
• 32. Tomar, S., 2006. Converting video formats with FFmpeg. Linux J. 2006 (146), 10. https://www.linuxjournal.com/article/8517
• 33. Trygonis, V., Sini, M., 2012. photoQuad: a dedicated seabed image processing software, and a comparative error analysis of four photoquadrat methods. J. Exp. Mar. Biol. Ecol. 424 (425), 99-108. https://doi.org/10.1016/j.jembe.2012.04.018
• 34. Urra, J., Palomino, D., Lozano, P., González-García, E., Farias, C., Mateo-Ramírez, Á., Fernández-Salas, L.M., López-González, N., Vila, Y., Orejas, C., Puerta, P., 2021. Deep-sea habitat characterization using acoustic data and underwater imagery in Gazul mud volcano (Gulf of Cádiz, NE Atlantic). Deep-Sea Res. Pt. I 169, 103458. https://doi.org/10.1016/j.dsr.2020.103458
• 35. Vaičiūt˙e, D., Bucas, M., Bresciani, M., 2012. Validation of MERIS bio-optical products with in situ data in the turbid Lithuanian Baltic Sea coastal waters. J. Appl. Remote Sens. 6 (1), 063568. https://doi.org/10.1117/1.JRS.6.063568
• 36. Wentworth, C.K., 1922. A scale of grade and class terms for clastic sediments. J. Geol. 30 (5), 377-392. https://doi.org/10.1086/622910
• 37. Yakubovskiy, P., 2019. Segmentation Models. GitHub repository.URL: https://github.com/qubvel/segmentation_models (accessed on 2 August 2023).
• 38. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J., 2017. Pyramid Scene Parsing Network. Proc. CVPR IEEE 6230-6239. https://doi.org/10.1109/CVPR.2017.660