Application of visual classification algorithms for identification of underwater audio signals

• Autorzy: Gnyś Piotr , Szczęsna Gabriela , Dominguez-Brito Antonio C. , Cabrera-Gámez Jorge

Identyfikatory

DOI

10.34808/wr60-jz83

• Języki publikacji: EN

Abstrakty

EN

An audio processing and classification pipeline is presented in this work. The main focus is on the classification of sounds in a marine acoustic environment, however, the presented approach can be applied to other audio data. Audio samples from heterogeneous sources automatically spliced, normalized and transformed into spectrogram based visual representation are tagged on the pipeline input. The said representation is then used to train a convolutional neural network that can identify the presented categories in future recordings.

Słowa kluczowe

EN

audio processing; audio classification; convolutional neural network

PL

przetwarzanie dźwięku; klasyfikacja dźwięku; splotowa sieć neuronowa

• Wydawca: Politechnika Gdańska
• Czasopismo: TASK Quarterly : scientific bulletin of Academic Computer Centre in Gdansk
• Rocznik: 2022
• Tom: Vol. 26, No 4
• Strony: 1--22
• Opis fizyczny: Bibliogr. 87 poz., rys., tab.

Twórcy

autor

Gnyś Piotr

• Department of Computer Science , Polish-Japanese Academy of Information Technology

• https://orcid.org/0000-0001-7266-9537

autor

Szczęsna Gabriela

• Department of Computer Science , Polish-Japanese Academy of Information Technology

autor

Dominguez-Brito Antonio C.

• Instituto Universitario SIANI and Departamento de Informática y Sistemas Universidad de las Palmas de Gran Canaria

• https://orcid.org/0000-0002-3252-5683

autor

Cabrera-Gámez Jorge

• Instituto Universitario SIANI and Departamento de Informática y Sistemas Universidad de las Palmas de Gran Canaria

• https://orcid.org/0000-0002-0260-4524

Bibliografia

• [1] M. Southworth, “The sonic environment of cities,” Environment and Behavior, vol. 1, 1969.
• [2] R. M. Schafer, The soundscape : our sonic environment and the tuning of the world. Destiny Books, 1994.
• [3] E. D. Franco, P. Pierson, L. D. Iorio, A. Calň, J. M. Cottalorda, B. Derijard, A. D. Franco, A. Galvé, M. Guibbolini, J. Lebrun, F. Micheli, F. Priouzeau, C. R. de Faverney, F. Rossi, C. Sabourault, G. Spennato, P. Verrando, and P. Guidetti, “Effects of marine noise pollution on mediterranean fishes and invertebrates: A review,” Marine Pollution Bulletin, vol. 159, 2020.
• [4] E. Spanier and D. Zviely, “Key environmental impacts along the mediterranean coast of israel in the last 100 years,” 2023.
• [5] A. D. Foote, Y. Liu, G. W. Thomas, T. Vinař, J. Alföldi, J. Deng, S. Dugan, C. E. V. Elk, M. E. Hunter, V. Joshi, Z. Khan, C. Kovar, S. L. Lee, K. Lindblad-Toh, A. Mancia, R. Nielsen, X. Qin, J. Qu, B. J. Raney, N. Vijay, J. B. Wolf, M. W. Hahn, D. M. Muzny, K. C. Worley, M. T. P. Gilbert, and R. A. Gibbs, “Convergent evolution of the genomes of marine mammals,” Nature Genetics, vol. 47, 2015.
• [6] A. Foote, Y. Liu, G. Thomas, T. Vinař, J. Alföldi, J. Deng, S. Dugan, C. Van elk, M. Hunter, V. Joshi, Z. Khan, C. Kovar, S. Lee, K. Lindblad-Toh, A. Mancia, R. Nielsen, X. Qin, J. Qu, B. Raney, and R. Gibbs, “Convergent evolution of the genomes of marine mammals,” Nature Genetics, 01 2015.
• [7] P. J. Clapham, S. B. Young, and R. L. Brownell, “Baleen whales: Conservation issues and the status of the most endangered populations,” Mammal Review, vol. 29, 1999.
• [8] S. Hooker, Toothed Whales, Overview, pp. 1173–1179. 12 2009.
• [9] M. C. Amorim, R. Vasconcelos, and P. Fonseca, Fish Sounds and Mate Choice, vol. 4, pp. 1–33. 03 2015.
• [10] I. Charrier, L. Jeantet, L. Maucourt, S. Régis, N. Lecerf, A. Benhalilou, and C. Damien, “First evidence of underwater vocalisations in green sea turtles chelonia mydas,” Endangered Species Research, vol. 48, pp. 31–41, 05 2022.
• [11] A. T. G. D. P. P. Thiebault A, Charrier I, “First evidence of underwater vocalisations in hunting penguins,” PeerJ, 2019.
• [12] Y. ZHANG, F. SHI, J. SONG, X. ZHANG, and S. YU, “Hearing characteristics of cephalopods: Modeling and environmental impact study,” Integrative Zoology, vol. 10, no. 1, pp. 141–151, 2015.
• [13] M.W. Johnson, F. A. Everest, and R.W. Young, “The role of snapping shrimp (crangon and synalpheus) in the production of underwater noise in the sea,” The Biological Bulletin, vol. 93, no. 2, pp. 122–138, 1947.
• [14] M. J. Vermeij, K. L. Marhaver, C. M. Huijbers, I. Nagelkerken, and S. D. Simpson, “Coral larvae move toward reef sounds,” PloS one, vol. 5, no. 5, p. e10660, 2010.
• [15] D. R. MacAyeal, E. A. Okal, R. C. Aster, and J. Bassis, “Seismic and hydroacoustic tremor generated by colliding icebergs,” Journal of Geophysical Research: Earth Surface, vol. 113, no. F3, 2008.
• [16] R. P. Dziak, E. T. Baker, A. M. Shaw, D. R. Bohnenstiehl, W. W. Chadwick, J. H. Haxel, H. Matsumoto, and S. L. Walker, “Flux measurements of explosive degassing using a yearlong hydroacoustic record at an erupting submarine volcano,” Geochemistry, Geophysics, Geosystems, vol. 13, 2012.
• [17] C. E. Nishimura, Monitoring whales and earthquakes by using SOSUS. Naval Research Laboratory Washington, DC, USA, 1994.
• [18] J. A. Hildebrand, “Impacts of anthropogenic sound,” Marine mammal research: conservation beyond crisis, pp. 101–124, 2005.
• [19] W. M. Zimmer, Passive acoustic monitoring of cetaceans. 2011.
• [20] I. M. Organization, Guidelines for the reduction of underwater noise from commercial shipping to address adverse impacts on marine life, 2014.
• [21] K. J. Reine, D. Clarke, and C. Dickerson, “Characterization of underwater sounds produced by hydraulic and mechanical dredging operations,” The Journal of the Acoustical Society of America, vol. 135, no. 6, pp. 3280–3294, 2014.
• [22] J. C. Wilson, M. Elliott, N. D. Cutts, L. Mander, V. Mendăo, R. Perez-Dominguez, and A. Phelps, “Coastal and offshore wind energy generation: Is it environmentally benign?,” Energies, vol. 3, 2010.
• [23] J. Tougaard, P. T. Madsen, and M. Wahlberg, “Underwater noise from construction and operation of offshore wind farms,” Bioacoustics, vol. 17, 2008.
• [24] P. T. Madsen, M. Wahlberg, J. Tougaard, K. Lucke, and P. Tyack, “Wind turbine underwater noise and marine mammals: implications of current knowledge and data needs,” Marine ecology progress series, vol. 309, pp. 279–295, 2006.
• [25] M. K. Pine, P. Schmitt, R. M. Culloch, L. Lieber, and L. T. Kregting, “Providing ecological context to anthropogenic subsea noise: Assessing listening space reductions of marine mammals from tidal energy devices,” Renewable and Sustainable Energy Reviews, vol. 103, 2019.
• [26] L. Palmer, D. Gillespie, J. D. MacAulay, C. E. Sparling, D. J. Russell, and G. D. Hastie, “Harbour porpoise (phocoena phocoena) presence is reduced during tidal turbine operation,” Aquatic Conservation: Marine and Freshwater Ecosystems, vol. 31, 2021.
• [27] G. Scowcroft, C. Knowlton, H. Morin, and D. Mclntire, “Discovery of sound in the sea,” University of Rhode Island, 2012.
• [28] K. J. Vigness-Raposa, G. Scowcroft, J. H. Miller, D. R. Ketten, and A. N. Popper, “Discovery of sound in the sea: Resources for educators, students, the public, and policymakers,” 2016.
• [29] J. Lynch, “The voices of marine mammals—william e. schevill and william a. watkins: Pioneers in bioacoustics,” The Journal of the Acoustical Society of America, vol. 148, 2020.
• [30] L. Sayigh, M. A. Daher, J. Allen, H. Gordon, K. Joyce, C. Stuhlmann, and P. Tyack, “The watkins marine mammal sound database: An online, freely accessible resource,” 2017.
• [31] D. Lavry, “Sampling theory for digital audio,” Lavry Engineering, Inc. Available online: http://www. lavryengineering.com/documents/Sampling_Theory. pdf (checked 24.5. 2010), 2004.
• [32] E. Por, M. van Kooten, and V. Sarkovic, “Nyquist–shannon sampling theorem,” Leiden University, vol. 1, no. 1, 2019.
• [33] J. Sola and J. Sevilla, “Importance of input data normalization for the application of neural networks to complex industrial problems,” IEEE Transactions on Nuclear Science, vol. 44, 1997.
• [34] “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, 2019.
• [35] L. Nanni, G. Maguolo, and M. Paci, “Data augmentation approaches for improving animal audio classification,” Ecological Informatics, vol. 57, 2020.
• [36] E. M. Stein and R. Shakarchi, Fourier analysis: an introduction, vol. 1. Princeton University Press, 2011.
• [37] H. Medwin and C. S. Clay, Fundamentals of Acoustical Oceanography. Academic Press, San Diego, 1998.
• [38] R. Mersereau and T. Speake, “A unified treatment of cooley-tukey algorithms for the evaluation of the multidimensional dft,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 29, no. 5, pp. 1011–1018, 1981.
• [39] W. Ellermeier, M. Eigenstetter, and K. Zimmer, “Psychoacoustic correlates of individual noise sensitivity,” The journal of the acoustical society of America, vol. 109, no. 4, pp. 1464–1473, 2001.
• [40] K. Rangra and M. Kapoor, “Exploring the mel scale features using supervised learning classifiers for emotion classification,” International Journal of Applied Pattern Recognition, vol. 6, no. 3, pp. 232–253, 2021.
• [41] F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, vol. 65, 1958.
• [42] S. Osowski, Sieci Neuronowe. Oficyna Wydawnicza Politechniki Warszawskiej, 1994.
• [43] W. S. McCulloch and W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” The Bulletin of Mathematical Biophysics, 1943.
• [44] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing coadaptation of feature detectors,” 7 2012.
• [45] “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014.
• [46] Tensorflow API Documentation for Python. [Online]. https://www.tensorflow.org/api_docs/python/tf.
• [47] A. Géron, Hands-On Machine Learing With Scikit-Learn & Tensor Flow. 2017.
• [48] L. Deng, “The mnist database of handwritten digit images for machine learning research,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012.
• [49] J. H. Steele, S. A. Thorpe, and K. K. Turekian, Encyclopedia of ocean sciences. Elsevier ScienceDirect, 2008.
• [50] R. Tadeusiewicz, Problemy Biocybernetyki. 1994.
• [51] F. e. a. Pedregosa, “Scikit learn: Machine learning in Python,” vol. 12, pp. 2825–2830, 2011.
• [52] L.Wyse, “Audio spectrogram representations for processing with convolutional neural networks,” 2017.
• [53] D. Gillespie, D. Mellinger, J. Gordon, D. Mclaren, P. Redmond, R. McHugh, P. Trinder, X.-Y. Deng, and A. Thode, “Pamguard: Semiautomated, open source software for real-time acoustic detection and localization of cetaceans.,” The Journal of the Acoustical Society of America, vol. 125, p. 2547, 05 2009.
• [54] T. P. Zieliński, Cyfrowe przetwarzanie sygnałów - od teorii do zastosowań. 2 ed., 2005.
• [55] H. A. Ghani, M. R. A. Malek, M. F. K. Azmi, M. J. Muril, and A. Azizan, “A review on sparse fast fourier transform applications in image processing,” 2020.
• [56] S. S. Stevens, J. Volkmann, and E. B. Newman, “A scale for the measurement of the psychological magnitude pitch,” Journal of the Acoustical Society of America, vol. 8, 1937.
• [57] J. A. Hildebrand, “Anthropogenic and natural sources of ambient noise in the ocean,” Marine Ecology Progress Series, vol. 395, 2009.
• [58] I. Herrera, M. Carrillo, M. C. de Esteban, and R. Haroun, “Distribution of cetaceans in the canary islands (northeast atlantic ocean): Implications for the natura 2000 network and future conservation measures,” Frontiers in Marine Science, vol. 8, 2021.
• [59] E. C. M. Parsons, “The negative impacts of whale-watching,” Journal of Marine Biology, vol. 2012, 2012.
• [60] S. Baulch and C. Perry, “Evaluating the impacts of marine debris on cetaceans,” Marine Pollution Bulletin, vol. 80, 2014.
• [61] A. Maglio, C. Soares, M. Bouzidi, Y. zabel, F. Souami, and G. Pavan, “Mapping shipping noise in the pelagos sanctuary (French part) through acoustic modelling to assess potential impacts on marine mammals.,” Sci. Rep. Port-Cros National Park, pp. 167–185.
• [62] M. Randone, M. Bocci, C. Castellani, C. Laurent, and C. Piante, “Safeguarding marine protected areas in the growing Mediterranean blue economy—recommendations for the maritime transport sector,” International Journal of Design and Nature and Ecodynamics, vol. Volume 14 (2019), Issue 4, no. 9/2020, p. 10.
• [63] W. M. X. Zimmer, Passive Acoustic Monitoring of Cetaceans. Cambridge University Press, 2021.
• [64] I. Herrera, M. Carrillo, and R. Haroun, “Conservación de cetáceos y planificación del espacio marino en las islas canarias,” Okeanos.
• [65] V. Sessions and M. Valtorta, “The effects of data quality on machine learning algorithms,” 2006.
• [66] P. Gnyś, “Mereogeometry based approach for behavioral robotics,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 70–80, 2017.
• [67] P. Gnyś, “Mereogeometry based approach for behavioral robotics,” pp. 70–80, 2017.
• [68] G. P and P. P, “Application of long short term memory neural networks for gps satellite clock bias prediction,” TASK Quarterly, vol. 25, pp. 381–395, 2021.
• [69] J. Cabrera-Gámez, A. C. Domínguez-Brito, F. Santana-Jorge, D. Gamo, D. Jiménez, A. Guerra, and J. J. Castro, “Acoustic detection of tagged angelsharks from an autonomous sailboat,” vol. 1092 AISC, 2020.
• [70] W. W. Chadwick, R. W. Embley, E. T. Baker, J. A. Resing, J. E. Lupton, K. V. Cashman, R. P. Dziak, V. Tunnicliffe, D. A. Butterfield, and Y. Tamura, “Spotlight 10: Northwest rota-1 seamount,” Oceanography, vol. 23, 2010.
• [71] W. W. Chadwick, K. V. Cashman, R. W. Embley, H. Matsumoto, R. P. Dziak, C. E. de Ronde, T. K. Lau, N. D. Deardorff, and S. G. Merle, “Direct video and hydrophone observations of submarine explosive eruptions at nw rota-1 volcano, mariana arc,” Journal of Geophysical Research: Solid Earth, vol. 113, 2008.
• [72] W. Europe, “Offshore statistics in europe.”
• [73] D. Risch, N. van Geel, D. Gillespie, and B. Wilson, “Characterisation of underwater operational sound of a tidal stream turbine,” The Journal of the Acoustical Society of America, vol. 147, 2020.
• [74] M. J. Kennish, Practical handbook of marine science. 2019.
• [75] P. Clapham, “Why do baleen whales migrate?,” Marine Mammal Science, vol. 17, pp. 432–436, 04 2001.
• [76] H. Medwin, C. S. Clay, and S. M. Flatte, “Fundamentals of acoustical oceanography,” Physics Today, vol. 52, 1999.
• [77] H. Urban, Handbook of Underwater Acoustic Engineering. STNAtlas- Elektronik GmbH, 2002.
• [78] J. D. Macaulay and D. Gillespie, “Pamguard: Open-source detection, classification, and localization software,” The Journal of the Acoustical Society of America, vol. 151, 2022.
• [79] T. Webber, D. Gillespie, T. Lewis, J. Gordon, T. Ruchirabha, and K. F. Thompson, “Streamlining analysis methods for large acoustic surveys using automatic detectors with operator validation,” Methods in Ecology and Evolution, vol. 13, 2022.
• [80] B. N. Korkmaz, R. Diamant, G. Danino, and A. Testolin, “Automated detection of dolphin whistles with convolutional networks and transfer learning,” Frontiers in Artificial Intelligence, vol. 6, 2023.
• [81] Microsoft, “Multimedia programming interface and data specifications 1.0,” International Business, 1991.
• [82] H. Fastl and E. Zwicker, Psychoacoustics: Facts and models. 2007.
• [83] K. S. Norris, R. M. Goodman, B. Villa-Ramirez, and L. Hobbs, “Behavior of california gray whale, eschrichtius robustus, in southern baja california, mexico,” Fish. Bull, vol. 75, no. 1, pp. 159–72, 1977.
• [84] C. J. Deutsch, J. P. Reid, R. K. Bonde, D. E. Easton, H. I. Kochman, and T. J. O’Shea, “Seasonal movements, migratory behavior, and site fidelity of west indian manatees along the atlantic coast of the united states,” Wildlife monographs, pp. 1–77, 2003.
• [85] Sub-Committee on Ship Design and Construction (SDC 8), 17-21January 2022, 2022.
• [86] E. Chou, B. L. Southall, M. Robards, and H. C. Rosenbaum, “Inter-national policy, recommendations, actions and mitigation efforts of anthropogenic underwater noise,”Ocean & Coastal Management, vol. 202, p. 105427, 2021.
• [87] J. Gordon, D. Thompson, D. Gillespie, M. Lonergan, S. Calderan,B. Jaffey, and V. Todd, “Assessment of the potential for acousticdeterrents to mitigate the impact on marine mammals of under-water noise arising from the construction of offshore windfarms,”Cowrie Ltd, July, 2007.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).