Dual Attention Graph Convolutional Neural Network to Support Mocap Data Animation

• Autorzy: Skublewska-Paszkowska Maria , Powroźnik Paweł , Barszcz Marcin , Dziedzic Krzysztof

Identyfikatory

DOI

10.12913/22998624/171592

• Języki publikacji: EN

Abstrakty

EN

The analysis of movements is one of the notable applications within the field of computer animation. Sophisticated motion capture techniques allow to acquire motion and store it in a digital form for further analysis. The combination of these two aspects of computer vision enables the presentation of data in an accessible way for the user. The primary objective of this study is to introduce an artificial intelligence-based system for animating tennis motion capture data. The Dual Attention Graph Convolutional Network was applied. Its unique approach consists of two attention modules, one for body analysis and the other for tennis racket alignment. The input to the classifier is a sequence of three dimensional data generated from the Mocap system and containing an object of a player holding a tennis racket and presenting fundamental tennis hits, which are classified with great success, reaching a maximum accuracy over 95%. The recognised movements are further processed using dedicated software. Movement sequences are assigned to the tennis player's 3D digital model. In this way, realistic character animations are obtained, reflecting the recognised moves that can be further applied in movies, video games and other visual projects.

Słowa kluczowe

EN

Tennis strokes recognition; graph networks; convolutional networks; computer animation

• Wydawca: Lublin University of Technology ; Polish Society of Ecological Engineering (PTIE), Branch of PTIE in Lublin
• Czasopismo: Advances in Science and Technology. Research Journal
• Rocznik: 2023
• Tom: Vol. 17, no 5
• Strony: 313--325
• Opis fizyczny: Bibliogr. 44 poz., fig., tab.

Twórcy

autor

Skublewska-Paszkowska Maria

• Faculty of Electrical Engineering and Computer Science, Department of Computer Science, Lublin University of Technology, ul. Nadbystrzycka 38 D, 20-618 Lublin, Poland

• https://orcid.org/0000-0002-0760-7126

autor

Powroźnik Paweł

• Faculty of Electrical Engineering and Computer Science, Department of Computer Science, Lublin University of Technology, ul. Nadbystrzycka 38 D, 20-618 Lublin, Poland

• https://orcid.org/0000-0002-5705-4785

autor

Barszcz Marcin

• Faculty of Electrical Engineering and Computer Science, Department of Computer Science, Lublin University of Technology, ul. Nadbystrzycka 38 D, 20-618 Lublin, Poland

• https://orcid.org/0000-0002-9061-4414

autor

Dziedzic Krzysztof

• Faculty of Electrical Engineering and Computer Science, Department of Computer Science, Lublin University of Technology, ul. Nadbystrzycka 38 D, 20-618 Lublin, Poland

• https://orcid.org/0000-0003-4909-9980

Bibliografia

• 1. Kadu H., Kuo C.C.J. Automatic human mocap data classification. IEEE Transactions on Multime-dia 2014; 16(8): 2191-2202.
• 2. Valencia-Marin C.K., Pulgarin-Giraldo J. D., Velasquez-Martinez L. F., Alvarez-Meza A. M., Castellanos-Dominguez G. An Enhanced Joint Hilbert Embedding-Based Metric to Support Mocap Data Classification with Preserved Interpretability. Sensors 2021; 21(13): 4443.
• 3. Shi M., Aberman, K., Aristidou A., Komura T., Lischinski D., Cohen-Or D., Chen B. Motionet: 3d human motion reconstruction from monocular video with skeleton consistency. ACM Transactions on Graphics (TOG) 2020; 40(1): 1-15.
• 4. Derlatka M., Borowska M., Ensemble of heterogeneous base classifiers for human gait recognition. Sensors 2023; 23(1): 508.
• 5. Kumar M.T.K., Kishore P.V.V., Madhav B.T.P., Kumar D.A., Kala N.S., Rao K.P.K., Prasad B. Can Skeletal Joint Positional Ordering Influence Action Recognition on Spectrally Graded CNNs: A Perspective on Achieving Joint Order Independent Learning. IEEE Access 2021; 9: 139611-139626.
• 6. Aggarwal J.K., Xia L. Human activity recognition from 3d data: A review. Pattern Recognition Letters 2014; 48: 70-80.
• 7. Sun Z., Ke Q., Rahmani H., Bennamoun M., Wang G., Liu J. Human action recognition from various data modalities: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence 2022; 45(3): 3200-3225.
• 8. Ma C., Fan J., Yao J., Zhang T. NPU RGB+ D Dataset and a Feature-Enhanced LSTM-DGCN Method for Action Recognition of Basketball Players. Applied Sciences 2021; 11(10): 4426.
• 9. Qi M., Wang Y., Li A., Luo J. Sports video captioning via attentive motion representation and group relationship modeling. IEEE Transactions on Circuits and Systems for Video Technology 2019; 30(8): 2617-2633.
• 10. Guo J., Liu H., Li X., Xu D., Zhang Y. An attention enhanced spatial–temporal graph convolutional LSTM network for action recognition in Karate. Applied Sciences 2021; 11(18): 8641.
• 11. Martinez B., Modolo D., Xiong Y., Tighe J. Action recognition with spatial-temporal discriminative filter banks. In: Proc. of the IEEE/CVF international conference on computer vision, Seul, Korea 2019, 5482-5491.
• 12. Nan M., Trăscău M., Florea A.M., Iacob C.C. Comparison between recurrent networks and temporal convolutional networks approaches for skeleton based action recognition. Sensors 2021; 21(6): 2051.
• 13. Yan S., Xiong Y., Lin D. Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proc. of the AAAI conference on artificial intelligence, New Orleans, USA 2018, 32 (1).
• 14. Chen Y., Tian Y., He M. Monocular human pose estimation: A survey of deep learning-based methods. Computer Vision and Image Understanding 2020, 192, #102897.
• 15. Liu J., Che Y. Action recognition for sports video analysis using part-attention spatio-temporal graph convolutional network. Journal of Electronic Imaging 2021; 30(3): 033017-033017.
• 16. Skublewska-Paszkowska M., Powroznik P. Temporal Pattern Attention for Multivariate Time Series of Tennis Strokes Classification. Sensors 2023; 23(5): 2422.
• 17. Mora S.V., Knottenbelt W.J. Deep learning for domain-specific action recognition in tennis. In: Proc. of the IEEE conference on computer vision and pattern recognition workshops, Honolulu, HI, USA, 2017, 114-122.
• 18. Mora S. V. Computer Vision and Machine Learning for In-Play Tennis Analysis: Framework, Algorithms and Implementation, Doctoral dissertation, Imperial College London, 2017.
• 19. Vainstein J., Manera J.F., Negri P., Delrieux C., Maguitman A. Modeling video activity with dynamic phrases and its application to action recognition in tennis videos. In: Proc. of 19th Iberoamerican Congress, CIARP 2014, Puerto Vallarta, Mexico 2014, 909-916.
• 20. Cai J., Hu J., Tang X., Hung T.Y., Tan Y.P. Deep historical long short-term memory network for action recognition. Neurocomputing 2020; 407: 428-438.
• 21. Ó Conaire C., Connaghan D., Kelly P., O’Connor N.E., Gaffney M., Buckley J. Combining inertial and visual sensing for human action recognition in tennis. In: Proc. of the first ACM international workshop on Analysis and retrieval of tracked events and motion in imagery streams, Firenze, Italy 2010, 51-56.
• 22. Skublewska-Paszkowska M., Powroznik P., Lukasik E. Learning three dimensional tennis shots using graph convolutional networks. Sensors 2020; 20(21): 6094.
• 23. Skublewska-Paszkowska M., Powroznik P., Karczmarek P., Lukasik E. Aggregation of Tennis Ground- strokes on the Basis of the Choquet Integral and Its Generalizations. In: Proc. of the 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Padua, Italy 2022.
• 24. Skublewska-Paszkowska M., Powroznik P., Lukasik E. Attention Temporal Graph Convolutional Network for Tennis Groundstrokes Phases Classification, In: Proc. of the 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Padua, Italy 2022.
• 25. de Souza C.R., Gaidon A., Cabon Y., Murray N., López A.M. Generating human action videos by coupling 3D game engines and probabilistic graphical models. International Journal of Computer Vision 2020; 128(5): 1505-1536.
• 26. Wang H. Research on design and implementation of computer 3D table tennis simulation animaion. Journal of Physics: Conference Series 2021; 1744(3): 032088.
• 27. Yang X. Table Tennis Animation Simulation Aided Teaching System based on Box Particle Filter Tracking. In: Proc. of the 2nd International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India 2021, 926-929.
• 28. Oshita M. Motion Volume: Visualization of Human Motion Manifolds. In: Proc. of the 17th International Conference on Virtual-Reality Continuum and its Applications in Industry, Brisbane QLD, Australia 2019, 1-7.
• 29. Ida H., Fukuhara K., Ishii M., Inoue T. Anticipatory judgements associated with vision of an opponent’s end-effector: An approach by motion perturbation and spatial occlusion. Quarterly Journal of Experimental Psychology 2019; 72(5): 1131-1140.
• 30. Loffing F., Sölter F., Hagemann N., Strauss B. Oncourt position and handedness in visual anticipation of stroke direction in tennis. Psychology of Sport and Exercise 2016; 27: 195-204.
• 31. Song P., Xu S., Fong W.T., Chin C.L., Chua G.G., Huang Z. An immersive VR system for sports education. IEICE TRANSACTIONS on Information and Systems 2012; 95(5): 1324-1331.
• 32. Chen F., Pan S., Jiang J., Huo H., Long G. DAGCN: dual attention graph convolutional networks. In: Proc. of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary 2019, 1-8.
• 33. Zhao L., Song Y., Zhang C., Liu Y., Wang P., Lin T., Li H. T-gcn: A temporal graph convolutional network for traffic prediction. IEEE transactions on inteligent transportation systems 2019; 21(9): 3848-3858.
• 34. Bai J., Zhu J., Song Y., Zhao L., Hou Z., Du R., Li H. A3t-gcn: Attention temporal graph convolutional network for traffic forecasting. ISPRS International Journal of Geo-Information 2021; 10(7).
• 35. Ganser A., Hollaus B., Stabinger S. Classification of tennis shots with a neural network approach. Sensors 2021; 21(17).
• 36. Ma K. A real time artificial intelligent system for tennis swing classification. In: Proc of 2021 IEEE 19th World Symposium on Applied Machine Intelligence and Informatics (SAMI), Herl’any, Slovakia 2021.
• 37. Pei W., Wang J., Xu X., Wu Z., Du X. An embedded 6-axis sensor based recognition for tennis stroke. In: Proc. of 2017 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV 2017, 55-58.
• 38. Ullah M., Mudassar Yamin M., Mohammed A., Daud Khan S., Ullah H., Alaya Cheikh F. Attention-based LSTM network for action recognition in sports. Electronic Imaging 2021; 2021(6): 302-1.
• 39. Gourgari S., Goudelis G., Karpouzis K., Kollias S. Thetis: Three dimensional tennis shots a human action dataset. In: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Portland, Oregon, USA 2013, pp. 676-681.
• 40. FarajiDavar N., De Campos T., Kittler J., Yan F. Transductive transfer learning for action recognition in tennis games. In: Proc. of 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain 2021, 1548-1553.
• 41. Zhu G., Xu C., Huang Q., Gao W., Xing L. Player action recognition in broadcast tennis video with applications to semantic analysis of sports game. In: Proc. of the 14th ACM international conference on Multimedia, anta Barbara, CA, United States 2006, 431-440.
• 42. Zhu G., Xu C., Gao W., Huang Q. Action recognition in broadcast tennis video using optical flow and support vector machine. In: Proc. of the Computer Vision in Human-Computer Interaction: ECCV 2006 Workshop on HCI, Graz, Austria, 2006, 89-98.
• 43. Zhu G., Xu C., Huang Q., Gao W. Action recognition in broadcast tennis video. In: Proc. of the 18th International Conference on Pattern Recognition (ICPR’06), Hong Kong, China 2006, vol. 1, 251-254.
• 44. Dziedzic K., Barszcz M., Paśnikowska-Łukaszuk M., Jankowska A. The role of computer animation in teaching technical subjects. Advances in Science and Technology Research Journal 2015; 9(28): 134-138.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).