Tracking of dynamic gesture fingertips positionin video sequence

• Autorzy: Grzejszczak Tomasz , Molle Reinhard , Roth Robert

Identyfikatory

DOI

10.24425/acs.2020.132587

• Języki publikacji: EN

Abstrakty

EN

The field of research of this paper combines Human Computer Interface, gesture recognition and fingertips tracking. Most gesture recognition algorithms processing color images are unable to locate folded fingers hidden inside hand contour. With use of hand landmarks detection and localization algorithm, processing directional images, the fingertips are tracked whether they are risen or folded inside the hand contour. The capabilities of the method, repeatibility and accuracy, are tested with use of 3 gestures that are recorded on the USB camera. Fingertips are tracked in gestures presenting a linear movement of an open hand, finger folding into fist and clenched fist movement. In conclusion, a discussion of accuracy in application to HCI is presented.

Słowa kluczowe

EN

hand landmarks; tracking; fingertip detection; hand gesture; gesture recognition; HMI; HCI

• Wydawca: Polish Academy of Sciences, Committee of Automatic Control and Robotics
• Czasopismo: Archives of Control Sciences
• Rocznik: 2020
• Tom: Vol. 30, No. 1
• Strony: 101--122
• Opis fizyczny: Bibliogr. 42 poz., rys., tab., wykr., wzory

Twórcy

autor

Grzejszczak Tomasz

• Silesian University of Technology, Faculty of Automatic Control, Electronics and Computer Science, Akademicka 16, 44-100 Gliwice, Poland

autor

Molle Reinhard

• Chair of Automation, Computer Science University of Wuppertal, Germany

autor

Roth Robert

• Chair of Automation, Computer Science University of Wuppertal, Germany

Bibliografia

• [1] Washef Ahmed, Kunal Chanda, and Soma Mitra: Vision based hand gesture recognition using dynamic time warping for indian sign language, In 2016 International Conference on Information Science (ICIS), pages 120–125. IEEE, 2016.
• [2] Z. Cao, G. Hidalgo,T. Simon, S. E. Wei, andY. Sheikh: Openpose: realtime multi-person 2d pose estimation using part affinity fields, arXiv preprint arXiv:1812.08008, 2018.
• [3] Ankit Chaudhary, J. L. Raheja, Karen Das, and Sonia Raheja: Intelligent approaches to interact with machines using hand gesture recognition in natural way: A survey, CoRR, abs/1303.2292, 2013.
• [4] Ming Jin Cheok, Zaid Omar, and Mohamed Hisham Jaward: A review of hand gesture and sign language recognition techniques, International Journal of Machine Learning and Cybernetics, 10(1) (2019), 131–153.
• [5] Zhiquan Feng, Bo Yang, Yuehui Chen, Yanwei Zheng, Tao Xu, Yi Li, Ting Xu, and Deliang Zhu: Features extraction from hand images based on new detection operators, Pattern Recognition, 44(5) (2011), 1089–1105.
• [6] T. Grzejszczak, A. Gałuszka, M. Niezabitowski, and K. Radlak: Comparison of hand feature points detection methods, In Luis M. Camarinha Matos, Nuno S. Barrento, and Ricardo Mendonca, editors, Technological Innovation for Collective Awareness Systems, pages 167–174, Berlin, Heidelberg, 2014, Springer Berlin Heidelberg.
• [7] T. Grzejszczak, M. Kawulok, and A. Galuszka: Hand landmarks detection and localization in color images, Multimedia Tools and Applications, 75(23) (2016), 16363–16387.
• [8] T. Grzejszczak, J. Nalepa, and M. Kawulok: Real-time wrist localization in hand silhouettes, In Robert Burduk, Konrad Jackowski, Marek Kurzynski, Michal Wozniak, and Andrzej Zolnierek, editors, Proc. International Conference on Computer Recognition Systems CORES 2013, volume 226 of Advances in Intelligent Systems and Computing, pages 439–449, Springer International Publishing, 2013.
• [9] M. Hagara and J. Pucik: Fingertip detection for virtual keyboard based on camera, In Radioelektronika (RADIOELEKTRONIKA), 2013 23rd International Conference, pages 356–360, April 2013.
• [10] G. J. Iddan and G. Yahav: Three-dimensional imaging in the studio and elsewhere, In Three-Dimensional Image Capture and Applications IV, volume 4298, pages 48–56, International Society for Optics and Photonics, 2001.
• [11] Feng Jiang, Shengping Zhang, Shen Wu, Yang Gao, and Debin Zhao: Multi-layered gesture recognition with kinect, In Gesture Recognition, pages 387–416, Springer, 2017.
• [12] M. Kawulok, J. Kawulok, and J. Nalepa: Spatial-based skin detection using discriminative skin-presence features, Pattern Recognition Letters, 41 (2014), 3–13.
• [13] M. Kawulok, J. A. Kawulok, J. Nalepa, and B. Smolka: Self-adaptive algorithm for segmenting skin regions, EURASIP Journal on Advances in Signal Processing, 2014(170) (2014).
• [14] M. Kawulok, J. Kawulok, J. Nalepa, and B. Smolka: Hybrid adaptation for detecting skin in color images, Intelligent Data Analysis, 20(s1) (2016), S121–S139.
• [15] M. Kawulok and J. Nalepa: Support vector machines training data selection using a genetic algorithm, In Georgy Gimel’farb, Edwin Hancock, Atsushi Imiya, Arjan Kuijper, Mineichi Kudo, Shinichiro Omachi, Terry Windeatt, and Keiji Yamada, editors, Structural, Syntactic, and Statistical Pattern Recognition, volume 7626 of Lecture Notes in Computer Science, pages 557–565, Springer Berlin Heidelberg, 2012.
• [16] M. Kawulok, J. Nalepa, and J. Kawulok: Skin detection and segmentation in color images, In M. Emre Celebi and Bogdan Smolka, editors, Advances in Low-Level Color Image Processing, volume 11 of Lecture Notes in Computational Vision and Biomechanics, pages 329–366, Springer Netherlands, 2014.
• [17] M. Kawulok and J. Szymanek: Precise multi-level face detector for advanced analysis of facial images, IET image processing, 6(2) (2012), 95–103.
• [18] S. Kolkur, D. Kalbande, P. Shimpi, C. Bapat, and J. Jatakia: Human skin detection using rgb, hsv and ycbcr color models, arXiv preprint arXiv:1708.02694, 2017.
• [19] Ngoc Le Ba, Sechang Oh, Dennis Sylvester, and Tony Tae-Hyoung Kim: A 256 pixel, 21.6 _w infrared gesture recognition processor for smart devices, Microelectronics Journal, 86 (2019), 49–56.
• [20] Bei Li, Ying Sun, Gongfa Li, Jianyi Kong, Guozhang Jiang, Du Jiang, Bo Tao, Shuang Xu, and Honghai Liu: Gesture recognition based on modified adaptive orthogonal matching pursuit algorithm, Cluster Computing, 22(1) (2019), 503–512, Jan.
• [21] Wen-Jeng Li, Chia-Yeh Hsieh, Li-Fong Lin, andWoei-Chyn Chu: Hand gesture recognition for post-stroke rehabilitation using leap motion. In 2017 International Conference on Applied System Innovation (ICASI), pages 386–388. IEEE, 2017.
• [22] Yi Li: Hand gesture recognition using kinect, In 2012 IEEE International Conference on Computer Science and Automation Engineering, pages 196–199, June 2012.
• [23] Hui Liang, Junsong Yuan, and D. Thalmann: 3D fingertip and palm tracking in depth image sequences, In Proceedings of the 20th ACM International Conference on Multimedia, MM ’12, pages 785–788, New York, NY, USA, 2012, ACM.
• [24] C. Lugaresi, Jiuqiang Tang, Hadon Nash, C. McClanahan, E. Uboweja, M. Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee,Wan-Teh Chang, Wei Hua, M. Georg, and M. Grundmann: Mediapipe: A framework for building perception pipelines, arXiv preprint arXiv:1906.08172, 2019.
• [25] G. Marin, F. Dominio, and P. Zanuttigh: Hand gesture recognition with leap motion and kinect devices, In Image Processing (ICIP), 2014 IEEE International Conference on, pages 1565–1569. IEEE, 2014.
• [26] J. Mazumder, L. N. Nahar, and M. U. Atique: Finger gesture detection and application using hue saturation value, International Journal of Image, Graphics & Signal Processing, 10(8) (2018).
• [27] A. Memo and P. Zanuttigh: Head-mounted gesture controlled interface for human-computer interaction, Multimedia Tools and Applications, 77(1) (2018) 27–53.
• [28] J. Molina, J. A. Pajuelo, and J. M. Martínez:Real-time motion-based hand gestures recognition from time-of-flight video, Journal of Signal Processing Systems, 86(1) (2017) 17–25.
• [29] M. Monisha and P. S. Mohan: A real-time embedded system for human action recognition using template matching, In 2017 IEEE International Conference on Electrical, Instrumentation and Communication Engineering (ICEICE), pages 1–5. IEEE, 2017.
• [30] S. S. Rautaray and A. Agrawal: Vision based hand gesture recognition for human computer interaction: a survey, Artificial Intelligence Review, pages 1–54, 2012.
• [31] Zhou Ren, Jingjing Meng, Junsong Yuan, and Zhengyou Zhang: Robust hand gesture recognition with kinect sensor, In Proceedings of the 19th ACM international conference on Multimedia, pages 759–760, ACM, 2011.
• [32] Zhou Ren, Junsong Yuan, Jingjing Meng, and Zhengyou Zhang: Robust part-based hand gesture recognition using kinect sensor, Multimedia, IEEE Transactions on, 15(5) (2013), 1110–1120, Aug.
• [33] A. S. Shirazi, Y. Abdelrahman, N. Henze, S. Schneegass, M. Khalilbeigi, and A. Schmidt: Exploiting thermal reflection for interactive systems, In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, pages 3483–3492, New York, NY, USA, 2014. ACM.
• [34] Y. Sato, Y. Kobayashi, and H. Koike: Fast tracking of hands and fingertips in infrared images for augmented desk interface, In Automatic Face and Gesture Recognition, 2000, Proceedings, Fourth IEEE International Conference on, pages 462–467, 2000.
• [35] Kabid Hassan Shibly, Samrat Kumar Dey, Md Aminul Islam, and Shahriar Iftekhar Showrav: Design and development of hand gesture based virtual mouse, In 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), pages 1–5. IEEE, 2019.
• [36] M. Sonka, V. Hlavac, and R. Boyle: Image processing, analysis, and machine vision, Cengage Learning, 2014.
• [37] Qinghui Wang, Ying Wang, Fenglin Liu, and Wei Zeng: Hand gesture recognition of arabic numbers using leap motion via deterministic learning, In 2017 36th Chinese Control Conference (CCC), pages 10823–10828, July 2017.
• [38] F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler: Analysis of the accuracy and robustness of the leap motion controller, Sensors, 13(5) (2013), 6380–6393.
• [39] G. Wu and W. Kang: Vision-based fingertip tracking utilizing curvature points clustering and hash model representation, IEEE Transactions on Multimedia, 19(8) (2017), 1730–1741.
• [40] K. Yadav, L. P. Saxena, B. Ahmed, and Y. K. Krishnan: Hand gesture recognition using improved skin and wrist detection algorithms for indian sign, Journal of Network Communications and Emerging Technologies (JNCET), 9(2) (2019), www.jncet.org.
• [41] Qingrui Zhang, Mingqiang Yang, Kidiyo Kpalma, Qinghe Zheng, and Xinxin Zhang: Segmentation of hand posture against complex backgrounds based on saliency and skin colour detection, IAENG International Journal of Computer Science, 45(3) (2018), 435–444.
• [42] Lian Deng and Shuhua Xu: Adaptation of human skin color in various populations, Hereditas, 155(1) (2018).

Uwagi

EN

1. The research was supported by Polish National Science Center under grant 02/010/PBU17/0090 (PBU/29/RAu1/2017/505). The calculations were performed with the use of IT infrastructure of GeCONiI.

PL

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2020).