Hand posture recognition using modified ensemble of shape functions and global radius-based surface descriptor

• Autorzy: Warchoł D.

Identyfikatory

DOI

10.7494/csci.2018.19.2.2718

• Języki publikacji: EN

Abstrakty

EN

This paper presents an approach to the recognition of static hand gestures based on data acquired from 3D cameras and point cloud descriptors: Ensemble of Shape Functions and Global Radius-based Surface Descriptor. We describe a recognition algorithm consisting of hand segmentation, noise removal and downsampling of point clouds, dividing point cloud bounding boxes to cells, feature extraction and normalization, and gesture classification. Modifications to the descriptors are proposed in order to increase the hand posture recognition rates while decreasing the quantity of used features as well as the computational cost of the algorithm. Experiments performed on four challenging datasets using cross-validation tests prove the usefulness of our approach.

Słowa kluczowe

EN

hand posture recognition; point cloud; descriptor; 3D camera

• Wydawca: Wydawnictwa AGH
• Czasopismo: Computer Science
• Rocznik: 2018
• Tom: Vol. 19 (2)
• Strony: 115--137
• Opis fizyczny: Bibliogr. 40 poz., rys., wykr., tab.

Twórcy

autor

Warchoł D.

• Rzeszów University of Technology, Faculty of Electrical and Computer Engineering, Department of Computer and Control Engineering, W. Pola 2, 35-959 Rzeszów, Poland

Bibliografia

• [1] Aldoma A., Marton Z.C., Tombari F., Wohlkinger W., Potthast C., Zeisl B., Rusu R.B., Gedikli S., Vincze M.: Tutorial: Point Cloud Library: Three- Dimensional Object Recognition and 6 DOF Pose Estimation. In: IEEE Robotics and Automation Magazine, vol. 9(3), pp. 80-91, 2012. ISSN 1070-9932. URL http://dx.doi.org/10.1109/MRA.2012.2206675.
• [2] Bay H., Ess A., Tuytelaars T., Gool L.V.: Speeded-Up Robust Features (SURF). In: Computer Vision and Image Understanding, vol. 110(3), pp. 346-359, 2008. ISSN 1077-3142. URL http://dx.doi.org/10.1016/j.cviu.2007.09.014.
• [3] Dalal N., Triggs B.: Histograms of Oriented Gradients for Human Detection. In: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR '05, pp. 886{893. IEEE Computer Society, Washington, DC, USA, 2005. ISBN 0-7695-2372-2. URL http: //dx.doi.org/10.1109/CVPR.2005.177.
• [4] Dominio F., Donadeo M., Zanuttigh P.: Combining Multiple Depth-based Descriptors for Hand Gesture Recognition. In: Pattern Recognition Letters, vol. 50(C), pp. 101-111, 2014. ISSN 0167-8655. URL http://dx.doi.org/10. 1016/j.patrec.2013.10.010.
• [5] Dong C., Leu M.C., Yin Z.: American Sign Language Alphabet Recognition Using Microsoft Kinect. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 44-52. 2015. ISSN 2160-7516. URL http://dx.doi.org/10.1109/CVPRW.2015.7301347.
• [6] Feng B., He F., Wang X.: Depth-Projection-Map-Based Bag of Contour Fragments for Robust Hand Gesture Recognition. In: IEEE Transactions on Human- Machine Systems, vol. PP(99), pp. 1-13, 2016. ISSN 2168-2305. URL http: //dx.doi.org/10.1109/THMS.2016.2616278.
• [7] Flasiński M., Myśliński S.: On the use of graph parsing for recognition of isolated hand postures of Polish Sign Language. In: Pattern Recognition, vol. 43(6), pp. 2249-2264, 2010. ISSN 0031-3203. URL http://dx.doi.org/10.1016/j. patcog.2010.01.004.
• [8] Jeong E., Lee J., Kim D.: Finger-gesture Recognition Glove using Velostat. In: 2011 11th International Conference on Control, Automation and Systems (IC- CAS). IEEE, 2011. ISSN 2093-7121.
• [9] Kapuściński T., Oszust M., Wysocki M.,Warcho l D.: Recognition of Hand Gestures Observed by Depth Cameras. In: International Journal of Advanced Robotic Systems, vol. 12(4), pp. 48-57, 2015. URL http://dx.doi.org/10.5772/60091.
• [10] Keskin C., Kirac F., Kara Y.E., Akarun L.: Real time hand pose estimation using depth sensors. In: ICCV Workshops, pp. 1228-1234. IEEE, 2011. ISBN 978-1-4673-0062-9. URL http://dx.doi.org/10.1109/ICCVW.2011.6130391.
• [11] Kuznetsova A., Leal-Taix L., Rosenhahn B.: Real-time sign language recognition using a consumer depth camera. In: IEEE International Conference on Computer Vision Workshops (ICCVW), 3rd Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV), 2013. URL http://dx.doi.org/10.1109/ICCVW. 2013.18.
• [12] Li Y.: Multi-scenario Gesture Recognition Using Kinect. In: Proceedings of the 2012 17th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGAMES), CGAMES '12, pp. 126-130. IEEE, Washington, DC, USA, 2012. ISBN 978-1-4673- 1120-5. URL http://dx.doi.org/10.1109/CGames.2012.6314563.
• [13] Luis-Prez F.E., Trujillo-Romero F., Martnez-Velazco W.: Control of a Service Robot Using the Mexican Sign Language. In: I.Z. Batyrshin, G. Sidorov, eds., MICAI (2), Lecture Notes in Computer Science, vol. 7095, pp. 419-430. Springer, 2011. ISBN 978-3-642-25329-4.
• [14] Marin G., Dominio F., Zanuttigh P.: Hand gesture recognition with leap motion and kinect devices. In: 2014 IEEE International Conference on Image Processing (ICIP). IEEE, 2014. ISSN 1522-4880. URL http://dx.doi.org/10.1109/ICIP. 2014.7025313.
• [15] Marin G., Dominio F., Zanuttigh P.: Hand Gesture Recognition with Jointly Calibrated Leap Motion and Depth Sensor. In: Multimedia Tools and Applications, vol. 75(22), pp. 14991-15015, 2016. ISSN 1380-7501. URL http: //dx.doi.org/10.1007/s11042-015-2451-6.
• [16] Marton Z.C., Pangercic D., Blodow N., Beetz M.: Combined 2D-3D Categorization and Classification for Multimodal Perception Systems. In: International Journal of Robotics Research, vol. 30(11), pp. 1378-1402, 2011. ISSN 0278-3649. URL http://dx.doi.org/10.1177/0278364911415897.
• [17] Marton Z.C., Pangercic D., Blodow N., Kleinehellefort J., Beetz M.: General 3D Modelling of novel Objects from a Single View. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Taipei, Taiwan, 2010. URL http://dx.doi.org/10.1109/IROS.2010.5650434.
• [18] Marton Z.C., Pangercic D., Rusu R.B., Holzbach A., Beetz M.: Hierarchical Object Geometric Categorization and Appearance Classification for Mobile Manipulation. In: Proceedings of the IEEE-RAS International Conference on Humanoid Robots. Nashville, TN, USA, 2010.
• [19] McCallum A.: Efficiently Inducing Features of Conditional Random Fields. In: Proceedings of the Nineteenth Conference on Uncertainty in Artificial Intelligence, UAI'03, pp. 403-410. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2003. ISBN 0-127-05664-5.
• [20] Molina J., Escudero-Vinolo M., Signoriello A., Pardas M., Ferran C., Bescos J., Marques F., Martinez J.M.: Real-time user independent hand gesture recognition from time-of-ight camera video using static and dynamic models. In: Machine Vision and Applications, vol. 24(1), 2013. ISSN 1432-1769. URL http://dx. doi.org/10.1007/s00138-011-0364-6.
• [21] Oprisescu S., Su C.R.B.: Automatic Static Hand Gesture Recognition Using Tof Cameras. In: Signal Processing Conference (EUSIPCO), 2012 Proceedings of the 20th European, pp. 2748-2751. Zenodo, 2012. URL http://dx.doi.org/10. 5281/zenodo.42821.
• [22] Pedersoli F., Benini S., Adami N., Leonardi R.: XKin: An Open Source Framework for Hand Pose and Gesture Recognition Using Kinect. In: The Visual Computer, vol. 30(10), pp. 1107-1122, 2014. ISSN 0178-2789. URL http://dx.doi.org/10.1007/s00371-014-0921-x.
• [23] Plouffe G., Cretu A.M.: Static and Dynamic Hand Gesture Recognition in Depth Data Using Dynamic Time Warping. In: IEEE Transactions on Instrumentation and Measurement, vol. 65(2), pp. 305-316, 2016. ISSN 1557-9662. URL http: //dx.doi.org/10.1109/TIM.2015.2498560.
• [24] Pugeault N., Bowden R.: Spelling it out: Real-time ASL fingerspelling recognition. In: ICCV Workshops, pp. 1114-1119. IEEE, 2011. ISBN 978-1-4673-0062-9. URL http://dx.doi.org/10.1109/ICCVW.2011.6130290.
• [25] Qin S., Zhu X., Yang Y., Jiang Y.: Real-time Hand Gesture Recognition from Depth Images Using Convex Shape Decomposition Method. In: Journal of Signal Processing Systems, vol. 74(1), pp. 47-58, 2014. ISSN 1939-8018. URL http: //dx.doi.org/10.1007/s11265-013-0778-7.
• [26] Ren Z., Yuan J., Meng J., Zhang Z.: Robust Part-Based Hand Gesture Recognition Using Kinect Sensor. In: IEEE Transactions on Multimedia, vol. 15(5), pp. 1110-1120, 2013. ISSN 1520-9210. URL http://dx.doi.org/10.1109/TMM. 2013.2246148.
• [27] Ren Z., Yuan J., Zhang Z.: Robust Hand Gesture Recognition Based on Fingerearth Mover's Distance with a Commodity Depth Camera. In: Proceedings of the 19th ACM International Conference on Multimedia, MM '11, pp. 1093-1096. ACM, New York, NY, USA, 2011. ISBN 978-1-4503-0616-4. URL http://dx. doi.org/10.1145/2072298.2071946.
• [28] Ribo A., Warcho l D., Oszust M.: An Approach to Gesture Recognition with Skeletal Data Using Dynamic TimeWarping and Nearest Neighbour Classifier. In: International Journal of Intelligent Systems and Applications, vol. 8(6), pp. 1-8, 2016. ISSN 2074-9058. URL http://dx.doi.org/10.5815/ijisa.2016.06.01.
• [29] Rusu R.B., Cousins S.: 3D is here: Point Cloud Library (PCL). In: IEEE International Conference on Robotics and Automation (ICRA). Shanghai, China, 2011. Software available at http://pointclouds.org.
• [30] Rusu R.B., Holzbach A., Bradski G., Beetz M.: Detecting and Segmenting Objects for Mobile Manipulation. In: Proceedings of IEEE Workshop on Search in 3D and Video (S3DV), held in conjunction with the 12th IEEE International Conference on Computer Vision (ICCV). Kyoto, Japan, 2009.
• [31] Sridevi K., Sundarambal M., Muralidharan K., Josephine R.L.: FPGA implementation of hand gesture recognition system using neural networks. In: 2017 11th International Conference on Intelligent Systems and Control (ISCO). IEEE, 2017. ISBN 978-1-5090-2717-0. URL http://dx.doi.org/10.1109/ISCO.2017. 7856017.
• [32] Tewari D., Srivastava S.K.: A Visual Recognition of Static Hand Gesture in Indian Sign Language based on Kohonen Self Organizing Map Algorithm. In: International Journal of Engineering and Advanced Technology, vol. 2(2), pp. 165-170, 2012. ISSN 2249-8958.
• [33] Uebersax D., Gall J., den Bergh M.V., Gool L.V.: Real-time sign language letter and word recognition from depth data. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). IEEE, 2011. ISBN 978-1- 4673-0063-6. URL http://dx.doi.org/10.1109/ICCVW.2011.6130267.
• [34] Wang C., Liu Z., Chan S.C.: Superpixel-Based Hand Gesture Recognition With Kinect Depth Camera. In: IEEE Transactions on Multimedia, vol. 17(1), pp. 29-39, 2015. URL http://dx.doi.org/10.1109/TMM.2014.2374357.
• [35] Wang Y., Yang R.: Real-time hand posture recognition based on hand dominant line using kinect. In: ICME Workshops. IEEE, 2013. ISBN 978-1-4799-1604-7. URL http://dx.doi.org/10.1109/ICMEW.2013.6618237.
• [36] Warchoł D., Wysocki M.: Recognition of Hand Posture Based on a Point Cloud Descriptor and a Feature of Extended Fingers. In: Journal of Automation, Mobile Robotics and Intelligent Systems, vol. 10(01), pp. 48-57, 2016. URL http://dx. doi.org/10.14313/JAMRIS_1-2016/7.
• [37] Wen Y., Hu C., Yu G., Wang C.: A robust method of detecting hand gestures using depth sensors. In: Proceedings of the 2012 IEEE International Workshop on Haptic Audio Visual Environments and Games, pp. 72-77. 2012. URL http: //dx.doi.org/10.1109/HAVE.2012.6374441.
• [38] Wohlkinger W., Vincze M.: Ensemble of shape functions for 3D object classification. In: ROBIO, pp. 2987-2992. IEEE, 2011. ISBN 978-1-4577-2136-6. URL http://dx.doi.org/10.1109/ROBIO.2011.6181760.
• [39] Xu B., Zhou Z., Huang J., Huang Y.: Static Hand Gesture Recognition Based on RGB-D Image and Arm Removal, pp. 180-187. Springer International Publishing, 2017. ISBN 978-3-319-59072-1. URL http://dx.doi.org/10.1007/ 978-3-319-59072-1_22.
• [40] Zhang C., Yang X., Tian Y.: Histogram of 3D facets: A characteristic descriptor for hand gesture recognition. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1-8. IEEE, 2013. URL http://dx.doi.org/10.1109/FG.2013.6553754.

Uwagi

PL

Opracowanie rekordu w ramach umowy 509/P-DUN/2018 ze środków MNiSW przeznaczonych na działalność upowszechniającą naukę (2018).