An autonomous system for identifying and tracking characters using neural networks

• Autorzy: Słomiński Sebastian , Sobaszek Magdalena

Identyfikatory

DOI

10.24425/bpasts.2023.147923

• Języki publikacji: EN

Abstrakty

EN

For the proper operation of intelligent lighting, the precise detection of a human silhouette on the scene is necessary. Correctly adjusting the light beam divergence requires locating the detected figure in virtual three-dimensional coordinates in real time. The market is currently dominated by the markers systems. This paper is focused on the advanced solution of the markerless system of identifying and tracking characters based on deep learning methods. Analyses of the selected pose detection, holistic detection (including BalzePose and MoveNet models), and body segmentation (BlazePose and tfbodypix) algorithms are presented. The BlazePose model was implemented for both pose tracking and body segmentation in the markerless dynamic lighting and mapping system. This article presents the results of the accuracy analysis of matching the displayed content to a moving silhouette. An assessment of the illumination precision was done as the function of the movement speed for the system with and without delay compensation.

Słowa kluczowe

EN

markerless tracking; deep learning detection; dynamic lighting; pose identification

PL

śledzenie bez znaczników; identyfikacja pozy; oświetlenie dynamiczne; głębokie uczenie; wykrywanie

• Wydawca: Polska Akademia Nauk, Wydział IV Nauk Technicznych
• Czasopismo: Bulletin of the Polish Academy of Sciences. Technical Sciences
• Rocznik: 2023
• Tom: Vol. 71, nr 6
• Strony: art. no. e147923
• Opis fizyczny: Bibliogr. 40 poz., rys., tab.

Twórcy

autor

Słomiński Sebastian

• Warsaw University of Technology, Electrical Power Engineering Institute, Lighting Technology Division, Poland

• https://orcid.org/0000-0003-0347-1601

autor

Sobaszek Magdalena

• Warsaw University of Technology, Electrical Power Engineering Institute, Lighting Technology Division, Poland

• https://orcid.org/0000-0002-2941-4958

Bibliografia

• [1] “Robe Lighting,” https://www.robelighting.com/ (accessed May 04, 2022).
• [2] “Blacktrax,” https://blacktrax.cast-soft.com/ (accessed May 04, 2022).
• [3] “OptiTrack,” Available: https://www.optitrack.com/, (accessed May 04, 2022).
• [4] “Xsens,” https://www.xsens.com/ (accessed May 04, 2022).
• [5] A.M. Ghonim, W.M. Salama, A.A.M. Khalaf, and M.H. Shalaby, “Indoor localization based on visible light communication and machine learning algorithms,” Opto-Electron. Rev., vol. 30, p. 140858, 2022, doi: 10.24425/opelre.2022.140858.
• [6] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, “Open-Pose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, pp. 172–186, Dec. 2021, doi: 10.1109/TPAMI.2019.2929257.
• [7] V. Bazarevsky, I. Grishchenko, K. Raveendran, T. Zhu, F. Zhang, and M. Grundmann, “BlazePose: On-device Real-time Body Pose tracking,” ArXiv, Jun. 2020, [Online]. Available: http://arxiv.org/abs/2006.10204.
• [8] S. Słomiński and M. Sobaszek, “Intelligent object shape and position identification for needs of dynamic luminance shaping in object floodlighting and projection mapping,” Energies (Basel), vol. 13, no. 23, p. 6442, Dec. 2020, doi: 10.3390/en13236442.
• [9] S. Kreiss, L. Bertoni, and A. Alahi, “OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 8, pp. 13498–13511, Aug. 2022, doi: 10.1109/TITS.2021.3124981.
• [10] J. Cheng, L. Zhang, Q. Chen, and R. Long, “Position detection for electric vehicle DWCS using VI-SLAM method,” Energy Rep., vol. 7, pp. 1–9, Nov. 2021, doi: 10.1016/j.egyr.2021.09.086.
• [11] K. Mohammed, A.S. Tolba, and M. Elmogy, “Multimodal student attendance management system (MSAMS),” Ain Shams Engineering Journal, vol. 9, no. 4, pp. 2917–2929, Dec. 2018, doi: 10.1016/j.asej.2018.08.002.
• [12] N. Aunsri and S. Rattarom, “Novel eye-based features for head pose-free gaze estimation with web camera: New model and low-cost device,” Ain Shams Eng. J., vol. 13, no. 5, p. 101731, Sep. 2022, doi: 10.1016/j.asej.2022.101731.
• [13] S. Sharma, K. Shanmugasundaram, and S.K. Ramasamy, “FAREC – CNN based efficient face recognition technique using Dlib,” in Proceedings of 2016 International Conference on Advanced Communication Control and Computing Technologies, ICACCCT 2016, IEEE, Jan. 2017, pp. 192–195, doi: 10.1109/ICACCCT.2016.7831628.
• [14] V. Bazarevsky, Y. Kartynnik, A. Vakunov, K. Raveendran, and M. Grundmann, “BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs,” ArXiv, Jul. 2019, [Online]. Available: http://arxiv.org/abs/1907.05047.
• [15] K. Zhang, Z. Zhang, and Qiao Yu, “Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks,” IEEE Signal Process. Lett., vol. 23, pp. 1499–1053, 2016, doi: 10.1109/LSP.2016.2603342.
• [16] A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks),” 2017 IEEE International Conference on Computer Vision (ICCV), 2017, doi: 10.1109/ICCV.2017.116.
• [17] Y. Kartynnik, A. Ablavatski, I. Grishchenko, and M. Grundmann, “Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs,” ArXiv, Jul. 2019, [Online]. Available: http://arxiv.org/abs/1907.06724.
• [18] “MoveNet,” https://www.tensorflow.org/hub/tutorials/movenet (accessed May 04, 2022).
• [19] A. Mankotia and M. Meenu Garg, “Real-time person segmentation,” Int. J. Creat. Res. Thoughts (IJCRT), vol. 9, no. 6, pp. 30–36, 2021, [Online]. Available: https://www.ijcrt.org/papers/IJCRT2106125.pdf.
• [20] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” 2017 IEEE International Conference on Computer Vision (ICCV), Mar. 2017, doi: 10.1109/ICCV.2017.322.
• [21] P. Viola and M. Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features,” Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, pp. 1–9, 2001, doi: 10.1109/CVPR.2001.990517.
• [22] J. Chmielińska and J. Jakubowski, “Detection of driver fatigue symptoms using transfer learning,” Bull. Pol. Acad. Sci. Tech. Sci., vol. 66, no. 6, pp. 869–874, 2018, doi: 10.24425/bpas.2018.125934.
• [23] S. Suwarno and K. Kevin, “Analysis of Face Recognition Algorithm: Dlib and OpenCV,” J. Inform. Telecomm. Eng., vol. 4, no. 1, pp. 173–184, Jul. 2020, doi: 10.31289/jite.v4i1.3865.
• [24] G. Anbarjafari, R.E. Haamer, I. LÜSi, T. Tikk, and L. Valgma, “3D face reconstruction with region based best fit blending using mobile phone for virtual reality based social media,” Bull. Pol. Acad. Sci. Tech. Sci., vol. 67, no. 1, pp. 125–132, 2019, doi: 10.24425/bpas.2019.127341.
• [25] H.-S. Fang, S. Xie, Y.-W. Tai, and C. Lu, “RMPE: Regional Multi-person Pose Estimation,” 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2353–2362, Nov. 2017, doi: 10.1109/ICCV.2017.256.
• [26] J. Li, C. Wang, H. Zhu, Y. Mao, H.-S. Fang, and C. Lu, “Crowd-Pose: Efficient Crowded Scenes Pose Estimation and A New Benchmark,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Dec. 2019, doi: 10.1109/CVPR.2019.01112.
• [27] “BodyPix,” https://blog.tensorflow.org/2019/11/updated-bodypix-2.html (accessed May 04, 2022).
• [28] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jan. 2018, doi: 10.1109/CVPR.2018.00474.
• [29] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Dec. 2015, doi: 10.1109/cvpr.2016.90.
• [30] A.G. Howard et al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” ArXiv, Apr. 2017, [Online]. Available: http://arxiv.org/abs/1704.04861.
• [31] J. Lin and G.H. Lee, “Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo,” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, doi: 10.1109/CVPR46437.2021.01171.
• [32] K. Takahashi, D. Mikami, M. Isogawa, and H. Kimata, “Human Pose as Calibration Pattern; 3D Human Pose Estimation with Multiple Unsynchronized and Uncalibrated Cameras,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1888–1895, 2018, doi: 10.1109/CVPRW.2018.00230.
• [33] J. Dong, W. Jiang, Q. Huang, H. Bao, and X. Zhou, “Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–10, Jan. 2019, doi: 10.1109/CVPR.2019.00798.
• [34] C. Huang et al., “End-to-end Dynamic Matching Network for Multi-view Multi-person 3d Pose Estimation,” European Conference on Computer Vision, pp. 477–493, 2020, doi: 10.1007/978-3-030-58604-1_29.
• [35] H. Chen, P. Guo, P. Li, G.H. Lee, and G. Chirikjian, “Multi-person 3D Pose Estimation in Crowded Scenes Based on Multi-View Geometry-Supplementary Material,” Lecture Notes in Computer Science, pp. 541–557, 2020, doi: 10.1007/978-3-030-58580-8_32.
• [36] C. Malleson, J. Collomosse, and A. Hilton, “Real-Time Multi-person Motion Capture from Multi-view Video and IMUs,” Int. J. Comput. Vis., vol. 128, no. 6, pp. 1594–1611, Jun. 2020, doi: 10.1007/s11263-019-01270-5.
• [37] K. Sun, B. Xiao, D. Liu, and J. Wang, “Deep High-Resolution Representation Learning for Human Pose Estimation,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, doi: 10.1109/CVPR.2019.00584.
• [38] Y. Chen, Z. Wang, Y. Peng, Z. Zhang, and G. Yu Jian Sun, “Cascaded Pyramid Network for Multi-Person Pose Estimation,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7103–7112, 2018, doi: 10.1109/CVPR.2018.00742.
• [39] S. Słomiński and M. Sobaszek, “Dynamic autonomous identification and intelligent lighting of moving objects with discomfort glare limitation,” Energies (Basel), vol. 14, no. 21, p. 7243, Nov. 2021, doi: 10.3390/en14217243.
• [40] K. Skarżyński and W. Żagan, “Improving the quantitative features of architectural lighting at the design stage using the modified design algorithm,” Energy Rep., vol. 8, pp. 10582–10593, Nov. 2022, doi: 10.1016/j.egyr.2022.08.203.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).