Multimodal robot programming interface based on RGB-D perception and neural scene understanding modules

• Autorzy: Kulecki Bartłomiej

Identyfikatory

DOI

10.14313/JAMRIS/3‐2023/20

• Języki publikacji: EN

Abstrakty

EN

In this paper, we propose a system for natural and intuitive interaction with the robot. Its purpose is to allow a person with no specialized knowledge or training in robot programming to program a robotic arm. We utilize data from the RGB-D camera to segment the scene and detect objects. We also estimate the configuration of the operator’s hand and the position of the visual marker to determine the intentions of the operator and the actions of the robot. To this end, we utilize trained neural networks and operations on the input point clouds. Also, voice commands are used to define or trigger the execution of the motion. Finally, we performed a set of experiments to show the properties of the proposed system.

Słowa kluczowe

EN

human-robot interface; robot programming; 3D perception

• Wydawca: Łukasiewicz Industrial Research Institute for Automation and Measurements PIAP
• Czasopismo: Journal of Automation Mobile Robotics and Intelligent Systems
• Rocznik: 2023
• Tom: Vol. 17, No. 3
• Strony: 29--37
• Opis fizyczny: Bibliogr. 26 poz., rys.

Twórcy

autor

Kulecki Bartłomiej

• Poznan University of Technology UL. Piotrowo 3A, 60‐965 Poznań, Poland, www.put.poznan.pl

• https://orcid.org/0000-0002-2820-8212

Bibliografia

• [1] R. Adamini, N. Antonini, A. Borboni, S. Medici, C. Nuzzi, R. Pagani, A. Pezzaioli, and C. Tonola. “User‐friendly human‐robot interaction based on voice commands and visual systems,” 2021 24th International Conference on Mechatronics Technology (ICMT), 2021, pp. 1–5.
• [2] A. Bochkovskiy, C. Wang, and H.M. Liao. “YOLOv4: Optimal speed and accuracy of object detection,” CoRR, vol. abs/2004.10934, 2020.
• [3] M. Chen, C. Liu, and G. Du. “A human–robot interface for mobile manipulator,” Intelligent Service Robotics, vol. 11, no. 3, 2018, pp. 269–278; doi: 10.1007/s11370‐018‐0251‐3.
• [4] P. de la Puente, D. Fischinger, M. Bajones, D. Wolf, and M. Vincze. “Grasping objects from the floor in assistive robotics: Real world implications and lessons learned,” IEEE Access, vol. 7, 2019, pp. 123725–123735.
• [5] W. Dudek and T. Winiarski. “Scheduling of a robot’s tasks with the tasker framework,” IEEE Access, vol. 8, 2020, pp. 161449–161471.
• [6] V. Dutta, and T. Zielińska. “Predicting human actions taking into account object affordances,” Journal of Intelligent & Robotic Systems, vol. 93, 2019, pp. 745–761.
• [7] V. Dutta, and T. Zielińska. “Prognosing human activity using actions forecast and structured database,” IEEE Access, vol. 8, 2020, pp. 6098–6116.
• [8] Q. Gao, J. Liu, Z. Ju, and X. Zhang. “Dual‐hand detection for human–robot interaction by a parallel network based on hand detection and body pose estimation,”IEEE Transactions on Industrial Electronics, vol. 66, no. 12, 2019, pp. 9663–9672.
• [9] M. Gualtieri, J. Kuczynski, A.M. Shultz, A. Ten Pas, R. Platt, and H. Yanco. “Open world assistive grasping using laser selection,” 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 4052–4057.
• [10] K. He, G. Gkioxari, P. Dollár, and R. Girshick. “Mask R‐CNN,” 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980–2988.
• [11] B. Kulecki. “Intuitive robot programming and interaction using RGB-D perception and CNN-based objects detection,” Automation 2022: New Solutions and Technologies for Automation, Robotics and Measurement Techniques, R. Szewczyk, C. Zieliński, and M. Kaliczyńska, eds., Cham, 2022, pp. 233–243.
• [12] B. Kulecki, and D. Belter. “Intuicyjny interfejs programowania robota z neuronowymi modułami do interpretacji sceny,” Postępy robotyki. T.2, C. Zieliński and A. Mazur, eds., Warszawa, 2022, pp. 89–98.
• [13] B. Kulecki, and D. Belter. “Robot programming interface with a neural scene understanding system,” Proceedings of the 3rd Polish Conference on Artificial Intelligence, April 25–27, 2022, Gdynia, Poland, 2022, pp. 130–133.
• [14] B. Kulecki, K. Młodzikowski, R. Staszak, and D. Belter. “Practical aspects of detection and grasping objects by a mobile manipulating robot,” Industrial Robot, vol. 48, no. 5, 2021, pp. 688–699.
• [15] T.‐Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C.L. Zitnick. “Microsoft COCO: Common objects in context,” Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds., Cham, 2014, pp. 740–755.
• [16] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.‐Y. Fu, and A.C. Berg. “SSD: Single shot multibox detector,” Computer Vision – ECCV 2016, B. Leibe, J. Matas, N. Sebe, and M. Welling, eds., Cham, 2016, pp. 21–37.
• [17] X. Lv, M. Zhang, and H. Li. “Robot control based on voice command,” 2008 IEEE International Conference on Automation and Logistics, 2008, pp. 2490–2494.
• [18] I. Maurtua, I. Fernández, A. Tellaeche, J. Kildal, L. Susperregi, A. Ibarguren, and B. Sierra. “Natural multimodal communication for human–robot collaboration,” International Journal of Advanced Robotic Systems, vol. 14, no. 4, 2017; doi: 10.1177/1729881417716043.
• [19] O. Mazhar, S. Ramdani, B. Navarro, R. Passama, and A. Cherubini. “Towards real-time physical human‐robot interaction using skeleton information and hand gestures,”IEEE/RSJ Int. Conf. On Int. Robots and Systems (IROS), 2018, pp. 1–6.
• [20] C. Nuzzi, S. Pasinetti, M. Lancini, F. Docchio,and G. Sansoni. “Deep learning-based hand gesture recognition for collaborative robots,” IEEE Instrumentation Measurement Magazine, vol. 22, no. 2, 2019, pp. 44–51.
• [21] K.‐B. Park, S. H. Choi, J. Y. Lee, Y. Ghasemi, M. Mohammed, and H. Jeong. “Hands‐freehuman-robot interaction using multimodal gestures and deep learning in wearable mixedreality,” IEEE Access, vol. 9, 2021, pp. 55448–55464.
• [22] P. Putthapipat, C. Woralert, and P. Sirinimnuankul. “Speech recognition gateway for home automation on open platform,” 2018 International Conference on Electronics, Information, and Communication (ICEIC), 2018, pp. 1–4.
• [23] S. Sharan, T. Q. Nguyen, P. Nauth, and R. Araujo. “Implementation and testing of voice control in a mobile robot for navigation,” 2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2019, pp. 145–150.
• [24] M. Simão, P. Neto, and O. Gibaru. “Natural control of an industrial robot using hand gesture recognition with neural networks,”IECON 2016 –42nd Annual Conference of the IEEE Industrial Electronics Society, 2016, pp. 5322–5327.
• [25] R. Staszak, B. Kulecki, W. Sempruch, and D. Belter. “What’s on the other side? A single‐view 3D scene reconstruction,” 2022 17th International Conference on Control, Automation, Robotics and Vision (ICARCV), 2022, pp. 173–180.
• [26] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.‐L. Chang, and M. Grundmann. “Mediapipe hands: On‐device real-time hand tracking,” arXiv e-prints, Jun 2020; doi: 10.48550/arXiv.2006.10214.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).