Fusion of Depth and Thermal Imaging for People Detection

• Autorzy: Gutfeter Weronika , Pacut Andrzej

Identyfikatory

DOI

10.26636/jtit.2021.155521

• Języki publikacji: EN

Abstrakty

EN

The methodology presented in this paper covers the topic of automatic detection of humans based on two types of images that do not rely on the visible light spectrum, namely on thermal and depth images. Various scenarios are considered with the use of deep neural networks being extensions of Faster R-CNN models. Apart from detecting people, independently, with the use of depth and thermal images, we proposed two data fusion methods. The first approach is the early fusion method with a 2-channel compound input. As it turned out, its performance surpassed that of all other methods tested. However, this approach requires that the model be trained on a dataset containing both types of spatially and temporally synchronized imaging sources. If such a training environment cannot be setup or if the specified dataset is not sufficiently large, we recommend the late fusion scenario, i.e. the other approach explored in this paper. Late fusion models can be trained with single-source data. We introduce the dual-NMS method for fusing the depth and thermal imaging approaches, as its results are better than those achieved by the common NMS.

Słowa kluczowe

EN

depth imaging; person detection; sensor fusion; thermal imaging

• Wydawca: Instytut Łączności - Państwowy Instytut Badawczy
• Czasopismo: Journal of Telecommunications and Information Technology
• Rocznik: 2021
• Tom: nr 4
• Strony: 53--60
• Opis fizyczny: Bibliogr. 26 poz., rys., tab.

Twórcy

autor

Gutfeter Weronika

• Biometric and Machine Intelligence Laboratory, Research and Academic Computer Network (NASK), ul. Kolska 12, Warsaw, Poland

• https://orcid.org/0000-0001-6359-8220

autor

Pacut Andrzej

• Biometric and Machine Intelligence Laboratory, Research and Academic Computer Network (NASK), ul. Kolska 12, Warsaw, Poland

• https://orcid.org/0000-0003-3489-8990

Bibliografia

• [1] A. Clapés, J. S. Jacques Junior, C. Morral, and S. Escalera, „Chalearn lap 2020 challenge on identity-preserved human detection: Dataset and results", in 15th IEEE Int. Conf. on Automatic Face and Gesture Recogn. (FG 2020), Buenos Aires, Argentina, pp. 859-866, 2020 (DOI: 10.1109/FG47880.2020.00135).
• [2] T.-Y. Lin et al., „Microsoft COCO: common objects in context", CoRR, abs/1405.0312, 2014 [Online]. Available: https://arxiv.org/pdf/1405.0312
• [3] Teledyne Flir LLC, „LWIR micro thermal camera module Lepton 3", 2018 [Online]. Available: https://www.ir.com/products/lepton/?model=500-0276-01 (accessed on: 01.01.2021).
• [4] J. Smisek, M. Jancosek, and T. Pajdla, 3D with Kinect, Consumer Depth Cameras for Computer Vision. Adv. in Computer Vision and Pattern Recogn., A. Fossati, J. Gall, H. Grabner, X. Ren, K. Konolige, Eds., pp. 3-25. Springer London, London: 2013 (DOI: 10.1007/978-1-4471-4640-7 1).
• [5] Intel Corporation, Intel RealSense Depth Module D400 Series CustomCalibration, 2019 [Online]. Available: https://www.intel.com/content/dam/support/ us/en/documents/emerging-technologies/ intel-realsense-technology/RealSense D400% 20 Custom Calib Paper.pdf (accessed on: 01.01.2021).
• [6] S. Kumar, T. K. Marks, and M. Jones, „Improving person tracking using an inexpensive thermal infrared sensor", in IEEE Conf. On Computer Vision and Pattern Recogn. Workshops, Columbus, OH, USA, pp. 217-224, 2014 (DOI: 10.1109/CVPRW.2014.41).
• [7] A. S. Charan, M. Jitesh, M. Chowdhury, and H. Venkataraman, „Abifn: Attention-based bi-modal fusion network for object detection at night time", Electronics Letters, vol. 56, no. 24, pp. 1309-1311, 2020 (DOI: 10.1049/el.2020.1952).
• [8] H. Haggag, M. Hossny, S. Nahavandi, and O. Haggag, „An adaptable system for RGB-D based human body detection and pose estimation: Incorporating attached props", in IEEE Int. Conf. on Systems, Man, and Cybernetics (SMC), Budapest, Hungary, pp. 1544-1549, 2016 (DOI: 10.1109/SMC.2016.7844458).
• [9] O. H. Jafari, D. Mitzel, and B. Leibe, „Real-time RGB-D based people detection and tracking for mobile robots and headworn cameras", in IEEE Int. Conf. on Robotics and Automation (ICRA), Hong Kong, China, 2014, pp. 5636-5643 (DOI: 10.1109/ICRA.2014.6907688).
• [10] M. Rasoulidanesh, S. Yadav, S. Herath, Y. Vaghei, and S. Payandeh, „Deep attention models for human tracking using RGBD", Sensors, vol. 19, no. 4, 2019 (DOI: 10.3390/s19040750).
• [11] H. S. Hadi, M. Rosbi, U. U. Sheikh, and S. H. M. Amin, „Fusion of thermal and depth images for occlusion handling for human detection from mobile robot", in 10th Asian Control Conf. (ASCC), Kota Kinabalu, Malaysia, pp. 1-5, 2015 (DOI: 10.1109/ASCC.2015.7244722).
• [12] D. J. Yeong, G. Velasco-Hernandez, J. Barry, and J. Walsh, „Sensor and sensor fusion technology in autonomous vehicles: A review", Sensors, vol. 21, no. 6, 2021 (DOI: 10.3390/s21062140). [13] V. F. Vidal et al., „Sensors fusion and multidimensional point cloud analysis for electrical power system inspection", Sensors, vol. 20, no. 14, pp. 40-42, 2020 (DOI: 10.3390/s20144042).
• [14] T. Alldieck, C. H. Bahnsen, and T. B. Moeslund, „Context-aware fusion of RGB and thermal imagery for traffic monitoring", Sensors, vol. 16, no. 11, 2016 (DOI: 10.3390/s16111947).
• [15] F. Farahnakian and J. Heikkonen, „Deep learning based multi-modal fusion architectures for maritime vessel detection", Remote Sensing, vol. 12, no. 16, 2020 (DOI: 10.3390/rs12162509).
• [16] A. Morfin-Santana et al., „Real-time people detection from thermal images by using an unmanned aerial system", in 16th Int. Conf. on Electric. Engineer., Comput. Sci. and Automatic Control (CCE), Mexico City, Mexico, pp. 1-6, 2019 (DOI: 10.1109/ICEEE.2019.8884561).
• [17] S. Chang, F. Yang, W. Wu, Y. Cho, and S. Chen, „Nighttime pedestrian detection using thermal imaging based on hog feature", in Proc. 2011 Int. Conf. on System Sci. and Engineer., Macau, China, pp. 694-698, 2011 (DOI: 10.1109/ICSSE.2011.5961992).
• [18] L. Spinello and K. O. Arras, „People detection in RGB-D data", in IEEE/RSJ Int. Conf. on Intell. Robots and Systems, San Francisco, CA, USA, pp. 3838-3843, 2011 (DOI: 10.1109/IROS.2011.6095074).
• [19] Ch. Herrmann, M. Ruf, and J. Beyerer, „CNN-based thermal infrared person detection by domain adaptation", in Proc. Autonomous Systems: Sensors, Vehicles, Security, and the Internet of Everything, M. C. Dudzik and J. C. Ricklin, Eds., Orlando, FL, USA, vol. 10643, 2018, pp. 38-43 (DOI: 10.1117/12.2304400).
• [20] Sh. Ren, K. He, R. Girshick, and J. Sun, „Faster R-CNN: Towards real-time object detection with region proposal networks", in Proc. of the 28th Int. Conf. on Neural Information Process. Systems, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds., vol. 1, pp. 91-99, 2015 [Online]. Available: http://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks.pdf
• [21] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, „Mask R-CNN", CoRR, abs/1703.06870, 2017 [Online]. Available: https://arxiv.org/pdf/1703.06870
• [22] W. Liu et al., „SSD: single shot multibox detector", CoRR, abs/1512.02325, 2015 [Online]. Available: https://arxiv.org/pdf/1512.02325
• [23] J. Redmon and A. Farhadi, „Yolov3: An incremental improvement", CoRR, abs/1804.02767, 2018 [Online]. Available: https://arxiv.org/pdf/1804.02767
• [24] K. He, X. Zhang, Sh. Ren, and J. Sun, „Deep residual learning for image recognition", CoRR, abs/1512.03385, 2015 [Online]. Available: https://arxiv.org/pdf/1512.03385
• [25] A. Karpathy et al., „Large-scale video classification with convolutional neural networks", 27th IEEE Conf. on Computer Vision and Pattern Recogn. (CVPR), Columbus, OH, USA, 2014 (DOI: 10.1109/CVPR.2014.223).
• [26] A. Neubeck and L. Van Gool, „Efficient non-maximum suppression", in 18th Int. Conf. on Pattern Recogn. (ICPR'06), Hong Kong, China, vol. 3, pp. 850-855, 2006 (DOI: 10.1109/ICPR.2006.479).

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).