Vehicle detection and masking in UAV images using YOLO to improve photogrammetric products

• Autorzy: Pargieła Karolina

Identyfikatory

DOI

10.2478/rgg-2022-0006

• Języki publikacji: EN

Abstrakty

EN

Photogrammetric products obtained by processing data acquired with Unmanned Aerial Vehicles (UAVs) are used in many fields. Various structures are analysed, including roads. Many roads located in cities are characterised by heavy traffic. This makes it impossible to avoid the presence of cars in aerial photographs. However, they are not an integral part of the landscape, so their presence in the generated photogrammetric products is unnecessary. The occurrence of cars in the images may also lead to errors such as irregularities in digital elevation models (DEMs) in roadway areas and the blurring effect on orthophotomaps. The research aimed to improve the quality of photogrammetric products obtained with the Structure from Motion algorithm. To fulfil this objective, the Yolo v3 algorithm was used to automatically detect cars in the images. Neural network learning was performed using data from a different flight to ensure that the obtained detector could also be used in independent projects. The photogrammetric process was then carried out in two scenarios: with and without masks. The obtained results show that the automatic masking of cars in images is fast and allows for a significant increase in the quality of photogrammetric products such as DEMs and orthophotomaps.

Słowa kluczowe

EN

deep learning; photogrammetry; structure from motion; roads; object detection

PL

uczenie głębokie; fotogrametria; wykrywanie obiektów

• Wydawca: Wydział Geodezji i Kartografii Politechniki Warszawskiej
• Czasopismo: Reports on Geodesy and Geoinformatics
• Rocznik: 2022
• Tom: Vol. 114
• Strony: 15--23
• Opis fizyczny: Bibliogr. 50 poz., rys., tab., wykr.

Twórcy

autor

Pargieła Karolina

• Department of Photogrammetry, Remote Sensing of Environment, and Spatial Engineering, Faculty of Geo-Data Science, Geodesy, and Environmental Engineering, AGH University of Science and Technology, Al. Mickiewicza 30, 30-059 Krakow, Poland

• https://orcid.org/0000-0002-5373-1240

Bibliografia

• [1] Andriyanov, N., Khasanshin, I., Utkin, D., Gataullin, T., Ignar, S., Shumaev, V., and Soloviev, V. (2022). Intelligent system for estimation of the spatial position of apples based on YOLOv3 and real sense depth camera d415. Symmetry, 14(1):148, doi:10.3390/sym14010148.10.3390/sym14010148
• [2] Bao, W., Ren, Y., Wang, N., Hu, G., and Yang, X. (2021). Detection of abnormal vibration dampers on transmission lines in UAV remote sensing images with PMA-YOLO. Remote Sensing, 13(20):4134, doi:10.3390/rs13204134.10.3390/rs13204134
• [3] Bianco, S., Ciocca, G., and Marelli, D. (2018). Evaluating the performance of structure from motion pipelines. Journal of Imaging, 4(8):98, doi:10.3390/jimaging4080098.10.3390/jimaging4080098
• [4] Campana, S. (2017). Drones in archaeology. State-of-the-art and future perspectives. International Journal of Archaeological Prospection, 24(4):275—-296, doi:10.1002/arp.1569.10.1002/arp.1569
• [5] Cardenal, J., Fernández, T., Pérez-García, J. L., and Gómez-López, J. M. (2019). Measurement of road surface deformation using images captured from UAVs. Remote Sensing, 11(12):1507, doi:10.3390/rs11121507.10.3390/rs11121507
• [6] Carrera-Hernández, J., Levresse, G., and Lacan, P. (2020). Is UAV-SfM surveying ready to replace traditional surveying techniques? International journal of remote sensing, 41(12):4820–4837, doi:10.1080/01431161.2020.1727049.10.1080/01431161.2020.1727049
• [7] Coombs, C., Hislop, D., Taneva, S. K., and Barnard, S. (2020). The strategic impacts of intelligent automation for knowledge and service work: An interdisciplinary review. The Journal of Strategic Information Systems, 29(4):101600, doi:10.1016/j.jsis.2020.101600.10.1016/j.jsis.2020.101600
• [8] Dainelli, R., Toscano, P., Di Gennaro, S. F., and Matese, A. (2021). Recent advances in unmanned aerial vehicles forest remote sensing — a systematic review. part i: A general framework. Forests, 12(3):327, doi:10.3390/f12030327.10.3390/f12030327
• [9] Delavarpour, N., Koparan, C., Nowatzki, J., Bajwa, S., and Sun, X. (2021). A technical study on UAV characteristics for precision agriculture applications and associated practical challenges. Remote Sensing, 13(6):1204, doi:10.3390/rs13061204.10.3390/rs13061204
• [10] Eltner, A. and Sofia, G. (2020). Structure from motion photogram-metric technique. In Developments in Earth surface processes, volume 23, pages 1–24. Elsevier, doi:10.1016/B978-0-444-64177-9.00001-1.10.1016/B978-0-444-64177-9.00001-1
• [11] Fiz, J. I., Martín, P. M., Cuesta, R., Subías, E., Codina, D., and Cartes, A. (2022). Examples and results of aerial photogrammetry in archeology with UAV: Geometric documentation, high resolution multispectral analysis, models and 3D printing. Drones, 6(3):59, doi:10.3390/drones6030059.10.3390/drones6030059
• [12] Ge, L., Li, X., and Ng, A. H.-M. (2016). UAV for mining applications: A case study at an open-cut mine and a longwall mine in New South Wales, Australia. In 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pages 5422–5425. IEEE, doi:10.1109/IGARSS.2016.7730412.10.1109/IGARSS.2016.7730412
• [13] Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, USA, pages 580–587. doi:10.1109/CVPR.2014.81.10.1109/CVPR.2014.81
• [14] Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2015). Fast R-CNN. In the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015, pages 7–13. doi:10.1109/ICCV.2015.169.10.1109/ICCV.2015.169
• [15] Gromada, K., Siemiątkowska, B., Stecz, W., Płochocki, K., and Woźniak, K. (2022). Real-time object detection and classification by UAV equipped with SAR. Sensors, 22(5):2068, doi:10.3390/s22052068.10.3390/s22052068891509935271213
• [16] Gruen, A. (2021). Everything moves: The rapid changes in photogrammetry and remote sensing. Geo-spatial Information Science, 24(1):33–49, doi:10.1080/10095020.2020.1868275.10.1080/10095020.2020.1868275
• [17] Han, X., Chang, J., and Wang, K. (2021). Real-time object detection based on YOLO-v2 for tiny vehicle object. Procedia Computer Science, 183:61–72, doi:10.1016/j.procs.2021.02.031.10.1016/j.procs.2021.02.031
• [18] Horzyk, A. and Ergün, E. (2020). YOLOv3 precision improvement by the weighted centers of confidence selection. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, doi:10.1109/IJCNN48605.2020.9206848.10.1109/IJCNN48605.2020.9206848
• [19] Iglhaut, J., Cabo, C., Puliti, S., Piermattei, L., O’Connor, J., and Rosette, J. (2019). Structure from motion photogrammetry in forestry: A review. Current Forestry Reports, 5(3):155–168, doi:10.1007/s40725-019-00094-3.10.1007/s40725-019-00094-3
• [20] Indolia, S., Goswami, A. K., Mishra, S. P., and Asopa, P. (2018). Conceptual understanding of convolutional neural network-a deep learning approach. Procedia computer science, 132:679–688, doi:10.1016/j.procs.2018.05.069.10.1016/j.procs.2018.05.069
• [21] Jiang, S., Jiang, C., and Jiang, W. (2020). Efficient structure from motion for large-scale UAV images: A review and a comparison of SfM tools. ISPRS Journal of Photogrammetry and Remote Sensing, 167:230–251, doi:10.1016/j.isprsjprs.2020.04.016.10.1016/j.isprsjprs.2020.04.016
• [22] Ju, M., Luo, H., Wang, Z., Hui, B., and Chang, Z. (2019). The application of improved YOLO v3 in multi-scale target detection. Applied Sciences, 9(18):3775, doi:10.3390/app9183775.10.3390/app9183775
• [23] Jurgiel, B. and Verchere, P. (2022). Profile Tool GitHub repository. Available online: https://github.com/etiennesky/profiletool, Last accessed April 2022.
• [24] Kaivosoja, J., Hautsalo, J., Heikkinen, J., Hiltunen, L., Ruuttunen, P., Näsi, R., Niemeläinen, O., Lemsalu, M., Honkavaara, E., and Salonen, J. (2021). Reference measurements in developing UAV systems for detecting pests, weeds, and diseases. Remote Sensing, 13(7):1238, doi:10.3390/rs13071238.10.3390/rs13071238
• [25] Koay, H. V., Chuah, J. H., Chow, C.-O., Chang, Y.-L., and Yong, K. K. (2021). YOLO-RTUAV: Towards real-time vehicle detection through aerial images with low-cost edge devices. Remote Sensing, 13(21):4196, doi:10.3390/rs13214196.10.3390/rs13214196
• [26] Koeva, M., Muneza, M., Gevaert, C., Gerke, M., and Nex, F. (2018). Using UAVs for map creation and updating. a case study in Rwanda. Survey Review, 50(361):312–325, doi:10.1080/00396265.2016.1268756.10.1080/00396265.2016.1268756
• [27] Li, C.-Y. and Lin, H.-Y. (2020). Vehicle detection and classification in aerial images using convolutional neural networks. In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Valletta, Malta, volume 5, pages 775–782. doi:10.5220/0008941707750782.10.5220/0008941707750782
• [28] Li, Y. and Liu, C. (2019). Applications of multirotor drone technologies in construction management. International Journal of Construction Management, 19(5):401–412, doi:10.1080/15623599.2018.1452101.10.1080/15623599.2018.1452101
• [29] Luo, X., Tian, X., Zhang, H., Hou, W., Leng, G., Xu, W., Jia, H., He, X., Wang, M., and Zhang, J. (2020). Fast automatic vehicle detection in UAV images using convolutional neural networks. Remote Sensing, 12(12):1994, doi:10.3390/rs12121994.10.3390/rs12121994
• [30] Nyimbili, P. H., Demirel, H., Seker, D., and Erden, T. (2016). Structure from motion (SfM) – approaches and applications. In Proceedings of the international scientific conference on applied sciences, Antalya, Turkey, pages 27–30.
• [31] Park, S. and Choi, Y. (2020). Applications of unmanned aerial vehicles in mining from exploration to reclamation: A review. Minerals, 10(8):663, doi:10.3390/min10080663.10.3390/min10080663
• [32] Pessacg, F., Gómez-Fernández, F., Nitsche, M., Chamo, N., Torrella, S., Ginzburg, R., and De Cristóforis, P. (2022). Simplifying UAV-based photogrammetry in forestry: How to generate accurate digital terrain model and assess flight mission settings. Forests, 13(2):173, doi:10.3390/f13020173.10.3390/f13020173
• [33] Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27–30 June 2016, pages 779–788. doi:10.48550/arXiv.1506.02640.10.1109/CVPR.2016.91
• [34] Redmon, J. and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, doi:10.48550/arXiv.1804.02767.
• [35] Ren, S., He, K., Girshick, R., and Sun, J. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, doi:10.1109/TPAMI.2016.2577031.10.1109/TPAMI.2016.257703127295650
• [36] Roberts, R., Inzerillo, L., and Di Mino, G. (2020). Using UAV based 3D modelling to provide smart monitoring of road pavement conditions. Information, 11(12):568, doi:10.3390/info11120568.10.3390/info11120568
• [37] Sahin, O. and Ozer, S. (2021). Yolodrone: Improved yolo architecture for object detection in drone images. In 2021 44th International Conference on Telecommunications and Signal Processing (TSP), pages 361–365. IEEE, doi: 10.1109/TSP52935.2021.9522653.10.1109/TSP52935.2021.9522653
• [38] Shapiro, S. S. and Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591–611, doi:10.2307/2333709.10.2307/2333709
• [39] Snavely, N., Seitz, S. M., and Szeliski, R. (2008). Modeling the world from internet photo collections. International journal of computer vision, 80(2):189–210, doi:10.1007/s11263-007-0107-3.10.1007/s11263-007-0107-3
• [40] Tan, L., Lv, X., Lian, X., and Wang, G. (2021). YOLOv4_Drone: UAV image target detection based on an improved YOLOv4 algorithm. Computers & Electrical Engineering, 93:107261, doi:10.1016/j.compeleceng.2021.107261.10.1016/j.compeleceng.2021.107261
• [41] Tan, Y. and Li, Y. (2019). UAV photogrammetry-based 3D road distress detection. ISPRS International Journal of Geo-Information, 8(9):409, doi:10.3390/ijgi8090409.10.3390/ijgi8090409
• [42] Wang, J., Su, S., Wang, W., Chu, C., Jiang, L., and Ji, Y. (2022). An object detection model for paint surface detection based on improved yolov3. Machines, 10(4):261, doi:10.3390/machines10040261.10.3390/machines10040261
• [43] Xiao, Y., Tian, Z., Yu, J., Zhang, Y., Liu, S., Du, S., and Lan, X. (2020). A review of object detection based on deep learning. Multimedia Tools and Applications, 79(33):23729–23791, doi:10.1007/s11042-020-08976-6.10.1007/s11042-020-08976-6
• [44] Xu, Z.-F., Jia, R.-S., Sun, H.-M., Liu, Q.-M., and Cui, Z. (2020). Light-YOLOv3: fast method for detecting green mangoes in complex scenes using picking robots. Applied Intelligence, 50(12):4670–4687, doi:10.1007/s10489-020-01818-w.10.1007/s10489-020-01818-w
• [45] Yahya, M. Y., Shun, W. P., Yassin, A. M., and Omar, R. (2021). The challenges of drone application in the construction industry. Journal of Technology Management and Business, 8(1):20–27, doi:10.30880/jtmb.2021.08.01.003.10.30880/jtmb.2021.08.01.003
• [46] Yang, C., Zhang, F., Gao, Y., Mao, Z., Li, L., and Huang, X. (2021). Moving car recognition and removal for 3D urban modelling using oblique images. Remote Sensing, 13(17):3458, doi:10.3390/rs13173458.10.3390/rs13173458
• [47] Zhao, Z.-Q., Zheng, P., Xu, S.-t., and Wu, X. (2019). Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems, 30(11):3212–3232, doi:10.1109/TNNLS.2018.2876865.10.1109/TNNLS.2018.287686530703038
• [48] Zhu, Q., Shang, Q., Hu, H., Yu, H., and Zhong, R. (2021). Structure-aware completion of photogrammetric meshes in urban road environment. ISPRS Journal of Photogrammetry and Remote Sensing, 175:56–70, doi:10.1016/j.isprsjprs.2021.02.010.10.1016/j.isprsjprs.2021.02.010
• [49] Zulkipli, M. A. and Tahar, K. N. (2018). Multirotor UAV-based photogrammetric mapping for road design. International Journal of Optics, 2018:7, doi:10.1155/2018/1871058.10.1155/2018/1871058
• [50] Šafář, V., Potůčková, M., Karas, J., Tlustý, J., Štefanová, E., Jančovič, M., and Cígler Žofková, D. (2021). The use of UAV in cadastral mapping of the Czech Republic. ISPRS International Journal of Geo-Information, 10(6):380, doi:10.3390/ijgi10060380.10.3390/ijgi10060380

Uwagi

PL

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2022-2023).