Exploring automated object detection methods for manholes using classical computer vision and deep learning for autonomous vehicles

• Autorzy: Rao Shika , Mitnala Nitya

Identyfikatory

DOI

10.22630/MGV.2023.32.1.2

• Języki publikacji: EN

Abstrakty

EN

Open, broken, and improperly closed manholes can pose problems for autonomous vehicles and thus need to be included in obstacle avoidance and lane-changing algorithms. In this work, we propose and compare multiple approaches for manhole localization and classification like classical computer vision, convolutional neural networks like YOLOv3 and YOLOv3-Tiny, and vision transformers like YOLOS and ViT. These are analyzed for speed, computational complexity, and accuracy in order to determine the model that can be used with autonomous vehicles. In addition, we propose a size detection pipeline using classical computer vision to determine the size of the hole in an improperly closed manhole with respect to the manhole itself. The evaluation of the data showed that convolutional neural networks are currently better for this task, but vision transformers seem promising.

Słowa kluczowe

EN

computer vision; object detection; size detection; convolutional neural networks; vision transformer; autonomous vehicles

• Wydawca: Institute of Information Technology of the Warsaw University of Life Sciences – SGGW
• Czasopismo: Machine Graphics and Vision
• Rocznik: 2023
• Tom: Vol. 32, No. 1
• Strony: 25--53
• Opis fizyczny: Bibliogr. 52 poz., rys., tab., wykr.

Twórcy

autor

Rao Shika

• Birla Institute of Technology and Science, Pilani - Hyderabad Campus, India

• https://orcid.org/0000-0002-3219-0326

autor

Mitnala Nitya

• Birla Institute of Technology and Science, Pilani - Hyderabad Campus, India
• Texas Instruments (India): Bengaluru, Karnataka, India

• https://orcid.org/0009-0001-1669-8103

Bibliografia

• [1] P. Adarsh, P. Rathi, and M. Kumar. YOLO v3-Tiny: object detection and recognition using one stage improved model. In 6th Int. Conf. Advanced Computing and Communication Systems (ICACCS 2020), pages 687-694, Coimbatore, India, 6-7 Mar 2020. doi:10.1109/ICACCS48705.2020.9074315.
• [2] A. Bochkovskiy. darknet. GitHub, 30 Oct 2021.https://github.com/AlexeyAB/darknet. [Accessed 17 Oct, 2022].
• [3] A. Bochkovskiy, C. Y. Wang, and H. Y. M. Liao. YOLOv4: Optimal speed and accuracy of object detection, 2020. arXiv:2004.10934. doi:10.48550/arXiv.2004.10934.
• [4] D. Boller, M. M. Vitry, J. D. Wegner, and J. P. Leitão. Automated localization of urban drainage infrastructure from public-access street-level images. Urban Water Journal, 16(7):480-493, 2019.doi:10.1080/1573062X.2019.1687743.
• [5] Can self-driving cars run on Indian roads? moneycontrol. [Accessed 17 Oct, 2022]. https://www.moneycontrol.com/news/driverless-cars/.
• [6] J. Canny. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(6):679-698, 1986. doi:10.1109/TPAMI.1986.4767851.
• [7] N. Carion, F. Massa, G. Synnaeve, et al. End-to-end object detection with transformers. In Proc. European Conf. Computer Vision (ECCV 2020), volume 12346 of Lecture Notes in Computer Science, pages 213-229, Glasgow, UK, 23-28 Aug 2020. doi:10.1007/978-3-030-58452-813.
• [8] Z. Chong and L. Yang. An algorithm for automatic recognition of manhole covers based on MMS images. In Proc. 11th Chinese Conf. Advances in Image and Graphics Technologies (IGTA 2016),volume 634 of Communications in Computer and Information Science. Springer, Beijing, China, 8-9 Jul 2016. doi:10.1007/978-981-10-2260-94.
• [9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv, 2020. arXiv:2010.11929v2. doi:10.48550/arXiv.2010.11929.
• [10] Y. Du, N. Pan, Z. Xu, et al. Pavement distress detection and classification based on YOLO network. International Journal of Pavement Engineering, 22(13):1659-1672, 2021. doi:10.1080/10298436.2020.1714047.
• [11] L. Eliot. Manhole covers and self-driving cars. In: Self-Driving Cars. Podcasts by Dr. Lance Eliot, 9 Sep 2021. https://ai-selfdriving-cars.libsyn.com/manhole-covers-and-self-driving-cars. [Accessed 17 Oct, 2022].
• [12] Y. Fang, B. Liao, X. Wang, et al. You Only Look at One Sequence: Rethinking transformer in vision through object detection. In Advances in Neural Information Processing Systems (NeurIPS 2021), volume 34, pages 26183-26197. 2021. https://proceedings.neurips.cc/paper/2021/hash/dc912a253d1e9ba40e2c597ed2376640-Abstract.html.
• [13] L. Fei-Fei, J. Deng, O. Russakovsky, A. Berg, and K. Li, editors. IMAGENET. 2021. [Accessed December 2022]. https://image-net.org.
• [14] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9):1627-1645, 2010-09. doi:10.1109/TPAMI.2009.167.
• [15] R. Ghosh and O. Smadi. Automated detection and classification of pavement distresses using 3D pavement surface images and deep learning. Transportation Research Record, 2675(9):1359-1374, 2021. doi:10.1177/03611981211007481.
• [16] E. Hildreth and D. Marr. Theory of edge detection. Proceedings of the Royal Society of London. Series B, Biological sciences, 207(1167):187-218, 1980. doi:10.1098/rspb.1980.0020.
• [17] S. Ji, Y. Shi, and Z. Shi. Manhole cover detection using vehicle-based multi-sensor data. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XXXIX-B3:281-284, 2012. doi:10.5194/isprsarchives-XXXIX-B3-281-2012.
• [18] G. Jocher, A. Chaurasia, A. Stoken, et al. ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Insurance Segmentation. Zenodo, Nov 2022. doi:10.5281/zenodo.7347926.
• [19] G. Jocher et al. YOLOv5 by Ultralytics. GitHub, 2020. https://github.com/ultralytics/yolov5.[Accessed 17 Oct, 2022].
• [20] E. Karimi, M. Rezanejad, B. Fiset, et al. Machine learning meets classical computer vision for accurate cell identification., 28 Feb 2022. doi:10.1101/2022.02.27.482183.
• [21] Y. M. Kim, Y. G. Kim, S. Y. Son, et al. Review of recent automated pothole-detection methods. Applied Sciences, 12(11):5320, 2022. doi:10.3390/app12115320.
• [22] C. Kumar. Preventable deaths: In India, at least 2 die each day due to open pits & manholes. TheTimes of India, 25 Nov 2021. [Accessed 9 Oct, 2022]. https://timesofindia.indiatimes.com/india/preventable-deaths-in-india-at-least-2-die-each-day-due-to-open-pits-manholes/articleshow/87917848.cms.
• [23] Label Studio community (originally created by Tzutalin). LabelImg. GitHub, 23 Sep 2022.https://github.com/heartexlabs/labelImg. [Accessed 17 Oct, 2022].
• [24] Y. Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In S. Avidan et al., editors, Proc. European Conf. Computer Vision (ECCV 2022),volume 13669 of Lecture Notes in Computer Science, pages 280-296, Tel Aviv, Israel, 23-27 Oct 2022. doi:10.1007/978-3-031-20077-917.
• [25] S. Liu, L. Qi, H. Qin, et al. Path Aggregation Network for instance segmentation. arXiv, 2018. arXiv:1803.01534v4. doi:10.48550/arXiv.1803.01534.
• [26] S. Maji and J. Malik. Object detection using a max-margin Hough transform. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2009), pages 1038-1045, Miami, FL, USA, 20-25 Jun 2009. doi:10.1109/CVPR.2009.5206693.
• [27] B. Mali, A. Shrestha, A. Chapagain, et al. Challenges in the penetration of electric vehicles in developing countries with a focus on Nepal. Renewable Energy Focus, 40:1-12, 2022. doi:10.1016/j.ref.2021.11.003.
• [28] A. Mohan and S. Poobal. Crack detection using image processing: A critical review and analysis. Alexandria Engineering Journal, 57(2):787-798, 2018. doi:10.1016/j.aej.2017.01.020.
• [29] U. Nepal and H. Eslamiat. Comparing YOLOv3, YOLOv4 and YOLOv5 for autonomous landing spot detection in faulty UAVs. Sensors, 22(2):464, 2022. doi:10.3390/s22020464.
• [30] H. Niigaki, J. Shimamura, and M. Morimoto. Circular object detection based on separability and uniformity of feature distributions using Bhattacharyya Coefficient. InProc. 21st Int. Conf. Pattern Recognition (ICPR2012), pages 2009-2012, Tsukuba, Japan, 11-15 Nov 2012. https://ieeexplore.ieee.org/abstract/document/6460553.
• [31] T. Ojala, M. Pietikainen, and T. Maenpaa. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7):971-987, 2002-07. doi:10.1109/TPAMI.2002.1017623.
• [32] J. Pasquet, T. Desert, O. Bartoli, et al. Detection of manhole covers in high-resolution aerial images of urban areas by combining two methods. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(5):1802-1807, 2016. doi:10.1109/JSTARS.2015.2504401.
• [33] S. Rao and N. Mitnala. ManholeDetection. GitHub, Dec 2022.https://github.com/sh-r/Manhole_Detection. [Accessed: Dec, 2022].
• [34] S. Rao, A. Quezada, S. Rodriguez, et al. Developing, analyzing, and evaluating vehicular lane keeping algorithms using electric vehicles. Vehicles, 4(4):1012-1041, 2022. doi:10.3390/vehicles4040055.
• [35] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You Only Look Once: Unified, real-time object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2016), pages 779-788, Las Vegas, NV, USA, 27-30 Jun 2016. doi:10.1109/CVPR.2016.91.
• [36] J. Redmon and A. Farhadi. YOLO9000: Better, faster, stronger. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2017), pages 7263-7271, Honolulu, HI, USA, 21-26 Jul 2017. doi:10.1109/CVPR.2017.690.
• [37] J. Redmon and A. Farhadi.YOLOv3: An incremental improvement. arXiv, 2018. arXiv:1804.02767v1. doi:10.48550/arXiv.1804.02767.
• [38] A. Santos, J. Marcato Junior, J. Andrade Silva, et al. Storm-drain and manhole detection. Geomatics and Computer Vision Datasets.https://sites.google.com/view/geomatics-and-computer-vision/home/datasets#h.sen6zve8r3ra. [Accessed Jan, 2022].
• [39] A. Santos, J. Marcato Junior, J. Andrade Silva, et al. Storm-drain and manhole detection using the RetinaNet method. Sensors, 20(16):4450, 2020. doi:10.3390/s20164450.
• [40] P. Saxena. Increase Frame Per Second (FPS) rate in the custom object detection step by step. Towards Data Science, 3 Sep 2020. https://towardsdatascience.com/no-gpu-for-your-production-server-a20616bb04bd. [Accessed 17 Oct, 2022].
• [41] M. Simonovsky. Ellipse detection using 1D Hough transform. In MATLAB Central File Exchange.2022. [Accessed 16 Oct, 2022]. https://www.mathworks.com/matlabcentral/fileexchange/33970-ellipse-detection-using-1d-hough-transform.
• [42] N. Tanaka and M. Mouri. A detection method of cracks and structural objects of the road surface image. In Proc. IAPR Workshop on Machine Vision Applications, pages 387-390, Tokyo, Japan,28-30 Nov 2000.http://b2.cvl.iis.u-tokyo.ac.jp/mva/proceedings/CommemorativeDVD/2000/papers/2000387.pdf.
• [43] R. Timofte and L. Gool. Multi-view manhole detection, recognition, and 3D localisation. In Proc. IEEE Int. Conf. Computer Vision Workshops (ICCV Workshops), pages 188-195, Barcelona, Spain, 16 Jan 2011. Workshops. doi:10.1109/ICCVW.2011.6130242.
• [44] H. Touvron, M. Cord, M. Douze, et al. Training data-efficient image transformers & distillation through attention. In M. Meila and T. Zhang, editors, Proc. 38th Int. Conf. Machine Learning (ICML 2021), volume 139 of Proceedings of Machine Learning Research, pages 10347-10357,Virtual Only, 18-24 Jul 2021. PMLR. https://proceedings.mlr.press/v139/touvron21a.html.
• [45] A. Vaswani, N. Shazeer, N. Parmar, et al. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS 2017), volume 30. 2017. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
• [46] C. Y. Wang, A. Bochkovskiy, and H. Y. M. Liao.YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv, 2022. arXiv:2207.02696v1. doi:10.48550/arXiv.2207.02696.
• [47] C.-Y. Wang, H.-Y. M Liao, Y.-H. Wu, et al. CSPNet: A new backbone that can enhance learning capability of CNN. InProc. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2020), pages 1571-1580, Seattle, WA, USA, 14-19 Jun 2020. doi:10.1109/CVPRW50498.2020.00203.
• [48] C.-Y. Wang, H.-Y. M. Liao, I.-H. Yeh, et al. CSPNet: A new backbone that can enhance learning capability of CNN. arXiv, 2019. arXiv:1911.11929v1. doi:10.48550/arXiv.1911.11929.
• [49] Y. Xie and Q. Ji. A new efficient ellipse detection method. In Proc. 2002 Int. Conf. Pattern Recognition (ICPR 2002), volume 2, pages 957-960, Quebec City, QC, Canada, 11-15 Aug 2002. doi:10.1109/ICPR.2002.1048464.
• [50] S. Yan. Manhole Cover Detection from Natural Images. PhD thesis, University of Dublin, Trinity College, Sep 2020. [Accessed 17 Oct, 2022]. https://www.scss.tcd.ie/publications/theses/diss/2020/TCD-SCSS-DISSERTATION-2020-111.pdf.
• [51] Z. Yang, Y. Liu, L. Liu, X. Tang, J. Xie, and X. Gao. Detecting small objects in urban settings using SlimNet model. IEEE Transactions on Geoscience and Remote Sensing, 57(11):8445-8457,2019. doi:10.1109/TGRS.2019.2921111.
• [52] Y. Yu, H. Guan, and Z. Ji. Automated detection of urban road manhole covers using mobile laser scanning data. IEEE Transactions on Intelligent Transportation Systems, 16(6):3258-3269, 2015.doi:10.1109/TITS.2015.2413812.

Uwagi

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2022-2023).