A novel method for fast generation of 3D objects from multiple depth sensors

• Autorzy: Szmuc Tomasz , Mrówka Rafał , Brańka Marek , Ficoń Jakub , Pięta Piotr

Identyfikatory

DOI

10.2478/jaiscr-2023-0009

• Języki publikacji: EN

Abstrakty

EN

Scanning real 3D objects face many technical challenges. Stationary solutions allow for accurate scanning. However, they usually require special and expensive equipment. Competitive mobile solutions (handheld scanners, LiDARs on vehicles, etc.) do not allow for an accurate and fast mapping of the surface of the scanned object. The article proposes an end-to-end automated solution that enables the use of widely available mobile and stationary scanners. The related system generates a full 3D model of the object based on multiple depth sensors. For this purpose, the scanned object is marked with markers. Markers type and positions are automatically detected and mapped to a template mesh. The reference template is automatically selected for the scanned object, which is then transformed according to the data from the scanners with non-rigid transformation. The solution allows for the fast scanning of complex and varied size objects, constituting a set of training data for segmentation and classification systems of 3D scenes. The main advantage of the proposed solution is its efficiency, which enables real-time scanning and the ability to generate a mesh with a regular structure. It is critical for training data for machine learning algorithms. The source code is available at https://github.com/SATOffice/improved_scanner3D.

Słowa kluczowe

EN

3D scan; LiDAR; point cloud; mesh; registration

• Wydawca: University of Social Sciences
• Czasopismo: Journal of Artificial Intelligence and Soft Computing Research
• Rocznik: 2023
• Tom: Vol. 13, No. 2
• Strony: 95--105
• Opis fizyczny: Bibliogr. 52 poz., rys.

Twórcy

autor

Szmuc Tomasz

• Department of Applied Computer Science, AGH University of Science and Technology, al. A. Mickiewicza 30, 30-059 Kraków, Poland

• https://orcid.org/0000-0003-4922-5369

autor

Mrówka Rafał

• Department of Applied Computer Science, AGH University of Science and Technology, al. A. Mickiewicza 30, 30-059 Kraków, Poland

• https://orcid.org/0000-0001-5143-3488

autor

Brańka Marek

• SynAppsTech, ul. I. Mościckiego 1, 24-110 Puławy, Poland

• https://orcid.org/0000-0002-4732-2176

autor

Ficoń Jakub

• PhD Student AGH University of Science and Technology, Kraków, Poland al. A. Mickiewicza 30, 30-059 Kraków, Poland

• https://orcid.org/0000-0001-8703-1413

autor

Pięta Piotr

• Department of Applied Computer Science, AGH University of Science and Technology, al. A. Mickiewicza 30, 30-059 Kraków, Poland

• https://orcid.org/0000-0001-8490-7117

Bibliografia

• [1] R. Ramm, M. Heinze, P. Kühmstedt, A. Christoph, S. Heist, G. Notni, Portable solution for high-resolution 3D and colour texture on-site digitization of cultural heritage objects, Journal of Cultural Heritage, vol. 53, pp. 165–175, Elsevier (2022). DOI 10.1016/j.culher.2021.11.006
• [2] J. Perez-Cerrolaza, J. Abella, L. Kosmidis, A.J. Calderon, F.J. Cazorla, J.L. Flores, GPU Devices for Safety-Critical Systems: A Survey, ACM Computing Surveys (CSUR), pp. 102 Tomasz Szmuc, Rafał Mrówka, Marek Branka, Jakub Fico ´ n, Piotr Pięta 1–35, New York, NY, USA (2022). DOI 10.1145/3549526
• [3] N. Li, C.P. Ho, J. Xue, L.W. Lim, G. Chen, Y.H. Fu, L.Y.T. Lee, A Progress Review on SolidState LiDAR and Nanophotonics-Based LiDAR Sensors, Laser & Photonics Reviews, vol. 16, 2100511, pp. 1–24, Wiley-VCH GmbH, Weinheim (2022). DOI 10.1002/lpor.202100511
• [4] B. Wang, J. Lan, J. Gao, LiDAR Filtering in 3D Object Detection Based on Improved RANSAC, Remote Sensing, Computational Intelligence in Remote Sensing, vol. 14, no. 9, 2110, pp. 1–18 (2022). DOI 10.3390/rs14092110
• [5] Y. Li, Z. Ge, G. Yu, J. Yang, Z. Wang, Y. Shi, J. Sun, Z. Li, BEVDepth: Acquisition of reliable depth for multi-view 3d object detection, pp. 1–12 (2022). arXiv preprint arXiv:2206.10092
• [6] M. Pandey, M. Fernandez, F. Gentile, O. Isayev, A. Tropsha, A.C. Stern, A. Cherkasov, The transformational role of GPU computing and deep learning in drug discovery, Nature Machine Intelligence, vol. 4, no. 3, pp. 211–221 (2022). DOI 10.1038/s42256-022-00463-x
• [7] L. You, H. Jiang, J. Hu, C.H. Chang, L. Chen, X. Cui, M. Zhao, GPU-accelerated Faster Mean Shift with euclidean distance metrics, In: IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 211–216 (2022). DOI 10.1109/COMPSAC54236.2022.00037
• [8] V. Tadic, A. Toth, Z. Vizvari, M. Klincsik, Z. Sari, P. Sarcevic, J. Sarosi, I. Biro, Perspectives of RealSense and ZED Depth Sensors for Robotic Vision Applications, Machines, vol. 10, no. 3, 183, pp. 1–16 (2022). DOI 10.3390/machines10030183
• [9] S. Cerfoglio, C. Ferraris, L. Vismara, G. Amprimo, L. Priano, G. Pettiti, M. Galli, A. Mauro, V. Cimolin, Kinect-Based Assessment of Lower Limbs during Gait in Post-Stroke Hemiplegic Patients: A Narrative Review, Sensors, vol. 22, no. 13, 4910, pp. 1–15 (2022). DOI 10.3390/s22134910
• [10] Z. Qiu, J. Martínez-Sánchez, V.M. Brea, P. López, P. Arias, Low-cost mobile mapping system solution for traffic sign segmentation using Azure Kinect, International Journal of Applied Earth Observation and Geoinformation, vol. 112, 102895, pp. 1–11 (2022). DOI 10.1016/j.jag.2022.102895
• [11] X. Xu, L. Zhang, J. Yang, C. Cao, W. Wang, Y. Ran, Z. Tan, M. Luo, A Review of Multi-Sensor Fusion SLAM Systems Based on 3D LiDAR, Remote Sensing, vol. 14, no. 12, 2835, pp. 1–27 (2022). DOI 10.3390/rs14122835
• [12] S. Zahia, B. Garcia-Zapirain, J. Anakabe, J. Ander, O. Jossa Bastidas, A. Loizate Totoricagüena, A Comparative Study between Scanning Devices for 3D Printing of Personalized Ostomy Patches, Sensors, vol. 22, no. 2, 560, pp. 1–20 (2022). DOI 10.3390/s22020560
• [13] S. Tavani, A. Billi, A. Corradetti, M. Mercuri, A. Bosman, M. Cuffaro, T. Seers, E. Carminati, Smartphone assisted fieldwork: Towards the digital transition of geoscience fieldwork using LiDAR-equipped iPhones, Earth-Science Reviews, vol. 227, 103969, pp. 1–15 (2022). DOI 10.1016/j.earscirev.2022.103969
• [14] P. Chemweno, R.J. Torn, Innovative safety zoning for collaborative robots utilizing Kinect and LiDAR sensory approaches, Procedia CIRP, vol. 106, pp. 209–214 (2022). DOI 10.1016/j.procir.2022.02.180
• [15] H.S. Tham, R. Hussin, R.C. Ismail, A Real-Time Distance Prediction via Deep Learning and Microsoft Kinect, In: IOP Conference Series: Earth and Environmental Science, vol. 1064, pp. 1–6 (2022). DOI 10.1088/1755-1315/1064/1/012048
• [16] M. Vogt, A. Rips, C. Emmelmann, Comparison of iPad Pro® ’s LiDAR and TrueDepth Capabilities with an Industrial 3D Scanning Solution, Technologies, vol. 9, no. 2, 25, pp. 1–13 (2021). DOI 10.3390/technologies9020025
• [17] B. Chen, S. Shi, J. Sun, W. Gong, J. Yang, L. Du, K. Guo, B. Wang, B. Chen, Hyperspectral LiDAR point cloud segmentation based on geometric and spectral information, OPTICS EXPRESS, vol. 27, no. 17, pp. 24043–24059 (2019)
• [18] T. Staffas, M. Brunzell, S. Gyger, L. Schweickert, S. Steinhauer, V. Zwiller, 3D scanning quantum LiDAR, In: 2022 Conference on Lasers and Electro-Optics (CLEO), pp. 1–2 (2022). DOI 10.1364/CLEO_AT.2022.AM2K.1
• [19] F. Di Stefano, S. Chiappini, A. Gorreja, M. Balestra, R. Pierdicca, Mobile 3D scan LiDAR: a literature review, Geomatics, Natural Hazards and Risk, vol. 12, no. 1, pp. 2387–2429, Taylor & Francis (2021). DOI 10.1080/19475705.2021.1964617
• [20] A. Notchenko, V. Ishimtsev, A. Artemov, V. Selyutin, E. Bogomolov, E. Burnaev, Scan2Part: Fine-grained and Hierarchical Part-level Understanding of Real-World 3D Scans, In: Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol 5, pp. 711–722.(2022). DOI 10.5220/0010848200003124
• [21] Y. Li, T. Harada, Non-rigid Point Cloud Registration with Neural Deformation Pyramid, pp. 1–19 (2022). arXiv preprint arXiv:2205.12796
• [22] P. Besl, H.D. McKay, A method for registration of 3-D shapes, IEEE Transactions on Pattern Analysis and Machine Intelligence vol. 14, pp. 239–256 1992). DOI 10.1109/34.121791
• [23] S. Gold, A. Rangarajan, C. Lu, S. Pappu, E. Mjolsness, New algorithms for 2D and 3D point matching: pose estimation and correspondence, Pattern Recognition, vol. 31, no. 8, pp. 1019–1031 (1998). DOI 10.1016/S0031-3203(98)80010-1.
• [24] Intel Corporation. Intel RealSense Product Overview. Retrieved October 30, 2022, from https://www.intelrealsense.com/
• [25] Intel Corporation. Learn About the Intel® RealSenseTM SDK 2.0. Retrieved October 30, 2022, from https://www.intelrealsense.com/intelrealsense- sdk-2-0/
• [26] Intel Corporation. Intel⃝R RealSenseTM LiDAR Camera L515. Retrieved October 30, 2022, from https://www.intelrealsense.com/LiDAR-cameral515/
• [27] Intel Corporation. Multi-Camera configurations with the Intel® RealSenseTM LiDAR Camera L515. Retrieved October 30, 2022, from https://dev.intelrealsense.com/docs/LiDARcamera- l515-multi-camera-setup
• [28] H. Sarmadi, R. Muñoz-Salinas, M.A. Berbís, A. Luna, R. Medina-Carnicer, Joint scene and object tracking for cost-effective augmented reality assisted patient positioning in radiation therapy, pp. 1–16 (2020). arXiv preprint arXiv:2010.01895.
• [29] The Gnomon Workshop. 3D scan and retopology for production. Retrieved October 30, 2022, from https://www.thegnomonworkshop.com/tutorials/3d-scan-and-retopology-for-production
• [30] ScanLab photogrammetry. 3D scanning service. Retrieved October 30, 2022, from https://scanlab.ca/services/3d-scanning/
• [31] Scan Engine. Studio. Retrieved October 30, 2022,from https://www.scan-engine.fr/
• [32] Shining 3D. 3D Digitizing Solutions. Product models. Retrieved October 30, 2022, from https://www.shining3d.com/3d-digitizing-solutions/
• [33] Artec 3D. Artec Leo 3D scanner. Retrieved October 30, 2022, from https://www.artec3d.com/portable-3d-scanners/artec-leo
• [34] Peel 3D. Peel 3 3D scanner. Retrieved October 30, 2022, from https://peel-3d.com/products/peel-3/
• [35] Calibry. Skaner 3D Calibry. Retrieved October 30, 2022, from https://calibry.pl/
• [36] A. Kulikajevas, Reconstruction algorithm of invisible sides of a 3D object for depth scanning systems, Doctoral dissertation (2022), Kauno technologijos Universitetas, KTU, Lithuania
• [37] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T.B. McHugh, V. Vanhoucke, Google Scanned Objects: A HighQuality Dataset of 3D Scanned Household Items, pp. 1–8 (2022). arXiv preprint arXiv:2204.11918
• [38] Google Research. Scanned Objects by Google Research: A Dataset of 3D-Scanned Common Household Items. Retrieved October 31, 2022,from https://ai.googleblog.com/2022/06/scannedobjects-by-google-research.html
• [39] Amazon. Amazon Berkeley Objects (ABO) Dataset. Retrieved October 31, 2022, from https://amazon-berkeley-objects.s3.amazonaws.com/index.html#home
• [40] J. Collins, S. Goel, K. Deng, A. Luthra, L. Xu, E. Gundogdu, T.F.Y. Vicente, T. Dideriksen, H. Arora, M. Guillaumin, J. Malik, ABO: Dataset and benchmarks for real-world 3d object understanding, In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21126-21136 (2021). DOI 10.1109/CVPR52688.2022.02045
• [41] Meta. Common Objects in 3D: LargeScale Learning and Evaluation of Real-life 3D Category Reconstruction. Retrieved October 31, 2022, from https://ai.facebook.com/datasets/CO3D-dataset/
• [42] J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, D. Novotny, Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction, In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10901–10911 (2021). DOI 10.1109/ICCV48922.2021.01072
• [43] Alibaba. Our Businesses. Retrieved October 31, 2022, from https://www.alibabagroup.com/enUS/about- alibaba-businesses
• [44] H. Fu, R. Jia, L. Gao, M. Gong, B. Zhao, S. Maybank, D. Tao, 3d-future: 3D furniture shape 104 Tomasz Szmuc, Rafał Mrówka, Marek Branka, Jakub Ficoń, Piotr Pięta with texture, International Journal of Computer Vision, vol. 129, no. 12, pp. 3313–3337 (2021). DOI 10.1007/s11263-021-01534-z
• [45] XRPro LLC (Structure). Structure Sensor Pro. Retrieved October 31, 2022, from https://structure.io/
• [46] Kaggle Inc. Datasets. Retrieved October 31, 2022, from https://www.kaggle.com/datasets
• [47] F.J. Romero-Ramirez, R. Muñoz-Salinas, R. Medina-Carnicer, Speeded up detection of squared fiducial markers, Image and Vision Computing, 76, pp. 38—47 (2018). DOI 10.1016/j.imavis.2018.05.004
• [48] Multi-Camera configurations with the Intel® RealSenseTM LiDAR Camera L515. Retrieved October 31, 2022 from https://dev.intelrealsense.com/docs/LiDARcamera-l515-multi-camera-setup
• [49] V. Kraevoy, A. Sheffer, Template-Based Mesh Completion, Eurographics Symposium on Geometry Processing (2005). DOI 10.2312/SGP/SGP05/013-022
• [50] J. Huang, T. Birdal, Z. Gojcic, L. Guibas, S. Hu, Multiway Non-rigid Point Cloud Registration via Learned Functional Map Synchronization, In: IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–18 (2022). DOI 10.1109/TPAMI.2022.3164653
• [51] Y. Li, T. Harada, Lepard: Learning partial point cloud matching in rigid and deformable scenes, pp. 1–17 (2021). DOI 10.48550/ARXIV.2111.12591
• [52] F. Poux, R. Billen, Voxel-based 3D point cloud semantic segmentation: unsupervised geometric and relationship featuring vs. deep learning methods, ISPRS International Journal of GeoInformation, vol. 8, no. 5, 213, pp. 1–34 (2019). DOI 10.3390/ijgi8050213

Uwagi

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2022-2023).