Segmentation-based Method of Increasing The Depth Maps Temporal Consistency

• Autorzy: Mieloch D. , Grzelka A.

Identyfikatory

DOI

10.24425/123521

• Języki publikacji: EN

Abstrakty

EN

In this paper, a modification of the graph-based depth estimation is presented. The purpose of proposed modification is to increase the quality of estimated depth maps, reduce the time of the estimation, and increase the temporal consistency of depth maps. The modification is based on the image segmentation using superpixels, therefore in the first step of the proposed modification a segmentation of previous frames is used in the currently processed frame in order to reduce the overall time of the depth estimation. In the next step, a depth map from the previous frame is used in the depth map optimization as the initial values of a depth map estimated for the current frame. It results in the better representation of silhouettes of objects in depth maps and in the reduced computational complexity of the depth estimation process. In order to evaluate the performance of the proposed modification the authors performed the experiment for a set of multiview test sequences that varied in their content and an arrangement of cameras. The results of the experiments confirmed the increase of the depth maps quality - the quality of depth maps calculated with the proposed modification is higher than for the unmodified depth estimation method, apart from the number of the performed optimization cycles. Therefore, use of the proposed modification allows to estimate a depth of the better quality with almost 40% reduction of the estimation time. Moreover, the temporal consistency, measured through the reduction of the bitrate of encoded virtual views, was also considerably increased.

Słowa kluczowe

EN

depth maps; depth map estimation; temporal consistency; image segmentation; free-viewpoint television; virtual view synthesis

• Wydawca: Polish Academy of Sciences, Committee of Electronics and Telecommunication
• Czasopismo: International Journal of Electronics and Telecommunications
• Rocznik: 2018
• Tom: Vol. 64, No. 3
• Strony: 293--298
• Opis fizyczny: Bibliogr. 37 poz., il., rys., tab., wykr.

Twórcy

autor

Mieloch D.

• Chair of Multimedia Telecommunications and Microelectronics, Poznań University of Technology, Poznań, Poland

autor

Grzelka A.

• Chair of Multimedia Telecommunications and Microelectronics, Poznań University of Technology, Poznań, Poland

Bibliografia

• [1] K. Muller, P. Merkle, and T. Wiegand, “3-D video representation using depth maps,” Proceedings of the IEEE, vol. 99, no. 4, pp. 643–656, April 2011.
• [2] O. Stankiewicz, M. Domański, A. Dziembowski, A. Grzelka, D. Mieloch, and J. Samelak, “A free-viewpoint television system for horizontal virtual navigation,” IEEE Transactions on Multimedia, pp. 1–1, 2018.
• [3] M. Tanimoto, “FTV standardization in MPEG,” in 2014 3DTVConference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), July 2014, pp. 1–4.
• [4] M. Tanimoto, M. P. Tehrani, T. Fujii, and T. Yendo, “FTV for 3-D spatial communication,” Proceedings of the IEEE, vol. 100, no. 4, pp. 905–917, April 2012.
• [5] M. Domański, A. Dziembowski, T. Grajek, A. Grzelka, K. Klimaszewski, D. Mieloch, R. Ratajczak, O. Stankiewicz, J. Siast, J. Stankowski, and K. Wegner, “Demonstration of a simple free viewpoint television system,” in 2017 IEEE International Conference on Image Processing (ICIP), Sept 2017, pp. 4589–4591.
• [6] A. Dziembowski, A. Grzelka, D. Mieloch, O. Stankiewicz, K. Wegner, and M. Domański, “Multiview synthesis - improved view synthesis for virtual navigation,” in 2016 Picture Coding Symposium (PCS), Dec 2016, pp. 1–5.
• [7] M. Camplani, T. Mantecon, and L. Salgado, “Depth-color fusion strategy for 3-D scene modeling with Kinect,” IEEE Transactions on Cybernetics, vol. 43, no. 6, pp. 1560–1571, Dec 2013.
• [8] J. Hernandez-Aceituno, R. Arnay, J. Toledo, and L. Acosta, “Using Kinect on an autonomous vehicle for outdoors obstacle detection,” IEEE Sensors Journal, vol. 16, no. 10, pp. 3603–3610, May 2016.
• [9] X. Suau, J. Ruiz-Hidalgo, and J. R. Casas, “Real-time head and hand tracking based on 2.5D data", IEEE Transactions on Multimedia, vol. 14, no. 3, pp. 575–585, June 2012.
• [10] L. Li, S. Zhang, X. Yu, and L. Zhang, “PMSC: PatchMatch-based superpixel cut for accurate stereo matching,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 3, pp. 679–692, March 2018.
• [11] O. Stankiewicz, K. Wegner, M. Tanimoto, and M. Domański, “Enhanced Depth Estimation Reference Software (DERS) for free-viewpoint television,” ISO/IEC JTC1/SC29/WG11, Doc. MPEG M31518, Geneva, 2013.
• [12] G. Lafruit, M. Domański, K. Wegner, T. Grajek, T. Senoh, J. Jung, P. Kovcs, P. Goorts, L. Jorissen, A. Munteanu, B. Ceulemans, P. Carballeira, S. Garca, and M. Tanimoto, “New visual coding exploration in MPEG: Super-MultiView and free navigation in free viewpoint TV,” in 2016 Proceedings of the Electronic Imaging Conference: Stereoscopic Displays and Application, February 2016, pp. 1–9.
• [13] C.-C. Lee, A. Tabatabai, and K. Tashiro, “Free viewpoint video (FVV) survey and future research direction,” vol. 4, 10 2015.
• [14] Y. S. Kang and Y. S. Ho, “High-quality multi-view depth generation using multiple color and depth cameras,” in 2010 IEEE International Conference on Multimedia and Expo, July 2010, pp. 1405–1410.
• [15] S. Xiang, L. Yu, Q. Liu, and Z. Xiong, “A gradient-based approach for interference cancelation in systems with multiple Kinect cameras,” in 2013 IEEE International Symposium on Circuits and Systems (ISCAS2013), May 2013, pp. 13–16.
• [16] D. Mieloch, A. Dziembowski, A. Grzelka, O. Stankiewicz, and M. Domański, “Temporal enhancement of graph-based depth estimation method,” in 2017 International Conference on Systems, Signals and Image Processing (IWSSIP), May 2017, pp. 1–4.
• [17] D. Mieloch, A. Dziembowski, A. Grzelka, O. Stankiewicz, and M. Domański, “Graph-based multiview depth estimation using segmentation,” in 2017 IEEE International Conference on Multimedia and Expo (ICME), July 2017, pp. 217–222.
• [18] V. Kolmogorov and R. Zabin, “What energy functions can be minimized via graph cuts?” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 2, pp. 147–159, Feb 2004.
• [19] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222–1239, Nov 2001.
• [20] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Ssstrunk, “Slic superpixels compared to state-of-the-art superpixel methods,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 11, pp. 2274–2282, Nov 2012.
• [21] O. Stankiewicz, M. Domański, and K. Wegner, “Estimation of temporally-consistent depth maps from video with reduced noise,” in 2015 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), July 2015, pp. 1–4.
• [22] M. Kppel, M. B. Makhlouf, M. Mller, and P. Ndjiki-Nya, “Temporally consistent adaptive depth map preprocessing for view synthesis,” in 2013 Visual Communications and Image Processing (VCIP), Nov 2013, pp. 1–6.
• [23] H. Jiang, G. Zhang, H. Wang, and H. Bao, “Spatio-temporal video segmentation of static scenes and its applications,” IEEE Transactions on Multimedia, vol. 17, no. 1, pp. 3–15, Jan 2015.
• [24] L. Sheng, K. N. Ngan, C. L. Lim, and S. Li, “Online temporally consistent indoor depth video enhancement via static structure,” IEEE Transactions on Image Processing, vol. 24, no. 7, pp. 2197–2211, July 2015.
• [25] N. Vretos and P. Daras, “Temporal and color consistent disparity estimation in stereo videos,” in 2014 IEEE International Conference on Image Processing (ICIP), Oct 2014, pp. 3798–3802.
• [26] M. Mueller, F. Zilly, C. Riechert, and P. Kauff, “Spatio-temporal consistent depth maps from multi-view video,” in 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTVCON), May 2011, pp. 1–4.
• [27] J. Lei, J. Liu, H. Zhang, Z. Gu, N. Ling, and C. Hou, “Motion and structure information based adaptive weighted depth video estimation,” IEEE Transactions on Broadcasting, vol. 61, no. 3, pp. 416–424, Sept 2015.
• [28] M. Sizintsev and R. P. Wildes, “Spatiotemporal stereo and scene flow via stequel matching,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 6, pp. 1206–1219, June 2012.
• [29] G. Nur, S. Dogan, H. K. Arachchi, and A. M. Kondoz, “Impact of depth map spatial resolution on 3D video quality and depth perception,” in 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video, June 2010, pp. 1–4.
• [30] O. Stankiewicz, K. Wegner, M. Tanimoto, and M. Doma´nski, “Enhanced view synthesis reference software (VSRS) for free-viewpoint television,” ISO/IEC JTC1/SC29/WG11, Doc. MPEG M31520, Geneva, 2013.
• [31] L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, “High-quality video view interpolation using a layered representation,” in ACM SIGGRAPH, vol. 23. Association for Computing Machinery, Inc., August 2004, p. 600608. [Online]. Available: https://www.microsoft.com/en-us/research/publication/highquality-video-view-interpolation-using-a-layered-representation/
• [32] P. Kovacs, “[FTV AHG] Big Buck Bunny light-field test sequences,” ISO/IEC JTC1/SC29/WG11, Doc. MPEG M35721, Geneva, 2015.
• [33] M. Domański, A. Dziembowski, M. Kurc, A. Luczak, D. Mieloch, J. Siast, O. Stankiewicz, and K. Wegner, “Poznan University of Technology test multiview video sequences acquired with circular camera arrangement ’Poznan Team’ and ’Poznan Blocks’ sequences,” ISO/IEC JTC1/SC29/WG11, Doc. MPEG M35846, Geneva, 2016.
• [34] M. Domański, A. Dziembowski, A. Grzelka, D. Mieloch, O. Stankiewicz, and K. Wegner, “Multiview test video sequences for free navigation exploration obtained using pairs of cameras,” ISO/IEC JTC1/SC29/WG11, Doc. MPEG M38247, Geneva, 2016.
• [35] M. Domański, O. Stankiewicz, K. Wegner, and T. Grajek, “Immersive visual media - MPEG-I: 360 video, virtual navigation and beyond,” in 2017 International Conference on Systems, Signals and Image Processing (IWSSIP), May 2017, pp. 1–9.
• [36] G. Bjøntegaard, “Calculation of average PSNR differences between RD986 curves,” ISO/IEC JTC1/SC29/WG11, Doc. MPEG M15378, Austin, 2001.
• [37] HEVC reference codec. [Online]. Available: https://hevc.hhi.fraunhofer.de/svn/svn HEVCSoftware

Uwagi

1. Opracowanie rekordu w ramach umowy 509/P-DUN/2018 ze środków MNiSW przeznaczonych na działalność upowszechniającą naukę (2018).

2. The presented work has been funded by the Polish Ministry of Science and Higher Education within the status activity task ”Theory and algorithms of multidimensional signal processing” (DS) in 2018.