Automatic Colorization of Digital Movies using Decolorization Models and SSIM Index

• Autorzy: Śluzek, Andrzej , Dudziński, Marcin , Świsłocki, Tomasz
• Języki publikacji: EN

Abstrakty

EN

Re-colorization of images or movies is a challenging problem due to the infinite RGB solutions for a monochrome object. In general, the process is assisted by humans, either by providing colorization hints or relevant training data for ML/AI algorithms. Our intention is to develop a mechanism for fully unguided (and with no training data used) colorization of movies. In other words, we aim to create acceptable colored counterparts of movies in domains where only monochrome visualizations physically exist (e.g. IR, UV, MRI, etc. data). Following our past approach to image colorization, the method assumes arbitrary rgb2gray models and utilizes a few probabilistic heuristics. Additionally, we maintain the temporal stability of colorization by locally using structural similarity (SSIM) between adjacent frames. The paper explains the details of the method, presents exemplary results and compares them to the state-of-the art solutions.

Słowa kluczowe

PL

informatyka; analiza koloru; obrazowanie rezonansu magnetycznego; modelowanie obliczeniowe; kryteria stabilności; dane treningowe; wizualizacja danych

EN

computer science; image color analysis; magnetic resonance imaging; computational modeling; stability criteria; training data; data visualization

• Czasopismo: Annals of Computer Science and Information Systems
• Rocznik: 2023
• Tom: Vol. 35
• Strony: 843--853
• Opis fizyczny: Bibliogr. 27 poz., il., wykr.

Twórcy

autor

Śluzek, Andrzej

• Warsaw University of Life Sciences - SGGW, Institute of Information Technology, Nowoursynowska 159, bld.34, 02-776 Warsaw, Poland,

autor

Dudziński, Marcin

• Warsaw University of Life Sciences - SGGW, Institute of Information Technology, Nowoursynowska 159, bld.34, 02-776 Warsaw, Poland,

autor

Świsłocki, Tomasz

• Warsaw University of Life Sciences - SGGW, Institute of Information Technology, Nowoursynowska 159, bld.34, 02-776 Warsaw, Poland,

Bibliografia

• 1. I. Zeger, S. Grgic, J. Vukovic, and G. Sisul, “Grayscale image colorization methods: Overview and evaluation,” IEEE Access, vol. 9, pp. 113 326–113 346, 2021. http://dx.doi.org/10.1109/ACCESS.2021.3104515
• 2. A. Salmona, L. Bouza, and J. Delon, “Deoldify: A review and implementation of an automatic colorization method,” Image Processing On Line, vol. 12, pp. 347–368, 2022. http://dx.doi.org/10.5201/ipol.2022.403
• 3. R. Irony, D. Cohen-Or, and D. Lischinski, “Colorization by example,” in Eurographics Symposium on Rendering (2005). The Eurographics Association, 2005. http://dx.doi.org/10.2312/EGWR/EGSR05/201-210. ISBN 3-905673-23-1. ISSN 1727-3463
• 4. R. Gupta, A. Chia, D. Rajan, E. Ng, and Z. Huang, “Image colorization using similar images,” in 20th ACM Int. Conf. on Multimedia (MM’12), 2012, pp. 369–378.
• 5. A. Levin, D. Lischinski, and Y. Weiss, “Colorization using optimization,” ACM Transactions on Graphics, vol. 23, pp. 689–694, 06 2004. http://dx.doi.org/10.1145/1015706.1015780
• 6. A. Popowicz and B. Smolka, “Fast image colourisation using the isolines concept,” Multimedia Tools and Applications, vol. 75, pp. 15 987–16 009, 2017. http://dx.doi.org/10.1007/s11042-016-3892-2
• 7. R. Zhang, P. Isola, and A. Efros, “Colorful image colorization,” in Computer Vision – ECCV 2016. Springer, 2016. http://dx.doi.org/10.1007/978-3-319-46487-9_40 pp. 649–666.
• 8. E. Farella, S. Malek, and F. Remondino, “Colorizing the past: Deep learning for the automatic colorization of historical aerial images,” Journal of Imaging, vol. 8, p. 269, 10 2022. http://dx.doi.org/10.3390/jimaging8100269
• 9. S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification,” ACM Transactions on Graphics, vol. 35, pp. 1–11, 07 2016. http://dx.doi.org/10.1145/2897824.2925974
• 10. J. Su, H. Chu, and J. Huang, “Instance-aware image colorization,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), jun 2020. http://dx.doi.org/10.1109/CVPR42600.2020.00799 pp. 7965–7974.
• 11. H. Lee, D. Kim, D. Lee, J. Kim, and J. Lee, “Bridging the domain gap towards generalization in automatic colorization,” in Computer Vision – ECCV 2022. Springer, 2022. http://dx.doi.org/10.1007/978-3-031-19790-1_32 pp. 527–543.
• 12. A. Deshpande, J. Rock, and D. Forsyth, “Learning large-scale automatic image colorization,” in 2015 IEEE International Conference on Computer Vision (ICCV), 2015. http://dx.doi.org/10.1109/ICCV.2015.72 pp. 567–575.
• 13. A. Royer, A. Kolesnikov, and C. Lampert, “Probabilistic image colorization,” in Proc. British Machine Vision Conference (BMVC). BMVA Press, September 2017. http://dx.doi.org/10.5244/C.31.85 pp. 85.1–85.12.
• 14. C. Lei and Q. Chen, “Fully automatic video colorization with self-regularization and diversity,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. http://dx.doi.org/10.1109/CVPR.2019.00387 pp. 3748–3756.
• 15. M. Hofinger, E. Kobler, A. Effland, and T. Pock, “Learned variational video color propagation,” in Computer Vision – ECCV 2022, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, Eds. Cham: Springer Nature Switzerland, 2022. ISBN 978-3-031-20050-2 pp. 512–530.
• 16. M. G. Blanch, N. O’Connor, and M. Mrak, “Scene-adaptive temporal stabilisation for video colourisation using deep video priors,” in Computer Vision – ECCV 2022 Workshops, L. Karlinsky, T. Michaeli, and K. Nishino, Eds. Cham: Springer Nature Switzerland, 2023. ISBN 978-3-031-25069-9 pp. 644–659.
• 17. A. Śluzek, “Do we always need ai for image colorization?” Proceedings of the 4rd Polish Conference on Artificial Intelligence, Lodz, Poland, April 2023, in print.
• 18. A. Śluzek, “On unguided automatic colorization of monochrome images,” in Proc. 31 Int. Conf. in Central Europe on Computer Graphics, Visualization and Computer Vision, WSCG 2023, Plzen, Czechia., May 2023, in print.
• 19. ITU-R, “Parameter values for the hdtv standards for production and international programme exchange,” Geneva, Recommendation BT.709-6, 2015.
• 20. E. Jessee and E. Wiebe, “Visual perception and the hsv color system: Exploring color in the communications technology classroom,” Technology Teacher, vol. 68, no. 1, pp. 7–11, 2008.
• 21. S.-Y. Chen, J.-Q. Zhang, Y.-Y. Zhao, P. L. Rosin, Y.-K. Lai, and L. Gao, “A review of image and video colorization: From analogies to deep learning,” Visual Informatics, vol. 6, no. 3, pp. 51–68, 2022. http://dx.doi.org/10.1016/j.visinf.2022.05.003
• 22. W.-S. Lai, J.-B. Huang, O. Wang, E. Shechtman, E. Yumer, and M.-H. Yang, “Learning blind video temporal consistency,” in Computer Vision – ECCV 2018, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds. Cham: Springer International Publishing, 2018. ISBN 978-3-030-01267-0 pp. 179–195.
• 23. K.-H. Thung and P. Raveendran, “A survey of image quality measures,” in 2009 International Conference for Technical Postgraduates (TECH-POS), 2009. http://dx.doi.org/10.1109/TECHPOS.2009.5412098 pp. 1–4.
• 24. P. Jagalingam and A. V. Hegde, “A review of quality metrics for fused image,” Aquatic Procedia, vol. 4, pp. 133–142, 2015. http://dx.doi.org/https://doi.org/10.1016/j.aqpro.2015.02.019
• 25. Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. http://dx.doi.org/10.1109/TIP.2003.819861
• 26. D. Hasler and S. Suesstrunk, “Measuring colorfulness in natural images,” in Human Vision and Electronic Imaging VIII, vol. 5007. SPIE, 2003. http://dx.doi.org/10.1117/12.477378 pp. 87–95.
• 27. E. Gebhardt and M. Wolf, “Camel dataset for visual and thermal infrared multiple object detection and tracking,” in 2018 15th IEEE Int. Conf. AVSS, 2018. http://dx.doi.org/10.1109/AVSS.2018.8639094 pp. 1–6.

Uwagi

1. Thematic Tracks Regular Papers

2. Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).

Identyfikatory

DOI

10.15439/2023F3017