High-quality synthesized face sketch using generative reference prior

• Autorzy: Mahfoud, Sami , Bengherabi, Messaoud , Daamouche, Abdelhamid , Boutellaa, Elhocine , Hadid, Abdenour
• Języki publikacji: EN

Abstrakty

EN

Face sketch synthesis (FSS) is considered an image-to-image translation problem, where a face sketch is generated from an input face photo. FSS plays a vital role in video/image surveillance-based law enforcement. In this paper, motivated by the recent success of generative adversarial networks (GAN), we consider conditional GAN (cGAN) to approach the problem of face sketch synthesis. However, despite the powerful cGAN model ability to generate fine textures, low-quality inputs characterized by the facial sketches drawn by artists cannot offer realistic and faithful details and have unknown degradation due to the drawing process, while high-quality references are inaccessible or even nonexistent. In this context, we propose an approach based on generative reference prior (GRP) to improve the synthesized face sketch perception. Our proposed model, which we call cGAN-GRP, leverages diverse and rich priors encapsulated in a pre-trained face GAN for generating highquality facial sketch synthesis. Extensive experiments on publicly available face databases using facial sketch recognition rate and image quality assessment metrics as criteria demonstrate the effectiveness of our proposed model compared to several state-of-the-art methods.

Słowa kluczowe

PL

sieci generatywne adwersarskie; synteza szkicu twarzy; generatywne odniesienie wcześniejsze

EN

face sketch synthesis; generative reference prior; generative adversarial network

• Czasopismo: Bulletin of the Polish Academy of Sciences. Technical Sciences
• Rocznik: 2024
• Tom: Vol. 72, nr 4
• Strony: art. no. e150109
• Opis fizyczny: Bibliogr. 45 poz., rys., tab.

Twórcy

autor

Mahfoud, Sami

• University of Algiers 3, Faculty of Economic, Commercial and Management Sciences, Laboratory of Governance and Modernization of Public Management, 02 Ahmed Ouaked Street Dely Ibrahim 16302, Algiers, Algeria,
• Center for Development of Advanced Technologies, Telecom Division, P.O. Box 17 Baba-Hassen 16303, Algiers, Algeria

autor

Bengherabi, Messaoud

• Center for Development of Advanced Technologies, Telecom Division, P.O. Box 17 Baba-Hassen 16303, Algiers, Algeria

autor

Daamouche, Abdelhamid

• SUniversity M’Hamed Bougara of Boumerdes, Institute of Electrical and Electronic Engineering, Laboratory of Signals and Systems, Boumerdes, 35000, Algeria

autor

Boutellaa, Elhocine

• SUniversity M’Hamed Bougara of Boumerdes, Institute of Electrical and Electronic Engineering, Laboratory of Signals and Systems, Boumerdes, 35000, Algeria

autor

Hadid, Abdenour

• Sorbonne University Abu Dhabi, Sorbonne Center for Artificial Intelligence, Abu Dhabi, UAE

Bibliografia

• [1] H. Bi, Z. Liu, L. Yang, K. Wang, and N. Li, “Face sketch synthesis: a survey,” Multimed. Tools Appl., vol. 80, no. 12, pp. 18 007–18 026, Feb 2021, doi: 10.1007/s11042-020-10301-0.
• [2] X. Wang and X. Tang, “Face photo-sketch synthesis and recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 11, pp. 1955–1967, Nov 2009, doi: 10.1109/TPAMI.2008.222.
• [3] P. Li, B. Sheng, and C.L. P. Chen, “Face sketch synthesis using regularized broad learning system,” IEEE Trans. Neural Netw. Learn. Syst., pp. 1–15, 2021, doi: 10.1109/TNNLS.2021.3070463.
• [4] X. Tang and X. Wang, “Face photo recognition using sketch,” in Proceedings. International Conference on Image Processing, vol. 1. IEEE, 2002, pp. I–I, doi: 10.1109/ICIP.2002.1038008.
• [5] Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, “A nonlinear approach for face sketch synthesis and recognition,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1. IEEE, 2005, pp. 1005–1010, doi: 10.1109/CVPR.2005.39.
• [6] S. Zhang, X. Gao, N. Wang, J. Li, and M. Zhang, “Face sketch synthesis via sparse representation-based greedy search,” IEEE Trans. Image Process., vol. 24, no. 8, pp. 2466–2477, 2015, doi: 10.1109/tip.2015.2422578.
• [7] H. Zhou, Z. Kuang, and K. K. Wong, “Markov weight fields for face sketch synthesis,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Jun 2012, pp. 1091–1097, doi: 10.1109/cvpr.2012.6247788.
• [8] Y. Song, L. Bao, Q. Yang, and M.-H. Yang, “Real-time exemplarbased face sketch synthesis,” in Computer Vision – ECCV 2014. Springer International Publishing, 2014, pp. 800–813, doi: 10.1007/978-3-319-10599-451.
• [9] A. Cichocki, T. Poggio, S. Osowski, and V. Lempitsky, “Deep learning: Theory and practice,” Bull. Pol. Acad. Sci. Tech. Sci., vol. 66, no. No 6 (Special Section on Deep Learning: Theory and Practice), pp. 757–759, 2018, doi: 10.24425/bpas. 2018.125923.
• [10] X. Yang, Y. Zhang, and D. Zhou, “Deep networks for image super-resolution using hierarchical features,” Bull. Pol. Acad. Sci. Tech. Sci., vol. 70, no. 1, p. e139616, 2022, doi: 10.24425/bpasts.2021.139616.
• [11] K. Hawari and I. Ismail, “The automatic focus segmentation of multi-focus image fusion,” Bull. Pol. Acad. Sci. Tech. Sci., vol. 70, no. 1, p. e140352, 2022, doi: 10.24425/bpasts.2022. 140352.
• [12] M. Grochowski, A. Kwasigroch, and A. Mikołajczyk, “Selected technical issues of deep neural networks for image classification purposes,” Bull. Pol. Acad. Sci. Tech. Sci., vol. 67, no. 2, pp. 363–376, 2019, doi: 10.24425/bpas.2019.128485.
• [13] L. Zhang, L. Lin, X. Wu, S. Ding, and L. Zhang, “End-to-end photo-sketch generation via fully convolutional representation learning,” in Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, Jun 2015, pp. 627–634, doi: 10.1145/2671188.2749321.
• [14] P. Isola, J.-Y. Zhu, T. Zhou, and A.A. Efros, “Image-to-image translation with conditional adversarial networks,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jul 2017, pp. 1125–1134, doi: 10.1109/cvpr.2017.632.
• [15] J.-Y. Zhu, T. Park, P. Isola, and A.A. Efros, “Unpaired image- toimage translation using cycle-consistent adversarial networks,” in 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, Oct 2017, pp. 2223–2232, doi: 10.1109/iccv.2017.244.
• [16] N. Wang, W. Zha, J. Li, and X. Gao, “Back projection: An effective postprocessing method for gan-based face sketch synthesis,” Pattern Recognit. Lett., vol. 107, pp. 59–65, 2018, doi: 10.1016/j.patrec.2017.06.012.
• [17] H. Bi, N. Li, H. Guan, D. Lu, and L. Yang, “A multi-scale conditional generative adversarial network for face sketch synthesis,” in 2019 IEEE international conference on image processing (ICIP). IEEE, 2019, pp. 3876–3880, doi: 10.1109/icip. 2019.8803629.
• [18] M. Zhu, N. Wang, X. Gao, J. Li, and Z. Li, “Face photo-sketch synthesis via knowledge transfer.” in IJCAI, 2019, pp. 1048–1054, doi: 10.24963/ijcai.2019/147.
• [19] M. Zhang, R.Wang, X. Gao, J. Li, and D. Tao, “Dual-transfer face sketch-photo synthesis,” IEEE Trans. Image Process., vol. 28, no. 2, pp. 642–657, 2019, doi: 10.1109/tip. 2018.2869688.
• [20] X. Deng, H. Zhang, and X. Li, “Hpg-gan: High-quality priorguided blind face restoration generative adversarial network,” Electronics, vol. 12, no. 16, p. 3418, 2023, doi: 10.3390/electronics12163418.
• [21] X. Wang, Y. Li, H. Zhang, and Y. Shan, “Towards realworld blind face restoration with generative facial prior,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9168–9178, doi: 10.1109/cvpr46437.2021.00905.
• [22] X. Guo, L. Yi, H. Zou, and Y. Gao, “Generative facial prior for large-factor blind face super-resolution,” in J. Phys.-Conf. Ser., vol. 2078, no. 1. Publishing, 2021, p. 012045, doi: 10.1088/1742-6596/2078/1/012045.
• [23] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8110–8119, doi: 10.1109/cvpr42600.2020.00813.
• [24] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint arXiv:1511.06434, 2015, doi: 10.48550/arXiv.1511.06434.
• [25] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241, doi: 10.1007/978-3-319-24574-428.
• [26] X. Wang, K. Yu, C. Dong, and C. C. Loy, “Recovering realistic texture in image super-resolution by deep spatial feature transform,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Jun 2018, pp. 606–615, doi: 10.1109/cvpr.2018.00070.
• [27] J. Gu, Y. Shen, and B. Zhou, “Image processing using multicode gan prior,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun. 2020, pp. 3012–3021, doi: 10.1109/cvpr42600.2020.00308.
• [28] X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, and P. Luo, “Exploiting deep generative prior for versatile image restoration and manipulation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 11, pp. 7474–7489, Nov. 2022, doi: 10.1109/ tpami.2021.3115428.
• [29] J. Zhu,Y. Shen, D. Zhao, and B. Zhou, In-DomainGAN Inversion for Real Image Editing. Springer International Publishing, 2020, pp. 592–608, doi: 10.1007/978-3-030-58520-4_35.
• [30] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for realtime style transfer and super-resolution,” in Computer Vision – ECCV 2016, Springer. Springer International Publishing, 2016, pp. 694–711, doi: 10.1007/978-3-319-46475-6-43.
• [31] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014, doi: 10.48550/arXiv.1409.1556.
• [32] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Commun. ACM, vol. 63, no. 11, pp. 139–144, 2020, doi: 10.1145/3422622.
• [33] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2019, pp. 4690–4699, doi: 10.1109/cvpr.2019.00482.
• [34] K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre, “Xm2vtsdb: The extended m2vts database,” in Second International Conference on Audio and Video-Based Biometric Person Authentication, vol. 964, 1999, pp. 965–966, Database: XM2VTSDB.
• [35] A.M. Martinez, “The ar face database,”CVCTechnicalReport24, Tech. Rep., 1998, AR Face database.
• [36] W. Wan, Y. Yang, and H. J. Lee, “Generative adversarial learning for detail-preserving face sketch synthesis,” Neurocomputing, vol. 438, pp. 107–121, May 2021, doi: 10.1016/j.neucom.2021.01.050.
• [37] N.Wang, X. Gao, L. Sun, and J. Li, “Bayesian face sketch synthesis,” IEEE Trans. Image Process., vol. 26, no. 3, pp. 1264–1274, Mar 2017, doi: 10.1109/TIP.2017.2651375.
• [38] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2019, pp. 4401–4410, doi: 10.1109/cvpr.2019.00453.
• [39] R. Zhang, P. Isola, A.A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Jun 2018, pp. 586–595, doi: 10.1109/cvpr.2018.00068.
• [40] D.-P. Fan, S. Zhang, Y.-H. Wu, Y. Liu, M.-M. Cheng, B. Ren, P. Rosin, and R. Ji, “Scoot: A perceptual metric for facial sketches,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct 2019, pp. 5612–5622, doi: 10.1109/iccv.2019.00571.
• [41] L.-F. Chen, H.-Y. M. Liao, M.-T. Ko, J.-C. Lin, and G.-J. Yu, “A new LDA-based face recognition system which can solve the small sample size problem,” Pattern Recognit., vol. 33, no. 10, pp. 1713–1726, Oct 2000, doi: 10.1016/s0031-3203(99) 00139-9.
• [42] X. Li, F. Gao, and F. Huang, “High-quality face sketch synthesis via geometric normalization and regularization,” in 2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE, Jul 2021, pp. 1–6, doi: 10.1109/ICME51207.2021.9428348.
• [43] N. Wang, X. Gao, and J. Li, “Random sampling for fast face sketch synthesis,” Pattern Recognit., vol. 76, pp. 215–227, Apr 2018, doi: 10.1016/j.patcog.2017.11.008.
• [44] N. Balayesu and H. K. Kalluri, “An extensive survey on traditional and deep learning-based face sketch synthesis models,” Int. J. Inf. Technol., vol. 12, no. 3, pp. 995–1004, Nov 2020, doi: 10.1007/s41870-019-00386-8.
• [45] S. Mahfoud, A. Daamouche, M. Bengherabi, and A. Hadid, “Hand-drawn face sketch recognition using rank-level fusion of image quality assessment metrics,” Bull. Pol. Acad. Sci. Tech. Sci., vol. 70, no. 6, 2022, doi: 10.24425/bpasts.2022.143554.

Identyfikatory

DOI

10.24425/bpasts.2024.150109