Mask face inpainting based on improved generative adversarial network

• Autorzy: Liu Qingyu , Juanatas Roben A.

Identyfikatory

DOI

10.35784/acs-2023-12

• Języki publikacji: EN

Abstrakty

EN

Face recognition technology has been widely used in all aspects of people's lives. However, the accuracy of face recognition is greatly reduced due to the obscuring of objects, such as masks and sunglasses. Wearing masks in public has been a crucial approach to preventing illness, especially since the Covid-19 outbreak. This poses challenges to applications such as face recognition. Therefore, the removal of masks via image inpainting has become a hot topic in the field of computer vision. Deep learning-based image inpainting techniques have taken observable results, but the restored images still have problems such as blurring and inconsistency. To address such problems, this paper proposes an improved inpainting model based on generative adversarial network: the model adds attention mechanisms to the sampling module based on pix2pix network; the residual module is improved by adding convolutional branches. The improved inpainting model can not only effectively restore faces obscured by face masks, but also realize the inpainting of randomly obscured images of human faces. To further validate the generality of the inpainting model, tests are conducted on the datasets of CelebA, Paris Street and Place2, and the experimental results show that both SSIM and PSNR have improved significantly.

Słowa kluczowe

EN

face inpainting; generative adversarial network; residual network; attention mechanism

• Wydawca: Polskie Towarzystwo Promocji Wiedzy ; Lublin University of Technology
• Czasopismo: Applied Computer Science
• Rocznik: 2023
• Tom: Vol. 19, no 2
• Strony: 25--42
• Opis fizyczny: Bibliogr. 29 poz., fig., tab.

Twórcy

autor

Liu Qingyu

• National University, College of Computing and Information Technologies, Philippines

• https://orcid.org/0009-0000-9774-808X

autor

Juanatas Roben A.

• Huainan Normal University, School of Computer Science, China

• https://orcid.org/0009-0000-0572-0785

Bibliografia

• [1] Ding, D., Ram, S., & Rodriguez, J. J. (2019). Image Inpainting Using Nonlocal Texture Matching and Nonlinear Filtering. IEEE Transactions on Image Processing, 28(4), 1705–1719. https://doi.org/10.1109/TIP.2018.2880681
• [2] Goodfellow, I. (2014). NIPS 2014 Tutorial: Generative Adversarial Networks. arXiv.
• [3] https://doi.org/10.48550/arXiv.1701.00160
• [4] He, L., Qiang, Z., Shao, X., Lin, H., Wang, M., & Dai, F. (2022). Research on High-Resolution Face Image Inpainting Method Based on StyleGAN. Electronics, 11(10), 1620. https://doi.org/10.3390/electronics11101620
• [5] Hore, A., & Ziou, D. (2010). Image Quality Metrics: PSNR vs. SSIM. 2010 20th International Conference on Pattern Recognition (pp. 2366–2369). IEEE. https://doi.org/10.1109/ICPR.2010.579
• [6] Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017). Globally and locally consistent image completion. ACM Transactions on Graphics, 36(4), 1–14. https://doi.org/10.1145/3072959.3073659
• [7] Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017). Imavolge-to-Image Translation with Conditional Adversarial Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 5967–5976). IEEE. https://doi.org/10.1109/CVPR.2017.632
• [8] Jaderberg, M., Simonyan, K., & Zisserman, A. (2016). Spatial Transformer Networks. arXiv.
• [9] https://doi.org/10.48550/arXiv.1506.02025
• [10] Jiang, Y., Yang, F., Bian, Z., Lu, C., & Xia, S. (2022). Mask removal: Face inpainting via attributes. Multimedia Tools and Applications, 81(21), 29785–29797. https://doi.org/10.1007/s11042-022-12912-1
• [11] Jia, J., & Tang, Ch.-K. (2003). Image repairing: Robust image synthesis by adaptive ND tensor voting. 2003 IEEE Computer Society Conference on Computer Vision springeand Pattern Recognition, 2003. Proceedings. (pp. I-I). IEEE. https://doi.org/10.1109/CVPR.2003.1211414
• [12] Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 105–114). IEEE. https://doi.org/10.1109/CVPR.2017.19
• [13] Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., & Catanzaro, B. (2018). Image Inpainting for Irregular Holes Using Partial Convolutions. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Computer Vision – ECCV 2018 (vol. 11215, pp. 89–105). Springer. https://doi.org/10.1007/978-3-030-01252-6_6
• [14] Mou, C., Wang, Q., & Zhang, J. (2022). Deep Generalized Unfolding Networks for Image Restoration. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 17378–17389). IEEE. https://doi.org/10.1109/CVPR52688.2022.01688
• [15] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context Encoders: Feature Learning by Inpainting. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2536–2544). IEEE. https://doi.org/10.1109/CVPR.2016.278
• [16] Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv. https://doi.org/10.48550/arXiv.1505.04597
• [17] Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1–4), 259–268. https://doi.org/10.1016/0167-2789(92)90242-F.
• [18] Sagong, M., Shin, Y., Kim, S., Park, S., & Ko, S. (2019). PEPSI: Fast Image Inpainting With Parallel Decoding Network. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 11352–11360). IEEE. https://doi.org/10.1109/CVPR.2019.01162
• [19] Shao, X., Qiang, Z., Dai, F., He, L., & Lin, H. (2022). Face Image Completion Based on GAN Prior. Electronics, 11(13), 1997. https://doi.org/10.3390/electronics11131997
• [20] Simakov, D., Caspi, Y., Shechtman, E., & Irani, M. (2008). Summarizing visual data using bidirectional similarity. 2008 IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (pp. 1–8). IEEE. https://doi.org/10.1109/CVPR.2008.4587842
• [21] Wang, Y., Tao, X., Qi, X., Shen, X., & Jia, J. (2018). Image Inpainting via Generative Multi-column Convolutional Neural Networks. arXiv. https://doi.org/10.48550/arXiv.1810.08771
• [22] Wu, H., Zhou, J., & Li, Y. (2020). Deep Generative Model for Image Inpainting with Local Binary Pattern Learning and Spatial Attention. arXiv. https://doi.org/10.48550/arXiv.2009.01031
• [23] Xie, C., Liu, S., Li, C., Cheng, M.-M., Zuo, W., Liu, X., Wen, S., & Ding, E. (2019). Image Inpainting With Learnable Bidirectional Attention Maps. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 8857–8866). IEEE. https://doi.org/10.1109/ICCV.2019.00895
• [24] Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2019). Free-Form Image Inpainting With Gated Convolution. arXiv. https://doi.org/10.48550/arXiv.1806.03589
• [25] Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018). Generative Image Inpainting With Contextual Attention. arXiv. https://doi.org/10.48550/arXiv.1801.07892
• [26] Zeng, Y., Fu, J., Chao, H.. & Guo, B. (2019). Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) ( pp.1486–1494). IEEE. https://doi.org/10.1109/CVPR.2019.00158
• [27] Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018). Image Super-Resolution Using Very Deep Residual Channel Attention Networks. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Computer Vision – ECCV 2018 (vol. 11211, pp. 294–310). Springer. https://doi.org/10.1007/978-3-030-01234-2_18
• [28] Zheng, C., Cham, T.-J., & Cai, J. (2019). Pluralistic Image Completion. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1438–1447). IEEE. https://doi.org/10.1109/CVPR.2019.00153
• [29] Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N., & Liang, J. (2018). UNet++: A Nested U-Net Architecture for Medical Image Segmentation. arXiv. http://arxiv.org/abs/1807.10165