MFFNet: A multi-frequency feature extraction and fusion network for visual processing

• Autorzy: Deng Jinsheng , Zhang Zhichao , Yin Xiaoqing

Identyfikatory

DOI

10.24425/bpasts.2022.140466

• Języki publikacji: EN

Abstrakty

EN

Convolutional neural networks have achieved tremendous success in the areas of image processing and computer vision. However, they experience problems with low-frequency information such as semantic and category content and background color, and high-frequency information such as edge and structure. We propose an efficient and accurate deep learning framework called the multi-frequency feature extraction and fusion network (MFFNet) to perform image processing tasks such as deblurring. MFFNet is aided by edge and attention modules to restore high-frequency information and overcomes the multiscale parameter problem and the low-efficiency issue of recurrent architectures. It handles information from multiple paths and extracts features such as edges, colors, positions, and differences. Then, edge detectors and attention modules are aggregated into units to refine and learn knowledge, and efficient multi-learning features are fused into a final perception result. Experimental results indicate that the proposed framework achieves state-of-the-art deblurring performance on benchmark datasets.

Słowa kluczowe

EN

deblurring; multi-feature fusion; deep learning; attention mechanism

PL

rozmywanie; fuzja wielu funkcji; głęboka nauka; mechanizm uwagi

• Wydawca: Polska Akademia Nauk, Wydział IV Nauk Technicznych
• Czasopismo: Bulletin of the Polish Academy of Sciences. Technical Sciences
• Rocznik: 2022
• Tom: Vol. 70, nr 3
• Strony: art. no. e140466
• Opis fizyczny: Bibliogr. 29 poz., rys., tab.

Twórcy

autor

Deng Jinsheng

• College of Advanced Interdisciplinary Studies, National University of Defense Technology, Changsha 410000, China

autor

Zhang Zhichao

• College of Computer, National University of Defense Technology, Changsha 410000, China

autor

Yin Xiaoqing

• College of Advanced Interdisciplinary Studies, National University of Defense Technology, Changsha 410000, China

Bibliografia

• [1] A. Cichocki, T. Poggio, S. Osowski, and V. Lempitsky, “Deep learning: Theory and practice,” Bull. Pol. Acad. Sci. Tech. Sci., vol. 66, no. 6, pp. 757–759, 2018.
• [2] C.J. Schuler, H. Christopher Burger, S. Harmeling, and B. Scholkopf, “A machine learning approach for non-blind image deconvolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013, pp. 1067–1074.
• [3] J. Zhang, J. Pan,W.-S. Lai, R.W. Lau, and M.-H. Yang, “Learning fully convolutional networks for iterative non-blind deconvolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 3817–3825.
• [4] C. Dhanamjayulu et al., “Identification of malnutrition and prediction of BMI from facial images using real-time image processing and machine learning,” IET Image Process., 2021, doi: 10.1049/ipr2.12222.
• [5] L. Xu, J. S. Ren, C. Liu, and J. Jia, “Deep convolutional neural network for image deconvolution,” in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 1790–1798.
• [6] O. Kupyn et al., “DeblurGAN: Blind motion deblurring using conditional adversarial networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 81838192.
• [7] O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, “DeblurGANv2: Deblurring (orders-of-magnitude) faster and better,” in Proc. 2019 IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Seoul, Korea (South), 2019, pp. 8877–8886, doi: 10.1109/ICCV.2019.00897.
• [8] S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 3883–3891.
• [9] G. Lin, A. Milan, C. Shen, and I. Reid, “RefineNet: Multipath refinement networks for high-resolution semantic segmentation,” in 2017 IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Honolulu, HI, USA, 2017, pp. 5168–5177, doi: 10.1109/CVPR.2017.549.
• [10] V. Nekrasov, C. Shen, and I. Reid, “Light-weight RefineNet for real-time semantic segmentation,” in Proc. BMVC, 2018.
• [11] H. Zhang, Y. Dai, H. Li, and P. Koniusz, “Deep stacked hierarchical multi-patch network for image deblurring,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2019, pp. 5978–5986.
• [12] X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, “Scale-recurrent network for deep image deblurring,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 8174–8182.
• [13] M. Ye, D. Lyu, and G. Chen, “Scale-iterative upscaling network for image deblurring,” IEEE Access, vol, 8, pp. 18316–18325, 2020.
• [14] T.R., Gadekallu, D.S. Rajput, M.P.K. Reddy, K. Lakshmanna, S. Bhattacharya, S. Singh, A. Jolfaei, and M. Alazab, “A novel PCA–whale optimization-based deep neural network model for classification of tomato plant diseases using GPU,” J. Real Time Image Process., vol. 18, pp. 1383–1396, 2021, doi: 10.1007/s11554-020-00987-8.
• [15] M. Hirsch, C.J. Schuler, S. Harmeling, and B. Schölkopf, “Fast removal of non-uniform camera shake,” in Proc. Int. Conf. Comput. Vis., Nov. 2011, pp. 463–470.
• [16] Z. Zhang, H. Chen, X. Yin, and J. Deng, “Joint generative image deblurring aided by edge attention prior and dynamic kernel selection,” Wirel. Commun. Mob. Comput., vol. 2021, 2021.
• [17] Z. Zhang, H. Chen, X. Yin, and J. Deng, “EAWNet: An edge attention-wise objector for real-time visual internet of things,” Wirel. Commun. Mob. Comput., vol. 2021, 2021.
• [18] Z. Zhang, H. Chen, J. Deng, and X. Yin, “A double feature fusion network with progressive learning for sharper inpainting,” in Proc. 2021 Int. Joint Conf. Neural Netw. (IJCNN), 2021, pp. 1–8, doi: 10.1109/IJCNN52387.2021.9534018.
• [19] Z. Zhang, J. Deng, H. Chen, and X. Yin, “Rotated YOLOv4 with attention-wise object detectors in aerial images,” in Proc. 2021 4th Int. Conf. Robot Systems and Applications, April 2021, pp. 1–6.
• [20] A. Naeem, A.R. Javed, M. Rizwan, S. Abbas, J.C.W. Lin, and T.R. Gadekallu, “DARE-SEP: A hybrid approach of distance aware residual energy-efficient SEP for WSN,” IEEE Trans. Green Comm. Netw., vol. 5, no. 2, pp. 611–621, 2021.
• [21] J. Dai, and Y.Wang, “Multiscale residual convolution neural network and sector descriptor-based road detection method,” IEEE Access, vol. 7, pp. 173377–173392, 2019.
• [22] K. Schelten, S. Nowozin, J. Jancsary, C. Rother, and S. Roth, “Interleaved regression tree field cascades for blind image deconvolution,” in Proc. IEEE Winter Conf. Appl. Comput. Vis., Jan. 2015, pp. 494–501.
• [23] K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep CNN denoiser prior for image restoration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 3929–3938.
• [24] Q. Wang, S. Shi, S. Zheng, K. Zhao, and X. Chu, “FADNet : A fast and accurate network for disparity estimation,” in Proc. ICRA, 2020.
• [25] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. 2016 IEEE Conf. Comput. Vis. and Pattern Recognit, (CVPR), 2016.
• [26] S. Zheng, Z. Zhu, J. Cheng, Y. Guo, and Y. Zhao, “Edge heuristic GAN for non-uniform blind deblurring,” IEEE Signal Process. Lett., vol. 26, no. 10, pp. 1546–1550, 2019.
• [27] J. Mei, Z.Wu, X. Chen, et al. “DeepDeblur: Text image recovery from blur to sharp,” Multimed. Tools Appl., vol. 78, pp. 18869–18885, 2019, doi: 10.1007/s11042-019-7251-y.
• [28] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in Proc. 2016 IEEE Conf. Comput. Vis. and Pattern Recognit, (CVPR), 2016.
• [29] P. Zhu et al., “VisDrone-VID2019: The vision meets drone object detection in video challenge results,” in Proc. 2019 IEEE/CVF Int. Conf. Comput. Vis. Workshop (ICCVW), Seoul, Korea (South), 2019, pp. 227–235, doi: 10.1109/ICCVW.2019.00031.

Uwagi

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2022-2023).