Pseudo-panchromatic image guided transformer model for multispectral image demosaicing

• Autorzy: Chen Peng , Wang Heng , Wei Cong , Yang Jiangnan , Su Xinyu , Wu Jingjun , Li Shuangli

Identyfikatory

DOI

10.37190/oa/193937

• Języki publikacji: EN

Abstrakty

EN

The multispectral imaging system using the filter array can capture the multispectral information of the scene in one snapshot and reconstruct the complete multispectral image by demosaicing. However, the sparse sampling rate makes image captured by demosaicing a challenging problem. Although a lot of demosaicing algorithms have been developed, the existing well-performing methods have limitations in modeling non-local dependencies which lead to artifacts. To solve this problem, this paper proposes a transformer-based multispectral image demosaicing model to address the problem. The proposed model comprises a pseudo-panchromatic image generation network and a transformer-based multispectral image reconstruction network. Additionally, we designed a fusion module to combine the pseudo-panchromatic image with the raw mosaic image captured by the camera, leveraging the correlation between the band of multispectral images to improve the performance of the model. The experimental results show that the proposed method has the advantages of high reconstruction precision, strong anti-noise interference ability, and small calculation amount, which provides a better image reconstruction solution for constructing a highquality multispectral imaging system applied to multiple scenes.

Słowa kluczowe

EN

multispectral image; demosaicing; pseudo-panchromatic image; deep learning; transformer

• Wydawca: Oficyna Wydawnicza Politechniki Wrocławskiej
• Czasopismo: Optica Applicata
• Rocznik: 2024
• Tom: Vol. 54, nr 4
• Strony: 551--565
• Opis fizyczny: Bibliogr. 29 poz., rys., tab.

Twórcy

autor

Chen Peng

• School of Information Engineering, Southwest University of Science and Technology, Mianyang, 621000, Sichuan, China

autor

Wang Heng

• Department of Engineering Physics, Tsinghua University, Beijing, 100084, Beijing, China

autor

Wei Cong

• School of Electronic and Optical, Nanjing University of Science and Technology, Nanjing, 210094, Jiangsu, China

autor

Yang Jiangnan

• School of Information Engineering, Southwest University of Science and Technology, Mianyang, 621000, Sichuan, China

autor

Su Xinyu

• School of Information Engineering, Southwest University of Science and Technology, Mianyang, 621000, Sichuan, China

autor

Wu Jingjun

• School of Electronic and Optical, Nanjing University of Science and Technology, Nanjing, 210094, Jiangsu, China

autor

Li Shuangli

• School of Information Engineering, Southwest University of Science and Technology, Mianyang, 621000, Sichuan, China

Bibliografia

• [1] PU H.B., LIN L., SUN D.-W., Principles of hyperspectral microscope imaging techniques and their applications in food quality and safety detection: A review, Comprehensive Reviews in Food Science and Food Safety 18(4), 2019: 853-866. https://doi.org/10.1111/1541-4337.12432
• [2] XIONG F., ZHOU J., QIAN Y., Material based object tracking in hyperspectral videos, IEEE Transactions on Image Processing 29, 2020: 3719-3733. https://doi.org/10.1109/TIP.2020.2965302
• [3] HAO W.-H., WEN D.-W., Multispectral imaging for plant food quality analysis and visualization, Comprehensive Reviews in Food Science and Food Safety 17(1), 2018: 220-239. https://doi.org/10.1111/1541-4337.12317
• [4] ELAKSHER A.F., Fusion of hyperspectral images and lidar-based dems for coastal mapping, Optics and Lasers in Engineering 46(7), 2008: 493-498. https://doi.org/10.1016/j.optlaseng.2008.01.012
• [5] KARAGIANNIS G., High resolution, in situ, multispectral, spectroscopic mapping imaging system applied in heritage science, Optics and Lasers in Engineering 174, 2024: 107971. https://doi.org/10.1016/j.optlaseng.2023.107971
• [6] LI W., LIU Y., LING L., SHENG Z., CHENG S., YI Z., WU P., ZENG Q., TANG B., AHMAD S., The tunable absorber films of grating structure of AlCuFe quasicrystal with high Q and refractive index sensitivity, Surfaces and Interfaces 48, 2024: 104248. https://doi.org/10.1016/j.surfin.2024.104248
• [7] LIANG S., XU F., LI W., YANG W., CHENG S., YANG H., CHEN J., YI Z., JIANG P., Tunable smart mid infrared thermal control emitter based on phase change material VO2 thin film, Applied Thermal Engineering 232, 2023: 121074. https://doi.org/10.1016/j.applthermaleng.2023.121074
• [8] LIANG S., CHENG S., ZHANG H., YANG W., YI Z., ZENG Q., TANG B., WU P., AHMAD S., SUN T., Structural color tunable intelligent mid-infrared thermal control emitter, Ceramics International 50(13), 2024: 23611-23620. https://doi.org/10.1016/j.ceramint.2024.04.085
• [9] LI W., ZHAO W., CHENG S., ZHANG H., YI Z., SUN T., WU P., ZENG Q., RAZA R., Tunable metamaterial absorption device based on Fabry–Perot resonance as temperature and refractive index sensing, Optics and Lasers in Engineering 181, 2024:108368. https://doi.org/10.1016/j.optlaseng.2024.108368
• [10] BRAUERS J., AACH T., A color filter array based multispectral camera, [In] 12. Workshop Farbbildverarbeitung, Ilmenau, 2006.
• [11] GUPTA M., RAM M., Weighted bilinear interpolation based generic multispectral image demosaicking method, Journal of Graphic Era University, 2019: 108-118.
• [12] MIHOUBI S., LOSSON O., MATHON B., MACAIRE L., Multispectral demosaicing using intensity-based spectral correlation, [In] 2015 International Conference on Image Processing Theory, Tools and Applications (IPTA), IEEE, 2015: 461-466. https://doi.org/10.1109/IPTA.2015.7367188
• [13] BIAN L., WANG Y., ZHANG J., Generalized MSFA engineering with structural and adaptive nonlocal demosaicing, IEEE Transactions on Image Processing 30, 2021: 7867-7877. https://doi.org/10.1109/TIP.2021.3108913
• [14] REDMON J., DIVVALA S., GIRSHICK R., FARHADI A., You only look once: Unified, real-time object detection, [In] Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016: 779-788. https://doi.org/10.1109/CVPR.2016.91
• [15] MANSOUR Y., HECKEL R., Zero-shot noise2noise: Efficient image denoising without any data, [In] Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023: 14018-14027. https://doi.org/10.1109/CVPR52729.2023.01347
• [16] WANG B., FENG Y., LIU H., Multi‐scale features fusion from sparse LiDAR data and single image for depth completion, Electronics Letters 54(24), 2018: 1375-1377. https://doi.org/10.1049/el.2018.6149
• [17] ZHAO B., ZHENG J., DONG Y., SHEN N., YANG J., CAO Y.L., CAO Y.P., PPI edge infused spatial–spectral adaptive residual network for multispectral filter array image demosaicing, IEEE Transactions on Geoscience and Remote Sensing 61, 2023: 5405214. https://doi.org/10.1109/TGRS.2023.3297250
• [18] LI S., LIU Y., Deep densely-connected residual learning for multispectral image demosaicing, [In] 2023 8th International Conference on Signal and Image Processing (ICSIP), IEEE, 2023: 768-772. https://doi.org/10.1109/ICSIP57908.2023.10270836
• [19] PAN Z., LI B., BAO Y., CHENG H., Deep panchromatic image guided residual interpolation for multispectral image demosaicking, [In] 2019 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), IEEE, 2019: 1-5. https://doi.org/10.1109/ WHISPERS.2019.8920868
• [20] SHOPOVSKA I., JOVANOV L., PHILIPS W., RGB-NIR demosaicing using deep residual U-Net, [In] 2018 26th Telecommunications Forum (TELFOR), IEEE, 2018: 1-4. https://doi.org/10.1109/TELFOR. 2018.8611819
• [21] LIU S., ZHANG Y., CHEN J., LIM K.P., RAHARDJA S., A deep joint network for multispectral demosaicking based on pseudo-panchromatic images, IEEE Journal of Selected Topics in Signal Processing 16(4), 2022: 622-635. https://doi.org/10.1109/JSTSP.2022.3172865
• [22] FENG K., ZHAO Y., CHAN J.C.-W., KONG S.G., ZHANG X., WANG B., Mosaic convolution-attention network for demosaicing multispectral filter array images, IEEE Transactions on Computational Imaging 7, 2021: 864-878. https://doi.org/10.1109/TCI.2021.3102052
• [23] VASWANI A., SHAZEER N., PARMAR N., USZKOREIT J., JONES L., GOMEZ A.N., KAISER L., POLOSUKHIN I., Attention is All You Need, arXiv:1706.03762 [cs.CL], 2017. https://doi.org/10.48550/arXiv.1706. 03762
• [24] DOSOVITSKIY A., BEYER L., KOLESNIKOV A., WEISSENBORN D., ZHAI X., UNTERTHINER T., DEHGHANI M., MINDERER M., HEIGOLD G., GELLY S., USZKOREIT J., HOULSBY N., An image is worth 16x16 words: Transformers for image recognition at scale, arXiv:2010.11929 [cs.CV], 2020. https://doi.org/10.48550/arXiv.2010.11929
• [25] ARAD B., TIMOFTE R., YAHEL R., MORAG N., BERNAT A., WU Y., WU X., FAN Z., XIA C., ZHANG F., LIU S., LI Y., FENG C., LEI L., ZHANG M., FENG K., ZHANG X., YAO J., ZHAO Y., MA S., HE F., DONG Y., YU S., QIU D., LIU J., BI M., SONG B., SUN W.F., ZHENG J., ZHAO B., CAO Y., YANG J., CAO Y., KONG X., YU J., XUE Y., XIE Z.., NTIRE 2022 spectral demosaicing challenge and data set, [In] 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW ), IEEE, 2022: 881-895. https://doi.org/10.1109/CVPRW56347.2022.00103
• [26] MIHOUBI S., LOSSON O., MATHON B., MACAIRE L., Multispectral demosaicing using pseudo-panchromatic image, IEEE Transactions on Computational Imaging 3(4), 2017: 982-995. https://doi.org/ 10.1109/TCI.2017.2691553
• [27] WANG Z., BOVIK A.C., SHEIKH H.R., SIMONCELLI E.P., Image quality assessment: From error visibility to structural similarity, IEEE Transactions on Image Processing 13(4), 2004: 600-612. https://doi.org/10.1109/TIP.2003.819861
• [28] KRUSE F.A., LEFKOFF A.B., BOARDMAN J.W., HEIDEBRECHT K.B., SHAPIRO A.T., BARLOON P.J., GOETZ A.F.H., The spectral image processing system (SIPS)—interactive visualization and analysis of imaging spectrometer data, Remote Sensing of Environment 44(2-3), 1993: 145-163. https://doi.org/10.1016/0034-4257(93)90013-N
• [29] MIAO L., QI H., RAMANATH R., SNYDER W.E., Binary tree-based generic demosaicking algorithm for multispectral filter arrays, IEEE Transactions on Image Processing 15(11), 2006: 3550-3558. https://doi.org/10.1109/TIP.2006.877476