Multi-scale local-global transformer with contrastive learning for biomarkers segmentation in retinal OCT images

• Autorzy: Liu Xiaoming , Ding Yuanzhe , Zhang Ying , Tang Jinshan

Identyfikatory

DOI

10.1016/j.bbe.2024.02.001

• Języki publikacji: EN

Abstrakty

EN

Quantitative analysis of biomarkers in Optical Coherence Tomography (OCT) images plays an import role in the diagnosis and treatment of retinal diseases. However, biomarker segmentation in retinal OCT images is very hard due to the large variations in size and shape of retinal biomarkers, blurred boundaries, low contrast, and speckle interference. We proposed a novel Multi-scale Local-Global Transformer network (MsLGT-Net) for biomarker segmentation in retinal OCT images. The network combines the proposed Multi-scale Fusion Attention (MFA) module, Local-Global Transformer (LGT) module, and Contrastive Learning Enhancement (CLE) module to tackle the challenges of biomarker segmentation. Specifically, the proposed MFA module aims to enhance the network’s ability to learn multi-scale features of retinal biomarkers by effectively combining the local detail information and contextual semantic information of biomarkers at different scales, and improve the representation ability for different classes of biomarkers. The LGT module is designed to learn local and global information adaptively from multi-scale fused features to address the challenge of small biomarker segmentation. In addition, to distinguish features between different types of retinal biomarkers, we propose the CLE module to enhance the feature representation of different biomarkers. Our proposed method is validated on one public dataset and one local dataset. The experimental results show that the proposed method is more effective than other state-of-theart methods.

Słowa kluczowe

EN

optical coherence tomography; OCT; retinal biomarker segmentation; multiclass segmentation; proposal contrastive learning

PL

optyczna tomografia koherentna; OCT; segmentacja wieloklasowa

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2024
• Tom: Vol. 44, no. 1
• Strony: 231--246
• Opis fizyczny: Bibliogr. 53 poz., rys., tab.

Twórcy

autor

Liu Xiaoming

• School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan 430070, China
• Hubei Province Key Laboratory of Intelligent Information Processing and Real-Time Industrial System, Wuhan 430070, China

autor

Ding Yuanzhe

• School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan 430070, China
• Hubei Province Key Laboratory of Intelligent Information Processing and Real-Time Industrial System, Wuhan 430070, China

autor

Zhang Ying

• Wuhan Aier Eye Hospital, Wuhan 430064, China

autor

Tang Jinshan

• Health Informatics, Department of Health Administration and Policy, College of Public Health, George Mason University, Fairfax, VA 22030, USA

Bibliografia

• [1] Mekjavić PJ, Balčiūniené VJ, Ćeklić L, Ernest J, Jamrichova Z, Nagy ZZ, et al. The burden of macular diseases in central and eastern Europe - implications for healthcare systems. Value Health Regional Issues 2019;19:1-6.
• [2] Nicolò M, Ferro Desideri L, Vagge A, Traverso CE. Faricimab: an investigational agent targeting the Tie-2/angiopoietin pathway and VEGF-A for the treatment of retinal diseases. Expert Opin Invest Drugs 2021;30:193-200.
• [3] Hassan B, Ahmed R, Li B, Noor A, Hassan Zu. A comprehensive study capturing vision loss burden in Pakistan (1990-2025): Findings from the Global Burden of Disease (GBD) 2017 study. PloS one. 2019;14:e0216492.
• [4] Harding S, Greenwood R, Aldington S, Gibson J, Owens D, Taylor R, et al. Grading and disease management in national screening for diabetic retinopathy in England and Wales. Diabet Med 2003;20:965-71.
• [5] Venhuizen FG, van Ginneken B, Liefers B, van Asten F, Schreur V, Fauser S, et al. Deep learning approach for the detection and quantification of intraretinal cystoid fluid in multivendor optical coherence tomography. Biomed Opt Express 2018;9: 1545-69.
• [6] de Sisternes L, Simon N, Tibshirani R, Leng T, Rubin DL. Quantitative SD-OCT imaging biomarkers as indicators of age-related macular degeneration progression. Invest Ophthalmol Vis Sci 2014;55:7093-103.
• [7] Ma Y, Hao H, Xie J, Fu H, Zhang J, Yang J, et al. ROSE: a retinal OCT-angiography vessel segmentation dataset and new model. IEEE Trans Med Imaging 2020;40: 928-39.
• [8] Liu X, Cao J, Wang S, Zhang Y, Wang M. Confidence-guided topology-preserving layer segmentation for optical coherence tomography images with focus-column module. IEEE Trans Instrum Meas 2020;70:1-12.
• [9] Kurup AR, Wigdahl J, Benson J, Martínez-Ramón M, Solíz P, Joshi V. Automated malarial retinopathy detection using transfer learning and multi-camera retinal images. Biocybernetics and Biomedical Engineering 2023;43:109-23.
• [10] Liu X, Zhang D, Yao J, Tang J. Transformer and convolutional based dual branch network for retinal vessel segmentation in OCTA images. Biomed Signal Process Control 2023;83:104604.
• [11] Yu Y, Zhu H. Transformer-based cross-modal multi-contrast network for ophthalmic diseases diagnosis. Biocybernetics and Biomedical Engineering 2023; 43:507-27.
• [12] Toğaçar M, Ergen B, Tümen V. Use of dominant activations obtained by processing OCT images with the CNNs and slime mold method in retinal disease detection. Biocybernetics and Biomedical Engineering 2022;42:646-66.
• [13] Xie S, Okuwobi IP, Li M, Zhang Y, Yuan S, Chen Q. Fast and automated hyperreflective foci segmentation based on image enhancement and improved 3D U-Net in SD-OCT volumes with diabetic retinopathy. Transl Vis Sci Technol 2020; 9:21.
• [14] Xi X, Meng X, Qin Z, Nie X, Yin Y, Chen X. IA-net: informative attention convolutional neural network for choroidal neovascularization segmentation in OCT images. Biomed Opt Express 2020;11:6122-36.
• [15] Gende M, de Moura J, Novo J, Ortega M. End-to-end multi-task learning approaches for the joint epiretinal membrane segmentation and screening in OCT images. Comput Med Imaging Graph 2022;98:102068.
• [16] Tennakoon R, Gostar AK, Hoseinnezhad R, Bab-Hadiashar A. Retinal fluid segmentation in OCT images using adversarial loss based convolutional neural networks. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018): IEEE; 2018. p. 1436-40.
• [17] Hu J, Chen Y, Yi Z. Automated segmentation of macular edema in OCT using deep neural networks. Med Image Anal 2019;55:216-27.
• [18] Zhang G, Fu DJ, Liefers B, Faes L, Glinton S, Wagner S, et al. Clinically relevant deep learning for detection and quantification of geographic atrophy from optical coherence tomography: a model development and external validation study. The Lancet Digital Health 2021;3:e665-75.
• [19] Varga L, Kovács A, Grósz T, Thury G, Hadarits F, Dégi R, et al. Automatic segmentation of hyperreflective foci in OCT images. Comput Methods Programs Biomed 2019;178:91-103.
• [20] He X, Fang L, Tan M, Chen X. Intra-and inter-slice contrastive learning for point supervised OCT fluid segmentation. IEEE Trans Image Process 2022;31:1870-81.
• [21] Asgari R, Waldstein S, Schlanitz F, Baratsits M, Schmidt-Erfurth U, Bogunović H. UNet with spatial pyramid pooling for drusen segmentation in optical coherence tomography. Ophthalmic Medical Image Analysis: 6th International Workshop, OMIA 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, Proceedings 6: Springer; 2019. p. 77-85.
• [22] Shen Y, Li J, Zhu W, Yu K, Wang M, Peng Y, et al. Graph Attention U-Net for Retinal Layer Surface Detection and Choroid Neovascularization Segmentation in OCT Images. IEEE Trans Med Imaging 2023.
• [23] Gao S-H, Cheng M-M, Zhao K, Zhang X-Y, Yang M-H, Torr P. Res2net: A new multiscale backbone architecture. IEEE Trans Pattern Anal Mach Intell 2019;43:652-62.
• [24] Zhao H, Shi J, Qi X, Wang X, Jia J. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition2017. p. 2881-90.
• [25] Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. European Conference on Computer Vision 2018.
• [26] Liu Y, Zhou J, Liu L, Zhan Z, Hu Y, Fu Y, et al. FCP-net: a feature-compression-pyramid network guided by game-theoretic interactions for medical image segmentation. IEEE Trans Med Imaging 2022;41:1482-96.
• [27] Meng Q, Wang L, Wang T, Wang M, Zhu W, Shi F, et al. MF-Net: Multi-Scale Information Fusion Network for CNV Segmentation in Retinal OCT Images. Front Neurosci 2021;15:743769.
• [28] Wang M, Zhu W, Shi F, Su J, Chen H, Yu K, et al. MsTGANet: Automatic drusen segmentation from retinal OCT images. IEEE Trans Med Imaging 2021;41: 394-406.
• [29] Xiang D, Yan S, Guan Y, Cai M, Li Z, Liu H, et al. Semi-supervised Dual Stream Segmentation Network for Fundus Lesion Segmentation. IEEE Trans Med Imaging 2022.
• [30] Xing G, Chen L, Wang H, Zhang J, Sun D, Xu F, et al. Multi-scale pathological fluid segmentation in OCT with a novel curvature loss in convolutional neural network. IEEE Trans Med Imaging 2022;41:1547-59.
• [31] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. Adv Neural Inf Proces Syst 2017;30.
• [32] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929. 2020.
• [33] Strudel R, Garcia R, Laptev I, Schmid C. Segmenter: Transformer for semantic segmentation. Proceedings of the IEEE/CVF international conference on computer vision2021. p. 7262-72.
• [34] Han K, Xiao A, Wu E, Guo J, Xu C, Wang Y. Transformer in transformer. Adv Neural Inf Proces Syst 2021;34:15908-19.
• [35] Chen C-FR, Fan Q, Panda R. Crossvit: Cross-attention multi-scale vision transformer for image classification. Proceedings of the IEEE/CVF international conference on computer vision2021. p. 357-66.
• [36] Wang W, Xie E, Li X, Fan D-P, Song K, Liang D, et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF international conference on computer vision2021. p. 568-78.
• [37] Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, et al. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference on computer vision2021. p. 10012-22.
• [38] Dong X, Bao J, Chen D, Zhang W, Yu N, Yuan L, et al. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2022. p. 12124-34.
• [39] Zhang Y, Liu H, Hu Q. Transfuse: Fusing transformers and cnns for medical image segmentation. Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part I 24: Springer; 2021. p. 14-24.
• [40] Wang W, Chen C, Ding M, Yu H, Zha S, Li J. Transbts: Multimodal brain tumor segmentation using transformer. Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part I 24: Springer; 2021. p. 109-19.
• [41] Wu F, Fan A, Baevski A, Dauphin YN, Auli M. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:190110430. 2019.
• [42] Wang W, Zhou T, Yu F, Dai J, Konukoglu E, Van Gool L. Exploring cross-image pixel contrast for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2021. p. 7303-13.
• [43] Lin T-Y, Goyal P, Girshick R, He K, Doll´ ar P. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision2017. p. 2980-8.
• [44] Sotoudeh-Paima S, Hajizadeh F, Soltanian-Zadeh H. Labeled Retinal Optical Coherence Tomography Dataset for Classification of Normal. Drusen, and CNV Cases Mendeley Data 2021.
• [45] Kermany DS, Goldbaum M, Cai W, Valentim CC, Liang H, Baxter SL, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 2018;172(1122-31):e9.
• [46] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. Cham: Springer International Publishing; 2015. p. 234-41.
• [47] Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, et al. Ce-net: Context encoder network for 2d medical image segmentation. IEEE Trans Med Imaging 2019;38: 2281-92.
• [48] Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, et al. Attention unet: Learning where to look for the pancreas. arXiv preprint arXiv:180403999. 2018.
• [49] Zhou Z, Rahman Siddiquee MM, Tajbakhsh N, Liang J. Unet++: A nested u-net architecture for medical image segmentation. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4: Springer; 2018. p. 3-11.
• [50] Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, et al. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:210204306. 2021.
• [51] Cao H, Wang Y, Chen J, Jiang D, Zhang X, Tian Q, et al. Swin-unet: Unet-like pure transformer for medical image segmentation. European conference on computer vision. Springer; 2022. p. 205-18.
• [52] Feng S, Zhao H, Shi F, Cheng X, Wang M, Ma Y, et al. CPFNet: Context pyramid fusion network for medical image segmentation. IEEE Trans Med Imaging 2020;39: 3008-18.
• [53] Hassan B, Qin S, Hassan T, Ahmed R, Werghi N. Joint segmentation and quantification of chorioretinal biomarkers in optical coherence tomography scans: A deep learning approach. IEEE Trans Instrum Meas 2021;70:1-17.

Uwagi

PL

Opracowanie rekordu ze środków MNiSW, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).