Automated detection of crystalline retinopathy via fundus photography using multistage generative adversarial networks

• Autorzy: Young Choi Eun , Han Seung Hoon , Hee Ryu Ik , Kuk Kim Jin , Sik Lee In , Han Eoksoo , Kim Hyungsu , Yul Choi Joon , Keun Yoo Tae

Identyfikatory

DOI

10.1016/j.bbe.2023.10.005

• Języki publikacji: EN

Abstrakty

EN

Purpose: Crystalline retinopathy is characterized by reflective crystal deposits in the macula and is caused by various systemic conditions including hereditary, toxic, and embolic etiologies. Herein, we introduce a novel application of deep learning with a multistage generative adversarial network (GAN) to detect crystalline retinopathy using fundus photography. Methods: The dataset comprised major classes (healthy retina, diabetic retinopathy, exudative age-related macular degeneration, and drusen) and a crystalline retinopathy class (minor set). To overcome the limited data on crystalline retinopathy, we proposed a novel multistage GAN framework. The GAN was retrained after CutMix combination by inputting the GAN-generated synthetic data as new inputs to the original training data. After the multistage CycleGAN augmented the data for crystalline retinopathy, we built a deep-learning classifier model for detection. Results: Using the multistage CycleGAN facilitated realistic fundus photography synthesis with the characteristic features of retinal crystalline deposits. The proposed method outperformed typical transfer learning, prototypical networks, and knowledge distillation for both multiclass and binary classifications. The final model achieved an area under the curve of the receiver operating characteristics of 0.962 for internal validation and 0.987 for external validation for the detection of crystalline retinopathy. Conclusion: We introduced a deep learning approach for detecting crystalline retinopathy, a potential biomarker of underlying systemic pathological conditions. Our approach enables realistic pathological image synthesis and more accurate prediction of crystalline retinopathy, an essential but minor retinal condition.

Słowa kluczowe

EN

crystalline retinopathy; multistage GAN; CutMix; deep learning; fundus photography

PL

retinopatia krystaliczna; GAN; CutMix; uczenie głębokie; obraz dna oka

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2023
• Tom: Vol. 43, no. 4
• Strony: 725--735
• Opis fizyczny: Bibliogr. 56 poz., rys., tab., wykr.

Twórcy

autor

Young Choi Eun

• Department of Ophthalmology, Gangnam Severance Hospital, Institute of Vision Research, Yonsei University College of Medicine, Seoul, South Korea

autor

Han Seung Hoon

• Department of Brain & Cognitive Engineering, Graduate School of Korea University, South Korea

autor

Hee Ryu Ik

• Research and Development Department, VISUWORKS, Seoul, South Korea
• Department of Ophthalmology, B&VIIT Eye Center, Seoul, South Korea

autor

Kuk Kim Jin

• Research and Development Department, VISUWORKS, Seoul, South Korea
• Department of Ophthalmology, B&VIIT Eye Center, Seoul, South Korea

autor

Sik Lee In

• Department of Ophthalmology, B&VIIT Eye Center, Seoul, South Korea

autor

Han Eoksoo

• Electronics and Telecommunications Research Institute (ETRI), Daejeon, South Korea

autor

Kim Hyungsu

• Yonsei Eye Clinic, Seoul, South Korea

autor

Yul Choi Joon

• Department of Biomedical Engineering, Yonsei University, Wonju, South Korea

autor

Keun Yoo Tae

• Research and Development Department, VISUWORKS, Seoul, South Korea
• Department of Ophthalmology, B&VIIT Eye Center, Seoul, South Korea

Bibliografia

• [1] Wagner SK, Fu DJ, Faes L, Liu X, Huemer J, Khalid H, et al. Insights into systemic disease through retinal imaging-based oculomics. Trans Vis Sci Tech 2020;9:6. https://doi.org/10.1167/tvst.9.2.6.
• [2] Aiello LP, Cahill MT, Wong JS. Systemic considerations in the management of diabetic retinopathy. Am J Ophthalmol 2001;132:760-76.
• [3] Oh E, Kim YH, Ryu IH, Yoo TK. The role of big data analysis in identifying a relationship between glaucoma and diabetes mellitus. Ann Transl Med 2022;10: 948. https://doi.org/10.21037/atm-22-4134.
• [4] Pereira KM, Mega J, Cesaro T, Hunter A. An algorithm to identify the aetiology of crystalline retinopathy. Clin Exp Optom 2019;102:551-5. https://doi.org/10.1111/cxo.12879.
• [5] Kovach JL, Garcia DP. Crystalline retinopathy and optical coherence tomography angiography: new insights in pathogenesis. Ann Eye Sci 2020;5:24. https://doi.org/10.21037/aes-20-89.
• [6] Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J Am Med Assoc 2016;316:2402-10. https://doi.org/10.1001/jama.2016.17216.
• [7] Pavithra KC, Kumar P, Geetha M, Bhandary SV. Computer aided diagnosis of diabetic macular edema in retinal fundus and OCT images: A review. Biocybernet Biomed Eng 2023;43:157-88. https://doi.org/10.1016/j.bbe.2022.12.005.
• [8] Burlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM. Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks. JAMA Ophthalmol 2017;135:1170-6. https://doi. org/10.1001/jamaophthalmol.2017.3782.
• [9] Tulsani A, Kumar P, Pathan S. Automated segmentation of optic disc and optic cup for glaucoma assessment using improved UNET++ architecture. Biocybernet Biomed Eng 2021;41:819-32. https://doi.org/10.1016/j.bbe.2021.05.011.
• [10] Liao Y, Xia H, Song S, Li H. Microaneurysm detection in fundus images based on a novel end-to-end convolutional neural network. Biocybernet Biomed Eng 2021;41: 589-604. https://doi.org/10.1016/j.bbe.2021.04.005.
• [11] Karkuzhali S, Manimegalai D. Robust intensity variation and inverse surface adaptive thresholding techniques for detection of optic disc and exudates in retinal fundus images. Biocybernet Biomed Eng 2019;39:753-64. https://doi.org/ 10.1016/j.bbe.2019.07.001.
• [12] Yoo TK, Choi JY, Kim HK. Feasibility study to improve deep learning in OCT diagnosis of rare retinal diseases with few-shot classification. Med Biol Eng Compu 2021;59:401-15. https://doi.org/10.1007/s11517-021-02321-1.
• [13] Quellec G, Lamard M, Conze P-H, Massin P, Cochener B. Automatic detection of rare pathologies in fundus photographs using few-shot learning. Med Image Anal 2020;61:101660. https://doi.org/10.1016/j.media.2020.101660.
• [14] Yoo TK, Kim SH, Kim M, Lee CS, Byeon SH, Kim SS, et al. DeepPDT-Net: predicting the outcome of photodynamic therapy for chronic central serous chorioretinopathy using two-stage multimodal transfer learning. Sci Rep 2022;12:18689. https://doi. org/10.1038/s41598-022-22984-6.
• [15] Xiao J, Xu H, Fang D, Cheng C, Gao H. Boosting and rectifying few-shot learning prototype network for skin lesion classification based on the internet of medical things. Wirel Netw 2021. https://doi.org/10.1007/s11276-021-02713-z.
• [16] Mai S, Li Q, Zhao Q, Gao M. Few-Shot Transfer Learning for Hereditary Retinal Diseases Recognition. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, Cham: Springer International Publishing; 2021:97-107. https://doi.org/10.1007/978-3-030-87237-3_10.
• [17] Rajendra Kurup A, Wigdahl J, Benson J, Martínez-Ramón M, Solíz P, Joshi V. Automated malarial retinopathy detection using transfer learning and multicamera retinal images. Biocybernet Biomed Eng 2023;43:109-23. https://doi.org/ 10.1016/j.bbe.2022.12.003.
• [18] Ramachandran S, Niyas P, Vinekar A, John R. A deep learning framework for the detection of Plus disease in retinal fundus images of preterm infants. Biocybernet Biomed Eng 2021;41:362-75. https://doi.org/10.1016/j.bbe.2021.02.005.
• [19] Pachade S, Porwal P, Thulkar D, Kokare M, Deshmukh G, Sahasrabuddhe V, et al. Retinal fundus multi-disease image dataset (RFMiD): A dataset for multi-disease detection research. Data 2021;6:14. https://doi.org/10.3390/data6020014.
• [20] Cen L-P, Ji J, Lin J-W, Ju S-T, Lin H-J, Li T-P, et al. Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks. Nat Commun 2021;12:4828. https://doi.org/10.1038/s41467-021-25138-w.
• [21] Kermany DS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 2018;172:1122-1131.e9. https://doi.org/10.1016/j.cell.2018.02.010.
• [22] Yun S, Han D, Chun S, Oh SJ, Yoo Y, Choe J. CutMix: Regularization strategy to train strong classifiers with localizable features. IEEE/CVF Int Conf Comput Vision (ICCV) 2019;2019:6022-31. https://doi.org/10.1109/ICCV.2019.00612.
• [23] Pacal I, Karaboga D. A robust real-time deep learning based automatic polyp detection system. Comput Biol Med 2021;134:104519. https://doi.org/10.1016/j. Compbiomed.2021.104519.
• [24] Huang Z, Zhang J, Zhang Y, Shan H. DU-GAN: Generative adversarial networks with dual-domain U-net-based discriminators for low-dose CT denoising. IEEE Trans Instrum Meas 2022;71:1-12. https://doi.org/10.1109/TIM.2021.3128703.
• [25] You A, Kim JK, Ryu IH, Yoo TK. Application of generative adversarial networks (GAN) for ophthalmology image domains: a survey. Eye and Vision 2022;9:6. https://doi.org/10.1186/s40662-022-00277-3.
• [26] Almahairi A, Rajeshwar S, Sordoni A, Bachman P, Courville A. Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data. Proceedings of the 35th International Conference on Machine Learning, PMLR; 2018:195-204.
• [27] Yoo TK, Ryu IH, Kim JK, Lee IS, Kim HK. A deep learning approach for detection of shallow anterior chamber depth based on the hidden features of fundus photographs. Comput Methods Programs Biomed 2022;219:106735. https://doi. org/10.1016/j.cmpb.2022.106735.
• [28] Zhu J-Y, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, 2017:2223-32.
• [29] Harms J, Lei Y, Wang T, Zhang R, Zhou J, Tang X, et al. Paired cycle-GAN-based image correction for quantitative cone-beam computed tomography. Med Phys 2019;46:3998-4009. https://doi.org/10.1002/mp.13656.
• [30] Jenkins AJ, Grant MB, Busik JV. Lipids, hyperreflective crystalline deposits and diabetic retinopathy: potential systemic and retinal-specific effect of lipid-lowering therapies. Diabetologia 2022;65:587-603. https://doi.org/10.1007/s00125-022-05655-z.
• [31] Yamamoto T, Kase S, Murata M, Ishida S. Serum advanced glycation end-products and αB-crystallin in diabetic retinopathy patients. Biomed Rep 2022;16:1-5. https://doi.org/10.3892/br.2022.1511.
• [32] Ren Z, Song J. Ameliorative effect of pioglitazone on glucose induced glycation of α-crystallin: Management of complications associated with diabetic retinopathy. Int J Biol Macromol 2022;209:107-16. https://doi.org/10.1016/j.ijbiomac.2022.03.202.
• [33] Mathew NE, Kozak J, Jayasundera KT, Jain N, Prabhu SS. Crystalline retinopathy in a 6-year-old boy with history of high-dose tamoxifen use. Retinal Cases and Brief Reports 2022:10.1097/ICB.0000000000001280. https://doi.org/10.1097/ ICB.0000000000001280.
• [34] Thomas K, Curry J. Longstanding crystalline retinopathy secondary to intravitreal triamcinolone injection. Optomet Clin Pract 2022;4:5. https://doi.org/10.37685/uiwlibraries.2575-7717.4.1.1026.
• [35] Zhang S, Wang L, Liu Z, Sun H, Li Q, Xing C, et al. Observation of the characteristics of the natural course of Bietti crystalline dystrophy by fundus fluorescein angiography. BMC Ophthalmol 2021;21:239. https://doi.org/10.1186/s12886-021-01999-z.
• [36] Montemagni M, Arrigo A, Battaglia Parodi M, Bianco L, Antropoli A, Malegori A, et al. Optical coherence tomography angiography in Bietti crystalline dystrophy. European Journal of Ophthalmology 2022:11206721221143156. https://doi.org/ 10.1177/11206721221143156.
• [37] Zhang SJ, Wang LF, Xiao Z, Liu ZQ, Xing C, Li Q, et al. Analysis of radial peripapillary capillary density in patients with Bietti crystalline dystrophy by optical coherence tomography angiography. Biomed Environ Sci 2022;35:107-14. https://doi.org/10.3967/bes2022.015.
• [38] Saatci AO, Atas¸ F, Çetin GO, Kayabas¸ı M. Diagnostic and management strategies of Bietti crystalline dystrophy: Current perspectives. Clin Ophthalmol 2023;17: 953-67. https://doi.org/10.2147/OPTH.S388292.
• [39] Lambert LH, Shaikh N, Marx JL, Ramsey DJ. End-stage crystalline maculopathy with retinal atrophy in Sjögren-Larsson syndrome: a case report and review of the literature. Therapeutic Advances in Rare Disease 2022;3:26330040221122496. https://doi.org/10.1177/26330040221122496.
• [40] Sambyal N, Saini P, Syal R, Gupta V. Modified U-Net architecture for semantic segmentation of diabetic retinopathy images. Biocybernet Biomed Eng 2020;40: 1094-109. https://doi.org/10.1016/j.bbe.2020.05.006.
• [41] Gu Z, Li Y, Wang Z, Kan J, Shu J, Wang Q. Classification of diabetic retinopathy severity in fundus images using the vision transformer and residual attention. Comput Intell Neurosci 2023;2023:e1305583. https://doi.org/10.1155/2023/1305583.
• [42] Li T, Bo W, Hu C, Kang H, Liu H, Wang K, et al. Applications of deep learning in fundus images: A review. Med Image Anal 2021;69:101971. https://doi.org/ 10.1016/j.media.2021.101971.
• [43] James SP, Abraham CD. Devignetting fundus images via Bayesian estimation of illumination component and gamma correction. Biocybernet Biomed Eng 2021;41: 107-92. https://doi.org/10.1016/j.bbe.2021.06.008.
• [44] Kumar A, Kalia A, Kalia A. ETL-YOLO v4: A face mask detection algorithm in era of COVID-19 pandemic. Optik (Stuttg) 2022;259:169051. https://doi.org/10.1016/j.ijleo.2022.169051.
• [45] Tsiknakis N, Theodoropoulos D, Manikis G, Ktistakis E, Boutsora O, Berto A, et al. Deep learning for diabetic retinopathy detection and classification based on fundus images: A review. Comput Biol Med 2021::104599. https://doi.org/10.1016/j. Compbiomed.2021.104599.
• [46] Punjabi OS, Riaz K, Mets MB. Crystalline retinopathy in primary hyperoxaluria. J Am Assoc Pediatr Ophthalmol Strabismus 2011;15:214-6. https://doi.org/ 10.1016/j.jaapos.2010.12.015.
• [47] Oster SF, Mojana F, Freeman WR. Visualization of retinal emboli with high-resolution optical coherence tomography. Arch Ophthalmol 2009;127:218. https://doi.org/10.1001/archophthalmol.2008.572.
• [48] Shah VA, Cassell M, Poulose A, Sabates NR. Talc retinopathy. Ophthalmology 2008;115:755-755.e2. https://doi.org/10.1016/j.ophtha.2007.10.043.
• [49] Mutasa S, Sun S, Ha R. Understanding artificial intelligence based radiology studies: What is overfitting? Clin Imaging 2020;65:96-9. https://doi.org/10.1016/j.clinimag.2020.04.025.
• [50] Peng D, Yang W, Liu C, Lü S. SAM-GAN: Self-attention supporting multistage generative adversarial networks for text-to-image synthesis. Neural Netw 2021; 138:57-67. https://doi.org/10.1016/j.neunet.2021.01.023.
• [51] Lee J, Liu C, Kim J, Chen Z, Sun Y, Rogers JR, et al. Deep learning for rare disease: A scoping review. J Biomed Inform 2022;135:104227. https://doi.org/10.1016/j. Jbi.2022.104227.
• [52] Bhati A, Gour N, Khanna P, Ojha A. Discriminative kernel convolution network for multi-label ophthalmic disease detection on imbalanced fundus image dataset. Comput Biol Med 2023;153:106519. https://doi.org/10.1016/j. Compbiomed.2022.106519.
• [53] Singh R, Bharti V, Purohit V, Kumar A, Singh AK, Singh SK. MetaMed: Few-shot medical image classification using gradient-based meta-learning. Pattern Recogn 2021;120:108111. https://doi.org/10.1016/j.patcog.2021.108111.
• [54] Guo X, Chen Z, Liu J, Yuan Y. Non-equivalent images and pixels: Confidence-aware resampling with meta-learning mixup for polyp segmentation. Med Image Anal 2022;78:102394. https://doi.org/10.1016/j.media.2022.102394.
• [55] Chai L, Wang Z, Chen J, Zhang G, Alsaadi FE, Alsaadi FE, et al. Synthetic augmentation for semantic segmentation of class imbalanced biomedical images: A data pair generative adversarial network approach. Comput Biol Med 2022;150: 105985. https://doi.org/10.1016/j.compbiomed.2022.105985.
• [56] Kim BR, Yoo TK, Kim HK, Ryu IH, Kim JK, Lee IS, et al. Oculomics for sarcopenia prediction: a machine learning approach toward predictive, preventive, and personalized medicine. EPMA J 2022. https://doi.org/10.1007/s13167-022-00292-3.