Learning-based local quality assessment of reflectance confocal microscopy images for dermatology applications

• Autorzy: Sikorska Miroslawa , Skalski Andrzej , Wodzinski Marek , Witkowski Alexander , Pellacani Giovanni , Ludzik Joanna

Identyfikatory

DOI

10.1016/j.bbe.2021.05.009

• Języki publikacji: EN

Abstrakty

EN

Background and objective: Skin cancer is one of the most common types of cancer and its early diagnosis significantly reduces patient morbidity and mortality. Reflectance confocal microscopy (RCM) is a modern and non-invasive method of diagnosis that is becoming popular amongst clinical dermatologists. The frequent occurrence of artifacts in the images is one of the most challenging factors in making a diagnosis based on RCM. It impedes the diagnosis process for the dermatologist and makes its automation difficult. In this work, we employ artificial neural networks to propose a local quality assessment system. It allows for the detection of artifacts and non-informative component images both retrospectively or in real-time during the examination. Methods: In this research we address the quality assessment issue by proposing an artificial intelligence-based solution. 612 RCM mosaics were divided into small component images and manually classified in order to train the ResNeXt model in the quality verification context. A trained network was used to create an application that marks individual classes of the component images on the mosaic. Results: We achieved the average classification precision of 0.98 both for the validation and test data sets. In addition, we present local quality assessment statistics of the 1540 cases of skin lesions to show which types of skin lesions most often present with artifacts in their RCM images. Conclusions: In this research we investigate the utility of the deep convolution neural networks for the local quality assessment of the RCM images. We propose an AI-based system that may be effectively used as real-time support for the dermatologist during a RCM examination and as a base for the automation of the diagnostic process.

Słowa kluczowe

EN

RCM; reflectance confocal microscopy; skin lesions; deep learning; image artifact

PL

RCM; zmiany skórne; uczenie głębokie; artefakt obrazu

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2021
• Tom: Vol. 41, no. 3
• Strony: 880--890
• Opis fizyczny: Bibliogr. 41 poz., rys., tab., wykr.

Twórcy

autor

Sikorska Miroslawa

• AGH University of Science and Technology, Department of Measurement and Electronics, Al. Mickiewicza 30, Cracow 30059, Poland

• https://orcid.org/0000-0002-7187-5651

autor

Skalski Andrzej

• AGH University of Science and Technology, Department of Measurement and Electronics, Cracow, Poland

• https://orcid.org/0000-0003-2299-458X

autor

Wodzinski Marek

• AGH University of Science and Technology, Department of Measurement and Electronics, Cracow, Poland

• https://orcid.org/0000-0002-8076-6246

autor

Witkowski Alexander

• Oregon Health and Science University, Department of Dermatology, Portland, Oregon, USA

autor

Pellacani Giovanni

• University of Modena and Reggio Emilia, Department of Dermatology, Modena, Italy

autor

Ludzik Joanna

• Oregon Health and Science University, Department of Dermatology, Portland, Oregon, USA; Jagiellonian University Medical College, Department of Bioinformatics and Telemedicine, Cracow, Poland

Bibliografia

• [1] Apalla Z, Lallas A, Sotiriou E, Lazaridou E, Ioannides D. Epidemiological trends in skin cancer. Dermatology Practical Conceptual 2017;7:1–6. https://doi.org/10.5826/dpc.0702a01.
• [2] Arora G, Dubey A, Jaffery Z, Rocha. Bag of feature and support vector machine based early diagnosis of skin cancer. Neural Comput Appl 2020;2020:1–8. https://doi.org/10.1007/s00521- 020-05212-y.
• [3] Bozkurt A, Kose K, Alessi-Fox C, Gill M, Dy J, Brooks D, Rajadhyaksha M. A multiresolution convolutional neural network with partial label training for annotating reflectance confocal microscopy images of skin. In: Frangi AF, Schnabel JA, Davatzikos C, Alberola-López C, Fichtinger G, editors. Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. Cham: Springer International Publishing; 2018. p. 292–9.
• [4] Braun RP, Rabinovitz HS, Oliviero M, Kopf AW, Saurat JH. Dermoscopy of pigmented skin lesions. J Am Acad Dermatol 2005;52:109–21. https://doi.org/10.1016/j.jaad.2001.11.001.
• [5] Celebi M, Iyatomi H, Schaefer G, Stoecker WV. Lesion border detection in dermoscopy images. Comput Med Imaging Graph 2009;33:148–53. https://doi.org/10.1016/ j.compmedimag.2008.11.002.
• [6] Cinotti E, Labeille B, Debarbieux S, Carrera C, Lacarrubba F, Witkowski A, Moscarella E, Arzberger E, Kittler H, Bahadoran P, Gonzalez S, Guitera P, Agozzino M, Farnetani F, HofmannWellenhof R, Ardigo M, Rubegni P, Tognetti L, Łudzik J, Perrot J. Dermoscopy versus reflectance confocal microscopy for the diagnosis of lentigo maligna. J Eur Acad Dermatol Venereol 2018;32. https://doi.org/10.1111/jdv.14791.
• [7] Curchin CES, Wurm EMT, Lambie DLJ, Longo C, Pellacani G, Soyer HP. First experiences using reflectance confocal microscopy on equivocal skin lesions in queensland. Australas J Dermatol 2011;52:89–97. https://doi.org/10.1111/ j.1440-0960.2011.00756.x.
• [8] Franceschini C, Persechino F, Ardigo M. In vivo reflectance confocal microscopy in general dermatology: How to choose the right indication. Dermatology Practical Conceptual 2020;10 . https://doi.org/10.5826/dpc.1002a32 e2020032.
• [9] Gill M, Fox CA, Kose K. Artifacts and landmarks: pearls and pitfalls for in vivo reflectance confocal microscopy of the skin using the tissue-coupled device. Dermatology Online J 2019;25.
• [10] Guida S, De Pace B, Ciardo S, Farnetani F, Pellacani G. Noninvasive imaging for skin cancers-the european experience. Current Dermatology Reports 2019;8:172–81. https://doi.org/ 10.1007/s13671-019-00269-y.
• [11] Gulati, S. and Bhogal, R., 2020. Serving the dermatologists: Skin diseases detection, in: Tuba, M. and Akashe, S.J.A. (Ed.), Information and Communication Technology for Sustainable Development. Advances in Intelligent Systems and Computing. Springer, Singapore. volume 933, pp. 799–822.
• [12] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 770–8. https://doi.org/ 10.1109/CVPR.2016.90.
• [13] Janda M, Cust AE, Neale RE, Aitken JF, Baade PD, Green AC, Khosrotehrani K, Mar V, Soyer HP, Whiteman D. Early detection of melanoma: a consensus report from the australian skin and skin cancer research centre melanoma screening summit. Aust N Z J Public Health 2020;44:111–5. https://doi.org/10.1111/1753-6405.12972.
• [14] Johansen T, Møllersen K, Ortega S, Fabelo H, Garcia A, Marrero Callico G, Godtliebsen F. Recent advances in hyperspectral imaging for melanoma detection. Wiley Interdisciplinary Rev: Comput Stat 2019. https://doi.org/ 10.1002/wics.1465.
• [15] Johnson JM, Khoshgoftaar T. Survey on deep learning with class imbalance. J Big Data 2019;6. https://doi.org/10.1186/ s40537-019-0192-5.
• [16] Kallipolitis A, Stratigos A, Zarras A, Maglogiannis I. Explainable fully connected visual words for the classification of skin cancer confocal images: Interpreting the influence of visual words in classifying benign vs malignant pattern. In: 11th Hellenic Conference on Artificial Intelligence. https://doi.org/10.1145/3411408.3411435.
• [17] Kaur P, Dana KJ, Cula GO, Mack MC. Hybrid deep learning for reflectance confocal microscopy skin images. In: 2016 23rd International Conference on Pattern Recognition (ICPR). p. 1466–71. https://doi.org/10.1109/ICPR.2016.7899844.
• [18] Kingma DP, Ba J. Adam: A method for stochastic optimization. International Conference on Learning Representations 2014. abs/1412.6980.
• [19] Koller S, Wiltgen M, Ahlgrimm-Siess V, Weger W, HofmannWellenhof R, Richtig E, Smolle J, Gerger A. In vivo reflectance confocal microscopy: automated diagnostic image analysis of melanocytic skin tumours. J Eur Acad Dermatol Venereol 2011;25:554–8. https://doi.org/10.1111/j.1468-3083.2010.03834.x.
• [20] Kolm, I., Braun, R.P., 2012. How reflectance confocal microscopy works, in: Hofman-Wellenhof, R., Pellacani, G., Malvehy, J., and H.P.S. (Eds.), Reflectance Confocal Microscopy for Skin Diseases. Springer, pp. 7–10.
• [21] Kose K, Alessi-Fox C, Gill M, Dy JG, Brooks DH, Rajadhyaksha M. A machine learning method for identifying morphological patterns in reflectance confocal microscopy mosaics of melanocytic skin lesions in-vivo. Photonic Therapeutics and Diagnostics XII 2016:11–8. https://doi.org/10.1117/12.2212978.
• [22] Kose K, Bozkurt A, Alessi-Fox C, Brooks DH, Dy JG, Rajadhyaksha M, Gill M. Utilizing machine learning for image quality assessment for reflectance confocal microscopy. J Investigative Dermatol 2020;140:1214–22. https://doi.org/ 10.1016/j.jid.2019.10.018.
• [23] Kose, K., Bozkurt, A., Alessi-Fox, C., Gill, M., Longo, C., Pellacani, G., G.Dy, J., H.Brooks, D., Rajadhyaksha, M., 2021. Segmentation of cellular patterns in confocal images of melanocytic lesions in vivo via a multiscale encoder-decoder network (med-net). Medical Image Analysis 67. 10.1016/ j.media.2020.101841.
• [24] Krombholz R, Lunter D. A new method for in-situ skin penetration analysis by confocal raman microscopy. Molecules 2020;25. https://doi.org/ 10.3390/molecules25184222. https://www.mdpi.com/1420-3049/25/18/4222.
• [25] Malvehy J, Pérez Anker J, Toll A, Pigem R, Garcia A, Alos L, Puig S. Ex vivo confocal microscopy: revolution in fast pathology in dermatology. Br J Dermatol 2020;183. https://doi. org/10.1111/bjd.19017.
• [26] Mazzilli S, Cosio T, Diluvio L, Vollono L, Gonzalez S, Di Prete M, Orlandi A, Bianchi L, Campione E. Dermoscopy and reflectance confocal microscopy in the diagnosis and management of nail fold squamous cell carcinomay. J Med Life 2020;13:107–11. https://doi.org/10.25122/jml-2019-0129.
• [27] Mittal R, Balu M, Krasieva T, Potma E, Elkeeb L, Zachary C, Wilder-Smith P. Evaluation of stimulated raman scattering microscopy for identifying squamous cell carcinoma in human skin. Lasers Surgery Med 2013;45. https://doi.org/ 10.1002/lsm.22168.
• [28] Pacheco AGC, Sastry CS, Trappenberg T, Oore S, Krohling RA. On out-of-distribution detection algorithms with deep neural skin cancer classifiers. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). p. 3152–61. https://doi.org/10.1109/ CVPRW50498.2020.00374.
• [29] Pellacani G, Pepe P, Casari A, Longo C. Reflectance confocal microscopy as a second-level examination in skin oncology improves diagnostic accuracy and saves unnecessary excisions: A longitudinal prospective study. Br J Dermatol 2014;171. https://doi.org/10.1111/bjd.13148.
• [30] Rajadhyaksha M, Marghoob A, Rossi A, Halpern AC, Nehal KS. Reflectance confocal microscopy of skin in vivo: From bench to bedside. Lasers Surg Med 2017. https://doi.org/10.1002/ lsm.22600.
• [31] Rueter L, Ramadori P, Ulrich M, Jung S, Kardorff B, Lademann J. Reflectance confocal microscopy for noninvasive examination of nonmelanocytic tumors and virus-associated skin lesions in organ transplant recipients. Skin Res Technol 2019;26:1–14. https://doi.org/10.1111/srt.12813.
• [32] Shahriari N, Grant-Kels J, Rabinovitz H, Oliviero M, Scope A. In vivo reflectance confocal microscopy image interpretation for the dermatopathologist. J Cutan Pathol 2017;45. https:// doi.org/10.1111/cup.13084.
• [33] Shahriari N, Grant-Kels JM, Rabinovitz H, Oliviero M, Scope A. Reflectance confocal microscopy: Principles, basic terminology, clinical indications, limitations, and practical considerations. J Am Acad Dermatol 2021;84:1–14. https://doi. org/10.1016/j.jaad.2020.05.153.
• [34] Shorten C, Khoshgoftaar TM. A survey on image data augmentation for deep learning. J Big Data 2019;6. https://doi. org/10.1186/s40537-019-0197-0.
• [35] Wodzinski M, Pajak M, Skalski A, Witkowski A, Pellacani G, Ludzik J. Automatic quality assessment of reflectance confocal microscopy mosaics using attention-based deep neural network. In: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC). p. 1824–7. https://doi.org/10.1109/ EMBC44109.2020.9176557.
• [36] Wodzinski, M., Skalski, A., and; Giovanni Pellacani, A.W., Ludzik, J., 2019. Convolutional neural network approach to classify skin lesions using reflectance confocal microscopy. IEEE EMBC 2019, 4754–475710.1109/EMBC.2019.8856731.
• [37] Wurm, E.M.T., Kolm, I., Ahlgrimm-Siess, V., 2012. A hands-on guide to confocal imaging, in: Hofman-Wellenhof, R., Pellacani, G., Malvehy, J., and H.P.S. (Eds.), Reflectance Confocal Microscopy for Skin Diseases. Springer, pp. 7–10.
• [38] Xie S, Girshick R, Dollár P, Tu Z, He K. Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 5987–95. https://doi.org/10.1109/CVPR.2017.634.
• [39] Xiong Y, Ma S, Li X, Zhong X, Duan C, Chen Q. A metaanalysis of reflectance confocal microscopy for the diagnosis of malignant skin tumours. J Eur Acad Dermatology Venereology 2016;30. https://doi.org/10.1111/jdv.13712.
• [40] Zakeri A, Hokmabadi A. Improvement in the diagnosis of melanoma and dysplastic lesions by introducing abcd-pdt features and a hybrid classifier. Biocybern Biomed Eng 2018;38:456–66. https://doi.org/10.1016/j.bbe.2018.03.005.
• [41] Łudzik J, Witkowski A, Roterman-Konieczna I, Bassoli S, Farnetani F, Pellacani G. Improving diagnostic accuracy of dermoscopically equivocal pink cutaneous lesions with reflectance confocal microscopy in telemedicine settings: Double reader concordance evaluation of 316 cases. PloS one 2016;11. https://doi.org/10.1371/journal.pone.0162495.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2021).