Deep learning on ultrasound images of thyroid nodules

• Autorzy: Sharifi Yasaman , Bakhshali Mohamad Amin , Dehghani Toktam , DanaiAshgzari Morteza , Sargolzaei Mahdi , Eslami Saeid

Identyfikatory

DOI

10.1016/j.bbe.2021.02.008

• Języki publikacji: EN

Abstrakty

EN

Due to safety, easy accessibility, noninvasively and cost-effectiveness of ultrasound imaging, this technology becomes one of the main contributors for analyzing thyroid nodules. However, interpretation of ultrasound images is a challenging task that subjects to the radiologist’s prior medical knowledge and observational skills. There is a significant need for reliable, objective, and automated approaches for the meaningful assessment of ultrasound images. Many areas of machine learning including computer vision and image processing have been revolutionized by the recent advances in the field of deep learning. The current study systematically reviews the existing literatures and evaluates technical characteristics of the deep learning applications on the ultrasound images of thyroid nodules. In this review, all of the included studies have been published from 2017 to 2020 indicating the recent growing interest in the utilization of deep learning-based techniques for assessment of ultrasound images of thyroid nodules. Although deep learning has demonstrated potential for analyzing thyroid nodules’ ultrasound images, this review highlights several existing barriers that need to be addressed in future works such as dealing with data limitation, generating public and valid datasets, and determining standard evaluation metrics. This survey outlines several methods (e.g., data augmentation and transfer learning) recently proposed to address similar challenges in other fields. Furthermore, to improve the diagnostic accuracy of the deep learning models, utilization of complementary information with multi-modal images are suggested.

Słowa kluczowe

EN

thyroid nodules; deep learning; ultrasonography; computer assisted diagnosis; medical diagnosis

PL

guzki tarczycy; uczenie głębokie; ultrasonografia; diagnoza wspomagana komputerowo; diagnoza medyczna

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2021
• Tom: Vol. 41, no. 2
• Strony: 636--655
• Opis fizyczny: Bibliogr. 82 poz., rys., tab., wykr.

Twórcy

autor

Sharifi Yasaman

• Medical Informatics, Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran

autor

Bakhshali Mohamad Amin

• Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran

autor

Dehghani Toktam

• Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran; Salman Higher Education Institute, Mashhad, Iran

autor

DanaiAshgzari Morteza

• Department of Computer, Faculty of Engineering, Islamic Azad University of Mashhad, Mashhad, Iran

autor

Sargolzaei Mahdi

• Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran

• https://orcid.org/0000-0001-7493-3071

autor

Eslami Saeid

• Pharmaceutical Research Center, Mashhad University of Medical Sciences, Mashhad, Iran; Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran; Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands

• https://orcid.org/0000-0003-3755-1212

Bibliografia

• [1] Khachnaoui H, Guetari R, Khlifa N. A review on Deep Learning in thyroid ultrasound Computer-Assisted Diagnosis systems. IEEE International Conference on Image Processing, Applications and Systems (IPAS). IEEE; 2018.
• [2] Cooper D. American Thyroid Association (ATA) guidelines taskforce on thyroid nodules and differentiated thyroid cancer. Revised American Thyroid Association management guidelines for patients with thyroid nodules and differentiated thyroid cancer. Thyroid 2009;19:1167–214.
• [3] Acharya UR, Swapna G, Sree SV, Molinari F, Gupta S, Bardales RH, et al. A review on ultrasound-based thyroid cancer tissue characterization and automated classification. Technol Cancer Res Treat 2014;13(4):289–301.
• [4] Zhuang Y, Li C, Hua Z, Chen K, Lin JL. A novel TIRADS of US classification. Biomed Eng Online, 17(1), 82.
• [5] Ren J, Baek JH, Chung SR, Choi YJ, Jung CK, Lee JH. Degenerating thyroid nodules: ultrasound diagnosis, clinical significance, and management. Korean J Radiol 2019;20 (6):947–55.
• [6] Lamartina L, Deandreis D, Durante C, Filetti S. ENDOCRINE TUMOURS: imaging in the follow-up of differentiated thyroid cancer: current evidence and future perspectives for a riskadapted approach. Eur J Endocrinol 2016;175(5):R185–202.
• [7] Ma L, Ma C, Liu Y, Wang X. Thyroid diagnosis from SPECT images using convolutional neural network with optimization. Comput Intelligence Neurosci 2019;2019.
• [8] Rajendra Acharya U, Swapna G, Vinitha Sree S, Molinari F, Gupta S, Bardales RH, et al. A review on ultrasound-based thyroid cancer tissue characterization and automated classification. Technol Cancer Res Treat 2014;13(4):289–301.
• [9] Ch WM, Rosairo Sh, Ratnatunga N, Wickramasinghe ND. Diagnostic accuracy of ultrasound characteristics in the identification of malignant thyroid nodules. BMC Res Notes 2019;12(193):1–7.
• [10] An JY, Unsdorfer KM, Weinreb JC. BI-RADS, C-RADS, CADRADS, LI-RADS, Lung-RADS, NI-RADS, O-RADS, PI-RADS, TIRADS: reporting and data systems. RadioGraphics 2019;39 (5):1435–6.
• [11] Koprowski R, Zieleźnik W, Wróbel Z, Małyszek J, Steôpień B, Wójcik W. Assessment of significance of features acquired from thyroid ultrasonograms in Hashimoto’s disease. Biomed Eng Online 2012;11(1):48.
• [12] Koprowski R, Korzyńska A, Wróbel Z, Zieleźnik W, Witkowska A, Małyszek J, et al. Influence of the measurement method of features in ultrasound images of the thyroid in the diagnosis of Hashimoto’s disease. Biomed Eng Online 2012;11 (1):91.
• [13] Prochazka A, Gulati S, Holinka S, Smutek D. Classification of thyroid nodules in ultrasound images using directionindependent features extracted by two-threshold binary decomposition. Technol Cancer Res Treatment. 2019;18:1533033819830748.
• [14] Nugroho HA, Frannita EL, Ardiyanto I, Choridah L. Computer aided diagnosis for thyroid cancer system based on internal and external characteristics. J King Saud Univ-Comp Inf Sci 2019.
• [15] Tamhane S, Gharib H. Thyroid nodule update on diagnosis and management. Clin Diab Endocrinol 2016;2(1):17.
• [16] Floridi C, Cellina M, Buccimazza G, Arrichiello A, Sacrini A, Arrigoni F, et al. Ultrasound imaging classifications of thyroid nodules for malignancy risk stratification and clinical management: state of the art. Gland Surgery 2019;8(Suppl 3): S233.
• [17] Yang R, Zou X, Zeng H, Zhao Y, Ma X. Comparison of diagnostic performance of five different ultrasound TI-RADS classification guidelines for thyroid nodules. Front Oncol 2020;10.
• [18] Namakshenas P, Mojra A. Numerical study of non-Fourier thermal ablation of benign thyroid tumor by focused ultrasound (FU). Biocybernet Biomed Eng 2019;39(3):571–85.
• [19] Guan Q, Wang Y, Du J, Qin Y, Lu H, Xiang J, et al. Deep learning based classification of ultrasound images for thyroid nodules: a large scale of pilot study. Ann Transl Med 2019;7 (7):137.
• [20] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521 (7553):436.
• [21] Yao AD, Cheng DL, Pan I, Kitamura F. Deep learning in neuroradiology: a systematic review of current algorithms and approaches for the new wave of imaging technology. Radiol Artif Intell 2020;2(2) e190026.
• [22] Fukushima K, Miyake S. Neocognitron: a new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognit 1982;15(6):455–69.
• [23] Bakator M, Radosav D. Deep learning and medical diagnosis: a review of literature. Multimodal Technol Interact 2018;2 (3):47.
• [24] Suzuki K. Overview of deep learning in medical imaging. Radiol Phys Technol 2017;10(3):257–73.
• [25] Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, et al. A comparison of deep learning performance against healthcare professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health 2019;1(6):e271–97.
• [26] Arthur F, Hossein KR. Deep learning in medical image analysis: a third eye for doctors. J Stomatol Oral Maxillofac Surg 2019.
• [27] Chen Y-C, Hong D-J-K, Wu C-W, Mupparapu M. The use of deep convolutional neural networks in biomedical imaging: a review. J Orofacial Sci 2019;11(1):3.
• [28] Cheng H-D, Shan J, Ju W, Guo Y, Zhang L. Automated breast cancer detection and classification using ultrasound images: a survey. Pattern Recognit 2010;43(1):299–317.
• [29] Huang Q, Luo Y, Zhang Q. Breast ultrasound image segmentation: a survey. Int J Comput Assist Radiol Surg 2017;12(3):493–507.
• [30] Jabarulla MY, Lee H-N. Computer aided diagnostic system for ultrasound liver images: a systematic review. Optik 2017;140:1114–26.
• [31] Sollini M, Cozzi L, Chiti A, Kirienko M. Texture analysis and machine learning to characterize suspected thyroid nodules and differentiated thyroid cancer: where do we stand? Eur J Radiol 2018;99:1–8.
• [32] Huang Q, Zhang F, Li X. Machine learning in ultrasound computer-aided diagnostic systems: a survey. Biomed Res Int 2018;2018.
• [33] Akkus Z, Cai J, Boonrod A, Zeinoddini A, Weston AD, Philbrick KA, et al, 2019. A survey of deep-learning applications in ultrasound: artificial intelligence-powered ultrasound for improving clinical workflow. J Am Coll Radiol, 16(9 Pt B), 1318–28.
• [34] Xu L, Gao J, Wang Q, Yin J, Yu P, Bai B, et al. Computer-aided diagnosis systems in diagnosing malignant thyroid nodules on ultrasonography: a systematic review and meta-analysis. Eur Thyroid J 2019;9:186–93.
• [35] Chambara N, Ying M. The diagnostic efficiency of ultrasound computer–aided diagnosis in differentiating thyroid nodules: a systematic review and narrative synthesis. Cancers 2019;11 (11):1759.
• [36] Chen J, You H, Li K. A review of thyroid gland segmentation and thyroid nodule segmentation methods for medical ultrasound images. Comput Methods Programs Biomed 2020;185 105329.
• [37] Liu S, Wang Y, Yang X, Lei B, Liu L, Li SX, et al, 2019. Deep learning in medical ultrasound analysis: a review. Engineering. 2019;5(2):261–275.
• [38] Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009;6(7) e1000097.
• [39] Malhotra R. A systematic review of machine learning techniques for software fault prediction. Appl Soft Comput 2015;27:504–18.
• [40] Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977:159–74.
• [41] Chi J, Walia E, Babyn P, Wang J, Groot G, Eramian M. Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network. J Digit Imaging 2017;30 (4):477–86.
• [42] Ma J, Wu F, Jiang T, Zhao Q, Kong D. Ultrasound image-based thyroid nodule automatic segmentation using convolutional neural networks. Int J Comput Assist Radiol Surg 2017;12 (11):1895–910.
• [43] Ma J, Wu F, Zhu J, Xu D, Kong D. A pre-trained convolutional neural network based method for thyroid nodule diagnosis. Ultrasonics 2017;73:221–30.
• [44] Ma J, Wu F, Jiang T, Zhu J, Kong D. Cascade convolutional neural networks for automatic detection of thyroid nodules in ultrasound images. Med Phys 2017;44(5):1678–91.
• [45] Seo JK, Kim YJ, Kim KG, Shin I, Shin JH, Kwak JY. Differentiation of the follicular neoplasm on the gray-scale US by image selection subsampling along with the marginal outline using convolutional neural network. Biomed Res Int 2017.
• [46] Song J, Chai YJ, Masuoka H, Park SW, Kim SJ, Choi JY, et al. Ultrasound image analysis using deep learning algorithm for the diagnosis of thyroid nodules. Medicine 2019;98(15) e15133.
• [47] Wang L, Yang S, Yang S, Zhao C, Tian G, Gao Y, et al. Automatic diagnosis for thyroid nodules in ultrasound images by deep neural networks. Med Image Anal 2020;61 101665.
• [48] Ko SY, Lee JH, Yoon JH, Na H, Hong E, Han K, et al. Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound. Head Neck 2019;41(4):885–91.
• [49] Wang Y, Ke W, Wan P. A method of ultrasonic image recognition for thyroid papillary carcinoma based on deep convolution neural network. NeuroQuantology 2018;16 (5):757–68.
• [50] Li H, Weng J, Shi Y, Gu W, Mao Y, Wang Y, et al. An improved deep learning approach for detection of thyroid papillary cancer in ultrasound images. Sci Rep 2018;8(1).
• [51] Li X, Zhang S, Zhang Q, Wei X, Pan Y, Zhao J, et al. Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol 2019;20 (2):193–201.
• [52] Buda M, Wildman-Tobriner B, Hoang JK, Thayer D, Tessler FN, Middleton WD, et al. Management of thyroid nodules seen on US images: deep learning may match performance of radiologists. Radiology 2019;292(3):695–701.
• [53] Liang X, Yu J, Liao J, Chen Z, 2020. Convolutional neural network for breast and thyroid nodules diagnosis in ultrasound imaging. Biomed Res Int, 2020.
• [54] Liu T, Guo Q, Lian C, Ren X, Liang S, Yu J, et al. Automated detection and classification of thyroid nodules in ultrasound images using clinical-knowledge-guided convolutional neural networks. Med Image Anal 2019;58 101555.
• [55] Wang L, Zhang L, Zhu M, Qi X, Yi Z. Automatic diagnosis for thyroid nodules in ultrasound images by deep neural networks. Med Image Anal 2020;61 101665.
• [56] Buda M, Wildman-Tobriner B, Castor K, Hoang JK, Mazurowski MA. Deep learning-based segmentation of nodules in thyroid ultrasound: improving performance by utilizing markers present in the images. Ultrasound Med Biol 2020;46(2):415–21.
• [57] Lee E, Ha H, Kim HJ, Moon HJ, Byon JH, Huh S, et al. Differentiation of thyroid nodules on US using features learned and extracted from various convolutional neural networks. Sci Rep 2019;9(1):1–11.
• [58] Nguyen DT, Pham TD, Batchuluun G, Yoon HS, Park KR. Artificial intelligence-based thyroid nodule classification using information from spatial and frequency domains. J Clin Med 2019;8(11).
• [59] Zhou H, Wang K, Tian J. Online transfer learning for differential diagnosis of benign and malignant thyroid nodules with ultrasound images. IEEE Trans Biomed Eng 2020.
• [60] Park VY, Han K, Seong YK, Park MH, Kim EK, Moon HJ, et al. Diagnosis of thyroid nodules: performance of a deep learning convolutional neural network model vs radiologists. Sci Rep 2019;9(1):17843.
• [61] Moussa O, Khachnaoui H, Guetari R, Khlifa N. Thyroid nodules classification and diagnosis in ultrasound images using fine-tuning deep convolutional neural network. Int J Imaging Syst Technol 2020;30(1):185–95.
• [62] Sai Sundar KV, Rajamani KT, Siva Sankara Sai S, 2019. Exploring image classification of thyroid ultrasound images using deep learning. Vol. 30. Lecture Notes in Computational Vision and Biomechanics (pp. 1635–41).
• [63] Liu T, Xie S, Yu J, Niu L, Sun W, 2017. Classification of thyroid nodules in ultrasound images using deep model based transfer learning and hybrid features. Paper presented at the 2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017.
• [64] Liu T, Xie S, Zhang Y, Yu J, Niu L, Sun W. Feature selection and thyroid nodule classification using transfer learning. 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), Melbourne, VIC, 2017, pp. 1096–99.
• [65] Sun J, Sun T, Yuan Y, Zhang X, Shi Y, Lin Y, 2019. Automatic Diagnosis of Thyroid Ultrasound Image Based on FCNAlexNet and Transfer Learning. 23rd IEEE International Conference on Digital Signal Processing, DSP 2018; 2019.
• [66] Mei X, Dong X, Deyer T, Zeng J, Trafalis T, Fang Y, 2018. Thyroid nodule benignty prediction by deep feature extraction. 2017 IEEE 17th International Conference on Bioinformatics and Bioengineering (BIBE), Washington, DC, 2017, pp. 241–5.
• [67] Zhu Y, Fu Z, Fei J, 2018. An image augmentation method using convolutional network for thyroid nodule classification by transfer learning. 2017 3rd IEEE International Conference on Computer and Communications (ICCC), Chengdu, 2017, pp. 1819–23.
• [68] Song W, Li S, Liu J, Qin H, Zhang B, Zhang S, et al. Multitask cascade convolution neural networks for automatic thyroid nodule detection and recognition. IEEE J Biomed Health Inf 2019;23(3):1215–24.
• [69] Zhou S, Wu H, Gong J, Le T, Wu H, Chen Q, et al., 2018. Markguided segmentation of ultrasonic thyroid nodules using deep learning. Paper presented at the 2nd International Symposium on Image Computing and Digital Medicine, ISICDM 2018.
• [70] Yi Z, Hao A, Song W, Li H, Li B. A novel intelligent thyroid nodule diagnosis system over ultrasound images based on deep learning. 7th International Conference on Virtual Reality and Visualization, ICVRV 2017; 2019.
• [71] Ke W, Wang Y, Wan P, Liu W, Li H. An Ultrasonic Image Recognition Method for Papillary Thyroid Carcinoma Based on Depth Convolution Neural Network. Vol. 10635 LNCS. 24th International Conference on Neural Information Processing, ICONIP 2017 (pp. 82–91).
• [72] Li X, Wang S, Wei X, Zhu J, Yu R, Zhao M, et al. Fully convolutional networks for ultrasound image segmentation of thyroid nodules. 20th International Conference on High Performance Computing and Communications, 16th IEEE International Conference on Smart City and 4th IEEE International Conference on Data Science and Systems, 2019.
• [73] Yu R, Liu K, Wei X, Zhu J, Li X, Wang J, et al, 2018. Localization of thyroid nodules in ultrasonic images. 13th International Conference on Wireless Algorithms, Systems, and Applications, WASA 20182018. pp. 635–46.
• [74] Ying X, Yu Z, Yu R, Li X, Yu M, Zhao M, et al, 2018. Thyroid nodule segmentation in ultrasound images based on cascaded convolutional neural network. 25th International Conference on Neural Information Processing, ICONIP 20182018. pp. 373–84.
• [75] Akkus Z, Boonrod A, Siddiquee MR, Philbrick KA, Stan MN, Castro RM, et al., 2019. Reduction of unnecessary thyroid biopsies using deep learning. Proc. SPIE 10949, Medical Imaging 2019: Image Processing.
• [76] Poudel P, Illanes A, Sadeghi M, Friebe M. Patch based texture classification of thyroid ultrasound images using convolutional neural network. Paper presented at the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2019.
• [77] Zhang H, Zhao C, Guo L, Li X, Luo Y, Lu J, et al., 2019. Diagnosis of Thyroid Nodules in Ultrasound Images Using Two Combined Classification Modules. 2019 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI); 2019: IEEE.
• [78] Ding J, Huang Z, Shi M, Ning C. Automatic thyroid ultrasound image segmentation based on U-shaped network. 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISPBMEI). 2019: IEEE; 2019.
• [79] Guo M, Du Y, editors, 2019. Classification of Thyroid Ultrasound Standard Plane Images using ResNet-18 Networks. Paper presented at the 2019 IEEE 13th International Conference on Anti-counterfeiting, Security, and Identification (ASID).
• [80] Xie S, Yu J, Liu T, Chang Q, Niu L, Sun W. Thyroid nodule detection in ultrasound images with convolutional neural 654 biocybernetics and biomedical engineering 41 (2021) 636–655 networks. 14th IEEE Conference on Industrial Electronics and Applications (ICIEA). 2019: IEEE; 2019.
• [81] Pedraza L, Vargas C, Narváez F, Durán O, Muñoz E, Romero E. An open access thyroid ultrasound image database. Paper presented at the 10th International Symposium on Medical Information Processing and Analysis, 2015.
• [82] Shi G et al. Knowledge-guided synthetic medical image adversarial augmentation for ultrasonography thyroid nodule classification. Comput Methods Programs Biomed 2020;196 105611.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2021).