Ultrasound spine image segmentation using multi-scale feature fusion Skip-Inception U-Net (SIU-Net)

• Autorzy: Banerjee Sunetra , Lyu Juan , Huang Zixun , Leung Frank H.F , Lee Timothy , Yang De , Su Steven , Zheng Yongping , Ling Sai Ho

Identyfikatory

DOI

10.1016/j.bbe.2022.02.011

• Języki publikacji: EN

Abstrakty

EN

Scoliosis is a 3D spinal deformation where the spine takes a lateral curvature, forming an angle in the coronal plane. Diagnosis of scoliosis requires periodic detection, and frequent exposure to radiative imaging may cause cancer. A safer and more economical alternative imaging, i.e., 3D ultrasound imaging modality, is being explored. However, unlike other radiative modalities, an ultrasound image is noisy, which often suppresses the image’s useful information. Through this research, a novel hybridized CNN architecture, multi-scale feature fusion Skip-Inception U-Net (SIU-Net), is proposed for a fully automatic bony feature detection, which can be further used to assess the severity of scoliosis safely and automatically. The proposed architecture, SIU-Net, incorporates two novel features into the basic U-Net architecture: (a) an improvised Inception block and (b) newly designed decoder-side dense skip pathways. The proposed model is tested on 109 spine ultrasound image datasets. The architecture is evaluated using the popular (i) Jaccard Index (ii) Dice Coefficient and (iii) Euclidean distance, and compared with (a) the basic U-net segmentation model, (b) a more evolved UNet++ model, and (c) a newly developed MultiResUNet model. The results show that SIU-Net gives the clearest segmentation output, especially in the important regions of interest such as thoracic and lumbar bony features. The method also gives the highest average Jaccard score of 0.781 and Dice score of 0.883 and the lowest histogram Euclidean distance of 0.011 than the other three models. SIU-Net looks promising to meet the objectives of a fully automatic scoliosis detection system.

Słowa kluczowe

EN

bony feature; convolutional neural network; scoliosis; ultrasound image; U-Net; feature fusion

PL

cechy kości; sieć neuronowa konwolucyjna; skolioza; obraz ultrasonograficzny; U-Net; łączenie funkcji

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2022
• Tom: Vol. 42, no. 1
• Strony: 341--361
• Opis fizyczny: Bibliogr. 69 poz., rys., tab., wykr.

Twórcy

autor

Banerjee Sunetra

• School of Biomedical Engineering, University of Technology Sydney, NSW, Australia

autor

Lyu Juan

• College of Information and Communication Engineering, Harbin Engineering University, Harbin, China

autor

Huang Zixun

• Department of Electronic and Information Engineering, Hong Kong Polytechnic University, Hong Kong, China

autor

Leung Frank H.F

• Department of Electronic and Information Engineering, Hong Kong Polytechnic University, Hong Kong, China

autor

Lee Timothy

• Department of Biomedical Engineering, Hong Kong Polytechnic University, Hong Kong, China

autor

Yang De

• Department of Biomedical Engineering, Hong Kong Polytechnic University, Hong Kong, China

autor

Su Steven

• School of Biomedical Engineering, University of Technology Sydney, NSW, Australia

autor

Zheng Yongping

• Department of Biomedical Engineering, Hong Kong Polytechnic University, Hong Kong, China

autor

Ling Sai Ho

• Building 11, University Of Technology, 15 Broadway, Ultimo, NSW 2007, Australia

Bibliografia

• [1] Kim H, Kim HS, Moon ES, Yoon C-S, Chung T-S, Song H-T, et al. Scoliosis imaging: what radiologists should know. Radiographics 2010;30(7):1823–42.
• [2] Liu D, Yang Y, Yu X, Yang J, Xuan X, Yang J, et al. Effects of specific exercise therapy on adolescent patients with idiopathic scoliosis: A prospective controlled cohort study. Spine 2020;45(15):1039–46.
• [3] Tsiligiannis T, Grivas T. Pulmonary function in children with idiopathic scoliosis. Scoliosis 2012;7:1–6.
• [4] Li S, Yang J, Zhu L, Li Y, Peng H, Lin Y, et al. Left ventricular mechanics assessed by 2-dimensional speckle tracking echocardiography in children and adolescents with idiopathic scoliosis. Clin Spine Surg 2017;30:E381–9.
• [5] Cobb J. Outline for the study of scoliosis. Instr Course Lect AAOS 1948;5:261–75.
• [6] Simony A, Hansen EJ, Christensen SB, Carreon LY, Andersen MO. Incidence of cancer in adolescent idiopathic scoliosis patients treated 25 years previously. Eur Spine J 2016;25:3366–70.
• [7] Lai K-L, Lee T-Y, Lee M-S, Hui J-H, Zheng Y-P. Validation of scolioscan air-portable radiation-free three-dimensional ultrasound imaging assessment system for scoliosis. Sensors 2021;21:2858.
• [8] Hwang BY, Mampre D, Ahmed AK, Suk I, Anderson WS, Manbachi A, et al. Ultrasound in traumatic spinal cord injury: a wide-open field. Neurosurgery 2021;89:372–82.
• [9] Tawfik NA, Ahmed AT, El-Shafei TE, Habba MR. Diagnostic value of spinal ultrasound compared to MRI for diagnosis of spinal anomalies in pediatrics. Egypt J Radiol Nucl Med 2020;51:1–11.
• [10] Zhang J, Cui X, Chen S, Dai Y, Huang Y, Zhang S. Ultrasound-guided nusinersen administration for spinal muscular atrophy patients with severe scoliosis: an observational study. Orphanet J Rare Dis 2021;16:1–8.
• [11] Kalagara H, Nair H, Kolli S, Thota G, Uppal V. Ultrasound imaging of the spine for central neuraxial blockade: a technical description and evidence update. Curr Anesthesiol Rep 2021;11:326–39.
• [12] Gai S, Zhang B, Yang C, Yu L. Speckle noise reduction in medical ultrasound image using monogenic wavelet and Laplace mixture distribution. Digital Signal Process 2018;72:192–207.
• [13] Jiang WW, Zhou GQ, Lai K-L, Hu SY, Gao QY, Wang XY, et al. A fast 3-D ultrasound projection imaging method for scoliosis assessment. Math Biosci Eng 2019;16:1067–81.
• [14] Cheung C-W, Zhou G-Q, Law S-Y, Mak T-M, Lai K-L, Zheng Y-P. Ultrasound volume projection imaging for assessment of scoliosis. IEEE Trans Med Imaging 2015;34:1760–8.
• [15] Zhou G-Q, Zheng Y-P. Assessment of scoliosis using 3-D ultrasound volume projection imaging with automatic spine curvature detection. In: IEEE International Ultrasonics Symposium (IUS). IEEE; 2015. p. 1–4.
• [16] Brignol A, Gueziri HE, Cheriet F, Collins DL, Laporte C. Automatic extraction of vertebral landmarks from ultrasound images: A pilot study. Comput Biol Med 2020;122:103838.
• [17] Zhou G-Q, Jiang W-W, Lai K-L, Zheng Y-P. Automatic measurement of spine curvature on 3-D ultrasound volume projection image with phase features. IEEE Trans Med Imaging 2017;36:1250–62.
• [18] Lee T-Y, Lai K-L, Cheng J-Y, Castelein RM, Lam T-P, Zheng Y-P. 3D ultrasound imaging provides reliable angle measurement with validity comparable to X-ray in patients with adolescent idiopathic scoliosis. J Orthop Transl 2021;29:51–9.
• [19] Pandey PU, Quader N, Guy P, Garbi R, Hodgson AJ. Ultrasound bone segmentation: a scoping review of techniques and validation practices. Ultrasound Med Biol 2020;46:921–35.
• [20] Zhang J, Li H, Lv L, Zhang Y. Computer-aided Cobb measurement based on automatic detection of vertebral slopes using deep neural network. Int J Biomed Imaging 2017;2017:1–6.
• [21] Krizhenvshky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional networks. Proceedings of the Conference Neural Information Processing Systems (NIPS). p. 1097-105.
• [22] Thong WE, Labelle H, Shen J, Parent S, Kadoury S. Stacked auto-encoders for classification of 3d spine models in adolescent idiopathic scoliosis. In: Recent Advances in Computational Methods and Clinical Applications for Spine Imaging. Springer; 2015. p. 13–25.
• [23] Liu S, Wang Yi, Yang X, Lei B, Liu Li, Li SX, et al. Deep learning in medical ultrasound analysis: a review. Engineering 2019;5 (2):261–75.
• [24] Hong R, Cheng W-H, Yamasaki T, Wang M, Ngo C-W. Advances in Multimedia Information Processing–PCM 2018: 19th Pacific-Rim Conference on Multimedia, Hefei, China, September 21-22, 2018, Proceedings, Part III: Springer; 2018.
• [25] Zeiler MD, Fergus R. Visualizing and understanding convolutional networks. In: European conference on computer vision. Springer; 2014. p. 818–33.
• [26] Kokabu T, Kanai S, Kawakami N, Uno K, Kotani T, Suzuki T, et al. An algorithm for using deep learning convolutional neural networks with three dimensional depth sensor imaging in scoliosis detection. Spine J 2021;21:980–7.
• [27] Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, et al. A survey on deep learning in medical image analysis. Med Image Anal 2017;42:60–88.
• [28] Ronneberger O, Fischer P, U-net BT. Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. Springer; 2015. p. 234–41.
• [29] Liu W, Li W, Gong W. Ensemble of fine-tuned convolutional neural networks for urine sediment microscopic image classification. IET Comput Vision 2020;14:18–25.
• [30] Amiri M, Brooks R, Behboodi B, Rivaz H. Two-stage ultrasound image segmentation using U-Net and test time augmentation. Int J Comput Assist Radiol Surg 2020;15:981–8.
• [31] Chen L, Tian Y, Deng Y, Abdulhay E. Neural network algorithm-based three-dimensional ultrasound evaluation in the diagnosis of fetal spina bifida. Sci Program 2021;2021:3605739.
• [32] Ungi T, Greer H, Sunderland KR, Wu V, Baum ZMC, Schlenger C, et al. Automatic spine ultrasound segmentation for scoliosis visualization and measurement. IEEE Trans Biomed Eng 2020;67:3234–41.
• [33] Huang Z, Wang L-W, Leung FH, Banerjee S, Yang D, Lee T, et al. Bone feature segmentation in ultrasound spine image with robustness to speckle and regular occlusion noise. arXiv preprint arXiv:201003740. 2020.
• [34] Lyu J, Bi X, Banerjee S, Huang Z, Leung FHF, Lee T-Y, et al. Dual-task ultrasound spine transverse vertebrae segmentation network with contour regularization. Comput Med Imaging Graph 2021;89:101896.
• [35] Ibtehaz N, Rahman MS. MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Networks 2020;121:74–87.
• [36] Bi L, Kim J, Kumar A, Fulham M, Feng D. Stacked fully convolutional networks with multi-channel learning: application to medical image segmentation. Visual Comput 2017;33:1061–71.
• [37] Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J. Unet++: A nested u-net architecture for medical image segmentation. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer; 2018. p. 3–11.
• [38] Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J. Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging 2020;39 (6):1856–67.
• [39] Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June. p. 1–9.
• [40] Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. Proceedings of the IEEE conference on computer vision and pattern recognition 2016. pp. 2818-26.
• [41] Szegedy C, Ioffe S, Vanhoucke V, Alemi A. Inception-v4, Inception-ResNet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence. February 2017; pp. 4278-84.
• [42] Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. pp. 4700-8.
• [43] Zhang Z, Wu C, Coleman S, Kerr D. DENSE-INception U-net for medical image segmentation. Comput Methods Programs Biomed 2020;192:105395.
• [44] Badrinarayanan V, Kendall A, Cipolla R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 2017;39:2481–95.
• [45] Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al. Tensorflow: A system for large-scale machine learning. In: 12th USENIX symposium on operating systems design and implementation (OSDI’16). p. 265–83.
• [46] Kingma DP, Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980. 2014.
• [47] Jaccard P. The distribution of the flora in the alpine zone. 1. New Phytol 1912;11:37–50.
• [48] Dice LR. Measures of the amount of ecologic association between species. Ecology 1945;26:297–302.
• [49] Abbasian Ardakani A, Bitarafan-Rajabi A, Mohammadzadeh A, Mohammadi A, Riazi R, Abolghasemi J, et al. A hybrid multilayer filtering approach for thyroid nodule segmentation on ultrasound images. J Ultrasound Med 2019;38:629–40.
• [50] Korez R, Putzier M, Vrtovec T. A deep learning tool for fully automated measurements of sagittal spinopelvic balance from X-ray images: performance evaluation. Eur Spine J 2020;29:2295–305.
• [51] Horng M-H, Kuok C-P, Fu M-J, Lin C-J, Sun Y-N. Cobb angle measurement of spine from X-Ray images using convolutional neural network. Comput Math Methods Med 2019;2019:6357171.
• [52] Li H, Luo H, Huan W, Shi Z, Yan C, Wang L, et al. Automatic lumbar spinal MRI image segmentation with a multi-scale attention network. Neural Comput Appl 2021;33 (18):11589–602.
• [53] Khandelwal P, Collins DL, Siddiqi K. Spine and individual vertebrae segmentation in computed tomography images using geometric flows and shape priors. Front Comput Sci 2021;3:66.
• [54] Zheng Y-P, Lee T-T-Y, Lai K-K-L, Yip B-H-K, Zhou G-Q, Jiang W-W, et al. A reliability and validity study for Scolioscan: a radiation-free scoliosis assessment system utilising 3D ultrasound imaging. Scoliosis Spinal Disord 2016;11:13.
• [55] Huang Q, Deng Q, Li L, Yang J, Li X. Scoliotic imaging with a novel double-sweep 2.5-dimensional extended field-of-view ultrasound. IEEE Trans Ultrason Ferroelectr Freq Control 2019;66(8):1304–15.
• [56] Huang Q, Zeng Z, Li X. 2.5-D extended field-of-view ultrasound. IEEE Trans Med Imaging 2018;37:851–9.
• [57] Chen H-B, Zheng R, Qian L-Y, Liu F-Y, Song S, Zeng H-Y. Improvement of 3D ultrasound spine imaging technique using fast reconstruction algorithm. IEEE Trans Ultrason Ferroelectr Freq Control 2021;68:3104–13.
• [58] Brink RC, Wijdicks SPJ, Tromp IN, Schlösser TPC, Kruyt MC, Beek FJA, et al. A reliability and validity study for different coronal angles using ultrasound imaging in adolescent idiopathic scoliosis. Spine J 2018;18:979–85.
• [59] Zheng R, Young M, Hill D, Le LH, Hedden D, Moreau M, et al. Improvement on the accuracy and reliability of ultrasound coronal curvature measurement on adolescent idiopathic scoliosis with the aid of previous radiographs. Spine 2016;41:404–11.
• [60] Young M, Hill DL, Zheng R, Lou E. Reliability and accuracy of ultrasound measurements with and without the aid of previous radiographs in adolescent idiopathic scoliosis (AIS). Eur Spine J 2015;24:1427–33.
• [61] Zeng H, Zheng R, Le LH, Ta D. Measuring spinous process angle on ultrasound spine images using the GVF segmentation method. In: IEEE International Ultrasonics Symposium (IUS). IEEE; 2019. p. 1477–80.
• [62] Zeng H-Y, Ge S-H, Gao Y-C, Zhou D-S, Zhou K, He X-M, et al. Automatic segmentation of vertebral features on ultrasound spine images using Stacked Hourglass Network. arXiv preprint arXiv:210503847. 2021.
• [63] de Reuver S, Brink RC, Lee TTY, Zheng Y-P, Beek FJA, Castelein RM. Cross-validation of ultrasound imaging in adolescent idiopathic scoliosis. Eur Spine J 2021;30:628–33.
• [64] Banerjee S, Ling SH, Lyu J, Su S, Zheng Y-P. Automatic segmentation of 3D ultrasound spine curvature using convolutional neural network. In: 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE; 2020. p. 2039–42.
• [65] Wu H-D, Liu W, Wong M-S. Reliability and validity of lateral curvature assessments using clinical ultrasound for the patients with scoliosis: a systematic review. Eur Spine J 2020;29(4):717–25.
• [66] Lian X, Pang Y, Han J, Pan J. Cascaded hierarchical atrous spatial pyramid pooling module for semantic segmentation. Pattern Recogn 2021;110:107622.
• [67] Banerjee S, Lyu J, Huang Z, Leung HFF, Lee TT-Y, Yang D, et al. Light-Convolution dense selection U-Net (LDS U-Net) for ultrasound lateral bony feature segmentation. Appl Sci. 2021;11:10180.
• [68] Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:180403999. 2018.
• [69] Lyu J, Ling SH, Banerjee S, Zheng JY, Lai KL, Yang D, et al. Ultrasound volume projection image quality selection by ranking from convolutional RankNet. Comput Med Imaging Graph 2021;89:101847.