A progressive and cross-domain deep transfer learning framework for wrist fracture detection

• Autorzy: Karam Christophe , El Zini Julia , Awad Mariette , Saade Charbel , Naffaa Lena , El Amine Mohammad

Identyfikatory

DOI

10.2478/jaiscr-2022-0007

• Języki publikacji: EN

Abstrakty

EN

There has been an amplified focus on and benefit from the adoption of artificial intelligence (AI) in medical imaging applications. However, deep learning approaches involve training with massive amounts of annotated data in order to guarantee generalization and achieve high accuracies. Gathering and annotating large sets of training images require expertise which is both expensive and time-consuming, especially in the medical field. Furthermore, in health care systems where mistakes can have catastrophic consequences, there is a general mistrust in the black-box aspect of AI models. In this work, we focus on improving the performance of medical imaging applications when limited data is available while focusing on the interpretability aspect of the proposed AI model. This is achieved by employing a novel transfer learning framework, progressive transfer learning, an automated annotation technique and a correlation analysis experiment on the learned representations. Progressive transfer learning helps jump-start the training of deep neural networks while improving the performance by gradually transferring knowledge from two source tasks into the target task. It is empirically tested on the wrist fracture detection application by first training a general radiology network RadiNet and using its weights to initialize RadiNetwrist, that is trained on wrist images to detect fractures. Experiments show that RadiNetwrist achieves an accuracy of 87% and an AUC ROC of 94% as opposed to 83% and 92% when it is pre-trained on the ImageNet dataset. This improvement in performance is investigated within an explainable AI framework. More concretely, the learned deep representations of RadiNetwrist are compared to those learned by the baseline model by conducting a correlation analysis experiment. The results show that, when transfer learning is gradually applied, some features are learned earlier in the network. Moreover, the deep layers in the progressive transfer learning framework are shown to encode features that are not encountered when traditional transfer learning techniques are applied. In addition to the empirical results, a clinical study is conducted and the performance of RadiNetwrist is compared to that of an expert radiologist. We found that RadiNetwrist exhibited similar performance to that of radiologists with more than 20 years of experience. This motivates follow-up research to train on more data to feasibly surpass radiologists’ performance, and investigate the interpretability of AI models in the healthcare domain where the decision-making process needs to be credible and transparent.

Słowa kluczowe

EN

deep learning; transfer learning; wrist fracture detection; medical informatics; progressive transfer learning

• Wydawca: University of Social Sciences
• Czasopismo: Journal of Artificial Intelligence and Soft Computing Research
• Rocznik: 2022
• Tom: Vol. 12, No. 2
• Strony: 101--120
• Opis fizyczny: Bibliogr. 86 poz., rys.

Twórcy

autor

Karam Christophe

• Department of Electrical and Computer Engineering, American University of Beirut

autor

El Zini Julia

• Department of Electrical and Computer Engineering, American University of Beirut

autor

Awad Mariette

• Department of Electrical and Computer Engineering, American University of Beirut

autor

Saade Charbel

• Department of Health Professions, Medical Imaging Sciences, Faculty of Health Sciences, American University of Beirut

autor

Naffaa Lena

• Department of Radiology, Faculty of Medicine, American University of Beirut

autor

El Amine Mohammad

• Department of Radiology, Faculty of Medicine, American University of Beirut

Bibliografia

• [1] W. Cooney, R. Bussey, J. Dobyns, and R. Linscheid, “Difficult wrist fractures. Perilunate fracture-dislocations of the wrist.,” Clinical Orthopaedics and Related Research, no. 214, pp. 136–147, 1987.
• [2] R. Lindsey, A. Daluiski, S. Chopra, A. Lachapelle, M. Mozer, S. Sicular, et al., “Deep neural network improves fracture detection by clinicians,” Proceedings of the National Academy of Sciences, vol. 115, no. 45, pp. 11591–11596, 2018.
• [3] C. M. Court-Brown and B. Caesar, “Epidemiology of adult fractures: A review,” Injury, vol. 37, pp. 691–697, Aug 2006.
• [4] C. A. Goldfarb, Y. Yin, L. A. Gilula, A. J. Fisher, and M. I. Boyer, “Wrist fractures: What the clinician wants to know,” Radiology, vol. 219, no. 1, pp. 11–28, 2001. PMID: 11274530.
• [5] H. R. Guly, “Injuries initially misdiagnosed as sprained wrist (beware the sprained wrist),” Emergency Medicine Journal, vol. 19, no. 1, pp. 41–42, 2002.
• [6] B. Petinaux, R. Bhat, K. Boniface, and J. Aristizabal, “Accuracy of radiographic readings in the emergency department,” The American Journal of Emergency Medicine, vol. 29, pp. 18–25, Jan 2011.
• [7] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, et al., “A survey on deep learning in medical image analysis,” Medical image analysis, vol. 42, pp. 60–88, 2017.
• [8] D. Kim and T. MacKinnon, “Artificial intelligence in fracture detection: Transfer learning from deep convolutional neural networks,” Clinical Radiology, vol. 73, 12 2017.
• [9] J. Olczak, N. Fahlberg, A. Maki, A. Razavian, A. Jilert, A. Stark, et al., “Artificial intelligence for analyzing orthopedic trauma radiographs: Deep learning algorithms—are they on par with humans for diagnosing fractures?,” Acta Orthopaedica, vol. 88, pp. 1–6, 07 2017.
• [10] R. Lindsey, A. Daluiski, S. Chopra, A. Lachapelle, M. Mozer, S. Sicular, et al., “Deep neural network improves fracture detection by clinicians,” Proceedings of the National Academy of Sciences, vol. 115, no. 45, pp. 11591–11596, 2018.
• [11] D. Soekhoe, P. van der Putten, and A. Plaat, “On the impact of data set size in transfer learning using deep neural networks,” Lecture Notes in Computer Science Advances in Intelligent Data Analysis XV, pp. 50–60, 2016.
• [12] H.-C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, et al., “Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning,” IEEE transactions on medical imaging, vol. 35, o. 5, pp. 1285–1298, 2016.
• [13] B. Q. Huynh, H. Li, and M. L. Giger, “Digital mammographic tumor classification using transfer learning from deep convolutional neural networks,” Journal of Medical Imaging, vol. 3, no. 3, p. 034501, 2016.
• [14] A. Van Opbroek, M. A. Ikram, M. W. Vernooij, and M. De Bruijne, “Transfer learning improves supervised image segmentation across imaging protocols,” IEEE transactions on medical imaging, ol. 34, no. 5, pp. 1018–1030, 2014.
• [15] V. Christen, A. Groß, and E. Rahm, “Approaches for annotating medical documents.,” in LWDA, pp. 227–232, 2016.
• [16] P. Klassen, F. Xia, and M. Yetisgen-Yildiz, “Annotating and detecting medical events in clinical otes,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pp. 3417–3421, 2016.
• [17] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
• [18] C. Karam, J. El Zini, and M. Awad, “X-ray wrist fracture classification,” 2019.
• [19] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, p. 436, 2015.
• [20] P. Lakhani and B. Sundaram, “Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks,” Radiology, vol. 284, no. 2, pp. 574–582, 2017.
• [21] M. P. McBee, O. A. Awan, A. T. Colucci, C. W. Ghobadi, N. Kadom, A. P. Kansagra, et al., “Deep learning in radiology,” Academic radiology, vol. 25, no. 11, pp. 1472–1480, 2018.
• [22] J. H. Thrall, X. Li, Q. Li, C. Cruz, S. Do, K. Dreyer, et al., “Artificial intelligence and machine learning in radiology: opportunities, challenges, pitfalls, and criteria for success,” Journal of the American College of Radiology, vol. 15, no. 3, pp. 504–508, 2018.
• [23] M. A. Mazurowski, M. Buda, A. Saha, and M. R. Bashir, “Deep learning in radiology: An overview f the concepts and a survey of the state of the art. with focus on mri,” Journal of Magnetic Resonance Imaging, vol. 49, no. 4, pp. 939–954, 2019.
• [24] A. S. Becker, M. Marcon, S. Ghafoor, M. C. Wurnig, T. Frauenfelder, and A. Boss, “Deep learning in mammography: diagnostic accuracy of a ultipurpose image analysis software in the detection of breast cancer,” Investigative radiology, vol. 52, no. 7, pp. 434–440, 2017.
• [25] J. Wang, X. Yang, H. Cai, W. Tan, C. Jin, and L. Li, “Discrimination of breast cancer with microcalcifications on mammography by deep learning,” Scientific reports, vol. 6, p. 27327, 2016.
• [26] D. Ribli, A. Horvath, Z. Unger, P. Pollner, and I. Csabai, “Detecting and classifying lesions in mammograms with deep learning,” Scientific reports, vol. 8, no. 1, p. 4165, 2018.
• [27] M. Araya-Polo, J. Jennings, A. Adler, and T. Dahlke, “Deep-learning tomography,” The Leading Edge, vol. 37, no. 1, pp. 58–66, 2018.
• [28] K.-L. Hua, C.-H. Hsu, S. C. Hidayati, W.-H. Cheng, and Y.-J. Chen, “Computer-aided classification of lung nodules on computed tomography images via deep learning technique,” OncoTargets and therapy, vol. 8, 2015.
• [29] T. Wurfl, F. C. Ghesu, V. Christlein, and A. Maier, ¨“Deep learning computed tomography,” in International conference on medical image computing and computer-assisted intervention, pp. 432–440, pringer, 2016.
• [30] H. Zhang, L. Li, K. Qiao, L. Wang, B. Yan, L. Li, et al., “Image prediction for limited-angle tomography via deep learning with convolutional neural network,” arXiv preprint arXiv:1607.08707, 2016.
• [31] M. H. Yap, G. Pons, J. Mart´ı, S. Ganau, M. Sent´ıs, R. Zwiggelaar, et al., “Automated breast ultrasound lesions detection using convolutional neural networks,” IEEE journal of biomedical and health informatics, vol. 22, no. 4, pp. 1218–1226, 2017.
• [32] K. Lekadir, A. Galimzianova, A. Betriu, M. del Mar Vila, L. Igual, D. L. Rubin, et al., “A convolutional neural network for automatic characterization of plaque composition in carotid ultrasound,” IEEE journal of biomedical and health informatics, vol. 21, no. 1, pp. 48–55, 2016.
• [33] P. Burlina, S. Billings, N. Joshi, and J. Albayda, “Automated diagnosis of myositis from muscle ultrasound: Exploring the use of machine learning and deep learning methods,” PloS one, vol. 12, no. 8, p. e0184059, 2017.
• [34] P. H. Kalmet, S. Sanduleanu, S. Primakov, G. Wu, A. Jochems, T. Refaee, A. Ibrahim, L. v. Hulst, P. Lambin, and M. Poeze, “Deep learning in fracture detection: a narrative review,” Acta orthopaedica, vol. 91, no. 2, pp. 215–220, 2020.
• [35] R. M. Jones, A. Sharma, R. Hotchkiss, J. W. Sperling, J. Hamburger, C. Ledig, R. O’Toole, M. Gardner, S. Venkatesh, M. M. Roberts, et al., “Assessment of a deep-learning system for fracture detection in musculoskeletal radiographs,” NPJ digital medicine, vol. 3, no. 1, pp. 1–6, 2020.
• [36] A. M. Raisuddin, E. Vaattovaara, M. Nevalainen, M. Nikki, E. Jarvenp ¨ a¨a, K. Makkonen, P. Pinola, ¨T. Palsio, A. Niemensivu, O. Tervonen, et al., “Critical evaluation of deep neural networks for 118 Christophe Karam, Julia El Zini, Mariette Awad, Charbel Saade, Lena Naffaa, Mohammad El Amine wrist fracture detection,” Scientific reports, vol. 11, no. 1, pp. 1–11, 2021.
• [37] B. Guan, G. Zhang, J. Yao, X. Wang, and M. Wang, “Arm fracture detection in x-rays based on improved deep convolutional neural network,” Computers & Electrical Engineering, vol. 81, p. 106530, 2020.
• [38] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2010.
• [39] R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng, “Self-taught learning: transfer learning from unlabeled data,” in Proceedings of the 24th international conference on Machine learning, pp. 759–766, ACM, 2007.
• [40] H. Ravishankar, P. Sudhakar, R. Venkataramani, S. Thiruvenkadam, P. Annangi, N. Babu, et al., “Understanding the mechanisms of deep transfer learning for medical images,” in Deep Learning and Data Labeling for Medical Applications, pp. 188–196, Springer, 2016.
• [41] N. Tajbakhsh, J. Y. Shin, S. R. Gurudu, R. T. Hurst, C. B. Kendall, M. B. Gotway, et al., “Convolutional neural networks for medical image analysis: Full training or fine tuning?,” IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1299–1312, 2016.
• [42] B. J. Erickson, P. Korfiatis, Z. Akkus, and T. L. Kline, “Machine learning for medical imaging,” Radiographics, vol. 37, no. 2, pp. 505–515, 2017.
• [43] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, pp. 1097–1105, 2012.
• [44] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in Thirty-First AAAI Conference on Artificial Intelligence, 2017.
• [45] S. Ren, K. He, R. Girshick, and J. Sun, “Faster rcnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems, pp. 91–99, 2015.
• [46] T. Urakawa, Y. Tanaka, H. Matsuzawa, K. Watanabe, and N. Endo, “Detecting intertrochanteric hip fractures with orthopedist-level accuracy using a deep convolutional neural network,” Journal of the International Skeletal Society A Journal of Radiology, Pathology and Orthopedics, vol. 42, pp. 239–244, 2019.
• [47] P. Rajpurkar, J. Irvin, A. Bagul, D. Ding, T. Duan, H. Mehta, et al., “Mura dataset: Towards radiologist-level abnormality detection in musculoskeletal radiographs,” 2017. Cite arxiv:1712.06957.
• [48] K. Gan, D. Xu, Y. Lin, Y. Shen, T. Zhang, K. Hu, et al., “Artificial intelligence detection of distal radius fractures: a comparison between the convolutional neural network and professional assessments,” Acta orthopaedica, pp. 1–12, 2019.
• [49] J. de Matos, A. de Souza Britto Jr., L. E. S. Oliveira, and A. L. Koerich, “Double transfer learning for breast cancer histopathologic image classification,” CoRR, vol. abs/1904.07834, 2019.
• [50] S. Christodoulidis, M. Anthimopoulos, L. Ebner,A. Christe, and S. G. Mougiakakou, “Multisource transfer learning with convolutional neural networks for lung pattern analysis,” CoRR, vol. abs/1612.02589, 2016.
• [51] J. Li, W. Wu, D. Xue, and P. Gao, “Multi-source deep transfer neural network algorithm,” Sensors (Basel, Switzerland), vol. 19, p. 3992, Sep 2019.31527437[pmid].
• [52] R. Gupta and L.-A. Ratinov, “Text categorization with knowledge transfer from heterogeneous data sources,” in AAAI, pp. 842–847, 2008.
• [53] Z. Yu, Z. Jin, L. Wei, J. Guo, J. Huang, D. Cai, X. He, and X.-S. Hua, “Progressive transfer learning for person re-identification,” Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Aug 2019.
• [54] W. Hu, Y. Jin, X. Wu, and J. Chen, “Progressive transfer learning for low frequency data prediction in full waveform inversion,” 2019.
• [55] Y. Gu, Z. Ge, C. P. Bonnington, and J. Zhou, “Progressive transfer learning and adversarial domain adaptation for cross-domain skin disease classification,” IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 5, pp. 1379–1393, 2020.
• [56] J. Antol´ık, “Automatic annotation of medical records,” Studies in health technology and informatics, vol. 116, pp. 817–822, 2005.
• [57] C. Ganoe, W. Wu, P. Barr, W. Haslett, M. Dannenberg, K. Bonasia, J. Finora, J. Schoonmaker, W. Onsando, J. Ryan, et al., “Natural language processing for automated annotation of medication mentions in primary care visit conversations,” medRxiv, 2021.
• [58] H. Li, B. Zhang, Y. Zhang, W. Liu, Y. Mao, J. Huang, and L. Wei, “A semi-automated annotation algorithm based on weakly supervised learning for medical images,” Biocybernetics and Biomedical Engineering, vol. 40, no. 2, pp. 787–802, 2020.
• [59] R. Bouslimi and J. Akaichi, “New approach for automatic medical image annotation using the bagof-words model,” in 2015 6th IEEE International Conference on Software Engineering and Service Science (ICSESS), pp. 1088–1093, 2015.
• [60] T. Gong, S. Li, J. Wang, C. L. Tan, B. Pang, T. Lim, C. Lee, Q. Tian, and Z. Zhang, “Automatic labeling and classification of brain ct images,” pp. 1581–1584, 09 2011.
• [61] A. R. Aronson and F.-M. Lang, “An overview of etamap: historical perspective and recent advances,” Journal of the American Medical Informatics Association : JAMIA, vol. 17, no. 3, pp. 229–236, 2010. PMC2995713[pmcid].
• [62] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” CoRR, vol. abs/1311.2901, 2013.
• [63] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” arXiv preprint arXiv:1703.01365, 2017.
• [64] M. T. Ribeiro, S. Singh, and C. Guestrin, “Anchors: High-precision model-agnostic explanations,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 2018.
• [65] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Object detectors emerge in deep scene cnns,” 2015.
• [66] B. Zhou, A. Khosla, L. A., A. Oliva, and A. Torralba, “Learning deep features for discriminative localization.,” CVPR, 2016.
• [67] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Mar 2018.
• [68] B. N. Patro, M. Lunayach, S. Patel, and V. P. Namboodiri, “U-cam: Visual explanation using uncertainty based class activation maps,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 7444–7453, 2019.
• [69] M. T. Ribeiro, S. Singh, and C. Guestrin, “”why should i trust you?”: Explaining the predictions of any classifier,” 2016.
• [70] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba, “Network dissection: Quantifying interpretability of deep visual representations,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541–6549, 2017.
• [71] R. Fong and A. Vedaldi, “Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8730–8738, 2018.
• [72] K. Leino, S. Sen, A. Datta, M. Fredrikson, and L. Li, “Influence-directed explanations for deep convolutional networks,” in 2018 IEEE International Test Conference (ITC), pp. 1–8, IEEE, 2018.
• [73] A. Mahendran and A. Vedaldi, “Understanding deep image representations by inverting them,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188–5196, 2015.
• [74] A. Dosovitskiy and T. Brox, “Inverting visual representations with convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4829–4837, 2016.
• [75] S. Bach, A. Binder, G. Montavon, F. Klauschen, K. Muller, and W. Samek, “On pixel-wise explanations for non-linear classifier decisions by layerwise relevance propagation,” PLoS ONE, vol. 10, 2015.
• [76] M. Bohle, F. Eitel, M. Weygandt, and K. Ritter, “Layer-wise relevance propagation for explaining deep neural network decisions in mri-based alzheimer’s disease classification,” Frontiers in aging neuroscience, vol. 11, pp. 194–194, Jul 2019. 31417397[pmid].
• [77] F. Eitel, E. Soehler, J. Bellmann-Strobl, A. U. Brandt, K. Ruprecht, R. M. Giess, J. Kuchling, S. Asseyer, M. Weygandt, J.-D. Haynes, M. S. l, F. Paul, and K. Ritter, “Uncovering convolutional neural network decisions for diagnosing multiple sclerosis on conventional mri using layer-wise relevance propagation,” NeuroImage: Clinical, vol. 24, p. 102003, 2019.
• [78] D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, “Canonical correlation analysis: An overview with application to learning methods,” Neural computation, vol. 16, no. 12, pp. 2639–2664, 2004.
• [79] D. Sussillo, M. M. Churchland, M. T. Kaufman, and K. V. Shenoy, “A neural network that finds a naturalistic solution for the production of muscle activity,” Nature neuroscience, vol. 18, no. 7, pp. 1025–1033, 2015.
• [80] M. Faruqui and C. Dyer, “Improving vector space word representations using multilingual correlation,” in Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 462–471, 2014.
• 81] M. Raghu, J. Gilmer, J. Yosinski, and J. SohlDickstein, “Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability,” in Advances in Neural Information Processing Systems, pp. 6076–6085, 2017.
• [82] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 8, pp. 1798–1828, 2013.
• [83] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?,” in Advances in neural information processing systems, pp. 3320–3328, 2014.
• [84] C. Castillo, T. Steffens, L. Sim, and L. Caffery, “The effect of clinical information on radiology reporting: A systematic review,” Journal of Medical Radiation Sciences, vol. 68, no. 1, pp. 60–74, 2021.
• [85] Theano Development Team, “Theano: A Python framework for fast computation of mathematical expressions,” arXiv e-prints, vol. abs/1605.02688, May 2016.
• [86] G. Huang, Z. Liu, and K. Q. Weinberger, “Densely connected convolutional networks,” CoRR, vol. abs/1608.06993, 2016.