Arabic and American Sign Languages Alphabet Recognition by Convolutional Neural Network

• Autorzy: Alshomrani Shroog , Aljoudi Lina , Arif Muhammad

Identyfikatory

DOI

10.12913/22998624/142012

• Języki publikacji: EN

Abstrakty

EN

Hearing loss is a common disability that occurs in many people worldwide. Hearing loss can be mild to complete deafness. Sign language is used to communicate with the deaf community. Sign language comprises hand gestures and facial expressions. However, people find it challenging to communicate in sign language as not all know sign language. Every country has developed its sign language like spoken languages, and there is no standard syntax and grammatical structure. The main objective of this research is to facilitate the communication between deaf people and the community around them. Since sign language contains gestures for words, sentences, and letters, this research implemented a system to automatically recognize the gestures and signs using imaging devices like cameras. Two types of sign languages are considered, namely, American sign language and Arabic sign language. We have used the convolutional neural network (CNN) to classify the images into signs. Different settings of CNN are tried for Arabic and American sign datasets. CNN-2 consisting of two hidden layers produced the best results (accuracy of 96.4%) for the Arabic sign language dataset. CNN-3, composed of three hidden layers, achieved an accuracy of 99.6% for the American sign dataset.

Słowa kluczowe

EN

convolutional neural network; deep learning; American sign language; Arabic sign language

PL

sieć neuronowa; głębokie uczenie; amerykański język migowy; arabski język migowy

• Wydawca: Lublin University of Technology ; Polish Society of Ecological Engineering (PTIE), Branch of PTIE in Lublin
• Czasopismo: Advances in Science and Technology. Research Journal
• Rocznik: 2021
• Tom: Vol. 15, no 4
• Strony: 136--148
• Opis fizyczny: Bibliogr. 52 poz., fig., tab.

Twórcy

autor

Alshomrani Shroog

• Department of Computer Science, College of Computer and Information Systems, Umm Alqura University, Kingdom of Saudi Arabia

autor

Aljoudi Lina

• Department of Computer Science, College of Computer and Information Systems, Umm Alqura University, Kingdom of Saudi Arabia

autor

Arif Muhammad

• Department of Computer Science, College of Computer and Information Systems, Umm Alqura University, Kingdom of Saudi Arabia

• https://orcid.org/0000-0003-0513-9872+

Bibliografia

• 1. Juan Pablo Bonet J.B. Reduccion de las letras y arte paraensen˜ar a hablar a los mudos. Madrid: Francisco abarca de angulo. Recuperado de biblioteca digital hispanica. Biblioteca nacional de espan˜a; 1620.
• 2. Groce N.E. Everyone here spoke sign language: Hereditary deafness on Martha’s Vineyard. Harvard University Press; 1985.
• 3. Luzerne Ray L. The abbe’de l’epee. American Annals of the Deaf and Dumb. 1848;1(2):69–76.
• 4. Shaw E., Delaporte Y. A historical and etymological dictionary of American Sign Language. Gallaudet University Press; 2014.
• 5. Brentari D. Sign languages. Cambridge University Press; 2010.
• 6. Abdel-Fattah M.A. Arabic sign language: a perspective. Journal of deaf studies and deaf education. 2005;10(2):212–221.
• 7. World Health Organization (WHO). 2021. Deafness and hearing loss.
• 8. Kumar P., Gauba H., Roy P.P., Dogra D.P. A multimodal framework for sensor based sign language recognition. Neurocomputing. 2017;259:21–38.
• 9. Ahmed M.A., Zaidan B.B., Zaidan A.A., Salih M.M., Bin Lakulu M.M. A review on systems-based sensory gloves for sign language recognition state of the art between 2007 and 2017. Sensors. 2018;18(7):2208.
• 10. Savur C., Sahin F. American sign language recognition system by using surface emg signal. In 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE 2016, 002872–002877.
• 11. Luqman H., Mahmoud S.A., et al. Transform-based arabic sign language recognition. Procedia Computer Science. 2017;117:2–9.
• 12. AlQattan D., Sepulveda F. Towards sign language recognition using eeg-based motor imagery brain computer interface. In 2017 5th International Winter Conference on Brain-Computer Interface (BCI), IEEE 2017, 5–8.
• 13. Hou J., Li X.Y., Zhu P., Wang Z., Wang Y., Qian J., Yang P.L. Signspeaker: A real-time, high-precision smartwatch-based sign language translator. In The 25th Annual International Conference on Mobile Computing and Networking 2019, 1–15.
• 14. Kolivand H., Joudaki S., Sunar M.S., Tully D. An implementation of sign language alphabet hand posture recognition using geometrical features through artificial neural network (part 2). Neural Computing and Applications. 2021;1–23.
• 15. Ravi S., Suman M., Kishore P.V.V., Eepuri K.K. Sign language recognition with multi feature fusion and ann classifier. Turkish Journal of Electrical Engineering and Computer Science. 2018;26(6):2871–2885.
• 16. Oyeniran O.A., Oyeniyi J.O., Sotonwa K.A., Ojo A.O. Review of the application of artificial intelligence in sign language recognition system. International Journal of Engineering and Artificial Intelligence. 2020;1(4):20–25.
• 17. Awwad S., Idwan S, Gharaibeh H. Real-time sign languages character recognition. International Journal of Computer Applications in Technology. 2021;65(1):36–44.
• 18. Joshi G., Singh S., Vig R. Taguchi-topsis based hog parameter selection for complex background sign language recognition. Journal of Visual Communication and Image Representation. 2020;71:102834.
• 19. Tyagi A., Bansal S. Feature extraction technique for vision-based indian sign language recognition system: A review. Computational Methods and Data Engineering. 2021;39–53.
• 20. Zhu Z., Jiang X., Zhang J. Sign language video classification based on image recognition of specified key frames. In International Conference on Multimedia Technology and Enhanced Learning. 2020;371–381.
• 21. Baumgartner L., Jauss S, Maucher J., Gottfried Zimmermann G. Automated sign language translation: The role of artificial intelligence now and in the future. CHIRA. 2020;170–177.
• 22. Sahoo J.P., Ari S., Patra S.K. Hand gesture recognition using pca based deep cnn reduced features and svm classifier. In 2019 IEEE International Symposium on Smart Electronic Systems (iSES) (Formerly iNiS), IEEE 2019, 221–224.
• 23. Diana Alejandra Contreras Alejo and Francisco Javier Gallegos Funes. Recognition of a single dynamic gesture with the segmentation technique hs-ab and principle components analysis (pca). En- tropy. 2019;21(11):1114.
• 24. Mohamed Deriche, Salihu O Aliyu, Mohamed Mohandes. An intelligent arabic sign language recognition system using a pair of lmcs with gmm based classification. IEEE Sensors Journal. 2019;19(18):8067–8078.
• 25. Rabeet Fatmi, Sherif Rashad, Ryan Integlia. Comparing ann, svm, and hmm based machine learning methods for american sign language recognition using wearable motion sensors. In 2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC), 2019;0290–0297. IEEE,
• 26. Nigus Kefyalew Tamiru, Menore Tekeba, and Ayodeji Olalekan Salau. Recognition of amharic sign language with amharic alphabet signs using ann and svm. The Visual Computer, pages 1–16, 2021.
• 27. Xianwei Jiang, Suresh Chandra Satapathy, Longxiang Yang, Shui-Hua Wang, and Yu-Dong Zhang. A survey on artificial intelligence in chinese sign language recognition. Arabian Journal for Science and Engineering. 2020;1–36.
• 28. Quinn M., Olszewska J.I. British sign language recognition in the wild based on multi-class svm. In 2019 Federated Conference on Computer Science and Information Systems (FedCSIS), IEEE 2019, 81–86.
• 29. Guo D., Zhou W., Li H., Wang M. Online early-late fusion based on adaptive hmm for sign language recognition. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM). 2017;14(1):1–18.
• 30. Roy P.P., Kumar P., Kim B.G. An efficient sign language recognition (slr) system using camshift tracker and hidden markov model (hmm). SN Computer Science. 2021;2(2):1–15.
• 31. Kumar P., Saini R., Roy P.P., Dogra D.P. A position and rotation invariant framework for sign language recognition (slr) using kinect. Multimedia Tools and Applications. 2018;77(7):8823–8846.
• 32. Dong C., Leu M.C., Yin Z. American sign language alphabet recognition using microsoft kinect. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2015;44–52.
• 33. Malik M.S.A., Kousar N., Abdullah T., Ahmed M., Rasheed F., Awais M. Pakistan sign language detection using pca and knn. International Journal of Advanced Computer Science and Applications. 2018;9(54):78–81.
• 34. Ching-Hua Chuan, Eric Regina, and Caroline Guardino. American sign language recognition using leap motion sensor. In 2014 13th International Conference on Machine Learning and Applications, IEEE 2014, 541–544.
• 35. Zakariya A.M., Jindal R. Arabic sign language recognition system on smartphone. In 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT), IEEE 2019, 1–5.
• 36. Hosseini M.P., Lu S., Kamaraj K., Slowikowski A., Venkatesh H.C. Deep learning architectures. In Deep learning: concepts and architectures. Springer; 2020;1–24.
• 37. Calin O. Deep Learning Architectures. Springer; 2020.
• 38. Gu J., Wang Z., Kuen J., Ma L., Shahroudy A., Shuai B., Liu T., Wang X., Wang G., Cai J., et al. Recent advances in convolutional neural networks. Pattern Recognition. 2018;77:354–377.
• 39. Alam M., Samad M.D., Vidyaratne L., Glandon A., Iftekharuddin K.M. Survey on deep neural networks in speech and vision systems. Neurocomputing. 2020;417:302–321.
• 40. Tubaiz N., Shanableh T., Assaleh K. Glove-based continuous arabic sign language recognition in user-dependent mode. IEEE Transactions on Human- Machine Systems. 2015;45(4):526–533.
• 41. Almasre M.A., Al-Nuaim H. Comparison of four svm classifiers used with depth sensors to recognize arabic sign language words. Computers. 2017;6(2):20.
• 42. Alzohairi R., Alghonaim R., Alshehri W., Aloqeely S., Alzaidan M., Bchir O. Image based arabic sign language recognition system. International Journal of Advanced Computer Science and Applications (IJACSA). 2018;9(3).
• 43. Bheda V., Radpour D. Using deep convolutional networks for gesture recognition in american sign language. arXiv preprint arXiv:1710.06836; 2017.
• 44. Gao Q., Ogenyi U.E., Liu J., Ju Z., Liu H. A two-stream cnn framework for american sign language recognition based on multimodal data fusion. In UK Workshop on Computational Intelligence. Springer; 2019;107–118.
• 45. Jalal M.A., Chen R., Moore R.K., Mihaylova L. American sign language posture understanding with deep neural networks. In 2018 21st International Conference on Information Fusion (FU- SION), IEEE 2018, 573–579.
• 46. Bantupalli K., Ying Xie. American sign language recognition using deep learning and computer vision. In 2018 IEEE International Conference on Big Data (Big Data), IEEE 2018, 4896–4899.
• 47. Daroya R., Peralta D., Naval P. Alphabet sign language image classification using deep learning. In TENCON 2018-2018 IEEE Region 10 Conference, IEEE 2018, 646–650.
• 48. Latif G., Mohammad N., Roaa A., Rawan A., Alghazo J., Khan M. An automatic arabic sign language recognition system based on deep cnn: An assistive system for the deaf and hard of hearing. International Journal of Computing and Digital Systems. 2020;9(4):715–724.
• 49. Elons A.S., Abull-Ela M., Tolba M.F. A proposed pcnn features quality optimization technique for pose-invariant 3d arabic sign language recognition. Applied Soft Computing. 2013;13(4):1646–1660.
• 50. Ghazanfar L., Jaafar A., Nazeeruddin M., Roaa A., Rawan A. Arabic alphabets sign language dataset (arasl). Mendeley Data, 2018;1.
• 51. Tecperson. Sign language mnist, 2017.
• 52. Latif G., Mohammad N., Alghazo J., Roaa A., Rawan A. Arasl: Arabic alphabets sign language dataset. Data in brief. 2019;23:103777.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2021).