The ASL Dataset for Real-Time Recognition and Integration with LLM Services

• Autorzy: Chwesiuk Michał , Popis Piotr

Identyfikatory

DOI

10.24425/ijet.2024.152513

• Języki publikacji: EN

Abstrakty

EN

This study aims to investigate the impact of hand gesture recognition techniques on the efficiency of American Sign Language (ASL) interpretation, addressing a critical gap in the existing literature. The research seeks new insights into the chal-lenges of automated sign language recognition, contributing to a deeper understanding of accessibility in communication for the deaf and hard-of-hearing community. The study employs a quan-titative approach, using a dataset comprising hand gesture images representing the static letters of the ASL alphabet collected from multiple users. Data were collected from various individuals to ensure diversity and analyzed using machine learning models to evaluate their effectiveness in recognizing ASL signs. The results reveal that the machine learning models implemented achieved a high accuracy rate in recognizing hand gestures, indicating that person-specific variations do not significantly hinder performance. These findings provide evidence that the proposed dataset and methodologies can improve the reliability of sign language recognition systems, offering significant impli-cations for the development of more inclusive communication technologies. This research offers a novel perspective on sign language recognition, providing valuable insight that extends the current understanding of gesture-based communication systems. The study’s findings contribute to advancements in accessibility technologies, highlighting areas for future research and practical applications in improving communication for the Deaf and hard of hearing community.

Słowa kluczowe

EN

dataset; gesture recognition; hand landmarks; machine learning; human-computer interaction; american sign language

• Wydawca: Polish Academy of Sciences, Committee of Electronics and Telecommunication
• Czasopismo: International Journal of Electronics and Telecommunications
• Rocznik: 2024
• Tom: Vol. 70, No. 4
• Strony: 1105--1112
• Opis fizyczny: Bibliogr., 30 poz., rys., fot., tab.

Twórcy

autor

Chwesiuk Michał

• Warsaw University of Technology

• https://orcid.org/0000-0002-6531-0232

autor

Popis Piotr

• Warsaw University of Technology

Bibliografia

• [1] H. Vaezi Joze and O. Koller, “Ms-asl: A large-scale data set and benchmark for understanding american sign language,” in The British Machine Vision Conference (BMVC), September 2019.
• [2] Tecperson, “Sign language mnist,” 2017. [Online]. Available: https://www.kaggle.com/datasets/datamunge/sign-language-mnist
• [3] A. A. Abdulhussein and F. A. Raheem, “Hand gesture recognition of static letters american sign language (asl) using deep learning,” Engineering and Technology Journal, vol. 38, no. 6A, 2024.
• [4] M. M. Zaki and S. I. Shaheen, “Sign language recognition using a combination of new vision based features,” Pattern Recognition Letters, vol. 32, no. 4, pp. 572-577, 2011.
• [5] A. N´u˜nez-Marcos, O. P. de Vi˜naspre, and G. Labaka, “A survey on sign language machine translation,” Expert Systems with Applications, vol. 213, p. 118993, 2023.
• [6] A. Gopi, S. D. P, S. T, J. Stephen, and B. VK, “Multilingual speech to speech mt based chat system,” in 2015 International Conference on Computing and Network Communications (CoCoNet), 2015, pp. 771-776.
• [7] J. T. S. Ru and P. Sebastian, “Real-time american sign language (asl) interpretation,” in 2023 2nd International Conference on Vision To-wards Emerging Trends in Communication and Networking Technologies (ViTECoN), 2023, pp. 1-6.
• [8] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, “Mediapipe hands: On-device real-time hand tracking,” 2020.
• [9] M. R. Chilukala and V. Vadalia, “Translating sign language to English text in real time using deep learning models,” in 2022 International Conference on Electronics and Renewable Systems (ICEARS), 2022, pp. 1296-1301.
• [10] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man´e, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Vi´egas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
• [11] C. Bhat, R. Rajeshirke, S. Chude, V. Mhaiskar, and V. Agarwal, “Two-way communication: An integrated system for american sign language recognition and speech-to-text translation,” in 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), 2023, pp. 1-7.
• [12] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee et al., “Mediapipe: A framework for building perception pipelines,” arXiv preprint arXiv:1906.08172, 2019.
• [13] P. Sahane, “Duplex sign language communicator,” International Journal for Research in Applied Science and Engineering Technology, 2021, a system utilizing NLP and CNN for sign language translation focusing on Indian Sign Language.
• [14] A. Dumbre, S. Jangada, S. Gosavi, and J. Gupta, “Classification of Indian sign language characters utilizing convolutional neural networks and transfer learning models with different image processing techniques,” in 2022 3rd International Conference on Advances in Computing, Communication, and Control (AIC), 2022.
• [15] M. N. Saiful, A. A. Isam, H. A. Moon, R. T. Jaman, M. Das, M. R. Alam, and A. Rahman, “Real-time sign language detection using cnn,” in 2022 International Conference on Data Analytics and Business Intelligence (ICDABI), 2022.
• [16] S. Targ, D. Almeida, and K. Lyman, “Resnet in resnet: Generalizing residual architectures,” arXiv preprint arXiv:1603.08029, 2016.
• [17] A. LeNail, “Nn-svg: Publication-ready neural network architecture schematics,” Journal of Open Source Software, vol. 4, no. 33, p. 747, 2019.
• [18] A. S. Dhanjal and W. Singh, “Tools and techniques of assistive technol-ogy for hearing impaired people,” in 2019 International conference on machine learning, big data, cloud and parallel computing (COMITCon). IEEE, 2019, pp. 205-210.
• [19] A. S. Lhoussain, G. Hicham, and Y. Abdellah, “Adaptating the leven-shtein distance to contextual spelling correction,” International Journal of Computer Science and Applications, vol. 12, no. 1, pp. 127-133, 2015.
• [20] A. L. Barczak, N. H. Reyes, M. Abastillas, A. Piccio, and T. Susnjak, “A new 2d static hand gesture colour image dataset for asl gestures,” 2011.
• [21] H. R. V. Joze and O. Koller, “Ms-asl: A large-scale data set and benchmark for understanding american sign language,” arXiv preprint arXiv:1812.01053, 2018.
• [22] Michal Chwesiuk and Piotr Popis, “Asldatacollector: Cli tool for managing and processing hand image datasets for asl recognition.” https://github.com/sqoshi/asldatacollector, 2024, accessed: 2024-10-20.
• [23] P. Michał Chwesiuk, “Asldatacollector: A python package for collecting asl image data,” 2024. [Online]. Available: https://pypi.org/project/asldatacollector/
• [24] Michal Chwesiuk and Piotr Popis, “Asl hands,” 2024. [Online]. Available: https://www.kaggle.com/datasets/piotrpopis/asl-hands
• [25] P. Popis, “hands-to-text: A web application and python package for converting sign language gestures into text.” https://github.com/sqoshi/hands-to-text , 2024, accessed: 2024-10-20
• [26] S. Ram´ırez, “Fastapi framework, high performance, easy to learn, fast to code, ready for production,” https://fastapi.tiangolo.com/ , 2018.
• [27] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” in Proceedings of the IEEE, vol. 86, no. 11, 1998, pp. 2278-2324.
• [28] Michal Chwesiuk and Piotr Popis, “htt-models: Package for training, processing, and versioning models for hands-to-text.” https://github.com/sqoshi/htt-models , 2024, accessed: 2024-10-20.
• [29] N. Bhavana and G. S. Shenoy, “Empowering communication: Harness-ing cnn and mediapipe for sign language interpretation,” in 2023 Inter-national Conference on Recent Advances in Science and Engineering Technology (ICRASET), 2023.
• [30] H. V. Joze and O. Koller, “Ms-asl: A large-scale data set and benchmark for understanding american sign language,” in The British Machine Vision Conference (BMVC), September 2019.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr POPUL/SP/0154/2024/02 w ramach programu "Społeczna odpowiedzialność nauki II" - moduł: Popularyzacja nauki (2025).