ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Human hand gesture identification framework using SIFT and knowledge-level technique

  • Muhammad Haroon (University Institute of Information Technology, Pir Mehr Ali Shah Arid Agriculture University) ;
  • Saud Altaf (University Institute of Information Technology, Pir Mehr Ali Shah Arid Agriculture University) ;
  • Zia-ur- Rehman (University Institute of Information Technology, Pir Mehr Ali Shah Arid Agriculture University) ;
  • Muhammad Waseem Soomro (Manukau Institute of Technology) ;
  • Sofia Iqbal (University Institute of Information Technology, Pir Mehr Ali Shah Arid Agriculture University)
  • Received : 2022.07.20
  • Accepted : 2022.10.11
  • Published : 2023.12.10
Download PDF
⟨ Previous Next ⟩

Abstract

In this study, the impact of varying lighting conditions on recognition and decision-making was considered. The luminosity approach was presented to increase gesture recognition performance under varied lighting. An efficient framework was proposed for sensor-based sign language gesture identification, including picture acquisition, preparing data, obtaining features, and recognition. The depth images were collected using multiple Microsoft Kinect devices, and data were acquired by varying resolutions to demonstrate the idea. A case study was designed to attain acceptable accuracy in gesture recognition under variant lighting. Using American Sign Language (ASL), the dataset was created and analyzed under various lighting conditions. In ASL-based images, significant feature points were selected using the scale-invariant feature transformation (SIFT). Finally, an artificial neural network (ANN) classified hand gestures using specified characteristics for validation. The suggested method was successful across a variety of illumination conditions and different image sizes. The total effectiveness of NN architecture was shown by the 97.6% recognition accuracy rate of 26 alphabets dataset with just a 2.4% error rate.

Keywords

References

  1. Y. S. Tan, K. M. Lim, C. Tee, C. P. Lee, and C. Y. Low, Convolutional neural network with spatial pyramid pooling for hand gesture recognition, Neural Comput. Applic. 33 (2021), 5339-5351. https://doi.org/10.1007/s00521-020-05337-0
  2. C. Dhiman and D. K. Vishwakarma, A robust framework for abnormal human action recognition using＼boldsymbol{＼mathcal{r}} -transform and zernike moments in depth videos, IEEE Sensors J. 19 (2019), 5195-5203.
  3. D. K. Vishwakarma, A two-fold transformation model for human action recognition using decisive pose, Cogn. Syst. Res. 61 (2020), 1-13. https://doi.org/10.1016/j.cogsys.2019.12.004
  4. D. K. Vishwakarma, R. Kapoor, R. Maheshwari, V. Kapoor, and S. Raman, Recognition of abnormal human activity using the changes in orientation of silhouette in key frames, (2015 2nd International Conference on Computing for Sustainable Global Development, New Delhi, India), 2015, pp. 336-341.
  5. A. Agarwal, A. Yadav, and D. K. Vishwakarma, Multimodal sentiment analysis via RNN variants, (IEEE International Conference on Big Data, Cloud Computing, and Data Science & Engineering, Honolulu, HI, USA), 2019, pp. 19-23.
  6. A. Yadav, A. Agarwal, and D. K. Vishwakarma, XRA-net framework for visual sentiments analysis, (IEEE Proceedings 5th international conference on multimedia big data, Singapore), 2019, pp. 219-224.
  7. D. Liang, X. Wu, J. Chen, and R. Setchi, Novel hand gesture recognition method based on illumination compensation and grayscale adjustment, In Human centred intelligent systems, Springer, Singapore, 2021, 115-125.
  8. D. K. Vishwakarma and R. Kapoor, An efficient interpretation of hand gestures to control smart interactive television, Int. J. Comp. Vis. Robot. 7 (2017), 454-471. https://doi.org/10.1504/IJCVR.2017.084991
  9. S. Wang, A. Wang, M. Ran, L. Liu, Y. Peng, M. Liu, G. Su, A. Alhudhaif, F. Alenezi, and N. Alnaim, Hand gesture recognition framework using a lie group based spatio-temporal recurrent network with multiple hand-worn motion sensors, Inform. Sci. 606 (2022), 722-741. https://doi.org/10.1016/j.ins.2022.05.085
  10. D. K. Vishwakarma and R. Kapoor, Simple and intelligent system to recognize the expression of speech-disabled person, (4th International Conference on Intelligent Human Computer Interaction, Kharagpur, India), 2012. https://doi.org/10.1109/IHCI.2012.6481804
  11. D. K. Vishwakarma, R. Kapoor, and A. Dhiman, Unified framework for human activity recognition: An approach using spatial edge distribution and ℜ-transform, AEU Int. J. Electron. Commun. 70 (2016), 341-353. https://doi.org/10.1016/j.aeue.2015.12.016
  12. D. K. Vishwakarma and R. Kapoor, Integrated approach for human action recognition using edge spatial distribution, direction pixel and R-transform, Adv. Robot. 29 (2015), 1553-1562. https://doi.org/10.1080/01691864.2015.1061701
  13. D. K. Vishwakarma and T. Singh, A visual cognizance based multi-resolution descriptor for human action recognition using key pose, AEU Int. J. Electron. Commun. 107 (2019), 157-169. https://doi.org/10.1016/j.aeue.2019.05.023
  14. N. S. Khan, A. Abid, and K. Abid, A novel natural language processing (NLP)-based machine translation model for English to Pakistan sign language translation, Cognit. Comput. 12 (2020), 748-765. https://doi.org/10.1007/s12559-020-09731-7
  15. H. Xiao, Y. Yang, K. Yu, J. Tian, X. Cai, Y. Zhao, K. Zhang, N. Guo, and J. Chen, Recognizing hand gesture in still infrared images by CapsNet, In Lecture notes in computer science international conference on web information systems engineering, Springer, Cham, 2021, 158-172.
  16. X. Yingxin, L. Jinghua, W. Lichun, and K. Dehui, A robust hand gesture recognition method via convolutional neural network, (6th International Conference on Digital Home, Guangzhou, China), 2016, pp. 64-67.
  17. L. Guo, L. Zongxing, and L. Yao, Human-machine interaction SensingTechnology based on hand gesture recognition: A review, IEEE Trans. Human-Mach. Syst. 51 (2021), 300-309. https://doi.org/10.1109/THMS.2021.3086003
  18. D. K. Vishwakarma, Hand Gesture Recognition using Shape and Texture evidences in Complex Background, (Proceedings of the International Conference on Inventive Computing and Informatics, Coimbatore, India), 2017, pp. 278-283.
  19. D. K. Vishwakarma and V. Grover, Hand gesture recognition in low-intensity environment using depth images, (International Conference on Intelligent Sustainable Systems, Palladam, Indai), 2018, pp. 429-433.
  20. D. K. Vishwakarma, R. Maheshwari, and R. Kapoor, An efficient approach for the recognition of hand gestures from very low resolution images, (Proceedings 5th International Conference on Communication Systems and NetworkTechnologies, Gwalior, India), 2015, pp. 467-471.
  21. X. Y. Wu, A hand gesture recognition algorithm based on DCCNN, Multimed. Tools Appl. 79 (2020), 9193-9205. https://doi.org/10.1007/s11042-019-7193-4
  22. X. Tang, Z. Yan, J. Peng, B. Hao, H. Wang, and J. Li, Selective spatiotemporal features learning for dynamic gesture recognition, Expert Syst. Appl. 169 (2021), 114499.
  23. G. Chen, Z. Xu, Z. Li, H. Tang, S. Qu, K. Ren, and A. Knoll, A novel illumination-robust hand gesture recognition system with event-based neuromorphic vision sensor, IEEE Trans. Autom. Sci. Eng. Published online 18 (2021), 508-520. https://doi.org/10.1109/TASE.2020.3045880
  24. Y. Huang and J. Yang, A multi-scale descriptor for real time RGB-D hand gesture recognition, Pattern Recognit. Lett. 144 (2021), 97-104. https://doi.org/10.1016/j.patrec.2020.11.011
  25. T. Song, H. Zhao, Z. Liu, H. Liu, Y. Hu, and D. Sun, Intelligent human hand gesture recognition by local-global fusing quality-aware features, Future Gener. Comput. Syst. 115 (2021), 298-303. https://doi.org/10.1016/j.future.2020.09.013
  26. C. K. M. Lee, K. K. H. Ng, C. H. Chen, H. C. W. Lau, S. Y. Chung, and T. Tsoi, American sign language recognition and training method with recurrent neural network, Expert Syst. Appl. 167 (2021), 114403.
  27. Y. Wang, A. Ren, M. Zhou, W. Wang, and X. Yang, A novel detection and recognition method for continuous hand gesture using fmcw radar, IEEE Access 8 (2020), 167264-167275. https://doi.org/10.1109/ACCESS.2020.3023187
  28. L. Xi, W. Chen, C. Zhao, X. Wu, and J. Wang, Image enhancement for remote photoplethysmography in a low-light environment, (IEEE International Conference on Automatic Face and Gesture Recognition, Buenos Aires, Argentina), 2020, pp. 761-764.
  29. A. Onan, Bidirectional convolutional recurrent neural network architecture with group-wise enhancement mechanism for text sentiment classification, J. King Saud Univ. Comput. Inf. Sci. 34 (2022), 2098-2117. https://doi.org/10.1016/j.jksuci.2022.02.025
  30. A. Onan, Consensus clustering-based undersampling approach to imbalanced learning, Sci. Prog. 2019 (2019), 1-14.
  31. A. Onan, Sentiment analysis on product reviews based on weighted word embeddings and deep neural networks, Concurrency Comput. Pract. Experience 33 (2021),
  32. A. Onan, S. Korukoglu, and H. Bulut, Ensemble of keyword extraction methods and classifiers in text classification, Expert Syst. Appl. 57 (2016), 232-247. https://doi.org/10.1016/j.eswa.2016.03.045
  33. A. Onan, S. Korukoglu, and H. Bulut, A hybrid ensemble pruning approach based on consensus clustering and multi-objective evolutionary algorithm for sentiment classification, Inf. Process. Manag. 53 (2017), 814-833. https://doi.org/10.1016/j.ipm.2017.02.008
  34. A. Onan, A two-stage topic extraction model for bibliometric data analysis based on word embeddings and clustering, IEEE Access 7 (2019), 145614-145633.
  35. A. Onanand and S. Korukoglu, A feature selection model based on genetic rank aggregation for text sentiment classification, J. Inf. Sci. 43 (2017), 25-38. https://doi.org/10.1177/0165551515613226
  36. A. Onan, Mining opinions from instructor evaluation reviews: A deep learning approach, Comput. Appl. Eng. Educ. 28 (2020), 117-138. https://doi.org/10.1002/cae.22179
  37. A. Onan, Sentiment analysis on massive open online course evaluations: A text mining and deep learning approach, Comput. Appl. Eng. Educ. 29 (2021), 572-589. https://doi.org/10.1002/cae.22253
  38. A. Onan, Biomedical text categorization based on ensemble pruning and optimized topic modelling, Comput. Math. Methods Med. 2018 (2018), 2497471.
  39. A. Onan, An ensemble scheme based on language function analysis and feature engineering for text genre classification, J. Inf. Sci. 44 (2018), 28-47. https://doi.org/10.1177/0165551516677911
  40. A. Onan and M. A. Tocoglu, A term weighted neural language model and stacked bidirectional LSTM based framework for sarcasm identification, IEEE Access 9 (2021), 7701-7722. https://doi.org/10.1109/ACCESS.2021.3049734
  41. A. Onan, Topic-enriched word embeddings for sarcasm identification, Adv. Intell. Syst. Comput. 984 (2019), 293-304. https://doi.org/10.1007/978-3-030-19807-7_29
  42. T. Fan, C. Ma, Z. Gu, Q. Lv, J. Chen, D. Ye, J. Huangfu, Y. Sun, C. Li, and L. Ran, Wireless hand gesture recognition based on continuous-wave Doppler radar sensors, IEEE Trans. Microwave Theory Tech. 64 (2016), 4012-4020. https://doi.org/10.1109/TMTT.2016.2610427
  43. Y. Wang, Y. Shu, X. Jia, M. Zhou, L. Xie, and L. Guo, Multifeature fusion-based hand gesture sensing and recognition system, IEEE Geosci. Remote Sens. Lett. 19 (2022), 1-5.
  44. K. Djunaidi, H. Bedi Agtriadi, D. Kuswardani, and Y. S. Purwanto, Gray level co-occurrence matrix feature extraction and histogram in breast cancer classification with ultrasonographic imagery, IJEECS 22 (2021), 795.
  45. D. G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vis. 60 (2004), 91-110. https://doi.org/10.1023/B:VISI.0000029664.99615.94
  46. I. R. Otero and M. Delbracio, Anatomy of the SIFT method, Image Process. On Line 4 (2016), 370-396. https://doi.org/10.5201/ipol.2014.82
  47. I. El Ouariachi, R. Benouini, K. Zenkouar, A. Zarghili, and H. El Fadili, RGB-D feature extraction method for hand gesture recognition based on a new fast and accurate multi-channel Cartesian Jacobi moment invariants, Multimed. Tools Appl. 81 (2022), 12725-12757. https://doi.org/10.1007/s11042-022-12161-2
  48. A. F. M. Saif and Z. R. Mahayuddin, An efficient method for hand gesture recognition using robust features vector, J. Inf. Syst. Technol. Manag. (JISTM) 6 (2021), 25-35.
  49. G. Chen, Z. Xu, Z. Li, H. Tang, S. Qu, K. Ren, and A. Knoll, A novel illumination-robust hand gesture recognition system with event-based neuromorphic vision sensor, IEEE Trans. Automat. Sci. Eng. 18 (2021), 508-520. https://doi.org/10.1109/TASE.2020.3045880
  50. G. Benitez-Garcia, J. Olivares-Mercado, G. Sanchez-Perez, and K. Yanai, IPN hand: A video dataset and benchmark for real-time continuous hand gesture recognition, (25th International Conference on Pattern Recognition, Milan, Italy), 2020, pp. 4340-4347.
  51. Y. S. Tan, K. M. Lim, and C. P. Lee, Hand gesture recognition via enhanced densely connected convolutional neural network, Expert Syst. Appl. 175 (2021), 114797.
  52. R. F. Pinto, C. D. B. Borges, A. M. A. Almeida, and I. C. Paula, Static hand gesture recognition based on convolutional neural networks, J. Electr. Comput. Eng. 2019 (2019), 1-12.  https://doi.org/10.1155/2019/4167890