Video-based HR measurement using adaptive facial regions with multiple color spaces

• Autorzy: Panigrahi Arpita , Sharma Hemant , Mukherjee Atin

Identyfikatory

DOI

10.1016/j.bbe.2023.12.001

• Języki publikacji: EN

Abstrakty

EN

Driven by the desire for feasible and convenient healthcare, non-contact heart rate (HR) monitoring based on consumer-grade cameras has gained significant recognition among researchers. However, this technology suffers from performance reliability and consistency in realistic situations of motion artifacts, illumination variations, and skin tones, limiting it to emerge as an alternative to conventional methods. Considering these challenges, this paper suggests an effective technique for HR measurement from facial RGB videos. The face being the region of interest (ROI) is divided into several small sub-ROIs of even size. A group of quality sub-ROIs is formed and weighted based on the fundamental periodicity coefficient to handle spatial non-uniform illumination and facial motions. Five different color spaces are considered, and the most suitable color component from each space is chosen to alleviate the influence of temporal illumination variation and other factors. The resultant color signals are denoised using the ensemble empirical mode decomposition and integrated using the principal component analysis to derive a pulsating component representing the blood volumetric changes for HR computation. Experiments are conducted over three standard datasets, namely PURE, UBFC, and COHFACE. The obtained mean absolute error values are 1.16 beats per minute (bpm), 1.56 bpm, and 2.10 bpm for PURE, UBFC, and COHFACE datasets, respectively, indicating the performance of the technique well above the clinically acceptable threshold. In comparison, the technique showed performance superiority over the state-of-art methods. These outcomes substantiate the potential of alternative color spaces for accurate and reliable HR monitoring from facial videos in challenging scenarios.

Słowa kluczowe

EN

noncontact PPG; iPPG; heart rate; color space; EEMD

PL

sygnał PPG; iPPG; tętno serca; EEMD

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2024
• Tom: Vol. 44, no. 1
• Strony: 68--82
• Opis fizyczny: Bibliogr. 80 poz., rys., tab., wykr.

Twórcy

autor

Panigrahi Arpita

• Department of Electronics & Communication Engineering, National Institute of Technology, Rourkela, 769008, India

autor

Sharma Hemant

• Department of Electronics & Communication Engineering, National Institute of Technology, Kurukshetra, 136119, India

autor

Mukherjee Atin

• Department of Electronics & Communication Engineering, National Institute of Technology, Rourkela, 769008, India

Bibliografia

• [1] Böhm M, Reil J-C, Deedwania P, Kim JB, Borer JS. Resting heart rate: risk indicator and emerging risk factor in cardiovascular disease. Am J Phys Med 2015;128(3):219-28.
• [2] Gilgen-Ammann R, Schweizer T, Wyss T. RR interval signal quality of a heart rate monitor and an ECG holter at rest and during exercise. Eur J Appl Physiol 2019;119(7):1525-32.
• [3] Li C, Wu Y, Lin H, Li J, Zhang F, Yang Y. ECG denoising method based on an improved VMD algorithm. IEEE Sens J 2022;22(23):22725-33.
• [4] Sharma H, Sharma K. Baseline wander removal of ECG signals using Hilbert vibration decomposition. Electron Lett 2015;51(6):447-9.
• [5] Qiu L, Cai W, Zhang M, Zhu W, Wang L. Two-stage ECG signal denoising based on deep convolutional network. Physiol Meas 2021;42(11):115002.
• [6] Karri M, Annavarapu CSR, Pedapenki KK. A real-time cardiac arrhythmia classification using hybrid combination of delta modulation, 1D-CNN and blended LSTM. Neural Process Lett 2023;55(2):1499-526.
• [7] Zhang Y, Yi J, Chen A, Cheng L. Cardiac arrhythmia classification by time-frequency features inputted to the designed convolutional neural networks. Biomed Signal Process Control 2023;79:104224.
• [8] Rahul J, Sharma LD. Automatic cardiac arrhythmia classification based on hybrid 1-D CNN and bi-LSTM model. Biocybern Biomed Eng 2022;42(1):312-24.
• [9] Wang T, Qin Y. A novel multi-scale convolutional network with attention-based bidirectional gated recurrent unit for atrial fibrillation discrimination. Biocybern Biomed Eng 2021;41(2):445-55.
• [10] Sharma H. Heart rate extraction from PPG signals using variational mode decomposition. Biocybern Biomed Eng 2019;39(1):75-86.
• [11] Motin MA, Karmakar CK, Palaniswami M. An EEMD-PCA approach to extract heart rate, respiratory rate and respiratory activity from PPG signal. In: 2016 38th annual international conference of the IEEE engineering in medicine and biology society. EMBC, IEEE; 2016, p. 3817-20.
• [12] Vicente-Samper JM, Tamantini C, Ávila-Navarro E, De La Casa-Lillo MÁ, Zollo L, Sabater-Navarro JM, et al. An ML-based approach to reconstruct heart rate from PPG in presence of motion artifacts. Biosensors 2023;13(7):718.
• [13] Dao D, Salehizadeh SM, Noh Y, Chong JW, Cho CH, McManus D, et al. A robust motion artifact detection algorithm for accurate detection of heart rates from photoplethysmographic signals using time-frequency spectral features. IEEE J Biomed Health Inform 2016;21(5):1242-53.
• [14] Ahmed R, Mehmood A, Rahman MMU, Dobre OA. A deep learning & fast wavelet transform-based hybrid approach for denoising of ppg signals. IEEE Sensors Lett 2023.
• [15] Ye Y, Cheng Y, He W, Hou M, Zhang Z. Combining nonlinear adaptive filtering and signal decomposition for motion artifact removal in wearable photoplethysmography. IEEE Sens J 2016;16(19):7133-41.
• [16] Sun Y, Thakor N. Photoplethysmography revisited: from contact to noncontact, from point to imaging. IEEE Trans Biomed Eng 2015;63(3):463-77.
• [17] Mehta AD, Sharma H. OPOIRES: A robust non-contact respiratory rate extraction based on optimal points-of-interest selection from an RGB camera. Biomed Signal Process Control 2023;86:105193.
• [18] Harford M, Catherall J, Gerry S, Young JD, Watkinson P. Availability and performance of image-based, non-contact methods of monitoring heart rate, blood pressure, respiratory rate, and oxygen saturation: a systematic review. Physiol Meas 2019;40(6):06TR01.
• [19] Odinaev I, Wong KL, Chin JW, Goyal R, Chan TT, So RH. Robust heart rate variability measurement from facial videos. Bioengineering 2023;10(7):851.
• [20] Mehta AD, Sharma H. Heart rate estimation from RGB facial videos using robust face demarcation and VMD. In: 2021 national conference on communications. IEEE; 2021, p. 1-6.
• [21] Gupta D, Etemad A. Privacy-preserving remote heart rate estimation from facial videos. 2023, arXiv preprint arXiv:2306.01141.
• [22] Chen M, Liao X, Wu M. PulseEdit: Editing physiological signals in facial videos for privacy protection. IEEE Trans Inf Forensics Secur 2022;17:457-71.
• [23] Hu J, Liao X, Wang W, Qin Z. Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network. IEEE Trans Circuits Syst Video Technol 2021;32(3):1089-102.
• [24] Liao X, Wang Y, Wang T, Hu J, Wu X. FAMM: Facial muscle motions for detecting compressed deepfake videos over social networks. IEEE Trans Circuits Syst Video Technol 2023.
• [25] Mehta AD, Sharma H. Cpulse: Heart rate estimation from RGB videos under realistic conditions. IEEE Trans Instrum Meas 2023.
• [26] Poh M-Z, McDuff DJ, Picard R. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express 2010;18(10):10762-74.
• [27] De Haan G, Jeanne V. Robust pulse rate from chrominance-based rPPG. IEEE Trans Biomed Eng 2013;60(10):2878-86.
• [28] Tsouri GR, Li Z. On the benefits of alternative color spaces for noncontact heart rate measurements using standard red-green-blue cameras. J Biomed Opt 2015;20(4):48002.
• [29] Panigrahi A, Sharma H. Non-contact HR extraction from different color spaces using RGB camera. In: 2022 national conference on communications. NCC, IEEE; 2022, p. 332-7.
• [30] Song R, Li J, Wang M, Cheng J, Li C, Chen X. Remote photoplethysmography with an eemd-mcca method robust against spatially uneven illuminations. IEEE Sens J 2021;21(12):13484-94.
• [31] Das M, Choudhary T, Bhuyan M, Sharma L. Non-contact heart rate measurement from facial video data using a 2d-vmd scheme. IEEE Sens J 2022;22(11):11153-61.
• [32] Kumar M, Veeraraghavan A, Sabharwal A. DistancePPG: Robust non-contact vital signs monitoring using a camera. Biomed Opt Express 2015;6(5):1565-88.
• [33] Li X, Chen J, Zhao G, Pietikainen M. Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2014, p. 4264-71.
• [34] Wang W, Den Brinker AC, Stuijk S, De Haan G. Algorithmic principles of remote PPG. IEEE Trans Biomed Eng 2016;64(7):1479-91.
• [35] Feng L, Po L-M, Xu X, Li Y, Ma R. Motion-resistant remote imaging photoplethysmography based on the optical properties of skin. IEEE Trans Circuits Syst Video Technol 2014;25(5):879-91.
• [36] Song R, Zhang S, Cheng J, Li C, Chen X. New insights on super-high resolution for video-based heart rate estimation with a semi-blind source separation method. Comput Biol Med 2020;116:103535.
• [37] Ryu J, Hong S, Liang S, Pak S, Chen Q, Yan S. A measurement of illumination variation-resistant noncontact heart rate based on the combination of singular spectrum analysis and sub-band method. Comput Methods Programs Biomed 2021;200:105824.
• [38] Sahin SM, Deng Q, Castelo J, Lee D, et al. Non-contact heart rate monitoring from face video utilizing color intensity. J Multimed Inf Syst 2021;8(1):1-10.
• [39] Martinez-Delgado GH, Correa-Balan AJ, May-Chan JA, Parra-Elizondo CE, Guzman-Rangel LA, Martinez-Torteya A. Measuring heart rate variability using facial video. Sensors 2022;22(13):4690.
• [40] Shi Y, Qiu J, Peng L, Han P, Luo K, Liu D. A novel non-contact heart rate measurement method based on EEMD combined with FastICA. Physiol Meas 2023;44(5):055002.
• [41] Yin R-N, Jia R-S, Cui Z, Yu J-T, Du Y-B, Gao L, et al. Heart rate estimation based on face video under unstable illumination. Appl Intell 2021;1-17.
• [42] Rahman H, Ahmed MU, Begum S. Non-contact physiological parameters extraction using facial video considering illumination, motion, movement and vibration. IEEE Trans Biomed Eng 2019;67(1):88-98.
• [43] Zhang Y, Dong Z, Zhang K, Shu S, Lu F, Chen J. Illumination variation-resistant video-based heart rate monitoring using LAB color space. Opt Lasers Eng 2021;136:106328.
• [44] Park J, Hong K. Robust pulse rate measurements from facial videos in diverse environments. Sensors 2022;22(23):9373.
• [45] Lie W-N, Le D-Q, Lai C-Y, Fang Y-S. Heart rate estimation from facial image sequences of a dual-modality RGB-NIR camera. Sensors 2023;23(13):6079.
• [46] Ryu J, Hong S, Liang S, Pak S, Chen Q, Yan S. Research on the combination of color channels in heart rate measurement based on photoplethysmography imaging. J Biomed Opt 2021;26(2):025003.
• [47] Niu X, Zhao X, Han H, Das A, Dantcheva A, Shan S, et al. Robust remote heart rate estimation from face utilizing spatial-temporal attention. In: 2019 14th IEEE international conference on automatic face & gesture recognition. FG 2019, IEEE; 2019, p. 1-8.
• [48] Hsu G-SJ, Xie R-C, Ambikapathi A, Chou K-J. A deep learning framework for heart rate estimation from facial videos. Neurocomputing 2020;417:155-66.
• [49] Qiu Y, Liu Y, Arteaga-Falconi J, Dong H, El Saddik A. EVM-CNN: Real-time contactless heart rate estimation from facial video. IEEE Trans Multimed 2018;21(7):1778-87.
• [50] Lokendra B, Puneet G. AND-rPPG: A novel denoising-rPPG network for improving remote heart rate estimation. Comput Biol Med 2022;141:105146.
• [51] Zhang X, Xia Z, Liu L, Feng X. Demodulation based transformer for rPPG generation and heart rate estimation. IEEE Signal Process Lett 2023.
• [52] Comas J, Ruiz A, Sukno F. Efficient remote photoplethysmography with temporal derivative modules and time-shift invariant loss. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, p. 2182-91.
• [53] Chen W, McDuff D. Deepphys: Video-based physiological measurement using convolutional attention networks. In: Proceedings of the European conference on computer vision. 2018, p. 349-65.
• [54] Špetlík R, Franc V, Matas J. Visual heart rate estimation with convolutional neural network. In: Proceedings of the British machine vision conference. 2018, p. 3-6.
• [55] Yu Z, Li X, Zhao G. Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. 2019, arXiv preprint arXiv:1905.02419.
• [56] Yu Z, Shen Y, Shi J, Zhao H, Cui Y, Zhang J, Torr P, Zhao G. Physformer++: Facial video-based physiological measurement with slowfast temporal difference transformer. Int J Comput Vis 2023;131(6):1307-30.
• [57] Song R, Zhang S, Li C, Zhang Y, Cheng J, Chen X. Heart rate estimation from facial videos using a spatiotemporal representation with convolutional neural networks. IEEE Trans Instrum Meas 2020;69(10):7411-21.
• [58] Zheng K, Ci K, Li H, Shao L, Sun G, Liu J, Cui J. Heart rate prediction from facial video with masks using eye location and corrected by convolutional neural networks. Biomed Signal Process Control 2022;75:103609.
• [59] Sabokrou M, Pourreza M, Li X, Fathy M, Zhao G. Deep-hr: Fast heart rate estimation from face video under realistic conditions. Expert Syst Appl 2021;186:115596.
• [60] Xiao H, Zhang W, Chen K, Zhu X, Li Y, Zhang D, Liu D. Facial video heart rate detection based on fast-ICA and LA-Res2Net. In: International conference on image, signal processing, and pattern recognition, vol. 12707. SPIE; 2023, p. 460-7.
• [61] Bousefsaf F, Pruski A, Maaoui C. 3D convolutional neural networks for remote pulse rate measurement and mapping from facial video. Appl Sci 2019;9(20):4364.
• [62] Ouzar Y, Djeldjli D, Bousefsaf F, Maaoui C. X-iPPGNet: A novel one stage deep learning architecture based on depthwise separable convolutions for video-based pulse rate estimation. Comput Biol Med 2023;154:106592.
• [63] Viola P, Jones M. Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, vol. 1. IEEE; 2001, p. I.
• [64] Tomasi C, Kanade T. Detection and tracking of point. Int J Comput Vis 1991;9:137-54.
• [65] Stricker R, Müller S, Gross H-M. Non-contact video-based pulse rate measurement on a mobile service robot. In: The 23rd IEEE international symposium on robot and human interactive communication. IEEE; 2014, p. 1056-62.
• [66] Bobbia S, Macwan R, Benezeth Y, Mansouri A, Dubois J. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognit Lett 2019;124:82-90.
• [67] Heusch G, Anjos A, Marcel S. A reproducible study on remote heart rate measurement. 2017, arXiv preprint arXiv:1709.00962.
• [68] Martin N, Mailhes C. About periodicity and signal to noise ratio-the strength of the autocorrelation function. In: CM 2010-MFPT 2010-7th international conference on condition monitoring and machinery failure prevention technologies. 2010, p. nc.
• [69] Ibraheem NA, Hasan MM, Khan RZ, Mishra PK. Understanding color models: a review. ARPN J Sci Technol 2012;2(3):265-75.
• [70] Yang Y, Liu C, Yu H, Shao D, Tsow F, Tao N. Motion robust remote photo-plethysmography in CIELab color space. J Biomed Opt 2016;21(11):117001.
• [71] Cheng J, Chen X, Xu L, Wang ZJ. Illumination variation-resistant video-based heart rate measurement using joint blind source separation and ensemble empirical mode decomposition. IEEE J Biomed Health Inform 2016;21(5):1422-33.
• [72] Balakrishnan G, Durand F, Guttag J. Detecting pulse from head motions in video. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2013, p. 3430-7.
• [73] Wang W, Stuijk S, De Haan G. A novel algorithm for remote photoplethysmography: Spatial subspace rotation. IEEE Trans Biomed Eng 2015;63(9):1974-84.
• [74] Qi L, Yu H, Xu L, Mpanda RS, Greenwald SE. Robust heart-rate estimation from facial videos using Project_ICA. Physiol Meas 2019;40(8):085007.
• [75] McDuff D, Blackford E. iphys: An open non-contact imaging-based physiological measurement toolbox. In: 2019 41st annual international conference of the IEEE engineering in medicine and biology society. IEEE; 2019, p. 6521-4.
• [76] Song R, Chen H, Cheng J, Li C, Liu Y, Chen X. PulseGAN: Learning to generate realistic pulse waveforms in remote photoplethysmography. IEEE J Biomed Health Inf 2021;25(5):1373-84.
• [77] Liu X, Hill B, Jiang Z, Patel S, McDuff D. Efficientphys: Enabling simple, fast and accurate camera-based cardiac measurement. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2023, p. 5008-17.
• [78] Li B, Zhang P, Peng J, Fu H. Non-contact PPG signal and heart rate estimation with multi-hierarchical convolutional network. Pattern Recognit 2023;139:109421.
• [79] Gupta AK, Kumar R, Birla L, Gupta P. RADIANT: Better rPPG estimation using signal embeddings and transformer. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2023, p. 4976-86.
• [80] Deshpande Y, Thapa S, Sarkar A, Abbott AL. Camera-based recovery of cardiovascular signals from unconstrained face videos using an attention network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023, p. 5974-83.

Uwagi

PL

Opracowanie rekordu ze środków MNiSW, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).