Non-contact breathing monitoring by integrating RGB and thermal imaging via RGB-thermal image registration

• Autorzy: Maurya Lalit , Mahapatra Prasant , Chawla Deepak

Identyfikatory

DOI

10.1016/j.bbe.2021.07.002

• Języki publikacji: EN

Abstrakty

EN

Altered breathing rate is an important sign of disease status. Currently used machine-based monitoring of the breathing rate includes contact with the body, which may result in pain and discomfort. In this paper, a non-contact breathing monitoring technique is proposed by integrating RGB and thermal imaging systems with RGB-thermal image registration. This method provides a linear mapping for automated selection of the region of interest (ROI) followed by tracking to extract the breathing rate. To evaluate the efficacy of the proposed approach and its robustness against motion, talking, varying breathing rate or rhythm, and high ambient temperature, this study was conducted in three phases. Validation of the proposed approach demonstrated a strong agreement with the reference method of breathing rate monitoring using a respiration belt. During normal breathing, the mean absolute error (MAE) reached 0.11 bpm (breaths per minute). While in more challenging conditions, defined by three phases, the MAE reached 1.46, 2.08, and 1.69 bpm, respectively. In short, the proposed method performance is a promising alternative to a contact-based method due to its strong agreement and might be useful in diverse applications such as sport studies, rehabilitation centres, quarantine centres, and in hospital or airport screening during the COVID 19 pandemic.

Słowa kluczowe

EN

breathing rate; noncontact breathing monitoring; thermal imaging

PL

częstość oddechu; monitorowanie oddechu; obrazowanie termiczne

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2021
• Tom: Vol. 41, no. 3
• Strony: 1107--1122
• Opis fizyczny: Bibliogr. 53 poz., rys., tab., wykr.

Twórcy

autor

Maurya Lalit

• Academy of Scientific and Innovative Research (AcSIR), Ghaziabad, India; CSIR-Central Scientific Instruments Organisation (CSIR-CSIO), Sector 30-C, Chandigarh, India

autor

Mahapatra Prasant

• V-2 (Biomedical Instrumentation Division), CSIR-Central Scientific Instruments Organisation, Sector 30 C, Chandigarh 160030, India; Academy of Scientific and Innovative Research (AcSIR), Ghaziabad, India

autor

Chawla Deepak

• Department of Neonatology, Government Medical College & Hospital (GMCH), Chandigarh, India

Bibliografia

• [1] I. Wheatley, Respiratory rate 3: how to take an accurate measurement n.d. https://www.nursingtimes.net/clinicalarchive/respiratory-clinical-archive/respiratory-rate-3-howto-take-an-accurate-measurement-25-06-2018/ (accessed April 14, 2021).
• [2] Edwards MO, Kotecha SJ, Kotecha S. Respiratory Distress of the Term Newborn Infant. Paediatr. Respir. Rev. 2013;14 (1):29–37. https://doi.org/10.1016/j.prrv.2012.02.002.
• [3] A.D. Droitcour, T.B. Seto, B.K. Park, S. Yamada, A. Vergara, C. El Hourani, et al. Non-contact respiratory rate measurement validation for hospitalized patients, in: Proc. 31st Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. Eng. Futur. Biomed. EMBC 2009, vol. 2009, IEEE Computer Society; 2009, p. 4812–5. https://doi. org/10.1109/IEMBS.2009.5332635.
• [4] Cretikos MA, Bellomo R, Hillman K, Chen J, Finfer S, Flabouris A. Respiratory rate: The neglected vital sign. Med. J. Aust. 2008;188:657–9. https://doi.org/10.5694/j.1326-5377.2008.tb01825.x.
• [5] Prince PGK, Immanuel Rajkumar R, Premalatha J. Novel NonContact Respiration Rate Detector for Analysis of Emotions. Cham: Springer; 2020. p. 157–78. https://doi.org/ 10.1007/978-3-030-35139-7_8.
• [6] Darzi A, Goršič M, Novak D. Difficulty adaptation in a competitive arm rehabilitation game using real-time control of arm electromyogram and respiration. IEEE Int. Conf. Rehabil. Robot., IEEE Comput. Soc. 2017:857–62. https://doi. org/10.1109/ICORR.2017.8009356.
• [7] Sun G, Nakayama Y, Dagdanpurev S, Abe S, Nishimura H, Kirimoto T, et al. Remote sensing of multiple vital signs using a CMOS camera-equipped infrared thermography system and its clinical application in rapidly screening patients with suspected infectious diseases. Int. J. Infect. Dis. 2017;55:113–7. https://doi.org/10.1016/j.ijid.2017.01.007.
• [8] Jiang Z, Hu M, Gao Z, Fan L, Dai R, Pan Y, et al. Detection of respiratory infections using RGB-infrared sensors on portable device. IEEE Sens. J. 2020;20:13674–81. https://doi.org/10.1109/ JSEN.2020.3004568.
• [9] Naranjo J, Centeno RA, Galiano D, Beaus M. A nomogram for assessment of breathing patterns during treadmill exercise. Br. J. Sports Med. 2005;39:80–3. https://doi.org/10.1136/ bjsm.2003.009316.
• [10] Flenady T, Dwyer T, Applegarth J. Accurate respiratory rates count: So should you! Australas Emerg. Nurs. J. 2017;20:45–7. https://doi.org/10.1016/j.aenj.2016.12.003.
• [11] Baharestani MM. An overview of neonatal and pediatric wound care knowledge and considerations. Ostomy Wound Manag. 2007;53:34–55.
• [12] Droitcour AD, Boric-Lubecke O, Kovacs GTA. Signal-to-noise ratio in doppler radar system for heart and respiratory rate measurements. IEEE Trans. Microw. Theory Tech. 2009;57:2498–507. https://doi.org/10.1109/ TMTT.2009.2029668.
• [13] Procházka A, Schätz M, Vyšata O, Vališ M. Microsoft kinect visual and depth sensors for breathing and heart rate analysis. Sensors 2016;16:996. https://doi.org/10.3390/ s16070996.
• [14] Wang Y, Hu M, Zhou Y, Li Q, Yao N, Zhai G, et al. Unobtrusive and automatic classification of multiple people’s abnormal respiratory patterns in real time using deep neural network and depth camera. IEEE Internet Things J. 2020;7:8559–71. https://doi.org/10.1109/JIOT.2020.2991456.
• [15] M. Bartula, T. Tigges, J. Muehlsteff, Camera-based system for contactless monitoring of respiration, in: Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS, vol. 2013, Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.; 2013, p. 2672–5. https://doi. org/10.1109/EMBC.2013.6610090.
• [16] van Gastel M, Stuijk S, de Haan G. Robust respiration detection from remote photoplethysmography. Biomed. Opt. Express 2016;7:4941. https://doi.org/10.1364/BOE.7.004941.
• [17] Pereira CB, Yu X, Czaplik M, Rossaint R, Blazek V, Leonhardt S. Remote monitoring of breathing dynamics using infrared thermography. Biomed. Opt. Express 2015;6:4378. https://doi. org/10.1364/BOE.6.004378.
• [18] Al-Naji A, Gibson K, Lee SH, Chahl J. Monitoring of cardiorespiratory signal: principles of remote measurements and review of methods. IEEE Access 2017;5:15776–90. https:// doi.org/10.1109/ACCESS.2017.2735419.
• [19] Massaroni C, Nicolo A, Sacchetti M, Schena E. Contactless methods for measuring respiratory rate: a review. IEEE Sens. J. 2021;21:12821–39. https://doi.org/10.1109/ JSEN.2020.3023486.
• [20] Z. Zhu, J. Fei, I. Pavlidis, Tracking human breath in infrared imaging, in: Proc. - BIBE 2005 5th IEEE Symp. Bioinforma. Bioeng., vol. 2005, 2005, p. 227–31. https://doi.org/10.1109/ BIBE.2005.55.
• [21] Murthy R, Pavlidis I. Noncontact measurement of breathing function. IEEE Eng. Med. Biol. Mag. 2006;25:57–67. https://doi. org/10.1109/MEMB.2006.1636352.
• [22] Abbas AK, Heiman K, Orlikowsky T, Leonhardt S. Noncontact respiratory monitoring based on real-time IRthermography. In: IFMBE Proc.. Springer Verlag; 2009. p. 1306–9. https://doi.org/10.1007/978-3-642-03882-2_346.
• [23] AL-Khalidi F, Saatchi R, Elphick H, Burke D, Hisham. An evaluation of thermal imaging based respiration rate monitoring in children. Am. J. Eng. Appl. Sci. 2011;4:586–97. https://doi.org/10.3844/ajeassp.2011.586.597.
• [24] Barbosa Pereira C, Yu X, Czaplik M, Blazek V, Venema B, Leonhardt S. Estimation of breathing rate in thermal imaging videos: a pilot study on healthy human subjects. J. Clin. Monit. Comput. 2017;31:1241–54. https://doi.org/10.1007/ s10877-016-9949-y.
• [25] Cho Y, Julier SJ, Marquardt N, Bianchi-Berthouze N. Robust tracking of respiratory rate in high- dynamic range scenes using mobile thermal imaging. ArXiv 2017;20:90782. https:// doi.org/10.1364/BOE.8.004480.
• [26] Jagadev P, Giri LI. Non-contact monitoring of human respiration using infrared thermography and machine learning. Infrared Phys. Technol. 2020;104:103117. https://doi. org/10.1016/j.infrared.2019.103117.
• [27] Ma J, Zhou Z, Wang B, Zong H. Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Phys. Technol. 2017;82:8–17. https://doi.org/10.1016/j.infrared.2017.02.005.
• [28] Wang J-G, Sung E. Facial feature extraction in an infrared image by proxy with a visible face image. IEEE Trans. Instrum. Meas. 2007;56:2057–66. https://doi.org/10.1109/ TIM.2007.904567.
• [29] G. Scebba, L. Tüshaus, W. Karlen, Multispectral camera fusion increases robustness of ROI detection for biosignal estimation with nearables in real-world scenarios, in: 2018 40th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2018, p. 5672-5.https://doi.org/10.1109/EMBC.2018.8513501.
• [30] Scebba G, Da Poian G, Karlen W. Multispectral video fusion for non-contact monitoring of respiratory rate and apnea. IEEE Trans. Biomed. Eng. 2021;68:350–9. https://doi.org/ 10.1109/TBME.1010.1109/TBME.2020.2993649.
• [31] Hu M-H, Zhai G-T, Li D, Fan Y-Z, Chen X-H, Yang X-K. Synergetic use of thermal and visible imaging techniques for contactless and unobtrusive breathing measurement. J. Biomed. Opt. 2017;22:1. https://doi.org/10.1117/1. JBO.22.3.036006.
• [32] Aguilera C, Barrera F, Lumbreras F, Sappa AD, Toledo R. Multispectral image feature points. Sensors 2012;12:12661–72. https://doi.org/10.3390/s120912661.
• [33] Mouats T, Aouf N, Sappa AD, Aguilera C, Toledo R. Multispectral stereo odometry. IEEE Trans. Intell. Transp. Syst. 2015;16:1210–24. https://doi.org/10.1109/ TITS.2014.2354731.
• [34] C.A. Aguilera, A.D. Sappa, R. Toledo, LGHD: A feature descriptor for matching across non-linear intensity variations, in: 2015 IEEE Int. Conf. Image Process., 2015, p. 178–81. https://doi.org/10.1109/ICIP.2015.7350783.
• [35] Ye Y, Shen L, Hao M, Wang J, Xu Z. Robust optical-to-SAR image matching based on shape properties. IEEE Geosci. Remote Sens. Lett. 2017;14:564–8. https://doi.org/10.1109/ LGRS.2017.2660067.
• [36] Li J, Hu Q, Ai M. RIFT: multi-modal image matching based on radiation-variation insensitive feature transform. IEEE Trans. Image Process. 2020;29:3296–310. https://doi.org/10.1109/ TIP.8310.1109/TIP.2019.2959244.
• [37] Hu M, Zhai G, Li D, Fan Y, Duan H, Zhu W, et al. Combination of near-infrared and thermal imaging techniques for the remote and simultaneous measurements of breathing and heart rates under sleep situation. PLoS ONE 2018;13. https:// doi.org/10.1371/journal.pone.0190466 e0190466.
• [38] L. Maurya, P. Mahapatra, D. Chawla, S. Verma, An automatic thermal and visible image registration using a calibration rig. Adv. Intell. Syst. Comput., vol. 1124, Springer; 2020, p. 67–76. https://doi.org/10.1007/978-981-15-2740-1_5.
• [39] Morrone MC, Owens RA. Feature detection from local energy. Pattern Recogn. Lett. 1987;6:303–13. https://doi.org/10.1016/ 0167-8655(87)90013-4.
• [40] Kovesi P. Phase congruency: A low-level image invariant. Psychol. Res. 2000;64:136–48. https://doi.org/10.1007/ s004260000024.
• [41] Horn BK. Robot Vision. 1st ed. McGraw-Hill Higher Education; 1986.
• [42] Rosten E, Porter R, Drummond T. Faster and better: A machine learning approach to corner detection. IEEE Trans. Pattern Anal. Mach. Intell. 2010;32:105–19. https://doi.org/ 10.1109/TPAMI.2008.275.
• [43] Wu Y, Ma W, Gong M, Su L, Jiao L. A novel point-matching algorithm based on fast sample consensus for image registration. IEEE Geosci. Remote Sens. Lett. 2015;12:43–7. https://doi.org/10.1109/LGRS.2014.2325970.
• [44] Zhang K, Zhang Z, Li Z, Qiao Y. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett. 2016;23:1499–503. https://doi.org/ 10.1109/LSP.2016.2603342.
• [45] Liu S, Liu D, Srivastava G, Połap D, Woźniak M. Overview and methods of correlation filter algorithms in object tracking. Complex Intell Syst 2020;1:3. https://doi.org/10.1007/s40747-020-00161-4.
• [46] M. Danelljan, G. Häger, F.S. Khan, M. Felsberg, Accurate scale estimation for robust visual tracking, in: BMVC 2014 - Proc. Br. Mach. Vis. Conf. 2014, British Machine Vision Association, BMVA; 2014. https://doi.org/10.5244/c.28.65.
• [47] Leys C, Ley C, Klein O, Bernard P, Licata L. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. J. Exp. Soc. Psychol. 2013;49:764–6. https://doi.org/10.1016/j.jesp.2013.03.013.
• [48] Rabiner L, Schafer R, Rader C. The chirp z-transform algorithm. IEEE Trans. Audio Electroacoust. 1969;17:86–92. https://doi.org/10.1109/TAU.1969.1162034.
• [49] Hsieh C-H, Chiu Y-F, Shen Y-H, Chu T-S, Huang Y-H. A UWB radar signal processing platform for real-time human respiratory feature extraction based on four-segment linear waveform model. IEEE Trans. Biomed. Circuits Syst. 2016;10:219–30. https://doi.org/10.1109/TBCAS.2014.2376956.
• [50] Go Direct Respiration Belt User Manual – Vernier n.d. https://www.vernier.com/manuals/gdx-rb/ (accessed April 14, 2021).
• [51] Maurer CR, Maciunas RJ, Fitzpatrick JM. Registration of head CT images to physical space using a weighted combination of points and surfaces [image-guided surgery]. IEEE Trans. Med. Imaging 1998;17:753–61. https://doi.org/10.1109/42.736031.
• [52] Chauvin R, Hamel M, Briere S, Ferland F, Grondin F, Letourneau D, et al. Contact-free respiration rate monitoring using a pan-tilt thermal camera for stationary bike telerehabilitation sessions. IEEE Syst. J. 2016;10:1046–55. https://doi.org/10.1109/JSYST.2014.2336372.
• [53] User’s manual FLIR Exx series n.d. https://www. flir.com/globalassets/imported-assets/document/flir-exxseries-user-manual.pdf (accessed April 14, 2021).

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2021).