Video-Based Physiological Parameters Measurement Technology and Research Advances

doi:10.3778/j.issn.1002-8331.2110-0050

Abstract

Abstract: The clinical monitoring of physiological parameters is of great significance for individual health management, disease prevention and tracking. Remote photoplethysmography（rPPG） is an emerging, camera-based, non-contact, fast and convenient technology for measurement of physiological parameters. The blood flow inside the microvascular vessels causes the slight changes in the color of the human skins. By amplifying and extracting this periodic change information, the physical parameters related to the function of the heart such as the heart rate and blood pressure can be analyzed. RPPG technology has developed rapidly since it was proposed in 2008, and its accuracy, robustness, and practicability have been greatly improved up to date. The rPPG detection process mainly starts with the acquisition of human facial data from the image sensor, and then completes the extraction of pulse waveforms and the calculation of related physical parameters based on image and signal processing technology. Starting from the basic detection process of rPPG, this article firstly introduces related hardware setup and processing algorithms, then presents its recent applications and finally the possible research directions of rPPG in the future are discussed.

Key words: remote photoplethysmography（rPPG）, non-contact measurement, physiological parameters, illumination, motion, deep learning

摘要： 生理体征参数的临床监测对于个体的健康管理、疾病的预防与跟踪具有重要的意义。远程光电容积描计法（remote photoplethysmography，rPPG）是一项新兴的、基于摄像头的、无接触方便快捷的生理体征参数测量技术。血管内部血液的流动造成人体表皮颜色的微弱变化，通过计算机对这一周期性变化的信息进行放大提取，便可以分析出与心脏运作相关的体征参数，如心率、血压等。RPPG自2008年提出之后得到了飞速的发展，其准确率、鲁棒性、实用性都得到了大幅度的提高。RPPG的检测流程主要是从图像传感器获取人体面部数据开始，随后基于图像及信号处理技术完成微弱脉冲波形的提取及相关体征参数的计算。从rPPG的基本检测流程出发，对rPPG的硬件采集设备进行介绍，重点描述从rPPG中提取心率的算法，进一步介绍rPPG相关的应用研究，就未来rPPG可能的研究方向进行了展望。

关键词: 远程光电容积描计法, 无接触测量, 生理体征参数, 光照, 运动, 深度学习

CHEN Jiatao, ZHANG Hongkai, HUANG Yanping, LAN Gongpu, XU Jingjiang, QIN Jia, AN Lin. Video-Based Physiological Parameters Measurement Technology and Research Advances[J]. Computer Engineering and Applications, 2022, 58(6): 58-68.

陈嘉涛, 张泓凯, 黄燕平, 蓝公仆, 许景江, 秦嘉, 安林. 基于视频的生理参数测量技术及研究进展[J]. 计算机工程与应用, 2022, 58(6): 58-68.

References

[1] VERKRUYSSE W，SVAASAND L O，NELSON J S.Remote plethysmographic imaging using ambient light[J].Optics Express，2008，16（26）：21434-21445.
[2] ROUAST P V，ADAM M T P，CHIONG R，et al.Remote heart rate measurement using low-cost RGB face video：a technical literature review[J].Frontiers of Computer Science，2018，12（5）：858-872.
[3] CHEN X，CHENG J，SONG R，et al.Video-based heart rate measurement：recent advances and future prospects[J].IEEE Transactions on Instrumentation and Measurement，2019，68（10）：3600-3615.
[4] POH M Z，MCDUFF D J，PICARD R W，et al.Non-contact，automated cardiac pulse measurements using video imaging and blind source separation[J].Optics Express，2010，18（10）：10762-10774.
[5] GIBSON K，AL-NAJI A，FLEET J，et al.Non-contact heart and respiratory rate monitoring of preterm infants based on a computer vision system：a method comparison study[J].Pediatric Research，2019，86（6）：738-741.
[6] HUANG P W，WU B J，WU B F.A heart rate monitoring framework for real-world drivers using remote photoplethysmography[J].IEEE Journal of Biomedical and Health Informatics，2021，25（5）：1397-1408.
[7] PARK S B，KIM G，BAEK H J，et al.Remote pulse rate measurement from near-infrared videos[J].IEEE Signal Processing Letters，2018，25（8）：1271-1275.
[8] WU B F，HUANG P W，CHUNG M L.Neural network based luminance variation resistant remote-photoplethysmography for driver’s heart rate monitoring[J].IEEE Access，2019，7：57210-57225.
[9] PARACCHINI M，MARCON M，VILLA F，et al.Biometric signals estimation using single photon camera and deep learning[J].Sensors，2020，20（21）：6102.
[10] YU Z，LI X，NIU X，et al.AutoHR：a strong end-to-end baseline for remote heart rate measurement with neural searching[J].IEEE Signal Processing Letters，2020，27：1245-1249.
[11] GHODRATIGOHAR M，GHANADIAN H，OSMAN H A.A remote respiration rate measurement method for non-stationary subjects using CEEMDAN and machine learning[J].IEEE Sensors Journal，2020，20（3）：1400-1410.
[12] VERKRUYSSE W，BARTULA M，BRESCH E，et al.Calibration of contactless pulse oximetry[J].Anesthesia & Analgesia，2017，124（1）：136-145.
[13] FAN X，YE Q，YANG X，et al.Robust blood pressure estimation using an RGB camera[J].Journal of Ambient Intelligence and Humanized Computing，2020，11（11）：4329-4336.
[14] VOLKOV M V，MARGARYANTS N B，POTEMKIN A V，et al.Video capillaroscopy clarifies mechanism of the photoplethysmographic waveform appearance[J].Scientific Reports，2017，7（1）：13298.
[15] VANDER KOOIJ K M，NABER M.An open-source remote heart rate imaging method with practical apparatus and algorithms[J].Behavior Research Methods，2019，51（5）：2106-2119.
[16] HU M，ZHAI G，LI D，et al.Combination of near-infrared and thermal imaging techniques for the remote and simultaneous measurements of breathing and heart rates under sleep situation[J].Plos One，2018，13（1）：e0190466.
[17] WANG W，DE BRINKER A C.Modified RGB cameras for infrared remote-PPG[J].IEEE Transactions on Biomedical Engineering，2020，67（10）：2893-2904.
[18] GAMBI E，AGOSTINELI A，BELLI A，et al.Heart rate detection using microsoft kinect：validation and comparison to wearable devices[J].Sensors，2017，17（8）：1776.
[19] MONNO Y，KIKUCHI S，TANAKA M，et al.A practical one-shot multispectral imaging system using a single image sensor[J].IEEE Transactions on Image Processing，2015，24（10）：3048-3059.
[20] ETHAN B B，JUSTIN R E.A multispectral testbed for cardiovascular sensing using imaging photoplethysmography[C]//Proceedings of SPIE.San Francisco，California，United States：SPIE，2017：100720R.
[21] NIU X，SHAN S，HAN H，et al.RhythmNet：end-to-end heart rate estimation from face via spatial-temporal representation[J].IEEE Transactions on Image Processing，2019，29：2409-2423.
[22] 韩静，吴俊林.CCD传感器工作原理及光强分布的测量[J].陕西师范大学学报（自然科学版），2006（S2）：44-45.
HAN J，WU J L.Principle of CCD and the measurement of luminous intensity distribution[J].Journal of Shaanxi Normal University（Natural Science Edition），2006（S2）：44-45.
[23] KWON S，KIM J，LEE D，et al.ROI analysis for remote photoplethysmography on facial video[C]//37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society，Milan，Italy，2015：4938-4941.
[24] BOCCIGNONE G，CONTE D，CUCULO V，et al.An open framework for remote-PPG methods and their assessment[J].IEEE Access，2020，8：216083-216103.
[25] LI P，BENEZETH Y，MACWAN R，et al.Video-based pulse rate variability measurement using periodic variance maximization and adaptive two-window peak detection[J].Sensors，2020，20（10）：2752.
[26] POH M Z，MCDUFF D J，PICARD R W.Advancements in noncontact，multiparameter physiological measurements using a webcam[J].IEEE Transactions on Biomedical Engineering，2011，58（1）：7-11.
[27] LEWANDOWSKA M,NOWAK J.Measuring pulse rate with a webcam[J].Journal of Medical Imaging & Health Informatics，2012，2（1）：87-92.
[28] CHEN D，WANG J，LIN K，et al.Image sensor-based heart rate evaluation from face reflectance using hilbert-huang transform[J].IEEE Sensors Journal，2015，15（1）：618-627.
[29] LEE D，KIM J，KWON S，et al.Heart rate estimation from facial photoplethysmography during dynamic illuminance changes[C]//Annual International Conference of the IEEE Engineering in Medicine and Biology Society，Milan，Italy，2015：2758-2761.
[30] CHENG J，CHEN X，XU L，et al.Illumination variation-resistant video-based heart rate measurement using joint blind source separation and ensemble empirical mode decomposition[J].IEEE Journal of Biomedical and Health Informatics，2016，21（5）：1422-1433.
[31] CHEN J，CHANG Z，QIU Q，et al.RealSense=real heart rate：Illumination invariant heart rate estimation from videos[C]//Sixth International Conference on Image Processing Theory，Tools and Applications（IPTA）.Oulu，Finland：IEEE Press，2016：1-6.
[32] XU L，CHENG J，CHEN X.Illumination variation interference suppression in remote PPG using PLS and MEMD[J].Electronics Letters，2017，53（4）：216-218.
[33] AL-NAJI A，PERERA A，HAHL J.Remote measurement of cardiopulmonary signal using an unmanned aerial vehicle[C]//AEROTECH VII-Sustainability in Aerospace Engineering and Technology.Putrajaya，Malaysia：IOP Press，2018：012001.
[34] CHENG J，WANG X，SONG R，et al.Exploring the feasibility of seamless remote heart rate measurement using multiple synchronized cameras[J].Multimedia Tools and Applications，2020，79（31/32）：23023-23043.
[35] RYU J，HONG S，LIANG S，et al.A measurement of illumination variation-resistant noncontact heart rate based on the combination of singular spectrum analysis and sub-band method[J].Computer Methods and Programs in Biomedicine，2020，200（10）：105824.
[36] LAURIE J，HIGGINS N，PEYNOT T，et al.Dedicated exposure control for remote photoplethysmography[J].IEEE Access，2020，8：116642-116652.
[37] TOUMI I，CALDARELLI S，TORRéSANI B.A review of blind source separation in NMR spectroscopy[J].Progress in Nuclear Magnetic Resonance Spectroscopy，2014，81：37-64.
[38] MACWAN R，BENEZETH Y，MANSOURI A.Heart rate estimation using remote photoplethysmography with multi-objective optimization[J].Biomedical Signal Processing and Control，2019，49：24-33.
[39] MACWAN R，BENEZETH Y，MANSOURI A.Remote photoplethysmography with constrained ICA using periodicity and chrominance constraints[J].Biomedical Engineering OnLine，2018，17（1）：1-22.
[40] LIU Y，QIN B，LI R，et al.Motion-robust multimodal heart rate estimation using BCG fused remote-PPG with deep facial ROI tracker and pose constrained Kalman filter[J].IEEE Transactions on Instrumentation and Measurement，2021，70：1-15.
[41] PO L M，FENG L，LI Y，et al.Block-based adaptive ROI for remote photoplethysmography[J].Multimedia Tools and Applications，2018，77（6）：6503-6529.
[42] BOBBIA S，MACWAN R，BENEZETH Y，et al.Unsupervised skin tissue segmentation for remote photoplethysmography[J].Pattern Recognition Letters，2019，124：82-90.
[43] BENEZETH Y，BOBBIA S，NAKAMURA K，et al.Probabilistic signal quality metric for reduced complexity unsupervised remote photoplethysmography[C]//13th International Symposium on Medical Information and Communication Technology，Oslo，Norway，2019：1-5.
[44] LAM A，KUNO Y.Robust heart rate measurement from video using select random patches[C]//IEEE International Conference on Computer Vision，Santiago，Chile，2015：3640-3648.
[45] KUMAR M，VEERARAGHAVAN A，SABHARWAL A.DistancePPG：robust non-contact vital signs monitoring using a camera[J].Biomedical Optics Express，2015，6（5）：1565-1588.
[46] YANG Y，LIU C，YU H，et al.Motion robust remote photoplethysmography in CIELab color space[J].Journal of Biomedical Optics，2016，21（11）：117001.
[47] DE HAAN G，JEANNE V.Robust pulse rate from chrominance-based rPPG[J].IEEE Transactions on Biomedical Engineering，2013，60（10）：2878-2886.
[48] RYU J，HONG S，LIANG S，et al.Research on the combination of color channels in heart rate measurement based on photoplethysmography imaging[J].Journal of Biomedical Optics，2021，26（2）：025003.
[49] CHO D，KIM J，LEE K J，et al.Reduction of motion artifacts from remote photoplethysmography using adaptive noise cancellation and modified HSI model[J].IEEE Access，2021，9：122655-122667.
[50] FENG L，PO L，XU X，et al.Motion-resistant remote imaging photoplethysmography based on the optical properties of skin[J].IEEE Transactions on Circuits and Systems for Video Technology，2015，25（5）：879-891.
[51] WANG W，DE BRINKER A，STUIJK S，et al.Algorithmic principles of remote PPG[J].IEEE Transactions on Biomedical Engineering，2017，64（7）：1479-1491.
[52] QI L，YU H，XU L，et al.Robust heart-rate estimation from facial videos using Project_ICA[J].Physiological Measurement，2019，40（8）：085007.
[53] KADO S，MONNO Y，YOSHIZAKI K，et al.Spatial-spectral-temporal fusion for remote heart rate estimation[J].IEEE Sensors Journal，2020，20（19）：11688-11697.
[54] FIN?GAR M，PODR?AJ P.A wavelet-based decomposition method for a robust extraction of pulse rate from video recordings[J].Peer J，2018，6：e5859.
[55] SONG R，LI J，WANG M，et al.Remote photoplethysmography with an EEMD-MCCA method robust against spatially uneven illuminations[J].IEEE Sensors Journal，2021，21（12）：13484-13494.
[56] ZHANG B，LI H，XU L，et al.Noncontact heart rate measurement using a webcam，based on joint blind source separation and a skin reflection model：for a wide range of imaging conditions[J].Journal of Sensors，2021：18.
[57] LI X，CHEN J，ZHAO G，et al.Remote heart rate measurement from face videos under realistic situations[C]//IEEE Conference on Computer Vision and Pattern Recognition.Ohio，USA：IEEE Press，2014：4264-4271.
[58] TARASSENKO L，VILLARROEL M，GUAZZI A，et al.Non-contact video-based vital sign monitoring using ambient light and auto-regressive models[J].Physiological Measurement，2014，35（5）：807.
[59] YIN R，JIA R，CUI Z，et al.Heart rate estimation based on face video under unstable illumination[J].Applied Intelligence，2021，51（8）：5388-5404.
[60] CHEN W，MCDUFF D.DeepPhys：video-based physiological measurement using convolutional attention networks[C]//European Conference on Computer Vision.Munich，Germany：Springer Press，2018：356-373.
[61] YU Z，PENG W，LI X，et al.Remote heart rate measurement from highly compressed facial videos：an end-to-end deep learning solution with video enhancement[C]//International Conference on Computer Vision.Seoul Korea：IEEE Press，2019：151-160.
[62] SONG R，ZHANG S，LI C，et al.Heart rate estimation from facial videos using a spatiotemporal representation with convolutional neural networks[J].IEEE Transactions on Instrumentation and Measurement，2020，69（10）：7411-7421.
[63] LIU S Q，YUEN P C.A general remote photoplethysmography estimator with spatiotemporal convolutional network[C]//2020 15th IEEE International Conference on Automatic Face and Gesture Recognition（FG 2020），2020：481-488.
[64] SHI J，ALIKHANI I，LI X，et al.Atrial fibrillation detection from face videos by fusing subtle sariations[J].IEEE Transactions on Circuits and Systems for Video Technology，2019，30（8）：2781-2795.
[65] SHAO D，LIU C，TSOW F，et al.Noncontact monitoring of blood oxygen saturation using camera and dual-wavelength imaging system[J].IEEE Transactions on Biomedical Engineering，2016，63（6）：1091-1098.
[66] DE ROSA A F G，BETINI R C.Noncontact SpO2 measurement using eulerian video magnification[J].IEEE Transactions on Instrumentation and Measurement，2020，69（5）：2120-2130.
[67] VANDERPOHL III，IRVIN J.Measuring multiple physiological parameters through blind signal processing of video parameters：14/202，567[P].2019.
[68] 许文媛，孟濬，赵夕朦.基于高速摄像机的动态血压非接触获取[J].浙江大学学报（工学版），2017，51（10）：2077-2083.
XU W Y，MENG R，ZHAO X M.Non-contact monitoring of ambulatory blood pressure based on high speed camera[J].Journal of Zhejiang University（Engineering Science），2017，51（10）：2077-2083.
[69] SHAO D，YANG Y，LIU C，et al.Noncontact monitoring breathing pattern，exhalation flow rate and pulse transit time[J].IEEE Transactions on Biomedical Engineering，2014，61（11）：2760-2767.
[70] FAN X，TJAHJADI T.Robust contactless pulse transit time estimation based on signal quality metric[J].Pattern Recognition Letters，2019，137：12-16.
[71] 李申龙，李毅彬，李洪阳，等.基于脉搏波相位差的无创连续血压测量方法[J].传感器与微系统，2016，35（1）：62-64.
LI S L，LI Y B，LI H Y，et al.Non-invasive continuous measurement method of blood pressure based on phase difference of pulse wave[J].Transducer and Microsystem Technologies，2016，35（1）：62-64.
[72] PEREIRA T，GADHOUMI K，MA M，et al.A supervised approach to robust photoplethysmography quality assessment[J].IEEE Journal of Biomedical and Health Informatics，2020，24（3）：649-657.
[73] ALGHOUL K，ALHARTHI S，OSMAN H A，et al.Heart rate variability extraction from videos signals：ICA vs EVM comparison[J].IEEE Access，2017，5：4711-4719.
[74] LIU Y，JOURABLOO A，LIU X.Learning deep models for face anti-spoofing：binary or auxiliary supervision[C]//Conference on Computer Vision and Pattern Recognition.Salt Lake City，USA：IEEE，2018：389-398.
[75] LI X，KOMULAINEN J，ZHAO G，et al.Generalized face anti-spoofing by detecting pulse from face videos[C]//23rd International Conference on Pattern Recognition.Cancún，Mexico：IEEE Press，2016：4244-4249.
[76] TSAI Y，LAI P，HUANG P，et al.Vision-based instant measurement system for driver fatigue monitoring[J].IEEE Access，2020，8：67342-67353.
[77] VILLARROEL M，CHAICHULEE S，JORGE J，et al.Non-contact physiological monitoring of preterm infants in the Neonatal Intensive care unit[J].NPJ Digital Medicine，2019，2：128.
[78] NAGY á，F?LDESY P，JáNOKI I，et al.Continuous camera-based premature-infant monitoring algorithms for NICU[J].Applied Sciences，2021，11（16）：7215.