2.1

Initial signal acquisition module

The initial signal acquisition module includes video acquisition time of N, video processing, face recognition, corner tracks, frame-by-frame interest area extraction, frame-by-frame interest area R, G, B channel average calculation, and removal of the first t seconds due to operating the mobile phone The shaking data is generated, and finally, we get the initial value signal.

The video acquisition duration is N means that the user captures a video with a duration of N and a frame rate of F, and uses the front camera of the mobile phone to take a picture of the face; the face is about 30 cm away from the mobile phone.

Video processing, to extract the required RGB three-channel data, an N-second video is decomposed into video frames whose total number is the product of N and F, and then a simple JPEG codec ffjpeg library is used to realize the conversion of video frames from NV21 to RGB24 code.

Face recognition uses face dynamic registration technology, which refers to face detection and face tracking. The face is tracked after the face is detected. It is to track the movement of the face instead of detecting it every frame. The face dynamic registration technology is mainly divided into two modules, the face detection module, and the face tracking module.

There are two main methods for ROI selection: the whole face area selection method and the partial area selection method. The local area selection has high computational complexity, there is a causal dilemma caused by the interdependence of skin detection and pulse waveform signal extraction, and the requirements for Spatio-temporal coherent local segmentation²³. We choose the whole face area selection method. The whole face area selection method refers to selecting the entire face or the face area except the eyes. For example, Li¹⁶ selects the face area except for the eyes. This method is the simplest ROI selection. The amount of calculation is relatively small.

The data of shaking caused by the operation of the mobile phone in the previous t seconds is removed because the error in the first two seconds is particularly large after calculation. The experimenter, with one arm tied to an Omron sphygmomanometer, and holding the phone in the other hand, when recording a video, you need to click the start button in the middle of the phone screen and find a comfortable angle that is 30 cm away from the phone and stay still at the beginning. An adaptation process is required. Checking the video recorded by the experimenter found that this adaptation process can be controlled within 2 seconds.

2.2

Moving window timing sampling

To reduce the root mean square error (RMSE), mean absolute error (MAE), and standard deviation (STD), the initial signal is intercepted by the piecewise function of different periods. For example, a 20-second video can get an 18-second initial signal, and because the experimenter’s hand-held mobile phone shakes more than a fixed camera. In this case, the longer the initial signal duration is, it is not conducive to the improvement of the accuracy of the heart rate value. The sampling time of the moving window sequence is too short, and the error of extracting the heart rate value is too large. This is that we sample T seconds through the moving window timing, and each time the moving window timing sampling is shifted to the right for 1 second, and these sampled signals are sent to further processing one by one, then we finally measure the heart rate value of N-T-t, that is, the temporary value of the heart rate H₁… H_N–T–t. Finally, the mean value of the temporary heart rate value is extracted as the heart rate value H.

2.3

Initial signal preprocessing module

The initial signal preprocessing module is divided into the initial signal segmented cubic Hermite interpolation and Butterworth low-pass filter.

Piecewise cubic Hermite interpolation. The signal is sampled with a sampling frequency of approximately twice the frame rate, and the segmented cubic Hermite interpolation method is used here.

Butterworth low-pass filter. After obtaining the PPG signal with the DC component removed, we need to digitally filter the PPG signal to remove low-frequency and high-frequency noise. 0.75-3.66 Hz. Because the frame rate of mobile phone recording is generally 25 or 30 frames per second, the sampling frequency is twice the frame rate. Through calculation and actual measurement, a sixth-order Butterworth low-pass filter with a normalized cut-off frequency of 0.04 is selected here. Because the Butterworth filter may cause extremely serious phase loss, we will adjust the phase of the signal after the Butterworth filter is processed. Realize zero-phase digital filtering by processing the input data forward and backward.

2.4

Blind source separation algorithm

In recent years, several core rPPG methods for extracting impulse signals from the video have been proposed. These include the following: blind source separation (BSS) (for example, principal component analysis)²⁴, based on independent component analysis (ICA)²⁵ and matrix completion²⁶), which use different standards to combine. The temporal RGB trajectories are separated into uncorrelated or independent signals to retrieve the source of the pulses; CHROM²⁷, which linearly combines the chrominance signals by assuming a standardized skin color to white balance the image; PBV²⁸, which uses blood at different wavelengths The feature of volume change clearly distinguishes the color change caused by the pulse and the motion noise in the RGB measurement; 2SR²⁹, which measures the temporal rotation of the subspace of the skin pixel space used for pulse extraction.

Blind source separation algorithms are still the mainstream. Blind source separation refers to the analysis of unobserved original signals from multiple observed mixed signals. Among many ICA algorithms, the fixed point algorithm also called Fast-ICA is widely used in the field of signal processing because of its fast convergence speed and good separation effect. The algorithm can well estimate the original signals that are statistically independent and mixed by unknown factors from the observed signals.

2.5

Heart rate calculation module

The heart rate calculation module includes fast Fourier transform, filter, and peak detection.

The first step is to perform a fast Fourier transform on the signal, and the second step is to filter the frequency signal with a bandpass filter of 0.75-3.66 Hz, and find the value with the largest power spectral density in the frequency range of the signal spectrum signal. The frequency corresponding to the number of points, and then multiply the frequency by 60 to get the experimenter’s heart rate value.

As for a normal person, the change of the heart rate value is not a sudden change process, so for the heart rate value determined in the adjacent time window, there will not be too much change, but there may be heart rate in the adjacent time window. A situation has changed too much. Considering this situation, we adopted moving window timing sampling to shorten the sample time for each calculation and measured multiple times to reduce the influence of changes in the heart rate value on the experimental results.

2.6

Extract heart rate value

Average the temporary heart rate H₁… H_N–T–t area and the final average of the temporary heart rate value is the heart rate value H.

4.1

Initial signal acquisition module experiment

Through the experimenter shooting a video of about 20 seconds, we proceed to the next processing of the video.

In this experiment, the skin detection partition ratio is 1, and the face detection frame is rectangular. The small white dots in the picture on the right are feature points. It can be seen that the edges of the face are well detected, and the portrait information can be extracted accurately to help us in the subsequent processing and application.

The human jitter or the brain is always slightly tilted, and the two eyes are not on the same horizontal line. At this time, we calculate the tilt angle of the two eyes and use a rotation function to rotate the rectangular frame of the eyes to completely cover the eyes area. As shown in the area of interest shown in Figure 3, it can be seen that the area of interest only contains the part of the human face without mixing background information.

Figure 3.

Figure with the area of interest. The right image is the result of region of interest extraction from the left image.

In the region of interest in Figure 3, the left image is the face recognition and feature value point tracking under the face dynamic registration technology just now, and the right image is the final obtained region of interest. If you look closely, you can see that the eye rectangle is not a horizontal rectangle. The eye rectangle has a slight, pixel-level tilt. It is very difficult for users to maintain an absolute level during use. In 20-second video shooting, tilt is inevitable. And we calculate the tilt angle and add the rotation matrix to further control the error and improve the accuracy.

The average value of the R, G, and B channels of the region of interest is calculated frame by frame, and the data that is shaken by the operation of the mobile phone in the first 2 seconds is removed, and finally, we obtain the initial value signal.

4.2

Moving window timing sampling experiment

We sample T seconds through the moving window time sequence, and each time the moving window time sequence sample is shifted to the right for 1 second, and these sampled signals are sent to further processing one by one, then we finally measure 18-T heart rate values, which are temporary heart rate values H₁… H_N–T-t, and finally, extract the mean value of the temporary heart rate value as the heart rate value H. The value of T is between 5 seconds and 16 seconds.

4.3

Initial signal preprocessing module experiment

Using the piecewise cubic Hermite interpolation is to divide the function domain interval into several intervals according to the piecewise interpolation method, and use the cubic Hermite interpolation on each interval.

A sixth-order Butterworth low-pass filter with a normalized cutoff frequency of 0.04 is selected. After filtering the data forward, the filtered sequence is inverted, and then it is passed through the filter again. Because the low-pass filter obtains the envelope of the signal, we will subtract this envelope from the signal to finally get a signal that reduces the effects of jitter and breathing, as shown in Figure 4.

Figure 4.

figure with before and after Butterworth filter processing.

4.4

Blind source separation algorithm experiment

As shown in Figure 5 before and after Fast-ICA processing, we can see the effect of the signal after the blind source separation algorithm is processed.

Figure 5.

figure with before and after Fast-ICA processing.

4.5

Heart rate calculation module experiment

Through the heart rate calculation module, after the fast Fourier transform, the frequency signal is filtered with a 0.75- 3.66Hz band-pass filter. After filtering the FFT signal, we can see the peak value. Finally, through peak detection, we extract the heart rate value.

4.6

Performance evaluation parameters

Root Mean Square (RMSE), Mean Absolute Error (MAE), and Standard Deviation (SD) are used to evaluate performance. Root Mean Square Error (RMSE): refers to the average square error between the output value and the true value. Mean Absolute Error (MAE): refers to the average error between the output value and the true value. Standard Deviation (SD): Refers to the arithmetic square root of the arithmetic mean of the squared deviation from the mean.

4.7.

Analysis of experimental results

Table 1 is error analysis of experimental data. N is 20 seconds of video recorded, the result after the initial signal is obtained after removing the data of 2 seconds before t is 2 seconds, where the value of T is 5 to 16 seconds, you can see the movement at 10 seconds. The window timing sampling effect is the best. Table 1 shows that the root means square error (RMSE), average absolute error (MAE), and standard deviation (STD) are the lowest, which are 6.6929, 5.1365, 6, 6165, respectively, compared with sampling without using a moving piecewise function. The errors are 13.53, 10.79, and 14.09.

Table 1.

Error analysis of experimental data.

Note: Errors are reduced by 50.5%, 52.4%, and 53.04%.