2.1.

Principle of the 4DSCANNER

The projected sine pattern is modulated by the object in both amplitude (by the texture) and phase (by the shape) and its image is registered by the camera. For a maximum measurement frequency a single fringe image is analyzed during the reconstruction of the phase modulation. The phase recovery is based on a 7-point SCPS method¹⁹ which investigates 7 consecutive pixels to evaluate local phase at the central pixel. In the subsequent step, we perform phase unwrapping to eliminate the $2\pi$ ambiguity. We guide the spanning tree algorithm to favor pixels that have high contrast, a low variation of local fringes period, and are far from object’s border, in order to avoid erroneous phase evaluation on the boundaries of the object (i.e., surface discontinuities), in high curvature areas (nonconstant period breaks SCPS), as well as regions that have texture features (SCPS requires constant fringes amplitude).

Because the desired phase distribution consists of absolute values, the information of fringe numbers needs to be carried onto the object. A single fringe of known absolute phase is modulated transversely for its recognition on the image to be straightforward. A random pixel on the stripe is chosen as the initial point of the spanning tree method. During unwrapping, all phase values are shifted by the absolute phase value.

The phase distribution is further scaled in $xyz$ coordinates with use of camera calibration,²⁰ which is called a local calibration. In the local calibration procedure each camera pixel is assigned to a line in the global coordinate system and further on, a phase-to-depth monotonic distribution is evaluated for each of these lines. As a result, for each camera pixel there is a known mapping from absolute phase value to a position on its line, thus to a certain 3-D point. Using this information, the phase map is transformed into a single-directional cloud of points, which represents the surface visible to a single camera.

A multiview measurement is possible due to the global calibration. This is performed using the same calibration board that has been used for the local calibration. The common global frame is fixed for all cameras and all resulting clouds of points can be merged into a single multidirectional cloud.

Figures 1Fig. 2–3 present exemplary intermediate results for the evaluated 3-D surface of a single frame: the key steps in phase evaluation (Fig. 1), the fringe images of an object observed from various points of view (Fig. 2), and the output clouds of points (Fig. 3).

Fig. 1

Phase modulation evaluation: (a) input fringe image, (b) wrapped phase (7-point SCPS), (c) quality map for spanning tree phase unwrapping algorithm, and (d) unwrapped phase distribution.

Fig. 2

Four simultaneous views of the object. The measured surfaces are merged into a single model.

Fig. 3

Resulting cloud of points: (a) single-directional cloud of points and (b) a complete cloud of points consisting of four single-directional clouds (note different coloring of each cloud) in a common coordinate system.

2.2.

Hardware Configuration

In this study, we used two digital projectors (Casio XJ-A230V) and four industrial cameras (FL3-FW-03S1M-C from Point Grey Research). One projector illuminated the subject with a blue light from the front and the other used a red light from the back; a pair of cameras registered the illuminated frontal surface of the body and another pair registered the rear surface, as shown in Fig. 4(b). Cameras and projectors were rigidly fixed to each other and to the ground [see Fig. 4(a)], and then calibrated in the 2-m-long section of the walkway between them.

Fig. 4

Hardware setup for gait registration: (a) dual-camera measurement head (single projector and two detectors) and (b) side-view configuration schematic. Lower cameras register the higher part of calibrated volume and upper cameras register the lower part, allowing measurement of both feet and trunk.

Image acquisition was performed at an approximate 60 Hz frame-rate. All cameras were triggered externally by a single custom microcontroller synchronized with one of the projectors using a photodiode. The microcontroller fired acquisition in a manner that guaranteed both full exploitation of available illumination and isolation of opposite cameras (see Fig. 5). Cameras on the side of the red-illuminating projector were triggered together as soon as the negative slope of the blue illumination was registered and the cameras on the side of the blue illumination were triggered after an appropriate amount of time to capture the following blue illumination.

Fig. 5

The concept of projector-camera synchronization that enables the isolation of cameras. Both identical DLP projectors are fed with the same video signal, so they work synchronously. Cameras are triggered by the microcontroller at fixed time intervals relative to the negative slope of the blue illumination for a fixed time period (i.e., shutter time), thus working synchronously with associated projector.

2.3.

Measurement Protocol

In a modestly darkened laboratory, the 4DSCANNER was calibrated within a volume of $1.5\times 2.0\times 2.0{\mathrm{m}}^3$. Each of the four cameras was calibrated independently with the use of a $1.5\times 2.0{\mathrm{m}}^2$ calibration board. The common coordinate system was provided by placing the board in the center of the measurement volume so that it is visible to all the cameras.

For the sake of comparison with the optical motion capture technique, the 4DSCANNER was calibrated inside the calibrated volume of the VICON system. Both systems were adjusted to work with the same acquisition frequency. To achieve synchronization (i.e., temporal alignment) of the data, a single marker was dropped on the floor during each recorded sequence. During the processing stage, the frame of the impact was evaluated for both marker trajectories and fringe video. To achieve a spatial alignment, the 4DSCANNER was calibrated to have the same origin as the VICON system, so the GRF measured with the force plates were expressed in coordinates aligned with output data of both motion capture systems.

This study included three healthy subjects performing a comfortable-speed walking exercise (see Table 1). During the exercise, both VICON and 4DSCANNER were recording the movement simultaneously, as well as a pair of force plates that measured GRF. Optical markers were labeled on VICON recordings, and their positions were evaluated. From the 4DSCANNER fringe image sequences the animated representation of the object surface was calculated. Based on the surface data, an equivalent, synthetic set of markers (so-called virtual markers) was built. The markers provided by both systems are the lower extremities markers suggested by the VICON’s Plug-In Gait (PIG) analysis module.²¹ As presented in the Fig. 6, the markers are given as:

Table 1

General information of the subjects used for validation.

Fig. 6

Markers chosen for the validation, forming a sufficient constellation in terms of musculoskeletal (M-S) modeling.

2.4.

Kinematics Extraction

In the TLEMsafe project, the M-S modeling is performed in the AMS.³ The generic M-S model in AMS software is personalized based on 3-D trajectories of markers attached to body segments coupled with plots of GRF.

For the purpose of preliminary assessment of the applicability of the presented 4-D scanner for kinematics evaluation, a method for emulating virtual markers based on 4-D point cloud sequence was developed (Fig. 7). The virtual markerset complies with the PIG markerset and the intention is to estimate the positions of virtual markers in close proximity to the physical ones. The method developed for locating virtual markers is composed of the following steps that have to be performed for each time-frame of the sequence to be measured:

Fig. 7

Kinematics extraction. (a) Input point cloud (point colors—cyan, green, magenta, and yellow—represent cameras that acquired each point), (b) initial slicing along the vertical axis and segmentation, (c) point cloud with thigh and $\text{shank}+\text{foot}$ segments sliced along their midlines, (d) circles fitted to leg segments, (e) virtual markers (magenta) as well as thigh and shank midlines (black), and (f) virtual markers and segment midlines location with regard to the point cloud.

The analysis described above is applied to every frame in the sequence resulting in a virtual markerset that is estimated for every point cloud. The trajectories of virtual markers are then exported to a C3D file²³ and together with GRF data are fed into the AMS for biomechanical analysis.

2.5.

Biomechanical Analysis

Clinical validation of the measurement system is aimed to assess whether the kinematic information acquired by the 4DSCANNER is sufficient to build a M-S model with reasonable quality. The reference kinematic data were acquired by the current gold standard, i.e., the VICON system⁷ (VICON Nexus version 1.6.1.57351). A set of M-S models were fed with kinematics from both systems using the AMS³ (version 5.2). A comparison of these models and their outcome parameters is considered highly relevant since M-S models carry the full information of the motor performance of the subject.

2.5.1.

Musculoskeletal model

Our M-S model was based on the TLEM implemented in the AMS. The model consisted of nine body segments: upper body (head, arms, trunk, and pelvis), right and left femur, patella, tibia, and foot. The fibula was considered as one unit with the tibia. Eight joints were modeled: left and right hip, knee, patella/femur, and ankle joints. Hip joints were modeled as a ball-and-socket, defined by a rotation center and three orthogonal rotation axes. The knee and ankle joints were defined as a hinge, with a fixed rotation center and axis. The patella could rotate with respect to the femur around a rotation axis with a fixed rotation center. The patellar ligament was defined as a nondeformable element that connected the patella to the tibia. Thus, without introducing an extra degree of freedom, the orientation and position of the patella depended solely on the knee flexion angle. The orientation and position of the center of mass of the pelvis with respect to a 3-D global frame, together with the joint rotations of the hip, knee, and ankle joints, resulted in a model with 16 degrees of freedom (DOFs) and 10 joint axes (Fig. 8). Each leg contained 56 muscle-tendon parts represented by three-element, Hill-type muscle in series with a tendon.²⁴

Fig. 8

(a) M-S model based on the Twente Lower Extremity Model (TLEM). (b) Inverse dynamics simulation, based on movement tracked by 4DSCANNER virtual markers and ground reaction forces (GRF) recorded by force plates.

3.1.

Kinematics: Joint Angles

The predicted time histories for hip flexion and knee flexion angle showed comparable patterns for all the three subjects. The 4DSCANNER-based hip flexion angle was generally lower than VICON-based, with a mean absolute error varying from 0.0660 rad (3.78 deg) for Subject 2 to 0.1964 rad (11.25 deg) for Subject 3. The 4DSCANNER-based knee flexion showed a good agreement around 30% to 40% of the gait cycle (heel strike), otherwise it was generally lower than VICON-based, with a mean absolute error varying from 0.0689 rad (3.94 deg) for Subject 1 to 0.1043 rad (5.98 deg) for Subject 3. The 4DSCANNER-based ankle dorsiflexion angle showed a good agreement in the second half of the gait cycle (stance phase), while larger differences were present around 30% to 50% of the gait cycle (heel strike), with a maximum absolute error up to 0.2387 rad (13.68 deg) for Subject 1.

3.2.

Dynamics: Joint Moments

The 4DSCANNER-based hip flexion moment showed a good agreement during swing phase (0% to 30% of the gait cycle) and stance phase (50% to 100% of the gait cycle), but larger differences with the VICON-based moment were present at heel strike (30% to 40% of the gait cycle), with a maximum absolute error of up to 35 Nm for Subject 1.

Also, the 4DSCANNER-based knee flexion moment showed a good agreement during swing phase (0% to 30% of the gait cycle), however, then the relative difference with VICON-based moment increased during the stance phase (40% to 90% of the gait cycle), with a maximum absolute error of up to 33 Nm for Subject 3.

On the other hand, the 4DSCANNER-based ankle plantarflexion moment showed good agreement with VICON-based moment, with a mean absolute error ranging from 1.3 Nm (for Subject 1) to 7.8 Nm (for Subject 3).

3.3.

Muscle Activity

The 4DSCANNER-based muscle activity of iliacus, gluteus maximus, biceps femoris caput longum, and rectus femoris showed good agreement with the VICON-based muscle activity, with mean absolute errors lower than 7%. The only exception is given by the muscle activity of biceps femoris caput longum for Subject 3, where the mean absolute error is 14% and maximum absolute error is 59%.

Tibialis anterior showed larger differences and variability, with a maximum absolute error varying from 42% for Subject 1 up to 89% for Subject 3.

In addition, the 4DSCANNER-based muscle activity of soleus showed good agreement with the VICON-based activity, with a mean absolute errors lower than 3%, while the 4DSCANNER-based muscle activity of gastrocnemius showed unrealistic values (around 200%) for Subject 3, with maximum absolute error equal to 100%.

4.1.

Summary

The OGX|4DSCANNER is the recently developed system for capturing a motion based on a full-field 3-D surface imaging. In particular, it is capable of registering kinematics of whole lower human body during gait, and feed the TLEM M-S model with an outcome satisfactory for most joints. As a main conclusion of the described research, the 4DSCANNER could potentially be used in clinical gait analysis instead of optical marker-based VICON system. This is an essential advantage that it does not require any physical artifacts fixed to the body, and, as an optical method of measurement, it is absolutely noninvasive. In addition, the hardware used for the measurement is relatively inexpensive (unlike current commercial motion capture systems) because it mostly consists of off-the-shelf components.

4.2.

Limitations

Current implementation of the 4DSCANNER requires a significant darkening of the room due to short cameras exposure time. This may be a discomforting factor for patients, but should be overcome in the future with more powerful projectors or more sensitive detectors.

The main limitation of the 4DSCANNER applied to gait analysis considers capturing feet around the heel strike stage of gait. No camera used for the measurement can capture the foot in this stage due to high ankle joint dorsiflexion, resulting in self-occlusion of the top foot surface. This issue affects mostly ankle flexion angle and tibialis anterior activity. It can be addressed by employing an additional projector-camera couple to measure feet from a higher altitude.

Hip flexion angles predicted by the M-S models showed very similar patterns when comparing the 4DSCANNER and the VICON system, but presented also an offset that could probably be eliminated by improving evaluation of pelvic tilt. In this case, a better estimation of virtual front pelvis markers, LASI and RASI, or back pelvis markers, LPSI and RPSI, is required.

The largest difference between models based on both systems was found for predicted activity of gastrocnemius, a biarticular muscle that acts both on ankle plantar flexion and on knee flexion. The unrealistic predicted values of around 200% present in the 4DSCANNER-based results were caused by an overestimated knee flexion moment. This represents an issue that needs to be addressed in future.

It is also important to point out the image acquisition bandwidth limitation present during the 4DSCANNER measurement in the current configuration. The projector used in the study was displaying $120\text{frames}/\mathrm{s}$, so with the developed technique it enabled measurement frequency of 120 Hz. Using two workstations to handle four 1394b cameras at full acquisition speed of 120 Hz revealed a bandwidth shortage. We were forced to capture every second frame, but this problem can be overcome by employing additional workstations or higher-bandwidth interface cameras, e.g., an USB3.0 interface.

4.3.

Further Research

In the study, a set of virtual markers was evaluated in order to assess the method for gait analysis. This was a limited usage of the data it provides. Since the surface is registered as a cloud of thousands of points (with the potential increase in future), a “swarm” of virtual markers can be evaluated and fed into a more reliable M-S model. We expect models built upon 1 to 2 levels of magnitude more markers to show smaller variation across trials. Also, geometrical results carry volumetric information of the object, which could be exploited in future to enhance M-S models.

The measurement frequency and spatial resolution should be improved. A higher resolution will enable the extraction and tracking of curvature features and will improve the evaluation of the virtual markers. A higher frequency will enable the ability to track surface transitions through time without curvature features involved and move the method to more statistical ground, making use of the amount of data being registered.

4.4.

Conclusion

We conclude that the 4DSCANNER has the potential to replace optical maker-based systems in clinical gait analysis, provided that overall accuracy is improved, particularly around the foot area. The advantage of the 4DSCANNER over OMC solutions is that it does not burden patients with time-consuming marker application and this study demonstrates the versatility of this multidirectional 4-D structured light measurement technique.