2.1.

Cap Preparation

The basic requirement of the method is good tracking of the cap-mounted head throughout the SfM process, which is facilitated by feature-rich captured objects. To enrich the cap-mounted head with features, we replace the solid-color connector sheets of the fNIRS cap with multiple two-color patterned plastic sheets [red and blue in the example in Fig. 1(a)]. We use Perlin noise²⁵ to generate distinct patterns. A script for generating this specific pattern is in the github.com/sagijaffedax/VideoRecon, but any distinct nontrivial color pattern will suffice, e.g., drawings or cartoons. In addition, we place six solid-color stickers [green in Fig. 1(a)] at specific positions on the cap: around the vertex, around the inion, between the vertex and the inion, and on the left, right, and front edges of the cap. These positions roughly correspond to Cz, Iz, Pz, T7, T8, and Fpz of the standard 10–20 system, respectively.

2.2.

Participant Preparation

We fit the cap on the subject and place three additional solid-color stickers [green in Fig. 1(c)] at specific anatomical positions on the participant fiducials: Nz, left preauricular, and right preauricular. In order to ensure other objects in the scene do not confuse our cropping method, we clear the surrounding of the participants from objects with similar color to the two-color plastic sheets and fiducial markers (e.g., no toys with similar colors, if the parent’s shirt had similar colors, we cover it with a dark barber cape, etc.). When the environment cannot be cleared of these objects (e.g., infant clothing or pacifier), we allow the experimenter to choose the color of the solid-color stickers to be distinct from the subject’s clothing, and manually mark the chosen color after the fact (see next steps).

With this visual information about the spatial positions of the cap relative to the participant’s head, our problem boils down to finding the spatial relation between the points on the head to those on the cap. Finding this relation is sufficient for exact positioning of the cap on the participant’s head.

2.3.

Video Capture

As discussed below, the core of our method is based on finding the 3-D positions of the sticker-marked fiducials using a computer vision algorithm. Since the 3-D relationships between the fiducials are required, it is imperative all of them are clearly visible in the captured video, and that the computer vision system’s tracking is not lost during video acquisition. Along with patient comfort, these are the driving concerns for the process of capturing the video. Hence, we suggest the following procedure to help ensure successful tracking of each fiducial. We record the cap placement on the subject’s head using a GoPro Hero 6 Black. We use the slow-motion (240 fps) feature to reduce motion blurring. Of course, any camera with a similar slow-motion option will suffice for the requirements of our method (e.g., iPhone 8 or higher). Furthermore, we attach a diffused flashlight to the camera to compensate for uneven lighting conditions across all head sides [see Fig. 1(b)]. Finally, we start and end the video recording at approximately the same position relative to the subject’s head. Note that depending on environmental conditions not all of, or even none of, the steps are necessary for success. These steps, however, have proven to be reliable and simple to implement. An example for such a video recording is available in Ref. 26.

2.4.

Cropping the Video Frames

Passing irrelevant information about the background of the given object-to-be-reconstructed to the SfM method often yields bad reconstructions, partial reconstructions, or falsely merged objects. These undesired reconstruction results are especially common in the presence of motion, i.e., when the object is moving during the video acquisition. Head movement and subsequent bad reconstruction are the major challenges that existing automated photogrammetry coregistration methods face with early developmental populations.

In order to avoid passing irrelevant information about the background to the SfM stage and to prevent the issues that arise from participants’ movements, we crop everything but the head and cap from each video frame. First, we maintain color consistency as much as possible, through white-balance correction of the video frames using OpenCV²⁷ package for MATLAB^®.²⁸ Then the user manually marks, from a single frame, the color of the stickers (green in Fig. 1) that were placed on the fiducial points. Manually selecting the color of the solid-color stickers allows flexibility regarding the choice of the used stickers—one could determine a fixed color for the fiducial stickers, which would render the last aforementioned step unnecessary. However, the sticker color should be distinctly different from any other color in the scene (see participant preparation stage).

After these steps, we use the following automatic processes to produce a cropping mask for each frame.

The rest of the frame is blackened [Fig. 1(e)], and frames in which more than 98% of the frame are blackened are rejected from further processing. These frames are either blurred frames due to fast motion of the camera or frames in which the participant’s head is out of frame.

2.5.

3-D Surface Reconstruction

In order to find the 3-D coordinates of the fiducial points, a 3-D model is automatically reconstructed. The cropped frames are passed to an SfM³² software (visual SfM³³). This widely used algorithm searches for shared features between frames (or correspondences³⁴) and estimates the camera position for each frame accordingly. Note that understanding this mechanism had enabled our key head-motion robustness contribution—when searching for correspondences between frames, a static background along with a moving head confuses the camera-position-estimation process. Hence, by removing the background in the cropping step, a moving head can be simply interpreted as additional camera movement, rendering accurate and successful reconstructions of our object of interest. We instruct the software to only search for correspondences that are less than two seconds apart. Searching for temporally distant correspondences is very computationally expensive and might yield faulty matches, since the video does not show the same parts for long periods. In addition, since the videos are taken such that they start and end the same way, matches are also searched for between the first and last frames of the video. This closed-loop helps in reducing accumulated drift and in cases where matches the tracking is lost midway through the video. A 3-D model can then be built [Fig. 1(f)] using the correspondences, camera positions, and simple geometric triangulations.

2.6.

Fiducial Extraction and Probe Positions Estimation

Our basic assumption in this part is that the cap undergoes limited deformation when worn by the participant (e.g., the cap does not stretch according to differences in head size as an EEG cap would). Instead, differences in head size, shape, cap placement, etc. are captured by the positions of the markers on the cap relative to the head (cap points and fiducial points, respectively, see below). After these points are extracted, we assume the cap can only rotate globally with respect to the head, and scale along the anterior-to-posterior and left-to-right axes. In this way, head size is implicitly considered in our reconstruction of probe positions. Therefore, capturing five points on the cap (along with the ones on the head) are sufficient for complete cap-head registration. The position of the probes can be then interpolated similarly to common photogrammetry-based methods.² In practice, we used six points for redundancy, with two in the difficult-to-capture back region (Pz and Iz), because infants are sitting on their parent’s lap.

From the reconstructed 3-D model, we identify the position of the fiducials by clustering nearby reconstructed vertices that hold the solid sticker color. Using RANSAC,³⁵ we determine the fiducial name associated with each aforementioned cluster (by looking at the expected relative position of all fiducials). In other words, we select three of the identified points and assign a labeling to them that is taken from the initial cap model. We then deform the head so that the three points match their counterparts and measure the distance of the rest of the markers from the ones they end up closest to. After choosing the labeling that minimizes the latter distances, we use the head fiducials (Nz, AR, AL, and Cz) to project the cap points (Cz, Pz, Iz, front, left, right of the cap; we are assuming Cz is relatively in the correct position) into MNI space by rotation and scaling.³⁶ The channel positions are then interpolated in MNI space based on the preregistered model cap relative to the cap points using “SPM for fNIRS” [Fig. 1(g)].³⁷ The interpolation is performed using barycentric coordinates: we represent each manually marked probe $p_j$ as a weighted sum of all fiducials $f_i,i=1\dots 9$. In other words, we find all $w_{i,j}$’s such that $p_j=\sum _{i=1}^9{w}_{i,j}·f_i$ in the preregistration step and simply use the same formula with the newly found fiducials to extract probe positions.

2.7.

3-D Digitizer Measurement

For comparison to our automated video-based method of 3-D reconstruction, we collected the 3-D coordinates of the head points and cap points using a 3-D digitizer (Fastrak, Polhemus, Colchester, VT) twice for each adult subject in each session. We then projected the channel positions into MNI space using the same predefined model cap and SPM for fNIRS.

2.8.

Estimation of Validity and Reliability of Channel Position Estimation

We compared the channel positions (139 channels; LABNIRS, Shimadzu inc., Kyoto, Japan) determined based on our new video-based method to the positions found using the dominant method in adult participants, the 3-D digitizer (Fastrak, Polhemus, Colchester, VT). We did the comparison twice for each subject ($N-10$) for the same session and compared the positions of the same channels both between the two methods (intermethod validity) and within each one (intramethod reliability). The intramethod comparison was used to estimate the error of each method independently (test–retest reliability).

In addition, we measured the channel position for a separate group of adult subjects ($N-10$) in two separate sessions (separate days, multiple cap placements) to estimate whether the video-based method captured the shifts in cap positioning on the same subject as compared to the 3-D digitizer.

2.9.

Feasibility for Early Developmental Populations

Finally, we show how this method can be used to estimate probe positions with early developmental populations. Specifically, we present results from a 6-month-old infant. The example video is available here.³⁸ Although it would be ideal to compare our video-based reconstruction methods as conducted in infants to other methods, we cannot conduct the same comparison as adults because it is not practically possible to collect the digitized positions of the fiducials and probes using the 3-D digitizer. After months of effort on this front and communication with other developmental cognitive neuroscience labs, we reached the conclusion that this method is too sensitive to motion to be reliable, and thus, is unusable for this population.

3.1.

Video-Based Method, Validated through the Standard 3-D Digitizer

We found that the validity of our new video-based method was highly comparable to the field’s standard method—the 3-D digitizer [Figs. 2(a) and 2(b) right]. Namely, the distances between the positions of the same channel as estimated by the two methods were similar, albeit slightly larger, to the distances between the positions of the same channel as measured by the 3-D digitizer twice (intermethod validity of $3.4\pm 0.9\mathrm{mm}$ compared to intramethod reliability of $2.6\pm 0.6\mathrm{mm}$; $\text{mean}\pm \mathrm{STD}$; $t_9=3.1$, $P<0.05$). Thus, there is a high correspondence between the channel positions estimated by our video-based methods and the 3-D digitizer.

Fig. 2

Spatial distribution of intermethod validity and intramethod reliability, comparing our method and the digitizer. (a) Distance from our (video) method to the 3-D digitizer one (intermethod validation; $3.4\pm 0.9\mathrm{mm}$). (b) Distance between two measurements of the 3-D digitizer (intramethod reliability; $2.6\pm 0.6\mathrm{mm}$). (c) Distance between two measurements of video-based method (intramethod reliability; $2.0\pm 0.5\mathrm{mm}$). Distances are represented by the color and the diameter of the patches for each channel position.

Moreover, the reliability of our video-based method was better than the reliability of the 3-D digitizer. The distances between the estimated positions of the same channel as measured twice by our video-based method [$2.0\pm 0.5\mathrm{mm}$; Fig. 2(c) right] was smaller than that of the 3-D digitizer ($t_9=2.6$, $P<0.05$; paired $t$-test). Overall, the magnitude of the errors that we found is much smaller than the radius of the channels, whose positions were estimated (typically on the order of centimeters).

Although the discrepancies between the two position estimation methods were spread evenly through the scalp, we did observe a tendency for larger distances in the anterior temporal channels [Fig. 2(a) left and middle]. This increased error at this position stems from smaller reliability of both methods in these areas [Figs. 2(b) and 2(c) left and middle]. The lower reliability of both methods in the anterior temporal lobe is probably a result of the low angle of camera or digitizer that is required to acquire the position of the fiducials in this region.

3.2.

Video-Based Method Captures Cap Positioning Variability

We now considered how well these two localization methods identify any change in position of the cap (i.e., shift or displacement of each channel) for the same participant across sessions. We found a significant regression coefficient for the size of the estimated shift in channel position (in mm) between the two methods [red line in Fig. 3(a); $F_{\mathrm{1,1388}}=5.86$ $p<0.05$; linear mixed effect model]. Namely, the shift size as estimated by the video-base method predicts the shift size as estimated by the 3-D digitizer method. As Fig. 3(a) clearly shows, there were many cases where the two estimation methods did not agree on the shift size between the two sessions [e.g., a high shift size was estimated by the 3-D digitizer and a small shift size was estimated by the video-based method for the points on the right-hand side; channels below the diagonal—black dashed line in Fig. 3(a)]. We manually examined a few cases of large discrepancies between the shift estimation of the two methods. In most cases, the video-based method was actually closer to estimating the shift magnitude correctly. For example, in Fig. 3(c), the shift between the sessions of channel 66 (circled in yellow) for participant 1 was estimated as 3.6 mm by the video-based method and 24.5 mm by the 3-D digitizer. Our manual estimation, based on image-space measurements scaled by known distances between channels, suggests a shift of 7 mm—a number much closer to the video-based estimation than to the 3-D digitizer-based one [Fig. 3(c)].

Fig. 3

Comparison of estimation of cap shifting between session through the video-based and the 3-D digitizer methods. (a) Estimation of shift size between the sessions of the video-based method versus the 3-D digitizer one. The color of each hexagon represents the number of channels in its range. We found a high correlation between the shifts as estimated by the two methods ($F_{\mathrm{1,1368}}=21.8$ $p<{10}^{-5}$; red line). (b) Angles between the shift as estimated by the video-based method and the shift as estimated by the 3-D digitizer method versus the size of the shift between sessions. The distribution of angles in skewed toward zero ($K=0.3$, $p<{10}^{-25}$; distribution along the ordinate). We found a negative correlation between the size of the angle between estimated shifts and the size of these shifts. When the estimated shifts are larger, the direction of the shifts as estimated by the two methods were similar (smaller angle). (c) Manual estimation of a case of large discrepancy between the two estimation methods. Left: photo of participant 1 in session 1. The distance of channel 66 (circled in yellow) from the left tragus (lower green sticker) was 49.3 mm (the length of the red arrow). Right: photo of participant 1 in session 2. The distance between channel 66 and the left tragus was 53.3 mm. The shift of the channel position between the two sessions was $\sim 4\mathrm{mm}$ and was closer to the video-based estimation (3.6 mm) than to the digitizer-based estimation (24.5 mm).

We also found that the shifts in channel positions were estimated in similar directions using the two methods. We measured the spatial angle between the direction of the shifts as estimated by the video method and the direction of the shift as estimated by the 3-D digitizer. The estimated shift of each channel between sessions can be represented as a line between the two 3-D estimated positions. Each method thus estimates such a line and we can measure the level of agreement between the methods by measuring the angle between these lines. The angle between the direction of the shifts, as estimated by the two methods, was small [closer to zero than a uniform distribution; distribution along the ordinate in Fig. 3(b); $K=0.3$, $p<{10}^{-25}$; Kolmogorov–Smirnov test; $\text{mean angle}\pm \mathrm{STD}$: $60.6\mathrm{deg}\pm 50.1\mathrm{deg}$]. Crucially, the similarity in shift direction was found to be greater in the larger shifts [red line in Fig. 3(a); $F_{\mathrm{1,1368}}=21.8$ $p<{10}^{-5}$; linear mixed effect model], where this similarity in direction matters most (i.e., when the cap shifts substantially and may potentially result in a change in the cortical region that the channel is assigned to).

Investigating these shifts in cap placement across sessions reveals how crucial it is for any coregistration to be accurate. Even if recordings of the same subject take place with the same cap mere days apart, these shifts in cap placement are large enough to change the cortical region under a channel. In our sample, more than 8% of the channels were shifted to a different lobe between sessions.

Additionally, our automatic method does not rely on the specific head size or model it specifically. Instead, it automatically adjusts for the head size by measuring the distance between the fiducial points and the cap directly, taking the head size implicitly into account (see Sec. 2.6). Thus, we expect no correlation between the error magnitude and head sizes. Unfortunately, we did not take independent measurements of our subjects’ head sizes (e.g., circumference, inion to Nz distance, etc.). However, we qualitatively compare the errors that were obtained for a distinctly larger head with the errors that were obtained for a smaller one. We did not find any systematic difference (Fig. 4). We also find in the visual comparison between these two head sizes that the probes are shifted down on the smaller head and up on the larger head reflecting the differences in the relative size of the cap to the participant’s head size. This logical transformation reflects the fact that head size is accommodated by our method through differences in the fiducial points and cap points and does not need to be included explicitly.

Fig. 4

Comparison of two subjects with relatively distinct head sizes. The subject with the smaller head (a) had a mean error of 2.7 mm. The subject with the larger head (b) had a mean error of 2.3 mm. Due to the rigidity of the fNIRS cap, the probe positions on the smaller head (c) were shifted slightly downward relative to the larger head (d). For example, channel 45 (circled in yellow) was estimated in STG on the smaller head size, but in postcentral gyrus on the larger head size.

3.3.

Automatic Estimation of Channel Position in Infants

We successfully reconstructed a 3-D model from video recordings of infant participants and estimated the channel positions on their scalp through our automated video-based method (using methods described above, Fig. 5 for one sample infant). Thus, here we present a proof of concept that the same method can be used for early developmental populations. In practice, we have employed this method for many dozens of infants at different ages. Since there can be no comparison with a 3-D digitizer, we focus this paper of the validity of the overall method in adult participants in comparison to established methods.

As is apparent in this example, the cap was slightly misplaced on the infant, tilting to the right [Fig. 5(a)]. Our probe position estimation method successfully estimated this misplacement [Fig. 5(d)].

Fig. 5

Infant channel position estimation using the video-based method. (a) The captured video. (b) The images were cropped to eliminate confounds of movements. (c) A 3-D model was reconstructed using visual SfM. Coordinates of fiducial points were extracted from the model. (d) Channel positions were interpolated from fiducial coordinates onto MNI space.