This contribution proposes a novel approach for image fusion of combined stereo and spectral series acquired
simultaneously with a camera array. To this purpose, nine cameras are equipped with spectral filters (50 nm
spectral bandwidth) such that the visible and near infrared parts of the spectrum (400-900 nm) are observed.
The resulting image series is fused in order to obtain two types of information: the 3D shape of the scene and
its spectral properties.
For the registration of the images, a novel region based registration approach which evaluates the gray
value invariant features (e.g. edges) of regions in segmented images is proposed. The registration problem is
formulated by means of energy functionals. The data term of our functional compares features of a region in
one image with features of an area in another image, such that an additional independency of the form and
size of the regions in the segmented images is obtained. As regularization, a smoothness term is proposed,
which models the fact that disparity discontinuities should only occur at edges in the images. In order to
minimize the energy functional, we use graph cuts. The minimization is carried out simultaneously over all
image pairs in the series.
Even though the approach is region based, a label (e.g. disparity) is assigned to each pixel. The result of
the minimization approach consists of a disparity map. By means of calibration, we use the disparity map to
compute a depth map. Once pixel depths are determined, the images can be warped to a common view, such
that a pure spectral series is obtained. This can be used to classify different materials of the objects in the
scene based on real spectral information, which cannot be acquired with a common RGB camera.
This contribution presents a fusion method for spectral series with the main purpose of obtaining 3D
information. The image series to be fused are combined stereo and spectral series gained with a camera
array. Therefore, in order to register them, features that are invariant with respect to the varying
gray values in the spectral images are extracted. The proposed approach is region based and uses
characteristics like size, position and shape for registration. The regions are identified using the watershed
transformation. The fusion problem is modeled using energy functionals that are to be optimized. They
take into consideration the size, position, shape and correlation of the regions. Using the implemented
algorithm, several scenes have been reconstructed. The experimental results show that the proposed
method delivers reliable and accurate dense depth maps of combined stereo and spectral series.