We describe OpenSpyrit, an open source ecosystem for reproducible research in hyperspectral single-pixel imaging. We outline the three elements of OpenSpyrit, which include a Python software for acquiring data from a single pixel, a collection of single-pixel hyperspectral images (SPIHIM), and a Python toolkit for reconstructing single-pixel images (SPYRIT). In particular, we examine the unrolled algorithms and the plug-and-play methods available in SPYRIT that we evaluate on various datasets of the SPIHIM collection.
We present a computational approach for hyperspectral computational Selective Plane Illumination Microscopy (SPIM), offering fast 3D imaging with reduced photobleaching. Inspired by Hadamard spectroscopy, our method employs structured light sheets via a digital micromirror device. A data-driven reconstruction strategy, implemented through an end-to-end trained neural network, demonstrates robust performance under varying noise levels. Leveraging non-negative least squares minimization, we obtain component maps, exemplifying applications such as autofluorescence removal in transgenic zebrafish and discrimination of closely matched red proteins. Our findings showcase the potential of computational strategies to advance hyperspectral SPIM in photonic research.
Spectral imaging is at the cornerstone of many fields, including astronomy, environmental monitoring, food processing, agriculture, and biomedical imaging. While most current spectral techniques lack imaging speed, we describe a computational approach that allows for fast high spectral resolution imaging.
Our set-up maps the image of the scene into the fiber of a compact spectrometer through a digital micromirror device (DMD), where a series of multiplexing patterns are uploaded. After a reconstruction step, the hypercube of the scene can be recovered. DMDs represent the fastest technology (e.g., >20 kHz) for spatial light modulators. The raw data is acquired for a (x,y,λ) hypercube of size 64x64x2048 in about 12 s, while the Hadamard inversion takes <1 s. The acquisition speed can be further reduced by limiting the number of Hadamard patterns; however, the resulting imaging reconstruction problem is turned into an underdetermined inverse problem, which requires regularization techniques to be used to obtain acceptable solutions, in particular, in the presence of noise.
Deep learning is a very efficient framework to solve inverse problems in imaging. Following a recent trend, several convolution neural network architectures have provided a link between deep and optimization-based image reconstruction methods. Contrary to the initially proposed “black box” networks, these deep-learning methods rely on a forward operator and lead to more interpretable networks. Here, we review deep architectures for single-pixel image reconstruction and show that the network can be trained easily, in a end-to-end manner, using databases such as STL-10 or ImageNet.
We present reconstruction results from simulated and experimental single-pixel acquisitions. We show that EM-Net generalizes very well to noise levels that are unseen during the training, despite having fewer learned parameters than alternative methods. The proposed EM-Net generally reconstructs images with fewer artifacts and with higher signal-to-noise ratios, particularly in high-noise situations.
Optical imaging has become an invaluable tool in life science. Among the variety of available techniques, selective plane illumination microscopy (SPIM) allows for fast (x,y,z) imaging of fluorescent samples with reduced photobleaching. SPIM directly acquire the (x,y) slice corresponding to a thin light sheet that illuminates the sample, while the third spatial dimension is scanned. Promoted by the open source SPIM project, many designs variants are now available. This enables the study of various samples such as fly embryos, zebrafish embryos and others.
The study of multi-labeled specimens implies to unmix the fluorophores, which usually relies on optical filters. As most of the light is rejected, this approach has a major drawback, as a large amount of information is lost (e.g., fluorophores with overlapping spectra cannot be unmixed). Therefore, there is a need for 3D imagers with hyperspectral capabilities, which can exploit the full-emission spectrum of a fluorescent sample.
We will describe a computational hyperspectral light sheet microscope inspired from Hadamard spectroscopy. We generate structured light sheets using a digital micromirror device and focus the fluorescence signal onto the entrance slit of an imaging spectrometer. Then, we reconstruct the full hypercube from the raw data acquired for multiple structured light patterns. Our technique enjoys excellent spectral resolution and allows resolving overlapping fluorophores with up to nanometer resolution. Furthermore, the Hadamard patterns used for illumination allow maximizing the collected signal compared to previous hyperspectral SPIM setups. To reduce the acquisition time, we consider undersampled measurements for which we will present reconstruction results obtained using an algorithm based on a deep convolutional network.
Single-pixel imaging is a paradigm that enables the capture of an image from a single point detector using a spatial light modulator. This approach is particularly interesting for optical set-ups where pixelated arrays of detectors are either too expensive or too cumbersome (e.g., multispectral, infrared imaging). It acquires the inner product between the image of the scene and a set of user-defined patterns that are sequentially uploaded onto the spatial light modulator. Compressed data acquisition reduces the acquisition time, although it leads to an ill-posed reconstruction problem, which is very challenging for real-time applications. Recently, neural networks have emerged as competitive alternatives to traditional reconstruction methods. Neural networks are parametric models that are trained by exploiting large datasets. Their noniterative nature allows for fast reconstructions, which opens the door to real-time image reconstruction from compressed acquisition. In this study, we evaluate the different networks for static and dynamic imaging. In particular, we introduce a recurrent neural network that is designed to exploit the spatiotemporal redundancy in videos via a memory state. We validate our algorithms on simulated data from the UCF-101 dataset, with a resolution of 128x128 pixels and a compression ratio of 98%. We also show experimentally that we can resolve small spectral differences in the spectrum of human skin measured in vivo.
We present the effects of using a single-pixel camera approach to extract optical properties with the single-snapshot spatial frequency-domain imaging method. We acquired images of a human hand for spatial frequencies ranging from 0.1 to 0.4 mm − 1 with increasing compression ratios using adaptive basis scan wavelet prediction strategy. In summary, our findings indicate that the extracted optical properties remained usable up to 99% of compression rate at a spatial frequency of 0.2 mm − 1 with errors of 5% in reduced scattering and 10% in absorption.
Pattern generalization was proposed recently as an avenue to increase the acquisition speed of single-pixel imaging setups. This approach consists of designing some positive patterns that reproduce the target patterns with negative values through linear combinations. This avoids the typical burden of acquiring the positive and negative parts of each of the target patterns, which doubles the acquisition time. In this study, we consider the generalization of the Daubechies wavelet patterns and compare images reconstructed using our approach and using the regular splitting approach. Overall, the reduction in the number of illumination patterns should facilitate the implementation of compressive hyperspectral lifetime imaging for fluorescence-guided surgery.
Recent progress in X-ray CT is contributing to the advent of new clinical applications. A common challenge for these applications is the need for new image reconstruction methods that meet tight constraints in radiation dose and geometrical limitations in the acquisition. The recent developments in sparse reconstruction methods provide a framework that permits obtaining good quality images from drastically reduced signal-to-noise-ratio and limited-view data. In this work, we present our contributions in this field. For dynamic studies (3D+Time), we explored the possibility of extending the exploitation of sparsity to the temporal dimension: a temporal operator based on modelling motion between consecutive temporal points in gated-CT and based on experimental time curves in contrast-enhanced CT. In these cases, we also exploited sparsity by using a prior image estimated from the complete acquired dataset and assessed the effect on image quality of using different sparsity operators. For limited-view CT, we evaluated total-variation regularization in different simulated limited-data scenarios from a real small animal acquisition with a cone-beam microCT scanner, considering different angular span and number of projections. For other emerging imaging modalities, such as spectral CT, the image reconstruction problem is nonlinear, so we explored new efficient approaches to exploit sparsity for multi-energy CT data. In conclusion, we review our approaches to challenging CT data reconstruction problems and show results that support the feasibility for new clinical applications.
While standard computed tomography (CT) data do not depend on energy, spectral computed tomography (SPCT) acquire energy-resolved data, which allows material decomposition of the object of interest. Decompo- sitions in the projection domain allow creating projection mass density (PMD) per materials. From decomposed projections, a tomographic reconstruction creates 3D material density volume. The decomposition is made pos- sible by minimizing a cost function. The variational approach is preferred since this is an ill-posed non-linear inverse problem. Moreover, noise plays a critical role when decomposing data. That is why in this paper, a new data fidelity term is used to take into account of the photonic noise. In this work two data fidelity terms were investigated: a weighted least squares (WLS) term, adapted to Gaussian noise, and the Kullback-Leibler distance (KL), adapted to Poisson noise. A regularized Gauss-Newton algorithm minimizes the cost function iteratively. Both methods decompose materials from a numerical phantom of a mouse. Soft tissues and bones are decomposed in the projection domain; then a tomographic reconstruction creates a 3D material density volume for each material. Comparing relative errors, KL is shown to outperform WLS for low photon counts, in 2D and 3D. This new method could be of particular interest when low-dose acquisitions are performed.
A time-resolved Diffuse Optical Tomography system based on multiple view
acquisition, pulsed structured light illumination and detection with spatial compression is
proposed. Reconstructions on heterogeneous tissue mimicking phantoms are presented.
Diffuse Optical Tomography (DOT) can be described as a highly multidimensional problem generating a huge data set with long acquisition/computational times. Biological tissue behaves as a low pass filter in the spatial frequency domain, hence compressive sensing approaches, based on both patterned illumination and detection, are useful to reduce the data set while preserving the information content. In this work, a multiple-view time-domain compressed sensing DOT system is presented and experimentally validated on non-planar tissue-mimicking phantoms containing absorbing inclusions.
Spectral computed tomography (CT) exploits the measurements obtained by a photon counting detector to reconstruct the chemical composition of an object. In particular, spectral CT has shown a very good ability to image K-edge contrast agent. Spectral CT is an inverse problem that can be addressed solving two subproblems, namely the basis material decomposition (BMD) problem and the tomographic reconstruction problem. In this work, we focus on the BMD problem, which is ill-posed and nonlinear. The BDM problem is classically either linearized, which enables reconstruction based on compressed sensing methods, or nonlinearly solved with no explicit regularization scheme. In a previous communication, we proposed a nonlinear regularized Gauss-Newton (GN) algorithm.1 However, this algorithm can only be applied to convex regularization functionals. In particular, the ℓp (p < 1) norm or the `0 quasi-norm, which are known to provider sparse solutions, cannot be considered. In order to better promote the sparsity of contrast agent images, we propose a nonlinear reconstruction framework that can handle nonconvex regularization terms. In particular, the ℓ1/ℓ2 norm ratio is considered.2 The problem is solved iteratively using the block variable metric forward-backward (BVMF-B) algorithm,3 which can also enforce the positivity of the material images. The proposed method is validated on numerical data simulated in a thorax phantom made of soft tissue, bone and gadolinium, which is scanned with a 90-kV x-ray tube and a 3-bin photon counting detector.
We propose to couple a single-pixel camera with a photon counting board in order to obtain an inexpensive time-resolved imaging system having a high spatial and temporal resolution. As an alternative to compressive sensing, a wavelet-based adaptive acquisition strategy is employed which allows high compression rates with little degradation of the image quality. The applicability of our approach is demonstrated for fluorescence lifetime imaging. Experimental results obtained by imaging samples embedding several fluorophores are provided. The proposed imaging system with the wavelet-based strategy can be suitable for a microscope in order to perform fluorescence lifetime imaging microscopy measurements.
Diffuse Optical Tomography (DOT) and Fluorescence Molecular Tomography (FMT) generally require a huge data set which poses severe limits to acquisition and computational time, especially with a multidimensional data set. The highly scattering behavior of biological tissue leads to a low bandwidth of the information spatial distribution and hence the sampling can be preferably carried out in the spatial frequency source/detector space. In this work, a time-resolved single pixel camera scheme combined with structured light illumination is presented and experimentally validated on phantoms measurements. This approach leads to a significant reduction of the data set while preserving the information content.
Fluorescence molecular tomography (FMT) is quite demanding in terms of acquisition/computational times due to the huge amount of data. Different research groups have proposed compression approaches regarding both illumination (wide field structured light instead of raster point scanning) and detection (compression of the acquired images). The authors have previously proposed a fast FMT reconstruction method based on the combination of a multiple-view approach with a full compression scheme. This method had been successfully tested on a cylindrical phantom and is being generalized in this paper to samples of arbitrary shape. The devised procedure and algorithms have been tested on an ex-vivo mouse.
In recent years, an increasing effort has been devoted to the optimization of acquisition and reconstruction schemes for fluorescence molecular tomography (FMT). In particular, wide-field structured illumination and compression of the measured images have enabled significant reduction of the data set and, consequently, a decrease in both acquisition and processing times. FMT based on this concept has been recently demonstrated on a cylindrical phantom with a rotating-view scheme that significantly increases the reconstruction quality. In this work, we generalize the rotating-view scheme to arbitrary geometries and experimentally demonstrate its applicability to murine models. To the best of our knowledge this is the first time that FMT based on a rotating-view scheme with structured illumination and image compression has been applied to animals.
Diffuse optical tomography (DOT) and Fluorescence mediated tomography
(FMT) are powerful in-vivo optical imaging techniques but they are affected
by long acquisition and computational times. Recently, the use of structured light has
been proposed in order to reduce the acquisition time and also the computational time
of the inverse problem. Additionally, it has been proposed to compress the measured
data set to reduce the reconstruction time. Here we present our experimental approach,
describing the instrument for structured illumination and wide field detection and we
discuss the advantages to use a finite elements based approach. Then, we introduce
the use of spatial wavelets. Our method is based on the projection of a small number
of wavelet patterns (Haar and Battle-Lemarie wavelets). The detected images are
wavelet transformed and the information content is compressed to achieve fast 3D
reconstruction. Experimental results are presented, showing fast reconstruction of
complex absorbing/fluorescent objects in thick diffusive samples. Implications for fast
small animal imaging are discussed.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.