This PDF file contains the front matter associated with SPIE Proceedings Volume 10937, including the Title Page, Copyright information, Table of Contents, Introduction and Author and Conference Committee lists

In the first part of this presentation, we will discuss recently emerging applications of the state-of-art deep learning methods on optical microscopy and microscopic image reconstruction, which enable image enhancement and new transformations among different modalities of microscopic imaging, driven entirely by image data. In this second part, we introduce a physical mechanism to perform machine learning by demonstrating a Diffractive Deep Neural Network architecture that can all-optically implement various functions following the deep learning-based design of passive layers that work collectively. We created 3D-printed diffractive networks that implement classification of images of handwritten digits and fashion products as well as the function of an imaging lens at terahertz spectrum. This passive diffractive network can perform, at the speed of light, various complex functions that computer-based neural networks can implement, and will find applications in all-optical image analysis, feature detection and object classification, also enabling new camera designs and optical components that perform unique tasks using diffractive neural networks.

KALYPSO is a novel detector operating at line rates above 10 Mfps. It consists of a detector board connected to FPGA based readout card for real time data processing. The detector board holds a Si or InGaAs linear array sensor, with spectral sensitivity ranging from 400 nm to 2600 nm, which is connected to a custom made front-end ASIC. A FPGA readout framework performs the real time data processing. In this contribution, we present the detector system, the readout electronics and the heterogeneous infrastructure for machine learning processing. The detector is currently in use at several synchrotron facilities for beam diagnostics as well as for single-pulse laser characterizations. Thanks to the shot-to-shot capability over long time scale, new attractive applications are open up for imaging in biological and medical research.

We first discuss the ultimate specifications of an augmented reality display that would saturate the human perception. Thereafter our study identifies fundamental limitations and trade-offs enforced by laws of optics for any augmented reality display that uses passive optical elements such as visors, waveguides, and meta-surfaces to deliver the image to the eye. The limitations are categorized into 7 rules that optics designers must consider when they are designing augmented reality glasses. These rules are directly drawn from Fermat's principle, perturbation theory, linear optics reciprocity, and human visual perception principles. Based on psychophysical theories we further work toward defining and quantizing levels of depth that would saturate the human depth perception. Our results indicate that passive optics acts as a passive system with less than unity pulse response function that would always reduce the performance of the original light source. Additionally, our investigations reveal the dynamics between allocation of depth levels and number of depth levels for ultimate lighfield experiences.

We experimentally create a neural network via a spatial light modulator, implementing connections between 2025 in parallel based on diffractive coupling. We numerically validate the scheme for at least 34.000 photonic neurons. Based on a digital micro-mirror array we demonstrate photonic reinforcement learning and predict a chaotic time-series via our optical neural network. The prediction error efficiently converges. Finally, we give insight based on the first investigation of effects to be encountered in neural networks physically implemented in analogue substrates.

I will discuss the application of learning techniques to imaging through complex media (scattering media or multimode fibers). I will show that learning methods use examples to carry out the inverse scattering operation but also to account for random fluctuations in the system that normally deteriorate the performance of conventional methods.

Deep learning is a class of machine learning techniques that uses multi-layered artificial neural networks for automated analysis of signals or data. The name comes from the general structure of deep neural networks, which consist of several layers of artificial neurons, each performing a nonlinear operation, stacked over each other. Beyond its main stream applications such as the recognition and labeling of specific features in images, deep learning holds numerous opportunities for revolutionizing image formation, reconstruction and sensing fields. In fact, deep learning is mysteriously powerful and has been surprising optics researchers in what it can achieve for advancing optical microscopy, and introducing new image reconstruction and transformation methods. From physics-inspired optical designs and devices, we are moving toward data-driven designs that will holistically change both optical hardware and software of next generation microscopy and sensing, blending the two in new ways. Today, we sample an image and then act on it using a computer. Powered by deep learning, next generation optical microscopes and sensors will understand a scene or an object and accordingly decide on how and what to sample based on a given task – this will require a perfect marriage of deep learning with new optical microscopy hardware that is designed based on data. For such a thinking microscope, unsupervised learning would be the key to scale up its impact on various areas of science and engineering, where access to labeled image data might not be immediately available or very costly, difficult to acquire. In this presentation, I will provide an overview of some of our recent work on the use of deep neural networks in advancing computational microscopy and sensing systems, also covering their biomedical applications.

Random projections are used in applications such as compressive sensing, speckle spectroscopy, and recurrent neural networks. Most prior work has used speckle in free-space systems. Here we report results using laser speckle in planar and cylindrical waveguides with the goal of integrating the whole system in a photonic integrated circuit. We demonstrate a compressive sensing RF receiver over the 2-19 GHz band that recovers the amplitude, phase and frequency of one or two RF sinusoids in a 4.5-ns time window. The RF signal is modulated on a wavelength-chirped optical field, derived from a dispersed mode-locked laser pulse, that propagates through a 5-m, 105-micron fiber. The output of the fiber is imaged onto a fiber bundle such that 32 independent measurements of the speckle pattern are made, and differential outputs of pairs of photodiodes are then digitized to give 16 compressive measurements. The frequency resolution in a single pulse is about 100 MHz, but the resolution can be improved to about 20 KHz by using 100 pulses at a 35 MHz rate. A simpler system uses a stable single-frequency laser diode and speckle in a planar waveguide to determine RF frequency to about 100 MHz. Finally, speckle in a multimode waveguide is used as the reservoir in a recurrent neural network to predict an RF time series.

Lock-in amplification is a technique used in many applications such as non-linear optics, spectroscopy and biomedical imaging. It is used to recover weak signals from noisy environments using a known carrier wave. Historically, this has been achieved using dedicated instruments, which suffer from slow connections to computer for further analysis. Here we present a technique that uses a PCIe waveform digitizer to stream data to a graphics processing unit, where digital lock-in amplification occurs. Briefly, the input signal is multiplied by an in-phase and quadrature reference before undergoing low-pass filtering with a sinc filter. The DC components of the in-phase and quadrature signals are then used to give the magnitude and phase of the signal. The bottleneck in this digital implementation is low-pass filtering due to the long filter length required to efficiently remove higher frequencies. To achieve the highest data throughput, convolution of the input signal with the low-pass filter was implemented in both time domain and frequency domain. Furthermore, decimation of the filtered data can further reduce processing time. Preliminary results show that the implemented digital lock-in amplification can sustain a continuous sampling rate of 3 billion samples per second. Using a 12 bit resolution analog to digital converter with an 800 mV input range, signals under 3 μV are reliably detectable. This GPU-based lock-in amplification implementation provides an ideal and low-cost solution for continuous and gapless signals acquisition.

In example-based super-resolution, the function relating low-resolution images to their high-resolution counterparts is learned from a given dataset. This data-driven approach to solving the inverse problem of increasing image resolution has been implemented with deep learning algorithms. In this work, we explore modifying the imaging hardware in order to collect more informative low-resolution images for better ultimate high-resolution image reconstruction. We show that this "physical preprocessing" allows for improved image reconstruction with deep learning in Fourier ptychographic microscopy. Fourier ptychographic microscopy is a technique allowing for both high resolution and high field-of-view at the cost of temporal resolution. In Fourier ptychographic microscopy, variable illumination patterns are used to collect multiple low-resolution images. These low-resolution images are then computationally combined to create an image with resolution exceeding that of any single image from the microscope. We use deep learning to jointly optimize the illumination pattern with the post-processing reconstruction algorithm for a given sample type, allowing for single-shot imaging with both high resolution and high field-of-view. We demonstrate that the joint optimization yields improved image reconstruction as compared with sole optimization of the post-processing reconstruction algorithm.

Medical imaging is fundamentally challenging due to absorption and scattering in tissues and by the need to minimize illumination of the patient with harmful radiation. Imaging modalities also suffer from low spatial resolution, limited dynamic range and low contrast. These predicaments have fueled interest in enhancing medical images using digital post processing. Recent progress in image super resolution using machine learning and in particular convolutional neural networks (CNNs) may offer new possibilities for improving the quality of medical images. However, the tendency of CNNs to hallucinate image details is detrimental for medical images as it may lead to false diagnostics. Also, these techniques require prohibitively large computational resource, a problem that is exacerbated by the large size of medical images. Rapid and Accurate Image Super Resolution (RAISR) method provides a computationally efficient solution for image upscaling. In this paper, we propose ARAISR, an improved variant of RAISR, which inherits the local features and regression model of RAISR but instead of utilizing cluster anchored points to represent image feature space. This algorithm combines the low computing complexity of RAISR with the feature enhancement advantage of phase stretch transform (PST), a new computational approach that is inspired by the physics of photonic time stretch technique. We obtain improved quality (i.e. maximum 1dB PSNR better than RAISR) and hallucination-free performance for medical images super resolution.

We present a deep learning-based framework for super-resolution image transformations across multiple fluorescence microscopy modalities. By training a neural network using a generative adversarial network (GAN), a single low-resolution image is transformed into a high-resolution image that surpasses the diffraction limit. The deep network’s output also demonstrates improved signal-to-noise ratio and extended depth-of-field. This framework is solely data-driven which means that it does not rely on any physical models of the imaging formation process, and instead learns a statistical transformation from the training image datasets. The inference process is non-iterative and does not require sweeping over parameters to achieve optimal results, in contrast to state-of-the-art deconvolution methods. The success of this framework is demonstrated by super-resolving wide-field images captured with low-numerical aperture objective-lenses to match the resolution of images captured with high-numerical aperture objectives. In another example, we demonstrate the transformation of confocal microscopy images into images that match the performance of stimulated emission depletion (STED) microscopy, by super-resolving the distributions of Histone 3 sites within cell nuclei. We also applied this framework to total-internal-reflection fluorescence (TIRF) microscopy and super-resolved TIRF images to match the resolution of TIRF-based structured illumination microscopy (TIRF-SIM). Our super-resolved TIRF images/movies reveal endocytic protein dynamics in SUM159 cells and amnioserosa tissues of a Drosophila embryo, providing a very good match to TIRF-SIM images/movies of the same samples. Our experimental results demonstrate that the presented data-driven super resolution approach generalizes to new types of images and super-resolves objects that were not present in the training stage.

Luminescence sensors are based on the determination of emitted intensity or decay time when a luminophore is in contact with its environment. Changes of the environment, like temperature or analyte concentration cause a change in the intensity and decay rate of the emission. Typically, since the absolute values of the measured quantities depend on the specific sensing element and scheme used, a sensor needs an analytical model to describe the dependence of the quantity to be determined, for example the oxygen concentration, from sensed quantity, for example the decay time. Additionally, since the details of this dependence are device specific, a sensor needs to be calibrated at known reference conditions. This work explores an entirely new artificial intelligence approach and demonstrates the feasibility of oxygen sensing through machine learning. The new developed neural network is used for optical oxygen sensing based on luminescence quenching. After training the neural network on synthetic data, it was tested on measured data to verify the prediction of the model. The results show a mean deviation of the predicted from the measured concentration of 0.5 % air, which is comparable to many commercial and low-cost sensors. The accuracy of the model predictions is limited by the ability of the generated data to describe the measured data, opening up future possibilities for significant improvement by performing the training on experimental data. In this work the approach is tested at different temperatures, showing its applicability in the entire range relevant for biological applications. This work demonstrates the applicability of this new approach based on machine learning for the development of a new generation of optical luminescence oxygen sensors without the need of an analytical model of the sensing element and sensing scheme.

Range sensors have drawn much interest for human activity related research since they provide explicit 3D information about the shape that is invariant to clothing, skin color and illumination changes. However, triangulationbased systems like structured-light sensors generate occlusions in the image when parts of the scene cannot be seen by both the projector and the camera. Those occlusions, as well as missing data points and measurement noise, depend on the structured-light system design. These artifacts add a level of difficulty to the task of human body segmentation that is typically not addressed in the literature. In this work, we design a segmentation model that is able to reason about 3D spatial information, to identify the different body parts in motion and is robust to artifacts inherent to the structured-light system, such as triangulation occlusions, noise and missing data. First, we build the first realistic sensor-specific training set by closely simulating the actual acquisition scenario with the same intrinsic parameters as our sensor and the artifacts it generates. Second, we adapt a state-of-the-art fully convolutional network to range images of the human body in order for it to transfer its learning toward 3D spatial information instead of light intensities. Third, we quantitatively demonstrate the importance of simulating sensor-specific artifacts in the training set to improve the robustness of the segmentation of actual range images. Finally, we show the capability of the model to accurately segment human body parts on real range image sequences acquired by our structured light sensor, with high inter-frame consistency and in real-time.

This paper reports a combined technique to enhance Rayleigh scattering signal in optical fiber and deep neutral network data analytics for distributed acoustic sensing (DAS). By using ultrafast laser direct writing technique, over 45 dB of Rayleigh backscattering enhancement can be achieved in silica fibers to improve backscattering signal to improve signal to noise ratio for optical fiber DAS based on phase-sensitive optical time-domain reflectometry (Φ-OTDR). The enhanced backscattering signal enhance detections of vibration signal to subsequent data analysis and classifications. Using deep neutral networks and through both supervised and unsupervised machine learnings, the distributed acoustic sensing system were used to detect and to identify human movements to achieve <90% identification accuracies. The applications of DAS and artificial intelligence in pipeline corrosion detection and damage classification is also discussed in this paper.

In optical filter based compressive sensing (CS) spectrometers, an input spectrum is multiplexed and modulated by a small number of optical filters which have different sensing patterns. Then, detectors read out the modulated signals called measurements. By exploiting the CS reconstruction algorithms that utilize the measurements and the sensing patterns of optical filters, the spectrum is recovered. However, there exists a drawback on CS reconstruction algorithms. The input spectrum should be a sparse signal or be sparsely represented by a pre-determined sparsifying basis. In practice, however, the input spectrum could not be sparse or be sparsely represented by the pre-determined sparsifying basis. Therefore, the performance of spectral recovery using the CS reconstruction algorithms is varying according to the sparsity of the input spectrum and the sparsifying basis. In this paper, we implement a convolutional neural networks (CNNs) structure to reconstruct the input spectrum from the measurements of the CS spectrometers. The CNNs structure learns the way of solving the inverse problem of the underdetermined linear system. As an input of the CNNs structure, a spectrum calculated by multiplying a fixed transform matrix and the measurements is used. We investigate the reconstruction performance of the CNNs structure comparing with the CS reconstruction algorithm with different sparsifying basis. The experiment results indicate the reconstruction performance of the CNNs structure is compatible with the CS reconstruction algorithm.