Artificial intelligence (AI) techniques have boosted the capability of optical imaging, sensing, and communication. Concurrently, photonics facilitate the tangible realization of deep neural networks, offering potential benefits in terms of latency, throughput, and energy efficiency. In this talk, I will discuss our efforts in AI photonics with two examples. The first involves employing a convolutional neural network for achieving single-shot full-field measurement of optical signals. The second example pertains to implementing a deep neural network with a multiple-scattering system featuring structural nonlinearity, thereby enabling nonlinear computations using linear optics.

As the rapid increase in the amount of information, there has been increasing demand on developing domain-specific computing hardware capable of addressing complex tasks that general-purpose von-Neumann computers cannot readily address. Here I show a kind of unconventional computing schemes, a probabilistic computing, which is promising to address various computationally hard problems, in particular the computational tasks for which probabilistic algorithms are often applied. Stochastic magnetic tunnel junction, an unconventional spintronics device utilizing the thermally activated random magnetization switching, plays a key role. I first describe the concept of the probabilistic computer with the stochastic magnetic tunnel junction and then show several proof-of-concepts of the probabilistic computer. I also discuss device physics and engineering for performing large-scale problems within a limited computation time.

3D Optical Neural Networks (ONN) is a promising solution to the energy, time, and area yearning AI hardware. The 3D additive manufacturing technique with Two-Photon Polymerization can be used to build these 3D dense ONNs. The hybrid waveguide circuit which fuses the polymer and air-clad waveguides is the important interconnect for the ONN. The polymer-cladded waveguide can support single mode and evanescent coupling while the air-clad can support tight bend for dense integration.

We report an all-optical phase conjugation (OPC) approach utilizing diffractive wavefront processors collectively optimized via deep learning. Our approach uses purely passive materials for OPC, thus obviating the need for external power sources or digital processing. Moreover, the resulting design is notably compact, axially spanning ~84 wavelengths between the input and output planes. We experimentally demonstrated its proof-of-concept using the terahertz spectrum, proving its efficacy in performing phase conjugation of waves perturbed by unknown distortions. Our method is expected to provide transformative OPC solutions that can be integrated into diverse imaging and sensing systems at different parts of the spectrum.

I will overview our recent work testing the absolute limits of optical computing, including realizations of optical neural networks that use vastly less than one photon per multiplication, operating deep in the stochastic regime dominated by quantum noise. I will also discuss how optical neural networks scale up, and how they may offer advantages (including 3-5 orders of magnitude more energy-efficient inference) for implementing large language models. Works referenced: Ma, S. Y., Wang, T., Laydevant, J., Wright, L. G., & McMahon, P. L. (2023). Quantum-noise-limited optical neural networks operating at a few quanta per activation. arXiv preprint arXiv:2307.15712 Anderson, M. G., Ma, S. Y., Wang, T., Wright, L. G., & McMahon, P. L. (2023). Optical transformers. arXiv preprint arXiv:2302.10360. Wang, T., Ma, S. Y., Wright, L. G., Onodera, T., Richard, B. C., & McMahon, P. L. (2022). An optical neural network using less than 1 photon per multiplication. Nature Communications, 13(1), 123.

Minimally invasive endoscopy using multicore fibers shows great potential for numerous applications in biomedical imaging. With a diffuser on the distal side of the fiber and image recovery using AI, single-shot 3D imaging is possible by encoding the image volume into 2D speckle patterns. In comparison to equivalent lens systems, a higher space-bandwidth product can be achieved. However, decoding the image with iterative algorithms is time-consuming. Thus, we propose utilizing a neural network for fast 2D and 3D image reconstruction at video rate. In this work, single-shot 3D fluorescence imaging with physics-informed neural network is presented, which is promising for calcium imaging at in vivo brain diagnostics.

We present a fast virtual-staining framework for defocused autofluorescence images of unlabeled tissue, matching the performance of standard virtual-staining models using in-focus label-free images. For this, we introduced a virtual-autofocusing network to digitally refocus the defocused images. Subsequently, these refocused images were transformed into virtually-stained H&E images using a successive neural network. Using coarsely-focused autofluorescence images, with 4-fold fewer focus points and 2-fold lower focusing precision, we achieved equivalent virtual-staining performance to standard H&E virtual-staining networks that utilize finely-focused images, helping us decrease the total image acquisition time by ~32% and the autofocusing time by ~89% for each whole-slide image.

Optical tweezers manipulate microscopic objects with light by exchanging momentum and angular momentum between particle and light, generating optical forces and torques. Understanding and predicting them is essential for designing and interpreting experiments. Here, we focus on geometrical optics and optical forces and torques in this regime, and we employ neural networks to calculate them. Using an optically trapped spherical particle as a benchmark, we show that neural networks are faster and more accurate than the calculation with geometrical optics. We demonstrate the effectiveness of our approach in studying the dynamics of systems that are computationally “hard” for traditional computation.

We analyse the collective behaviours of Escherichia coli (E. coli) active matter. The individual movements of these E. coli can be accurately tracked and examined using a recently developed machine learning software called DeepTrack (Midvedt et al., 2021). This provides greater insight into the chaotic dynamics of E. coli swarms as well as the potential to critically assess current theoretical models. DeepTrack analysis can also be applied in more complex environments including interactions with microstructures made with photolithography. Analysing the movements of E. coli active matter with DeepTrack has promising implications in engineering and biomedical applications.

Information processing is vital for living systems and involves complex networks of active processes. These systems have influenced various forms of modern machine learning, including reservoir computing. Reservoir computing utilizes networks of nodes with fading memory to perform computations and make complex predictions. Reservoirs can be implemented on computer hardware or unconventional physical substrates like mechanical oscillators, spins, or bacteria, known as physical reservoir computing. We demonstrate physical reservoir computing with a synthetic active microparticle system that self-organizes from an active and passive component into inherently noisy nonlinear dynamical units. The self-organization and dynamical response of the unit is the result of a delayed propulsion of the microswimmer to a passive target. A reservoir of such units with a self-coupling via the delayed response can perform predictive tasks despite the strong noise resulting from the Brownian motion of the microswimmers. To achieve efficient noise suppression, we introduce an architecture that uses historical reservoir states for output. We discuss the node and collective reservoir dynamics.

Fluorescence microscopy is the gold standard for investigation of biological samples. While powerful, this technology has its limitations, including phototoxicity, technical difficulties in introducing fluorescent markers, and limited simultaneous labeling of different structures due to the need for spectral separation. Here, artificial intelligence-driven virtual staining can provide label-free imaging. Deep neural networks are trained to learn the correlation between a label-free image and a ground-truth fluorescence image. However, the main challenge for the transition from traditional to virtual staining is the acquisition of huge datasets including labeled and unlabeled data. We have performed proof-of-concept experiments to investigate the possibilities of transfer learning in virtual staining. Therefore, U-Net architectures were pretrained on a larger dataset and later on trained again on a smaller and more specific dataset. We show that transfer learning decreases the needed dataset size and may even improve prediction quality. Furthermore, the interpretability of the trained networks was studied.This was investigated using guided backpropagation and modified input images.

We introduce a deep learning-based approach utilizing pyramid sampling for the automated classification of HER2 status in immunohistochemically (IHC) stained breast cancer tissue images. Our deep learning-based method leverages pyramid sampling to analyze features across multiple scales from IHC-stained breast tissue images, managing the computational load effectively and addressing the challenges of HER2 expression heterogeneity by capturing detailed cellular features and broader tissue architecture. Upon application to 523 core images, the model achieved a classification accuracy of 85.47%, demonstrating the ability to counteract staining variability and tissue heterogeneity, which might improve the accuracy and timeliness of breast cancer treatment planning.

This presentation covers the opportunities and challenges of photonic-electronic chip-based machine intelligence acceleration hardware. We will start with a review of device-level component performance specifications such as footprint, energy consumption, reconfiguration speed, for instance. Emerging materials, when integrated monolithically into photonic waveguide circuits, show promise for next generation opto-electronic components offering high FOMs, however have high barrier to entry in for foundry PDKs. Beyond devices, we will explore a variety of architectural choices, known as co-design optimization. Parallelization strategies, smart routing, optical hardware function implementation (e.g. Fourier transformation on-chip) will be covered. Next, we explore chip packaging options including ADK and digital-twin hardware in the loop optimizations thereof. Finally, examples of prototyping will be shared and application options discussed.

Deep learning has emerged as a powerful tool in medical imaging, fundamentally transforming the field by automatically learning intricate patterns and features from large datasets. Its application spans various tasks such as image classification, segmentation, registration, reconstruction, and synthesis across diverse modalities including MRI, CT, PET/CT, PET/MRI, ultrasound, and X-ray.

Functional magnetic resonance imaging data suggest that temporal correlations in cortical networks vary across distinct behavioral states, providing for the dynamic reorganization of patterned activity. However, these data generally lack the temporal resolution to establish links between cortical signals and the continuously varying fluctuations in spontaneous behavior observed in awake animals. To understand the relationship between neural activity, functional connectivity and behavior, we used wide-field mesoscopic calcium imaging to monitor cortical dynamics in awake mice. I will present an approach we developed to quantify rapidly time-varying functional connectivity. Our "graph of graphs" approach, relying on Riemannian geometry, extracts the latent dynamics of time-varying functional connectivity in neuronal networks. Our analysis demonstrates that spontaneous behaviors are represented by fast changes in the correlational structure of cortical networks that are distinct from fluctuations in the magnitude of network activity. Combining mesoscopic imaging with simultaneous cellular-resolution two-photon microscopy demonstrated that correlations among neighboring neurons and between local and large-scale networks also encode behavior.

We present a rapid, stain-free, and automated viral plaque assay utilizing deep learning and time-lapse holographic imaging, which can significantly reduce the time needed for plaque-forming unit (PFU) detection and entirely bypass the chemical staining and manual counting processes. Demonstrated with vesicular stomatitis virus (VSV), our system identified the first PFU events as early as 5 hours of incubation and detected >90% of PFUs with 100% specificity in <20 hours, saving >24 hours compared to the traditional viral plaque assays that take ≥48 hours. Furthermore, our method was proven to adapt seamlessly to new types of viruses by transfer learning.

The integration of artificial intelligence (AI) in healthcare, particularly using advanced deep learning and generative methods, presents substantial challenges and opportunities. Implementation of AI in clinical settings, such as radiology, requires evaluation beyond algorithm development. Measuring the effectiveness of AI is complex: should success metrics be based on the number of AI uses in radiology, the variety of pathologies analyzed, throughput, user engagement, or clinical quality measures like diagnostic accuracy or study turnaround time? Economic factors like return on investment are also critical. The author, drawing on multi-year practical experience in large-scale AI deployment in radiology, proposes methods to evaluate and enhance the value of AI in medical imaging. These include AI-PROBE, which assesses radiologists' performance with and without AI, and can provide advances like revised radiology Quality Assurance (QA) programs, or improved multi-institutional public health tracking. The presentation will conclude with introducing a multi-speaker session, covering in-house AI system development, the use of AI in clinical research, and FDA regulatory aspects.

Artificial intelligence in radiology has advanced rapidly over the last decade. Radiology is particularly amenable to AI because of its focus on imaging. In this presentation, I will review some of the latest advances in radiology AI, including the use of transformers and large language models. I will provide examples of radiology AI for diverse clinical applications including cardiovascular, cancer, and liver disease.

Rapid advances in artificial intelligence (AI) and machine learning have enabled rapid development of decision support tools for application in broad health care areas. Many of the AI systems target medical imaging applications. The availability of public datasets additionally stimulated the advancement of the medical AI systems and the ability of research groups to contribute to the field. In-house AI system development is crucial for exploring new ideas, and new research directions and applications which strongly contribute to the generation of fundamental knowledge by a broad research community in broad clinical areas. Number of in-house computer-aided AI decision support applications will be presented. It is essential to properly train and rigorously validate a clinical decision support tool and verify its generalizability and reliability prior to translation for patient care in the clinic. Best practices for the development and performance assessment of computer-aided AI decision support systems will be discussed.

As the number and range of new AI medical devices continues to grow, they pose novel challenges for regulators, including the FDA, to balance safety and effectiveness while allowing for industry to quickly innovate. This presentation will provide an overview of the current regulatory environment for AI-enabled medical devices in radiology, gastroenterology, dermatology, and pathology. It will also include a description of some of the important advances coming from our regulatory science research here at FDA focused on improving and streamlining the development, assessment, and long-term monitoring of medical AI. Our efforts include research addressing the limitations of small and fragmented patient datasets, identifying AI bias and their sources, reducing algorithm performance differences across subpopulations, assessing novel and continuous-learning AI, developing assessment pipelines that account for factors such as heterogeneity in the reference standard, and developing effective post-market monitoring of AI devices.

Hyperdimensional computing is an emerging computing paradigm that leverages distributed representations of input data. The hyperdimensional distributed representations facilitate energy-efficient, low-latency, and noise-robust computations using low-precision and basic arithmetic operations. In this presentation, we introduce optical hyperdimensional distributed representations using laser speckles for adaptive, efficient, and low-latency optical in-sensor processing. We focus on applications for optical soft-touch interfaces and tactile sensors. We demonstrate that this optical approach achieves high accuracy in touch or tactile recognition while significantly reducing the amount of training data and computational burdens compared to traditional deep learning-based sensing approaches.

A high-performance photonic reservoir, which utilizes injection locking of a semiconductor multimode laser (SML), will be developed. This innovative design allows for fully parallel and high-bandwidth operation at telecommunication wavelength. The output of this system is projected in space and imaged onto a digital micromirror device, which provides a readout and facilitates the hardware integration of programmable output weights. By using a highly multimodal semiconductor laser, injection locking enables a large number of modes to be simultaneously locked to the high frequency modulated injection laser that provides the input signal, resulting in high dimensionality of the reservoir. The hardware integration of programmable output weights enables the system to be optimized for specific tasks, improving performance and reducing power consumption.

Reservoir computing (RC) is a neuromorphic machine learning paradigm that is ideal for temporal signal processing and is suitable for analog implementations in physical substrates, including photonic devices. RC has been extended to quantum systems due to the enhanced capabilities provided by an enlarged Hilbert space. In that regard, quantum reservoir computing (QRC) has the advantage of avoiding barren plateaus during training. Photonic architectures have already been studied for QRC applications. In our research, we propose a scalable quantum photonic platform for QRC that is suitable for solving temporal tasks. The physical substrate of our reservoir is an optical pulse, which recirculates through an optical cavity with losses, thus creating a quantum memory. The dissipation device (a beam-splitter) also allows the injection of external information and the weak monitoring of the reservoir. Our work focuses on the ability to process classical signals in real time by creating a physical ensemble of identical pulses inside a fiber and the noise robustness of our architecture by tuning the squeezing produced inside the optical cavity.

Disorder is a pervasive characteristic of natural systems, offering a wealth of non-repeating patterns. In this study, we present a novel storage method that harnesses naturally-occurring random structures to store an arbitrary pattern in a memory device. This method, the Stochastic Emergent Storage (SES), builds upon the concept of emergent archetypes, where a training set of imperfect examples (prototypes) is employed to instantiate an archetype in a Hopfield-like network through emergent processes. We demonstrate this non-Hebbian paradigm in the photonic domain by utilizing random transmission matrices, which govern light scattering in a white-paint turbid medium, as prototypes. Through the implementation of programmable hardware, we successfully realize and experimentally validate the capability to store an arbitrary archetype and perform recognition at the speed of light. Leveraging the vast number of modes excited by mesoscopic diffusion, our approach enables the simultaneous storage of thousands of memories without requiring any additional fabrication effort, moreover these memories can be grouped to realize higher level classes thus realizing patterns classification.

We present an all-optical image denoiser based on spatially-engineered diffractive layers. Following a one-time training process using a computer, this analog processor composed of fabricated passive layers achieves real-time image denoising by processing input images at the speed of light and synthesizing the denoised results within its output field-of-view, completely bypassing digital processing. Remarkably, these designs achieve high output diffraction efficiencies of up to 40%, while maintaining excellent denoising performance. The effectiveness of this diffractive image denoiser was experimentally validated at the terahertz spectrum, successfully removing salt-only noise from intensity images using a 3D-fabricated denoiser that axially spans <250 wavelengths.

Optical computing provides a compelling avenue to sustain the rapid growth of computing power needed by the rise of AI. However, it still struggles to efficiently implement the all-optical nonlinearities required to achieve effectively deep neural networks, a prerequisite for modern performance. Here, we exploit purely a linear optical setup to perform optical nonlinear mapping for information processing and compression. The essential ingredient is to encode information on a spatial light modulator embedded in a multiple scattering cavity. Light exiting the cavity encodes a much richer information than its linear counterpart. Fed into a very simple neural network as a digital decoder, we demonstrate its potential for various machine learning tasks include detecting pedestrians for self-driving cars, defining a new state-of-the-art in optical computing.

Coupled systems with multiple interacting degrees of freedom provide a fertile ground for emergent dynamics, which is otherwise inaccessible in their solitary counterparts. Particularly, nonlinearity and non-equilibrium dynamics enable new opportunities in coupled photonic systems that are not present in their linear and equilibrium counterparts that can have profound consequences in sensing and computing. In this talk, I will overview recent experimental progress on accessing such dynamics in time-multiplexed networks of nonlinear resonators towards computing and sensing applications. I will present demonstrations of topological dissipation, non-equilibrium spectral phase transitions, topological mode-locked lasers, non-Hermitian topologically enhanced sensing, and photonic elementary cellular automata. I will also overview the progress on integrated optical parametric oscillators (OPOs) and their networks in lithium niobate (LN) nanophotonics for classical and quantum information processing applications.

Fluorescence lifetime imaging microscopy (FLIM) measures fluorescence lifetimes of fluorescent probes to investigate molecular interactions. However, conventional FLIM systems often requires extensive scanning that is time-consuming. To address this challenge, we developed a novel computational imaging technique called light field tomographic FLIM (LIFT-FLIM). Our approach acquires volumetric fluorescence lifetime images in a highly data-efficient manner, significantly reducing the number of scanning steps. We demonstrated LIFT-FLIM using a single-photon avalanche diode array on various biological systems. Additionally, we expanded to spectral FLIM and demonstrated high-content multiplexed imaging of lung organoids. LIFT-FLIM can open new avenues in the biomedical research.

The traditional histochemical staining of autopsy tissue samples usually suffers from staining artifacts due to autolysis caused by delayed fixation of cadaver tissues. Here, we introduce an autopsy virtual staining technique to digitally convert autofluorescence images of unlabeled autopsy tissue sections into their hematoxylin and eosin (H&E) stained counterparts through a trained neural network. This technique was demonstrated to effectively mitigate autolysis-induced artifacts inherent in histochemical staining, such as weak nuclear contrast and color fading in the cytoplasmic-extracellular matrix. As a rapid, reagent-efficient, and high-quality histological staining approach, the presented technique holds great potential for widespread application in the future.

We present subwavelength imaging of amplitude- and phase-encoded objects based on a solid-immersion diffractive processor designed through deep learning. Subwavelength features from the objects are resolved by the collaboration between a jointly-optimized diffractive encoder and decoder pair. We experimentally demonstrated the subwavelength-imaging performance of solid immersion diffractive processors using terahertz radiation and achieved all-optical reconstruction of subwavelength phase features of objects (with linewidths of ~λ/3.4, where λ is the wavelength) by transforming them into magnified intensity images at the output field-of-view. Solid-immersion diffractive processors would provide cost-effective and compact solutions for applications in bioimaging, sensing, and material inspection, among others.

We introduce a diffractive super-resolution display system combining an electronic encoder and a diffractive decoder network to project super-resolved images using a low-resolution spatial light modulator (SLM). This deep learning-enabled display system achieves ~4x super-resolution, corresponding to a ~16x increase in the space-bandwidth product, which was also experimentally demonstrated using 3D-fabricated diffractive decoders that operate at the THz spectrum. The design principles of this diffractive super-resolution display were also used to project high-resolution color images using a low-resolution SLM. Diffractive super-resolution image projection paves the way for developing compact, low-power, and computationally efficient high-resolution image and video display systems.

Lucendi has developed Aqusens - an AI-based holographic imaging flow cytometer platform that includes a lens-less instrument enabling rapid screening of liquids and characterization of microobjects for many different applications. In contrast with conventional lens-based cytometers, Aqusens achieves much higher throughput and dynamic range, while built in a robust and portable package that can be used for in-lab or in-field deployments. Aqusens operations are enabled by different applications of neural networks that are used for autofocusing and phase recovery, as well as for advanced data analytics. The use of AI in Aqusens enables optimization of computing that allows practical deployments of the system for real applications.

We present a method for accurately performing complex-valued linear transformations with a Diffractive Deep Neural Network (D2NN) under spatially incoherent illumination. By employing 'mosaicing' and 'demosaicing' techniques, complex data are encoded into optical intensity patterns for all-optical diffractive processing, and then decoded back into the complex domain at the output aperture. This framework not only enhances the capabilities of D2NNs for visual computing tasks but also opens up new avenues for applications in image encryption under natural light conditions to demonstrate the potential of diffractive optical networks in modern visual information processing needs.

We demonstrate a reconfigurable diffractive deep neural network (termed R‑D2NN) with a single physical model performing a large set of unique permutation operations between an input and output field-of-view by rotating different layers within the diffractive network. Our study numerically demonstrated the efficacy of R‑D2NN by accurately approximating 256 distinct permutation matrices using 4 rotatable diffractive layers. We experimentally validated the proof-of-concept of reconfigurable diffractive networks using terahertz radiation and 3D-printed diffractive layers, achieving high concordance with numerical simulations. The reconfigurable design of R‑D2NN provides scalability with high computing speed and efficient use of materials within a single fabricated model.

We present GedankenNet, a self-supervised learning framework designed to eliminate reliance on experimental training data for holographic image reconstruction and phase retrieval. Analogous to thought (Gedanken) experiments in physics, the training of GedankenNet is guided by the consistency of physical laws governing holography without any experimental data or prior knowledge regarding the samples. When blindly tested on experimental data of various biological samples, GedankenNet performed very well and outperformed existing supervised models on external generalization. We further showed the robustness of GedankenNet to perturbations in the imaging hardware, including unknown changes in the imaging distance, pixel size and illumination wavelength.

Mode decomposition is a quantitative technique for analyzing multimode fibers. With pre-knowledge of the eigenmodes, the phase and amplitude weights of each mode can be extracted from the optical field. In this paper, we introduce a simple deep learning-based mode decomposition method by integrating a physical model with a deep neural network. We demonstrate that this method can decompose up to thousands of modes based on pure-intensity images.

Multiple scattering enables optical encryption: the input information (i.e., plaintext) is scrambled and only speckles (i.e., cyphertext) are output for access. The decryption, however, is ineffective with conventional methods. Recent intervention of deep learning (DL) decrypts information from speckles with much higher fidelity, driven by massive data. We developed a complex-valued platform to decrypt complex information (e.g., human face) from the speckles, with sufficient-high fidelity for face recognition. Continuous efforts endeavor for higher stability and security. By introducing a spin-multiplexing disordered metasurface as an ultra-stable speckle generator, the system demonstrates excellent decryption efficiency over extended periods in noisy environment with numerous encryption channels and a proposed double-secure scheme provides robust protection for the plaintext with a security key. Also, breaking inherent correlation among the speckles via a speckle modulation network can further boost the security and enable hierarchical authentication encryption. Collectively, speckle-based cryptosystem via DL is promising towards practice.

We introduce the enhanced Fourier Imager Network (eFIN), an end-to-end deep neural network that synergistically integrates physics-based propagation models with data-driven learning for highly generalizable hologram reconstruction. eFIN overcomes a key limitation of existing methods by performing seamless autofocusing across a large axial range without requiring a priori knowledge of sample-to-sensor distances. Moreover, eFIN incorporates a physics-informed sub-network that accurately infers unknown axial distances through an innovative loss function. eFIN can also achieve a three-fold pixel super-resolution, increasing the space-bandwidth product by nine-fold and enabling substantial acceleration of image acquisition and processing workflows with a negligible performance penalty.

We introduce an information hiding-decoding system, which employs a passive diffractive processor as the front-end and an electronic decoder as the back-end, offering a fast, energy-efficient, and scalable solution for protecting visual information. This diffractive processor all-optically transforms arbitrary input messages into deceptive output patterns, decipherable only through a jointly-trained electronic decoder neural network. This method can successfully hide infinitely many input messages into ordinary-looking patterns at its output field-of-view, which can be subsequently decoded by an electronic network. We experimentally validated the feasibility of our information-hiding camera by 3D-printing a physical diffractive system and testing it under terahertz illumination.

Photonic systems offer a promising platform for analog neuromorphic computing and machine learning acceleration, boasting advantages such as massive parallelism, low latency, and energy efficiency. Disordered photonic media have been utilized for implementing neural networks (NNs) architectures with simultaneous coding and processing of information, overcoming digital NNs' bottleneck of data transfer between memory and processor. I explore second-order nonlinear disordered photonic media assembled from oxide nanoparticles, particularly barium titanate and lithium niobate nanocrystals. Thanks to the simultaneous linear scattering and second-harmonic generation, these media enable multiple implementation of the activation function in the optical neural network, facilitating deep multi-layer operation. Experimental demonstrations showcase the potential of these media, particularly a deep two-layer optical neural network based on a nonlinear disordered multiple-scattering slab of lithium niobate nanocrystals, enhancing computing performance for various machine learning tasks including image classification and regression.

This talk will describe new microscopes and space-time algorithms that enable 3D or super-resolution fluorescence microscopy and phase measurement with high resolution on dynamic samples. Traditional model-based image reconstruction algorithms work together with neural networks to optimize the inverse problem solver and the data capture strategy in order to account for sample motion during the capture time of a multi-shot computational imaging method.

In this work we propose a new training scheme for analog neural networks. Instead of complexifying the training algorithm by trying to incorporate more information about the non-idealities of the system, we shift the complexity to the training data itself. We train the analog system to extract invariant representation vectors given two noisy views of the same input. This approach allows for learning locally and on the analog system itself. We experimentally demonstrate our approach with a free-space optical vector-matrix multiplier. We successfully trained in-situ a linear optical layer followed by a digital non-linearity and a linear classifier achieve state of the art test accuracy on par with digital simulations on MNIST. Next, we show that our method allows to continuously train the optical neural network while we apply some perturbations to it (misalignment, blurring, ...) while methods such as digital pre-training or Physics-Aware training cannot succeed at this task.

A foundational component of vision processing is edge and feature detection. In the human eye, this is carried out powerfully via retinal ganglion cells which fight to suppress neuronal firing of neighbouring cells - a process termed 'lateral inhibition'. Such spatially-distributed activity competition leads to strong nonlinear enhancement of key image features such as edges, enabling more complex vision functionality including object recognition & motion detection. Software convolutional neural networks draw inspiration from this process and also begin with edge-detection, but in software this functionality is slow & the process intrinsically linear (matrix multiplications between the input image & convolutional kernels), with nonlinearity forced in via subsequent computationally-expensive activation functions such as ReLu. Here, we present a physical system which reproduces the strongly nonlinear lateral inhibition used in the retina. Using spatially-distributed mode-competition in nanoscale random network lasers, we demonstrate cutting-edge feature detection on complex images, and leverage this for a retinomorphic photonic convolutional neural network with strong performance.

Computation exploiting physics can provide outstanding energy efficiency, but physical systems tend to suffer from noise and unpredictability. This is for example the case of emerging memory devices. They can be used as artificial synapses for low-energy AI, but they suffer from variability, making them functionally analogous to random variables. In machine learning, Bayesian approaches are designed to operate with random variables. In this talk, we show that they can be an excellent way to exploit emerging memory devices without suffering from their drawbacks. We present three experimental realizations of Bayesian systems exploiting emerging memory devices: a Bayesian reasoning machine, which provides explainable decision-making, a Bayesian neural network, capable of quantifying the certainty of its predictions, and a Bayesian learning system. As these systems fully embrace, and sometimes exploit, device variability, they show outstanding robustness. We finally discuss the potential of Bayesian approaches to exploit other types of physics.

To counter the exponential growth of computing power requirements for machine learning, efficient sailing of the integrated photonic processors has become a fundamental issue to be addressed. However, it remains challenging to properly calibrate the circuit imperfections, such as fabrication errors and crosstalk originating from both thermal and electric effects, which drastically affects the performance as the circuit size becomes larger. We demonstrate a silicon-photonics 16×16 Clements-type photonic vector-matrix multiplier. The degradation of fidelity caused by crosstalk and fabrication error was successfully compensated using our proposed machine learning based tuning method and deterministic calibration. The first experimental 10-digit MNIST classification was performed, which defines the classification results directly corresponding to the optical output ports. Furthermore, we also fabricated an 8  8 MZI-mesh photonic processor based on the planar lightwave circuit (PLC) technique which can realize low wavelength dependence operation due to low fabrication errors. This structure achieves the efficient throughput due to the O(N2W) operation, where N and W denote the number of spatial and wavelength channels, respectively. A high fidelity of 0.99 at 1550 nm and >0.96 over the C band was achieved, demonstrating the feasibility of the matrix-matrix multiplication operation with a combination of MZI-mesh and WDM.

Graphs are used to model complex relationships in various domains, such as interacting particles or neural connections within a brain. Efficient analysis and classification of graphs pose significant challenges due to their inherent structural complexity and variability. Here, an approach is presented to address these challenges through the development of a Graph Autoencoder Uncovering Descriptive Information (GAUDI). GAUDI effectively summarizes graph structures while preserving important topological details through multiple hierarchical pooling steps. This enables the extraction of physical parameters describing the graphs. The performance of GAUDI is demonstrated across diverse graph data originating from complicated systems, including the classification of protein assembly structures from single--molecule localization microscopy data, as well as the analysis of collective behavior and correlations between brain connections and age. This approach holds great promise for examining diverse systems, enhancing our comprehension of various forms of graph data.