This PDF file contains the front matter associated with SPIE Proceedings Volume 11931, including the Title Page, Copyright information, Table of Contents, and Conference Committee listings.

A see-through Maxwellian display with aberration-free pupil steering is proposed and experimentally demonstrated. The system uses a polarization-dependent off-axis lens coupler set fabricated with cholesteric liquid crystal. Electrically addressable polarization converters are used to switch among pupils.

For daily use of AR technology, the development of smart glasses which look like ordinary eyeglasses has been accelerated, and eye-tracking devices to support their video expression and natural human-machine interfaces have been attracting attention. Over the past few years, we have been developing a non-video-based eye-tracking system, that consists of a VCSEL array and a position-sensitive detector (PSD), and is implemented on small glass devices without preventing device design and their appearance. In wearable eye-tracking devices, calibration is frequently required, when misalignments of the device and the user switching occur. The most common calibration method is having the user gazes at multiple fixed points, but it interrupts user’s activities and causes stress. To eliminate the calibration stress, a novel algorithm has been proposed, that estimates the shape and position of user’s eyes from continuously detected data and corrects the gaze direction while wearing glasses. The fundamental principle of this algorithm is that such eye parameters affect spatial characteristics of detected laser beam spot on the PSD, reflected on the eye surface. The Bayesian estimation is used to updates the probability distribution of unconscious eye movements, and the eye parameters are identified with the help of the canonical correlation analysis. In this paper, the details of the gaze detection algorithm with autonomous calibration mechanism have been described, and a ray-tracing simulation has been performed for a proof of concept. The results show the applicability of our proposed algorithm to provide an eye-tracking module without any user stress of calibration.

We present a new hyperchromatic laser-based multifocal display. In the proposed design multiple full-colour virtual image planes can be displayed simultaneously at different depths in front of the observer through wavelength (de-)multiplexing. Each depth plane is displayed through its own combination of red, green and blue lasers at specific wavelengths. Hyperchromatic displays can be useful for augmented (AR) and mixed reality (MR) applications where real and virtual objects are located at different depths in front of the observer because they allow the human eye to focus on virtual objects and reduce the vergence-accommodation conflict (VAC). We present a laboratory demonstrator where the images generated by two red lasers (630 nm and 960 nm) were separated by more than 3 meters. We also discuss the applicability of the technology to the automotive head-up-display (HUD) systems and present an HUD system based on low-cost off-the-shelf components.

Augmented/mixed reality has many potential applications in the medical field. We sought to determine whether a mixed reality ruler capable of measuring the distance between a user’s fingers is sufficiently accurate for making measurements during clinical procedures. A custom virtual ruler application was deployed to the Microsoft HoloLens headset. The custom app calculates the linear distance between the user's fingers. The accuracy of the virtual ruler was tested under various conditions including room brightness and background textures. A set of wires of known length were measured with the virtual ruler. The accuracy of the virtual ruler was dependent on the measurement length. Measurements between 2-15 cm had an error less than 0.5 cm, between 15-30 cm had an error less than 1 cm, and between 30-50 cm had an error less than 1.6 cm. A mixed reality based hand-ruler is sufficiently accurate for making measurements during maximally sterile clinical procedures.

Most research works featuring AR and Brain-Computer Interface (BCI) systems are not taking advantage of the opportunities to integrate the two planes of data. In this project, we propose a prototype which combines an existing AR headset HoloLens 2 with a Brain-Computer Interfaces (BCI) system and we perform several tasks to validate this concept. In the first experiment, we reduced the distraction of the user by including information about the current attentional state. A simple game was designed for the Microsoft HoloLens 2, which changed in real time according to the user’s state of attention measured via electroencephalography (EEG). The system only responded if the attentional orientation was classified as "external." Fourteen users tested the attention-aware system; we show that the augmentation of the interface improved the usability of the system. We conclude that more systems would benefit from clearly visualizing the user’s ongoing attentional state as well as further efficient integration of AR and BCI headsets. In the second prototype we propose a system based on covert visuospatial attention (CVSA) – the process of focusing attention on different regions of the visual field without overt eye movements. In the experiment we did not rely on any stimulusdriven responses. Fourteen participants were able to test the system over the course of two days. To the best of our knowledge, this system is the first AR EEG-BCI integrated prototype that explores the complementary features of the AR headset like the HoloLens 2 and the CVSA paradigm.

100 microseconds response liquid crystal technology has been introduced showing whole the switching process with in-plane only retardation switching. This in-plane only retardation switching characteristic property enables keeping clean wavefront preservation for incoming image light flux with several hundred microseconds response time, which would provide higher image quality for AR/VR/MR applications.

Transcranial Magnetic Stimulation (TMS) is a neurostimulation technique based on the principle of electromagnetic induction of an electric field in the brain with both research and clinical applications. To produce an optimal neuro-modulatory effect, TMS coil must be placed on the head and oriented accurately with respect to the region of interest within the brain. A robotic method can enhance the accuracy and facilitate the procedure for TMS coil placement. This work presents two system improvements for robot-assisted TMS (RA-TMS) application. Previous systems have used outside-in tracking method where a stationary external infrared (IR) tracker is used as a reference point to track the head and TMS coil positions. This method is prone to losing track of the coil or the head if the IR camera is blocked by the robotic arm during its motion. To address this issue, we implemented an inside-out tracking method by mounting a portable IR camera on the robot end-effector. This method guarantees that the line of sight of the IR camera is not obscured by the robotic arm at any time during its motion. We also integrated a portable projection mapping device (PPMD) into the RA-TMS system to provide visual guidance during TMS application. PPMD can track the head via an IR tracker, and can project a planned contact point of TMS coil on the head or overlay the underlying brain anatomy in real-time.

A promising path towards consumer electronics-ready manufacturing of optical waveguide combiners is via large-area nanoimprinted surface relief gratings on high index glass substrates. Presently, this is realized through equipment and substrates based on wafer format (up to 12-inch). In this work, we present a way to produce waveguides with surface relief gratings utilizing the entire value chain from design to mastering to replication on panel-level nanoimprint equipment using rectangular high refractive index glass substrates and high refractive index resins. This is demonstrated on a greater than Gen 3 panel size (550 mm x 650 mm). The fabricated waveguides are optically tested to validate the design and the value chain. We demonstrate that the quality of the large area imprints is similar to present wafer-level imprints. Thus, we introduce a new approach towards high volume and low-cost manufacturing of waveguides based on surface relief gratings.

We demonstrate a quadrant detector-based method for aligning image sensors to the optical axis of augmented reality (AR) and virtual reality (VR) head mounted displays (HMDs). The sensor location that gives optimal image quality is known as the eyepoint. While a wide variety of methods have been explored for determining the eyepoint, the sensitivity and results of these methods depend on the optical design of the HMD. In our quadrant detector-based method, a test pattern consisting of dots is displayed on the HMD and imaged by a camera. The signal from the camera is divided into four quadrants and the camera is positioned at the eyepoint when the signal difference between the quadrants is minimized. This method has the advantage of giving the direction of the eyepoint from a single measurement without mapping the eyebox of the HMDs. The measurements were performed using an FDA developed setup for characterizing HMDs with the flexibility to incorporate a variety of light measuring devices and HMDs using commercially available components, which is described in detail. These results demonstrate a generalizable eyepoint alignment method, which allows for simpler automation of aligning the camera to the eyepoint.

Bayfol® HX photopolymer films prove themselves as easy-to-process recording materials for volume holographic optical elements (vHOE’s) and are available in customized grade at industrial scale. Their full-color (RGB) recording and replay capabilities are two of their major advantages. Moreover, the adjustable diffraction efficiency, tunable angular and spectral selectivity of vHOE’s recorded into Bayfol® HX as well as their unmatched optical clarity enables superior invisible “off Bragg” optical functionality. As a film product, the replication of vHOE’s in Bayfol® HX can be carried out in a highly cost-efficient and purely photonic roll-to-roll (R2R) process. Utilizing thermoplastic substrates, Bayfol® HX was demonstrated to be compatible to state-of-the-art plastic processing techniques like thermoforming, film insert molding and casting, which opened up using a variety of industry-proven integration technologies for vHOE’s. Therefore, Bayfol® HX makes its way in applications in the field of augmented reality such as Head-up-Displays (HUD) and Head-mountedDisplays (HMD), in free-space combiners, in plastic optical waveguides, and in transparent screens. Also, vHOE’s made from Bayfol® HX are utilized in highly sophisticated spectrometers in astronomy as well as in narrow band notch filters for eyeglasses against laser strikes. See through applications such as, HMD and HUD, have demanding performance requirements on combiner and imaging technologies such as efficiency, optical function, and clarity. The expanded properties of Bayfol® HX make it well suited to solve these challenges: In primary display applications such as free space combiners and waveguide based smart glasses applications, as well as integrated applications such as near-infrared imaging as used in eye-tracking and remote sensing. The transparency and performance characteristics of Bayfol® HX make it perfectly suited for maintaining the requirements on optical performance as demanded in see through applications. We demonstrate practical examples of Bayfol® HX vHOE’s in see through applications.

For a diffractive waveguide-based augmented reality display, we demonstrate a rapid and straightforward method to define the most compact layout of an exit pupil expander (EPE) from an analysis of diffractive optical elements in kspace. The idea is to consider all k-vectors coming from in-coupling to out-coupling areas according to the grating theory, and with the geometrical tracing of the EPE region to find the optimal layout. For a working wavelength of 532 nm, 40 degrees field of view display, our method is able to fit the layout into a circular shaped area with a minimal radius or to optimize for the minimal EPE area. Furthermore, a web-based interactive application demonstrating the EPE layout calculation is provided as a layout design tool.

Spatial Light Modulators (SLMs) are widely used in several fields of optics such as adaptive optics or holographic displays. SLMs based on Liquid Crystal (LC) devices allow a dynamic and easy representation of two-dimensional phase maps. However, these devices have two main drawbacks, their elevated cost and large dimensions, which prevents them to be used in applications where compactness and low prices are a must. Here we present a more affordable and compact approach based on vertical aligned LC devices, with characteristics of phase modulation very similar to the widely used parallel aligned LC devices. We study the maximal Field of View for visual correction in a see-through system, where the displayed phase map is used for the correction of visual disorders, from refractive errors, to high order aberrations. To conclude we discuss the potential of using this SLM technology and approach as a key component in smart glasses, paving the way for the development of economic, compact and reliable smart glasses for vision correction, among other applications.

Light measurement devices (LMD) are essential tools for developing novel display technologies as well as for quality assurance in display mass production. More complex displays, e.g. near-eye displays (NEDs), might require various LMDs in different development and production phases. Here we examine the comparability of complementary measurement principles and LMDs with respect to sharpness measurements of NEDs. Scanning goniometric MTF analysis with narrow field of view (FOV) and high angular resolution (300 pixel per degree) are compared to one-shot MTF analyses with wide FOV and moderate angular resolution (30 pixel per degree). The two different measurement principles can provide identical or complementary MTF results, depending on the sharpness of the device under test (DUT). Every component within each measurement system has an associated modulation transfer function and, as a result, contributes to the overall MTF of the system. Here, we examine the possibilities to remove the MTF contribution of the measurement devices to achieve fast and accurate sharpness measurements of the DUT. Experimental data from these measurement principles are compared, emphasizing sharpness measurements on LCD and OLED displays of Virtual Reality (VR) headsets. Different sharpness regimes are discussed with respect to comparability and sharpness characterization of DUTs using measurement devices with high and moderate angular resolution.

We demonstrate a hologram-based image projector that expands the field of view (FOV) via wavelength multiplexing while retaining the flexibility of the high-resolution image displayed by a Digital Micromirror Device (DMD). Current Near-to-Eye Displays (NED) for AR/VR/MR applications are often limited in Etendue of the display devices and of the image transfer optics. Moreover, angular resolution and FOV are typically coupled, therefore a large physical package volume is also needed for large FOV optics. Our group previously reported increasing Etendue by custom pulsed illumination sources synchronized to a single DMD. To effectively utilize the increased Etendue along with the often angularly band-limited optical channels, for example, a TIR image guide, additional means to effectively increase Etendue of image transfer medium is needed. To address this challenge, wavelength multiplexed image transfer with extraction volume holograms proved to effectively increase the FOV at least by factor of 2. Along with the volume hologram, the prototype projector system employed 100ns pulsed dual lasers as sources, with a 0.7” DMD operating at 20kHz, coupled with a reflection type extraction volume hologram. In this talk, we address the optical architecture of wavelength multiplexed and FOV expanded projector, and are undergoing quantitative assessments of image quality, such as uniformity, resolution, contrast, and crosstalk.

Waveguide-type optical combiners with holographic optical element couplers are advantageous in augmented reality field, especially for near-to-eye displays, with their structural features such as optical see-through capability, compact form factor, and high diffraction efficiency. The structural features of the waveguide-type optical combiner can also be utilized when it is adopted for floating displays. In this paper, waveguide-type floating display using holographic optical elements with convergent power is proposed as a new method of adopting waveguide-type optical combiner structure. The waveguide configuration enables highly compact form factor. Changing the depth of the floating images can be conveniently applied by using the output holographic optical element coupler recorded with different convex lens. Experimental results demonstrate that waveguide-type floating display with convergent power is feasible as a new method of applying waveguide optical combiner.

In AR, VR, and MR (XR) devices, display quality is determined by user perception of the display when observed as designed into the device. Measurement systems emulate the viewing conditions of an XR user by applying imaging with optics that position the entrance pupil (aperture) at the eye position within the headset. This viewing proximity is common across XR displays, but there is no standard guiding XR device design. Display technologies, projection methods, and headsets continue to evolve. New display measurement equipment must be developed continuously to address the unique viewing conditions of new devices—different angular fields of view (FOV), resolutions, focus ranges, and hardware form factors. Many XR devices require measurement systems with costly custom optics. There is no one-size-fits-all solution and, once designed, a measurement system is not easily modified for different use cases.

The speed of XR innovation calls for equipment that helps manufacturers deploy device-specific display measurement as quickly and easily as possible. Described in this paper, a novel optical design greatly simplifies the process of building imaging systems for near-eye display (NED) metrology. Using commercially available modular components, a variety of optical configurations can be produced and, if needed, reconfigured to match unique specifications of most XR headsets with minimal development time. Features such as electronic focus control and folded (“periscope”) geometries can be incorporated easily to accommodate different focus ranges and form factors. This paper introduces this optical design concept and provides examples of the performance of a modular system for NED measurement.

A glasses-like augmented reality (AR) display with an extended field-of-view (FOV) using a liquid crystal polarization-dependent combiner (PDC) is proposed. The novel PDC consists of two polarization-selective volume lenses (PVLs) that can control the beam path depending on the polarization states. The viewing angle limited by the system etendue can be extended twice. Besides, the PVL is based on patterned liquid crystal which can be thin and flat. The proposed configuration provides a promising solution for the limited FOV issue in glasses-like AR displays.

Augmented reality environments allow users to interact naturally with 3D objects, including robots. Many robots have been used in the automated sector for painting, picking, packing, and palletizing tasks. The Baxter robot is an example of an industrial robot ideal for research and education. Baxter robots can offer multiple benefits compared with regular robots. In this study, we designed an augmented reality system that makes users intuitively interact in 3D environments by using a Leap Motion controller as a hand tracker and performing a basic human-robot coordination task with the Baxter robot. And the Baxter robot with a stereo camera is connected to a Linux computer, which was programmed with python language. The augmented reality world was programmed in the Unity software. The human robot-coordination task consisted of an augmented reality alignment. We asked the subject to wear the head-mounted display and move the hands. Every hand motion was translated into the robot’s limb motion in real-time. The subject and robot had to align two augmented reality markers. Here are two different experimental conditions: visual information activated and deactivated. The subject performed three trials under each condition. The experimental results showed that the subject under the visual information activated mode improved the average time by 70.63 %.

This paper introduces a new method of detecting the rotational direction of an electrostatic, resonant MEMS mirror with in-plane comb (angular vertical comb) fingers driven by parametric excitation, in which its rotational direction strongly depends on its initial condition due to the nature of its actuation method. In AR applications, this ambiguity on its rotational direction could cause a projected image to be flipped in its scanning direction. To avoid this ambiguity, its rotational direction has to be determined before image projection. To do that, its motion-induced current is measured with a transimpedance amplifier (TIA), and the phase of the motion-induced current at its resonance, f0, is determined through quadrature demodulation, while the resonant MEMS mirror is driven at twice its torsional resonance, 2f0, through parametric excitation. We also validate this concept of phase measurement through a series of experiments with a Laser Doppler Vibrometer (LDV) that optically measures its rotational direction and is used as a reference, as well as a Lock-In Amplifier (LIA) that electrically measures the phase of its motion-induced current at its resonance, f0, with respect to the optical reference signal from LDV.

This paper presents a new, low-voltage, and small-form-factor analog front-end (AFE) circuit that measures the torsional angle of an electrostatically-actuated, quasi-static MEMS mirror, while it operates at a frame rate of 50 Hz to 60 Hz or higher, following a given angle profile, such as a sawtooth or triangle profile. The quasi-static MEMS mirror is actuated with two high voltage (HV) differential signals. To enforce this quasi-static MEMS mirror to track a given angle profile or trajectory with minimum error, the MEMS mirror is driven by feedback control that requires a sensor capable of measuring its mechanical or optical angle in real time with high enough bandwidth and sensitivity. Hence, we design and implement the angle sensing circuit (ASC) that meets the low-power, low-cost, and small-form-factor requirements to reduce power consumption, size, and weight for AR applications. This ASC consists of a variable gain amplifier (VGA) and an envelope detector that operates at 3.3 V and draws about 1 mA during operation.

We demonstrate for the first time the replication of a pupil in a curved waveguide with high resolution. Typically, replication of a pupil in a curved waveguide is hampered by split images, occurring because rays from the same pixel hit the eye pupil at different angles. We have identified and couple into eigen modes of the curved waveguide. As a result, light inside the waveguide has a continuous wavefront and pupil replication is achieved without split-images. The curved waveguide is used in an AR demonstrator, sustaining high resolution, great transparency, low distortion seethrough. The waveguide combiner is compatible with all projector techniques, in RGB, and can have any radius of curvature. As such, the waveguide can be integrated into slim prescription glasses using Trulife's patent-pending curved design.

In ever-evolving XR device designs, constant improvements are being made to the performance of the various optical and sensing components. One of the key materials to enable these improvements are ultra-high refractive index resins that not only provide a boost in efficiency in the visual light spectrum (400 – 700 nm), but also near-infrared (NIR) wavelengths (800 – 1550 nm). Today, using Pixelligent’s PixNIL™ Titanium Dioxide (TiO2) nanocrystal formulations in acrylate-based resins systems, one can create optical architectures and planarized films with refractive index values of 1.95 to 2.00 at 550 nm. These formulations reach such high RI values with the proper tuning of material components and process conditions at specific film thicknesses. Films can be deposited by way of spin-coating and inkjet printing, and, depending on the type of formulation, are imprintable to yield specific structures for various diffractive optical elements such as waveguides. In addition, these films maintain very high transparency in the visible light spectrum and demonstrate very low haze and low absorbance.

We propose and demonstrate a Field of View (FOV) expander, realized by wavelength-division multiplexing (WDM) by using volume holographic gratings (VHG). This holographic based FOV expander is low cost and easy to manufacture, which we demonstrate through our updated in-plane writing setup. We present our experimental results of diffraction efficiency measurements and image analysis which show high SNR and angular resolution. This holographic based FOV expander based on VHGs have potentials and can be implemented into AR/VR systems.

AR/VR/MR systems challenge projector systems to increase their refresh rates, resolutions, and Field of View (FOV). Projecting a 1080p image with a 100Hz frame rate by a Laser-Based Scanner (LBS) projector necessitates a very fast MEMS scanning mirror (with a scanning frequency of over 100kHz). Designing and manufacturing such MEMS devices for raster scan mode is extremely challenging, yet to be achieved. As a result, Lissajous scanning mode was suggested, presenting an improved pixel coverage of the FOV with relatively lower frame rates. Nevertheless, even with reduced frame rates, the additional challenge raised by using Lissajous scan mode with existing laser drivers switching times. At such scan rates, a pixel ‘on’ time is about 2.5ns. For common 200MHz lasers drivers, this implies a horizontal jitter of about +/-0.5 of a pixel. Addressing the challenges of projecting HD with MEMS mirrors, a new paradigm enabling to project of HD video using a feasible MEMS scanning mirrors technology is suggested by using interlaced raster scan mode.

Interlace raster projecting allows using a MEMS mirror with a much lower scanning frequency (of a few tens of kHz) while having improved pixel coverage of the FOV. At such scanning rates, the pixel jitter level goes to 0.05 of a pixel for a 200MHz laser driver. Moreover, the interlace projection allows increasing the vertical resolution on the expanse of the frame rate. However, using interlacing with ramp scan mode for the vertical scan could affect the non-uniformity of pixel coverage due to line crosshatching. The same phenomenon is observed in Lissajous scan mode. The presented new paradigm suggests an interlace raster projection, with parallel lines of each projected field, with low vertical jitter. For this purpose, the slow vertical ramp motion is replaced with a fast stepped motion for each projected line. Maradin presents a novel MEMS mirror controller applying the interlace scan approach to enable the extreme requirements of the AR/VR/MR displays.

XR (Extended Reality) glasses have many problems to overcome, such as MTP (Motion-To-Photon) delay and optical artifacts. We introduced a hardware and software co-design approach to solving the MTP issue. We applied it successfully to FSC (Field Sequential Color) LCoS (Liquid Crystal on Silicon) micro-display controller SoC (System on a Chip) design to reduce the MTP latency significantly down to less than 5 msec and to mitigate color break-up. A commercial AR (Augmented Reality) device available in the market embeds our solution. Our near-zero MTP delay display solution shows that smartphone/compute device tethered XR viewers or HMDs (Head Mounted Displays) could enhance user experience along with SLAM (simultaneous localization and mapping) or image processor. A localized optic artifact compensation scheme has also been applied to a waveguide optic, reducing AR optic design iterations.

Humans can flexibly adapt to visual stimulation, such as spatial inversion, in which a person wears glasses that display images upside down for long periods. As with adaptation to spatial inversion, it is assumed that humans can also adapt when given the ability to control both eyes independently. To investigate the flexible adaptation of the human visual system, we developed a system, “Virtual Chameleon,” that provides independent fields of view to both eyes of a human user. The Virtual Chameleon consists of two independently controlled CCD cameras, a head-mounted display and a controller. The use of Virtual Chameleon posed binocular rivalry to visions of the users based on oral reports from users. In addition to it, the users often stated that the use of Virtual Chameleon induces exhaustion and malaise. In this study, we developed a new Virtual Chameleon for fundamental experiments, which is equipped with an omnidirectional mobile device and can be controlled by touchpad devices to investigate workload induced using Virtual Chameleons. We conducted a fundamental investigation using critical flicker fusion frequency as evaluation criteria. Six healthy volunteers (21-22 years old) with normal or corrected-to-normal vision participated. The results show the use of Virtual Chameleon gives the user exhaustion with lapse time and binocular rivalry induced by similar two different images especially causing greater fatigue.

The ultimate challenge for next generation XR devices, like smart glasses, is the integration of optics and electronics in a lightweight and unobtrusive device. On this path, miniaturization is an essential key factor, in which adhesives and functional polymers play a vital role. The latest generation of adhesives and functional polymers are not only used for conventional bonding tasks but can exhibit special properties to fulfill additional jobs such as active alignment of projector devices or in- and outcoupling of light into the optical waveguide by imprinting a polymer grating structure directly on the glass substrate. Next to the additional functionality, the clear advantage of a polymer is the combination of a comparably low mass to minimize weight, dispensing in arbitrary shapes for maximum flexibility and miniaturization and finally easy and fast processing, which results in high yield and units per hour in a mass production process.