Objectives of new infrared smart sensor systems-and their focal plane processing requirements for target detection are presented. Various image processing techniques for enhancement of target-to-noise ratio by performing background clutter suppression are surveyed, including a family of statistical target estimation techniques being developed at the Naval Postgraduate School. Highlights of their results in processing a set of multiple frame infrared test images are presented using statistical spatial filters for single frames of image and statistical temporal filters and combined statistical spatial/temporal filters for multiple frames of images.

By applying the concept of analogue matched filtering to spatially correlated background scenes and making the assumption that they are characterized by a first order Markov process, a simple but powerful digital filter is derived. Direct inspection of this filter results in some useful insights into the process of spatial filtering. The usefulness of this filter is further demonstrated by showing its clutter reduction capability on a set of five measured infrared scenes representing a wide variety of terrain and weather conditions. Also, matched digital filters are obtained directly for each of the five measured scenes making no assumption about the spectral content of the backgrounds. Although the resulting filters sometimes differ in form from those obtained using the first order Markov process assumption, their performance is shown to be nearly identical, proving that this assumption provides a useful model for IR backgrounds. Finally, a technique is developed to construct matched filters to detect resolved targets whose size and intensity distributions are known only statistically. Examples of such targets are constructed and the clutter reduction properties of these filters are quantitatively evaluated.

A critical problem in the design of signal processing systems for mosaic sensors is the development of algorithms suitable to the detection of dim targets in the face of several significant noise components. These algorithms must provide near optimum performance for given constraints on memory and computing capacity. A general frequency domain theory of the detection of dim targets against an earth background by passive mosaic sensors is given. Simple temporal filtering algorithms are discussed and the results of comparative analysis of these algorithms are presented.

Constant-False-Alarm Rate (CFAR) processors are used when the interference characteristics are not known a priori or change with time. When the unknown characteristic is only the level of the interference, a common CFAR implementation is the normalizer. This processor obtains an estimate of the interference level by arithmetically averaging the outputs of the resolution cells adjacent to the test cell. The test cell output is divided by the average and the normalized output is independent of the interference level. Therefore, a CFAR action is obtained. The penalty associated with the need to estimate the interference level is that the signal-to-interference ratio (SIR) required for target detection increases over the SIR value required for a known level noise background. Conventional normalizing C FAR circuits use the same configurations when detecting targets in interference regions and in clear regions. The CFAR penalty incurred in the clear region can be reduced by using processors that recognize the region is clear so that normalization is not necessary. An analysis of the target detection performance for a particular modified CFAR processor is given. It is shown that the decreased CFAR penalty in the clear is coupled with an increase of false alarm rate in the clutter regions.

A new least-mean-square filter technique is defined for signal detection problems. The technique is applied to a scanning infrared surveillance system operating in poorly characterized but primarily low-frequency noise clutter environments. Near optimal performance is predicted both for continuous time and sampled data systems.

The maximum likelihood method is applied to the problem of extracting the correct sequence of spatial patterns from the corresponding sequence of measurement frames corrupted by background noise. Each measurement frame is modeled as the sum of a pattern matrix and a background noise matrix. The model proposed for the statistics of the sequence of background matrices results in jointly Gaussian elements whose correlations are product separable in row, column, and frame indices. The logarithm of the likelihood function is computed and involves a matched filtering operation on the measurement frames, which acts to suppress the background relative to the pattern. Because of the product separability in the back-ground element correlations, this matched filtering operation is accomplished by pre- and post-multiplying the measurement frames by the inverses of the background row and column correlation matrices, respectively. Thus, the measurement frames are operated on in their original matrix format without resorting to stacking. Applications include the detection of targets in background noise.

This paper presents an overview of target tracking techniques. It includes an assessment of presently fielded military trackers and their shortcomings; present state-of-the-art (SOA) tracker capabilities and their shortcomings; and capabilities of future "intelligent" target trackers.

Under contract to the Army's Night Vision and Electro-Optics Laboratory, Westinghouse has been investigating the design, test, and implementation of a set of algorithms to perform intelligent target tracking and intelligent target homing on FLIR and TV imagery. The focus is a system of algorithms which can quantitively identify a target and its present background, but can acknowledge the approach of a new background in the target's path, characterize this new region, and intelligently predict the target's signature before it enters a new domain. The operating environment for the system is a high clutter background associated with ground targets. Three types of obscurations are considered: (1) the target enters a background very similar in gray scale to it, (2) the target passes behind an obscuration such that some identifiable portion is always visible, and (3) where the target passes behind a thin screen of obscuration and portions of the target can be seen on each frame but portions vary from frame to frame. The system may be viewed as a target cuer operating in conjunction with a target signature predictor and high speed frame to frame registration. The review describes some of the work performed in the first 6 months of a 19-month program.

The Information Adaptive System (IAS) is an element of the NASA End-to-End Data System (NEEDS) Phase II, and is focused toward on-board image processing for the NEEDS program. Because the IAS is a data processing system, it is directly applicable to "smart sensor" programs and, as such, represents a preliminary step toward the development of smart sensors. Response nonuniformity correction, geometric correction, data set selection, data formatting, image feature analysis and adaptive system control are some of the functions planned for the IAS. The paper will present the preliminary design of the Information Adaptive System and will discuss plans for its development and demonstration.

Currently, NASA is planning to develop a new generation earth resources survey sensor which uses solid-state line arrays of detectors in a pushbroom scan mode. Precision radiometric calibration of data from detector arrays with thousands of elements presents one of the more difficult systems challenges. In the general case, each element exhibits its own characteristic signal at dark, and its own transfer function (volts out vs. light in). For monolithic arrays of detectors, this pattern of response variations must be accepted as a fact of life; i.e., individual detectors are not accessible to be individually "tweaked" to match responses. A program was undertaken to design, fabricate, and test a real-time hardwired data preprocessor which applies a calibration normalization to each detector in a 576-element linear photodiode array. The raw video data was quantized to 8 bits. The particular algorithm selected was driven by the large element-to-element variations for this particular array. The most significant bit in dynamic range was lost for a majority of the elements leaving an effective dynamic range of 127 counts. A limited number of elements had a dynamic range as low as 43 counts. After normalization and scaling back to 255 counts, various calibration problems were uncovered: (1) due to system noise in recording the calibration tables; (2) due to thermal drift; and (3) due to the original quantization process. In this experiment, noise and thermal drift led to fixed errors in the normalization of responses on the order of ±10 counts out of 255 counts for many of the detectors. This level of coherent error is of course readily observable as a stripe in a pushbroom scan image. Quantization thresholding was a second order error, and was not separable in the test images. These results lead to recommending: focal plane cooling to minimize dark leakage currents and improve dynamic range; the use of some form of multiline averaging to reduce the noise in the calibration tables; and in the case of systems with very large elemental offsets, to use analog offset corrections ahead of digitization to maximize dynamic range for the high offset elements. Application of this experiment to potential spaceflight hardware indicates a significant increase in electronic hardware complexity, as well as some power penalty.

This paper discusses the two-dimensional infrared focal plane and signal-processing technology that will make thermoelectrically cooled staring arrays available for smart-sensor applications. Infrared detector materials being considered are mercury-cadmium telluride and indium arsenide-antimonide. The operation of these materials at elevated temperatures (195 K) and the interface with the focal-plane processor are described. The detector must have high zero-bias resistance and low noise when reverse biased in order to achieve a high signal-to-noise ratio and efficient processor interfacing. The focal-plane processor must accommodate both the saturation current from an elevated-temperature detector and the large background current associated with along integrationtime. Focal-plane component uniformity is critical to the full real-ization of the sensitivity advantage afforded by staring sensors. Low-power compensation electronics are required to equalize any residual nonuniformities in both gain and offset. The status of the technology in each of these areas and current programs to address the significant problems are examined.

To utilize the full performance advantages of staring infrared imaging systems currently under development, it is necessary to compensate for the characteristic fixed pattern "noise" which is present at the output from these IR focal planes. Since many of the applications for staring sensor systems require low power dissipiation configurations, it is necessary to develop automatic non-uniformity compensation electronics which have much lower power dissipation requirements than conventional digital compensation techniques. This paper discusses the sources of the non-uniformities and describes the typical characteristics of elevated temperature staring arrays. An analysis is given which shows how detector/CCD electrical coupling techniques strongly influence the compensation implementation and finally a review of circuit configurations for the compensation function will be given which show that very low power dissipation circuitry can be developed which meet the performance power dissipation requirements.

Nonscanned (staring) thermal imagers have many inherent performance advantages if the fixed-pattern noise caused by elemental nonuniformities can be compensated. Staring array systems are more efficient than mechanically-scanned FLIRS because all detector elements are always integrating photons from the scene, except during short readout periods. The successful application of staring arrays in imaging systems requires compensation of the elemental dark current and responsivity nonuniformities in real time. A novel approach to implementation of the compensation electronics has been developed using a microprocessor based system. The system is programmable to allow for applications to many types of staring sensors and system applications where other special functions are also needed. This unit has been demonstrated using simulated video noise and has shown compensation of array nonuniformities to better than 0.1 percent.

A smart sensor is defined as one that measures the appropriate quantities with the required accuracy and extracts the maximum amount of information from those measurements. In that context, an imaging sensor system, consisting of an imaging sensor, a reference image, and an image matcher, is discussed. The interaction of the elements of the system are explored, highlighting technology shortfalls and the problems of realizing the design in hardware are touched on. The basic deficiency in developing an optimum sensor is the lack of an analytical model for the images and the matching process.

ENSCO, Inc. is developing a sensor for the Maryland State Highway Administration that will measure road surface texture from aboard a moving vehicle. In the future this sensor will enable real-time classification of pavement texture, which enables the speed dependency of skid resistance to be determined. The sensor consists of a projector, detector, and electronic processor. The projector projects an extremely short-duration slit of light vertically downwards onto the road surface. This results in an illuminated single-line profile of the surface texture. This texture profile is detected by a vidicon camera that views the surface at a 45 degree angle. The camera image is digitized by the processor and stored on digital tape to facilitate subsequent computer processing of the texture data. The sensor system is designed to measure four-inch long samples of surface texture with a resolution of 0.01 inches at vehicle speeds up to 40 mph.

The Night Vision & Electro-Optics Laboratory recently completed an extensive field test at Grafenwoehr, Germany near the West German-Czechoslovakia border. The major objective of this test was to derive performance criteria for various infrared sensors operating in typical, non-benign battlefield environments. The scenarios included a moving and stationary tank target at several kilometers range with the viewing path degraded by atmospheric conditions and annihilation or hard point strength artillery barrages. The artillery impact area was directly forward of the sensor platform. Imagery taken from a TV compatible FLIR will be processed by local area contrast enhancement algorithms in an attempt to permit continuous target detection during munition's explosions. The imagery will also be processed by several target extraction (segmentation plus low level classification) techniques to ascertain algorithm performance under typical battlefield conditions. Degradations in the viewing path include atmospherics, explosives, earthen particles, airborne dust, and explosion craters.

The Rockwell Pattern Matcher (RPM) is a feature based technique which has been demonstrated on multi-wavelength imagery. Feature extraction and image matches have been performed on imagery from 3 cm Synthetic Aperture Radar (SAR), 3.2 mm active radar, 10.6 u active laser, 8-12 u passive IR and optical photographs. The feature detection, recognition and image matches were performed on imagery of the same wavelength as well as on those using a different wavelength as the reference scene. The capability of the RPM algorithm to operate on images generated from a wide spectrum of wavelengths allows its utilization for a variety of applications. The versatility and robustness of the Rockwell algorithm gives rise to the advent of "smart sensors" to achieve functions not previously attainable.

The equivalence of 3 x 3 linear discrete convolution filtering to spatial frequency domain multiplication is reviewed in this paper. DPCM (Differential Pulse Code Modulation) is simply a low pass filter in the frequency domain. More complex 3 x 3 filters allow the choice of a band stop locus in the frequency domain in order to reduce unwanted spatial frequency components. This real time spatial filtering is readily implemented using a multitapped CCD delay line coefficients on-chip with a CCD imager.

Bandwidth compression schemes have found useful application in preventing the jamming of transmitted information. Such data reduction methods are particularly needed in the case of RPV imagery transmission. The Night Vision & Electro-Optics Laboratory has initiated funding of separate studies to access the feasibility of obtaining 1000 to 1 and 10,000 to 1 compression ratios. The achievement of such ratios necessitates a considerable degree of local intelligence in which specially selected scene information only, is transmitted. The 1000 to 1 concept involves sending with fidelity, just such information that is deemed to be of probable value to the RPV mission. In the case of 10,000 to 1 ratio the image is segmented, classified and only coordinates and image descriptors are transmitted.

A sensor has been designed to provide real-time detection and recognition of 3mm wires at a range of 300 meters during nighttime helicopter flight operations. An Army-sponsored program to demonstrate such automatic wire detection and warning for Nap-of-the-Earth (NOE) helicopter missions is currently in progress. Wire or wire-like objects are electro-optically detected and then recognized by a pattern recognition technique. The recognition algorithm is accomplished within 50 msec of the first wire detection indication. A flyable exploratory development WOWS model, consisting of a scanning laser transmitter, electro-optical receiver, real-time processor and display unit is described.

Applications of image processing to FLIR systems include image enhancement, to improve the imagery displayed to the observer and automatic target cueing, to reduce the operator's search time. The Prototype Automatic Target Screener performs automatic real time detection, recognition and cueing of tactical targets. It also incorporates DC restoration to correct for artifacts introduced by the common module FLIR; and adaptive contrast enhancement to optimally use the display dynamic range and eliminate the need for continual operator adjustments. The PATS system is designed to interface to standard 525-line and 875-line TV formats and perform real-time processing. Decisions on target classification and location are updated every 1/10 second and displayed by means of symbology overlays on the operator's display. Three levels of classification provide optimum performance at low computational cost. The first level rejects clutter. Potential targets are further classified into one of six categories at the second level. These decisions are continually correlated over several frames and the combined decisions are displayed as cues. The PATS architecture features charge-coupled devices to perform many of the high speed functions required for image segmentation and first level feature extraction. It incorporates a bit-slice micro-programmable digital processor and frame memory for speed and flexibility in second level feature extraction and classification.

Needs for smart sensing in terrestrial and atmospheric remote sensing are discussed and related to current technology research and a scheduled Shuttle experiment. Space Shuttle offers unique opportunities to evaluate new concepts in sensor development and methods for superimposing data from different types of sensors taken either simultaneously or at different times. A time-phased technology approach is outlined involving a series of Shuttle-borne experiments to develop Earth feature identification and tracking technology. The first phase includes a Feature Identification and Location Experiment (FILE), undergoing fabrication and scheduled for flight on the NASA Shuttle (STS2/flight OSTA-1) in 1980. The experiment objective is to evaluate a technique for autonomously classifying Earth features into four categories: bare land; water; vegetation; and clouds, snow, or ice. The experiment package, experiment concepts, and plans for evolution of the FILE-related technology to provide discrimination among clouds, snow, and ice are described. The technology development plan, beyond feature identification/classification and cloud detection/discrimination, leads to capabilities for pointing instruments to predetermined sites, reacquiring Earth features or landmarks, and tracking features such as coastlines or rivers. Technology concepts are discussed relative to an overall system transfer function, and the technology development status is outlined.

A new two-dimensional DEFT sensor (for Direct Electronic Fourier Transform) for multi-processing will be described. Operating at a center frequency of 100 MHz, it is capable of resolving 1800 unique spatial Fourier image components in a random access mode. As a Fourier transformer it can identify vector spatial frecuencies, their amplitude and phase. By suitable change of input functions, the device can be operated as a spatial raster scanner, Hadamard transformer, matrix multiplier, or convolver. The device achieves this flexibility by virtue of producing the spatial integral of the product of two electronic signals with an optical signal.

It is possible to form an optical analog of a set of N analog voltages by passing an optical plane wave, confined in an electrooptic waveguide, under a set of N electrodes to which the voltages are applied. In the limit in which diffraction is ignored, the wavefront of the emerging guided wave will have superimposed upon it N discrete phase shifts. We have designed and are in the process of fabricating a number of processors which operate upon analog voltages encoded in this manner. These include a simple comparator in which incoming data are compared to a holographic record of the optical analog of a reference set, a device which is designed to perform an identification function by comparing each set of data to a library of reference sets, and a "smart" system based upon holographic self-subtraction, in which the processor is able to independently adapt to changes in background information. A laboratory model of the first of these preprocessors has been built and tested, the second is under development and experiments designed to define the operating parameters of the self-subtraction preprocessor are in progress. The principles of operation, potential applications and progress towards the implementation of the devices in an integrated optical configuration are described.

This paper outlines the main features of a design presently being completed in study form for a new generation of Polaris star sensors. The sensors are intended for attitude sensing in geosynchronous satellites; the study is being conducted at Sira Institute on behalf of the European Space Agency. The sensors will use a CCD imager for compactness, reliability and geometrical stability. The signal processing includes background subtraction, response and dark-level non-uniformity correction, and selective acquisition of digitised image data. The digitised data are used to calculate energy distribution in the image allowing star position to be located to greater accuracy than permitted by the geometric resolution of the array. The target accuracy for the sensor is 30 seconds of arc in a 4 degree field of view, to be capable of improvement to 10 seconds of arc with calibration: this has to be achieved within very tight constraints on mass and power consumption. High reliability is required over a five-year mission life. Some of the trade-offs leading to an optimisation of the design within these limitations are discussed. The results of experimental evaluation of critical detector characteristics carried out at the Institute are outlined.

A panoramic scanner using Time Delay and Integration Charge Coupled Device (TDI-CCD) technology that is presently directed towards a submarine periscope mission scenario is discussed. The system utilizes a large format TDI device coupled with real-time video processing techniques, i.e., dark signature removal, response non-uniformity correction, on-chip exposure control and background subtraction. This data processing, in combination with a real-time, digital falling raster display, produces a man-machine system capable of detecting and recognizing target silhouettes against the horizon under starlight illumination. The background subtraction techniques also provide the capability for achieving low contrast image enhancement under daylight illumination conditions. System details and experimental performance data are presented.

The potential use of charge couple device (CCD) programmable digital/analog correlators for multispectral data classification is discussed. CCD digital/analog correlator technology and the experimental evaluation of a 32-stage 4-bit test device are presented. The design of an IC for use in a multispectral classification system for 16 sensors and 8 bit accuracy is reviewed.

Surface acoustic waves excited in a Si-SiO2-ZnO layered structure can produce a traveling electric field in the silicon substrate. Charges stored in the traveling potential wells can be transferred at high speed and density and with less complexity than conventional charge coupled devices. The monolithic structure under investigation for the SAW-charge transfer device consists of a silicon substrate, a thin silicon dioxide insulating layer on top of which a ZnO piezoelectric film is deposited by sputtering. The surface acoustic waves are excited by interdigital transducers. The signal charge is injected into traveling potential wells that travel with the velocity of sound. The presence of a thin shorting plate placed on the ZnO film, over the charge transfer region can enhance the acoustoelectric potential at the Si-SiO2 interface, thus resulting in a more efficient device. An 80 MHz, 2μ-second SAW-CCD has successfully been fabricated. An optical application utilizing such a structure is proposed. It can be used in place of a conventional interline transfer design. Surface acoustic waves are launched before the charges are transferred from the sensor region to the transport region.