1.

INTRODUCTION / PROBLEM DESCRIPTION

Pushbroom scanners use a linear image sensor in the focal plane of the optics. This sensor scans the image perpendicular to the flight path. To scan the image in the flight direction, the orbital motion of the satellite is used, which results in the corresponding motion of the projected Earth image in the focal plane of the camera. Compared with area or matrix scan cameras, pushbroom scanners have following advantages:

At the same time, pushbroom scanners have one significant disadvantage: they require high attitude stability during scanning of the image. If the satellite attitude is not stable, the image motion in the focal plane will not be steady what results in geometrical distortions of the obtained images. For example, an image with ground resolution of 2.5 m taken from satellite on a 500 km orbit in presence of attitude change by 5 μrad (1 arc second) will show noticeable image distortion (0.5 pixel). The spectrum of attitude disturbances spreads from 1/orbital period to few hundreds Hz. Low frequency disturbances are mainly caused by thermal deformations of the satellite structure, atmospheric drag variations and other factors, related to orbital position. High frequency disturbances (vibrations) are caused by unbalance of reaction and momentum wheels and operations of other mechanical systems onboard the satellite.

To some extent, the distortions can be corrected using the record of attitude disturbances during image scan, measured by high precision Inertial Measurement Unit (IMU) onboard the satellite. Such solution, however, will significantly increase the mission cost. Besides, it is not suitable for correction of high frequency distortions: high accuracy of IMU is usually obtained at a cost of bandwidth limitation; moreover non rigidity of satellite structure can result in significant differences of high frequency attitude records taken at non collocated mounting positions of gyros and the pushbroom camera.

Image distortions can be also corrected by geo-referencing of the obtained images by matching them with the previously made images of the same area. This solution does not require installation of additional devices onboard, it generally provides high correction accuracy (considering also the ground relief variations and perspective distortions), but it is suitable only for correction of low frequency distortions. Matching of the images (determination of local shifts) with high accuracy generally requires 2D correlation of sufficiently large image blocks. This determines the averaged shifts of blocks of lines without the possibility to measure the individual lines shifts. Large high frequency disturbances can also decrease the accuracy of matching or even makes the correlation of image blocks impossible.

To solve this problem the SMARTSCAN imaging concept has been developed by TU Dresden, Institute of Automation [1, 2], which is based on the recording of the actual motion of the focal plane image during the frame scan with additional focal plane image sensors and subpixel image motion tracking in real-time using an onboard optical correlator. The image distortions, caused by the satellite attitude instability, are then corrected on base of this image motion records by a ground computer.

The proposed solution allows to expand the area of pushbroom scanners application to satellites with moderately attitude stability, micro satellites or to use them as secondary imaging payload for low orbit communication satellites or the Space Station.

2.

SMARTSCAN PRINCIPLE

The principle of SMARTSCAN operation is based on real time recording of the actual image motion in the image plane. Such record, made directly in the focal plane of the camera, automatically considers in-situ all factors, which cause the unsteadiness of image motion and corresponding image distortions. Its accuracy is linked with the resolution of the camera – the errors of such record are measured in image pixels (not in arc seconds) and will be the same for cameras with any ground resolution.

The image motion record is a record of the focal plane image motion with respect to the linear sensor. To simplify the explanation, it is possible to suppose, that the image is fixed and image sensors are moving with respect to it. Then the image motion record can be interpreted as the trajectory of the linear sensor in the plane of the fixed image (with respect to the first line) or as a sequence of linear sensor positions in the moments of the lines exposure (Fig. 1).

Fig. 1.

Image motion record

With the actual position of every line is known, the actual position of every pixel of the distorted image with respect to the first line is calculated and a correct image (corresponding to steady image motion) can easily be reconstructed by standard 2D interpolation.

Such simple approach works correctly in absence of perspective distortions (non-distorting nadir oriented camera, no surface relief). In real case the “ideal” (steady) image motion at a certain point of the image depends on the local distance to imaged surface. If the camera is not nadir pointing and/or the imaged area is not flat, attitude and surface relief data should be considered for accurate image correction.

Best results can be obtained by combining the SMARTSCAN approach with matching of the obtained images with previously obtained, geo-referenced images of the same area. In this case first the SMARTSCAN image motion record can be used to correct the high frequency component of image distortions and to make the obtained image suitable for block-matching with the reference one. Low frequency distortions will be then corrected on base of matching with reference images. If no reference images of the target area are available, low frequency distortions can be corrected with high accurate attitude information from an IMU. Required high accuracy of attitude data can be obtained in this case at a cost of bandwidth limitation, by using the slow and accurate IMU or by filtering of attitude data.

Practically the recording of the focal plane image motion can be realized by 2D correlation of sequential images, taken by auxiliary matrix image sensors in the focal plane. A matrix sensor will be exposed at the same moments as the linear sensor, i.e. for each image line taken by linear sensor, each of matrix sensors will produce a small 2D image. To measure the (yaw) rotation of focal plane image, at least two matrix sensors should be used. The number of sensors can be further increased to provide some redundancy, reasonably four such sensors can be used (Fig. 2).

Fig. 2.

Possible configuration of the image sensors in the focal plane

The image shift between the exposure moments t0 and t1 of neighboring lines can be determined by 2D spatial correlation of the matrix sensor images, taken in the same moments as the linear sensor exposing (Fig. 3).

Fig. 3.

Image shift determination with 2D correlation

The position of the correlation peak corresponds to the image shift during the time interval t₀ - t₁. The image motion record is then recalculated from the shift vectors by geometrical transformations. Image shift determination based on correlation methods is extremely noise-resistant, gives subpixel accuracy and does not require any specific features on the image. As a drawback, the 2D correlation of the images requires a very large amount of numerical calculations.

3.

REQUIREMENTS TO IMAGE CORRELATOR

Necessity of image shift determination for every line of the linear scanner image sets challenging requirements to the correlation rate. A satellite on a 500 km orbit has a ground track velocity of 7059 m/s [3]. With ground sampling distance of 2.5 m this corresponds to a lines frequency of 2824 lines/s. With four matrix image sensors this number should be multiplied by 4, what results in processing rate requirements of 11300 correlations/s.

Essential factors are also the correlation stability, the accuracy of the image shifts determination and the range of measurable shifts. All these characteristics are closely linked with the size of the correlated fragment.

Correlation stability here is understood as an ability to determine the shift between the correlated fragments for possibly larger range of image content (also with weak image textures and in presence of noise) and to detect reliably the cases when the correlation is not possible. Both abilities are linked with the size of correlated fragments. Generally a weak image texture results in degradation of the correlation peak which becomes finally comparable with the surrounding correlation image content and therefore difficult to detect. Increasing of the input fragments size the makes the peak thinner and higher with respect to correlation image, what improves dramatically its detectability and therefore the stability of correlation.

Accuracy of the shift determination also improves with a larger size of correlated fragments as the higher peak is much less affected by correlation image noise.

Finally the measurable shifts range is directly linked with the fragment size: with large fragments proportionally large mutual shifts can be allowed with the same percentage of images overlapping.

Considering the above mentioned properties, the size of correlated fragments should be as large as possible. For practical realization reasons, however the data transmission capabilities should be also considered. For a high speed SpaceWire link the data rate is limited by 400 Mbps [4]. With one SpaceWire connection per matrix image sensor, the frames rate of 2824 frames/s and radiometric resolution of 8 bits/pixel a fragment size of 128x128 pixels is possible.

Taking into account these high requirements to data processing rate (11300 correlations of image fragments of 128x128 pixels per second – 4x400 Mbps input data rate) and the limitations of data processing resources it is not possible to produce this image motion record by digital data processing onboard the satellite. It is also not possible to transmit the matrix sensors images to the ground station for further processing due to very high data rate (1600 Mbps).

To provide the required real-time onboard performance, it has been proposed [1, 2] to perform the image correlation with an onboard optical correlator.

4.

OPTICAL CORRELATOR

A Joint Transform Optical correlator is an optoelectronic device, capable of extremely fast image processing due to application of high parallel optical computing technology. Its operation is based on the natural feature of the lens to produce a 2D Fourier transform of the image. This diffraction-based phenomenon is used to perform 2D correlation of two images by two sequential optical Fourier transforms (Fig. 4), according to the Joint Transform Correlation principle [5].

Fig. 4.

Optical correlator principle

With the 2D-Fourier transform performed optically, the time and power are consumed mainly for entering the images into the optical system and reading the results of the optical transform. This makes possible to reach a high image processing rate with limited power consumption. This advanced technology and its applications have been studied during last years at the Institute of Automation of the Technische Universität Dresden [6, 7]. Special design solutions have been developed to make the optical correlator robust against mechanical loads and to eliminate the need for precise assembling and adjustment [8, 9].

To prove the feasibility of SMARTSCAN concept a hardware model of optical correlator has been manufactured (Fig. 5).

Fig. 5.

Hardware model of optical correlator

To cope with the limited funding and to save development time, it uses standard video cameras as the image sensors. This limits the image processing rate to 60 correlations per second. The model has been successfully tested in airborne flight test campaign during summer 2002 [10].

Recent research has given solutions for a very compact realization of such a device, suitable for onboard installation on a spacecraft. The realization concept of the optical correlator module is based on customized miniature optoelectronic components: a reflective Spatial Light Modulator (SLM) and a Spectrum/Correlation Image Sensor (SCIS). Both optical Fourier transforms will be performed sequentially with a single optical Fourier processor. To reduce the size, a folded optical system design on the base of glass block is currently being considered (Fig. 6).

Fig. 6.

Optical system layout

The coherent light, emitted by a laser diode, reflects from the aluminized side of the block and illuminates the SLM surface via the embedded lens. The beam, reflected from SLM, is modulated by the input image (pair of image fragments to be correlated). It is focused by the same lens and forms (after intermediate reflection) the image of the Fourier spectrum of input image on the SCIS surface. This image is read and sent to SLM for a second optical Fourier transform, after which the correlation image is obtained. Mutual shift of correlated fragments is calculated from the correlation peaks positions within the correlation image. This operation will be performed directly inside the SCIS chip, what makes possible to simplify the digital electronics and to reduce significantly power consumption. Using unpackaged optoelectronic components and Chip-on-Board mounting, the optical system can be realized within the volume of 80x60x30 mm. Digital image processing operations (interfaces, preparing the image fragments for correlation, control) will be realized on a single FPGA and 3 RAM modules. Expected performances of the whole optical correlator module are summarized in Table 1.

Table 1.

Expected performances of the onboard optical correlator

5.

AIRBORNE TESTING

Airborne flight tests have been performed with the model of SMARTSCAN system, including the camera and optical processor models, a portable PC and special software for control and image processing. The general configuration of the test equipment onboard the airplane is shown in Fig. 7.

Fig. 7.

Test equipment configuration onboard the plane

The tests have been performed at the DLR (Deutsches Zentrum für Luft- und Raumfahrt) facilities in Oberphaffenhofen near Munich [10]. The test equipment has been mounted onboard a small, single engine turboprop aircraft (Cessna Grand Caravan – Fig. 8). The camera model was mounted in one of the bottom port of aircraft, all other equipment – in a special rack in the cabin (Fig. 9).

Fig. 8.

Cessna Grand Caravan

Fig. 9.

Test equipment mounted onboard the plane

The flight altitude was approximately 2400 m with a velocity – 240 km/h (67 m/s). The ground resolution of the camera model was 0.45 m per pixel.

During imaging the airplane produced considerable attitude disturbances and vibrations. The distorted image from the linear sensor has been loaded to the PC. Two streams of small images from matrix sensors have been processed in real time by the optical processor; the resulting image motion record has been loaded to PC too. After finishing of the imaging session the distorted linear sensor image has been corrected on base of the stored image motion record.

As a direct result of the tests, the complete end–to–end functionality of SMARTSCAN imaging system has been demonstrated with real remotely sensed Earth images under airborne flight conditions. Totally nine imaging sessions have been performed, each with a linear sensor image (2048 x 2048 pixels) and the corresponding image motion record produced by real time processing of the matrix sensor images by optical processor onboard the airplane. The linear sensor images, obtained during the tests, are distorted due to airplane attitude instability, vibrations and flight direction and velocity changes. An example of such a distorted raw image is shown in Fig. 10.

Fig. 10.

Distorted linear sensor image

The image motion records for each imaging session represent the motion of both ends of the linear sensor with respect to the image. The starting points of these records coincide with the sensor position at the moment of exposure of the first image line. Fig. 11 shows the dependency of the position of one end of the linear sensor from the number of obtained lines (two components – along and perpendicular to the flight direction). The motion record for the other end of the linear sensor is not presented in Fig. 11 as it is practically equivalent to the first one (the rotational – yaw - component of image motion was relatively small). The record, presented in Fig. 11, corresponds to the distorted image in Fig. 10.

Fig. 11.

Image motion record (IMR)

In the ideal case, the image should advance in flight direction by exactly one pixel with any obtained line, so the ideal image motion record in the flight direction should be a straight line with a 45° slope. The real record in the flight direction however has a different slope due to the airplane velocity and local variations caused by attitude instability and vibrations.

Perpendicular to the flight direction there should be no image motion in the ideal undisturbed case, so the corresponding component of image motion record should coincide with the horizontal axis. Actually there were considerably large deviations from the ideal case due to attitude instability and vibrations also in this case.

The correction of the distorted images has been made after the flight on base of the image motion records produced in-flight. Fig. 12 shows the example of such an image correction result.

Fig. 12.

Example of image correction on base of an image motion record (fragment of large image)

Some residual distortions of the corrected image are caused mainly by vibration components above the Nyquist frequency. Such distortions can be eliminated by increasing of the correlation rate. A certain degree of smoothing of the corrected image is caused by the interpolation procedure itself and (on some parts of the image) by a high local velocity of image motion due to high vibration amplitudes (which are actually not expected on a satellite).

6.

CONCLUSIONS

This paper has presented the novel SMARTSCAN concept for robust pushbroom imaging on shaky platforms. It allows to get high resolution images without any mechanical compensation mechanisms. The sole optoelectronic image motion compensation uses advanced optical correlator technology, which allows to measure image motion records with subpixel accuracy in realtime with a compact and low power device. The concept has been demonstrated successfully with airborne tests.