One of the major deficiencies in current robot control schemes is the lack of high-level knowledge in the feedback loop. Typically, the sensory data acquired are fed back to the robot controller with minimal amount of processing. However, by accumulating useful sensory data and processing them intelligently, one can obtain invaluable information about the state of the task being performed by the robot. This paper presents a method based on the screw theory for interpreting the position and force sensory data into high-level assembly task constraints. The position data are obtained from the joint angle encoders of the manipulator and the force data are obtained from a wrist force sensor attached to the mounting plate of the manipulator end-effector. The interpretation of the sensory data is divided into two subproblems: representation problem and interpretation problem. Spatial and physical constraints based on the screw axis and force axis of the manipulator are used to represent the high-level task constraints. Algorithms which yield least-squared error results are developed to obtain the spatial and physical constraints from the position and force data. The spatial and physical constraints obtained from the sensory data are then compared with the desired spatial and physical constraints to interpret the state of the assembly task. Computer simulation and experimental results for verifying the validity of the algorithms are also presented and discussed.

A robot system that integrates the fusion of computer based voice recognition and computer vision, as well as artificial neural networks and knowledge is discussed. Our goal is to develop a flexible system with a broad user bandwidth that can be used in hazardous environments such as those that occur in military applications. Three major components of this system, which are voice recognition module, the real-time visual image precessing module, and the command control module, will be addressed. The system is designed to function (at least initially) in a known environment. Future studies will be made about unknown, dynamic environment.

The problem of finding a path for a mobile robot in two dimensions among stationary obstacles is examined, using a free space graph representation of the robot's workspace. One problem with solutions of this type has been the time it takes to construct a graph from the image of the workspace. Our solution seeks to resolve this situation by constructing the graph representation "on the fly" in a scanline fashion by processing the image one line at a time from top to bottom. One module breaks the obstacles in the image into pseudo obstacles; another module analyzes the pseudo obstacles and locates maximal overlap regions (MOvs) of prime convex regions (PCRs) in the free space; a third constructs a graph representation utilizing the MOvs and PCRs as nodes and edges respectively. The modularity lends itself toward pipelined processing to reach the solution.

In this paper we are going to describe an ongoing research project intended to integrate a full vision system in a flexible robot programming environment. The use of the vision system sensors, allows the robot to derive a description of the work cell. This description is used for the collision avoidance problem of robot manipulators. The work cell in assembly context can include moving objects. Without any previous knowledge of the work space, the vision system thus immediately determines the work cell map in its entirely. Successively this map is used as input for the path planner process to find the collision-free path. During the assembly robot operation, the vision system is activated to reflect any changes in the robot environment. In this way the path planner works recursively, updating the collision free path until the goal is reached.

This paper discusses the integration of model based computer vision with a robot planning system. The vision system deals with structured objects with several movable parts (the "Task Panel"). The robot planning system controls a T3-746 manipulator that has a gripper and a wrist mounted camera. There are two control functions: move the gripper into position for manipulating the panel fixtures (doors, latches, etc.), and move the camera into positions preferred by the vision system. This paper emphasizes the issues related to repositioning the camera for improved viewpoints.

The objective of the automatic nesting problem is to find an arrangement for cutting irregular Shaped pieces that most efficiently utilizes an available space in a reasonable amount of time automatically. The available space, in this case, is highly irregular. Highly irregular resources not only have irregular boundaries but also defective areas that cannot be utilized. To solve this problem physical objects and mental concepts were represented in a framework called object-oriented representation. A recursive lookahead approach was also used, a novel way of segmenting an image in order to localize the search space. An abstract heuristic hill-climbing search was combined with a best-first search using a limited backtracking method to create a hybrid search technique. This system has been developed and the preliminary result is satisfactory. The testing has been performed by comparing the system against a human expert. The average yield difference has been within five percent.

In this paper, the advantages of using an object-oriented environment as a tool for general purpose simulation are discussed, emphasising its particular relevance for the simulation of perception and navigation for mobile robots. In this context, an object-oriented mobile robot simulator - MOBILE, designed using Smalltalk, is presented. The flexibility and modularity gained by using an object-oriented design philosophy are illustrated by a simple example consisting of the simulation of a control strategies for a vision-guided mobile platform. The objects for modeling the vehicle, and the on board vision system are presented, as well as the graphical output results, where the simulated vehicle trajectory can be visualized.

To facilitate experiments in bipedal walking, we have developed a method for fast autonomous recognition of footholds marked by a distinctive color. An image patch is taken to be part of a foot-hold if its color is within a specified region of red-green-blue space and the patch is sufficiently large. The method is suitable for continuous use while the machine is walking and is robust with respect to variations in ambient lighting.

A World Modeling System for operating a remote robotic servicing system should provide an operator with the same degree of flexibility and feedback possible from teleoperators, but under a supervisory control constraint. Thus, a sophisticated software system must exist on both the remote servicer and the local platform. This system must provide simulations and offline programming of the remote servicer (and its environment) for the operator to compose new commands (and test them). It must provide communications to and from the remote platform (within bandwidth limitations). It must allow interaction at multiple levels of abstraction (for instance at the task level, the robotic path planning level, and the robot joint level), and it must provide sensors, sensor processing, and model matching capabilities required to direct remote autonomous operations (like tracking or grip point identification) and to keep the local simulated servicer environment up to date. This concept of a World Modeling System encompasses perception and control of robotic systems.

Six dimensional target information is produced by a target tracking vision system for use in real time target tracking by a robotic system. The vision system described in this paper produces three axes of position data and three axes of orientation data using a single camera which views a three dimensional target. The system performs target detection, target discrimination and determines the target position and orientation relative to the camera. As the target information is determined, it is communicated to a computer which is controlling the robot motion in real time. Details of the image processing algorithms and image processing hardware used in the vision system are discussed in the paper.

Computer vision technology offers potential application for automation of space operations. The merits of computer vision and automation for earth bound operations are magnified when they are applied to space. Human involvement may be greatly reduced for routine operations and the risk to human life may be virtually eliminated. Two applications of this technology are described; one is applied to automatic orbital docking and the other is focused on automation of routine operations within the space station. Both application concepts use video cameras as the primary sensor, however they each employ different techniques. Orbital docking is the more time critical of the two applications and a syntactic image analysis technique for this use is outlined in this report.

Traditionally, a programmer's role has been reduced to designing computer vision algorithms and coding them in a machine independent language since the knowledge of the architecture has been implicit in the compiler. However, advances in the design of computing systems have resulted in the appearance of a wide range of different architectures ranging from high performance, single chip ALUs to large systems of communicating processors. The efficient use of each of these machines requires detailed knowledge of the architecture. Presently, the programmer must employ detailed knowledge of the architecture in writing the high level language program to run on any of these novel architectures. As a result, these programs are tailored to a specific computer and must be rewritten for other computers. Until now, architectural knowledge has been implicit in the high level language compilers and is not easily changed. A more promising approach is to develop compilers which use heuristic advice and architectural descriptions to guide code generation. With such a system, the high level statement of the program will be substantially machine independent and the compiler will resume its proper role as the architectural expert. In this paper we present a compiler in which architectural information is represented as a rule base and show examples of code generation for low level image processing algorithms including edge relaxation labeling, FFT, and histogram based thresholding. The input language is C supplemented with a distributed data storage class and concurrent assignments. The examples shown illustrate that the algorithms can be developed without knowledge of the target processor(s) or their interconnection network. Only a few additional declarations have to be added to any program to specify the target architecture. Currently, the compiler generates message passing C code suitable for distributed memory parallel computers.

Scene analysis and scene understanding are acknowledged to require not only the matching of individual object shapes and texture, but also to involve reasoning about the spatial and contextual relationships among objects, so as to provide the basis for a general, real-time knowledge-based interpretation system1. In this paper, semantic information and pragmatics are used to identify and classify both individual objects and sub-scenes (concepts) in the scene analysis process. The use of an interactive concept classifier for scene analysis permits the study of adaptive concept classification and model-directed reasoning for image understanding. A reasoning engine that automatically generates (proposes) recognition strategies is now in prospect based on this approach.

An algorithm capable of estimating the image motion of a moving object in real-time is described. Our method is based on the image correlation technique. But unlike previous approaches, we estimate the motion along the image gradient direction only. As the result we obtain the normal flow vector. This approach allows us to avoid difficulties often arise due to the multiple matching. Normal flow vectors are used to estimate the actual flow vector (often called full flow vector) by assuming that the 2-D motion is locally constant in a small neighborhood. This process requires solving a linear system of order 2 only. This entire process operates uniformly on several spatial resolutions of image sequence. Each resolution is tuned to a specific range of image motion, and the correct resolution is determined by comparing the confidence which has been computed in the course of full flow estimation.

This paper describes the implementation of a hierarchical robot simulation which supports the design of robots with vision and mobility. A seeing robot applies a classification expert system for visual identification of laboratory objects. The visual data acquisition algorithm used by the robot vision system has been developed to exploit multiple viewing distances and perspectives. Several different simulations have been run testing the visual logic in a laboratory environment. Much work remains to integrate the vision system with the rest of the robot system.

This paper presents a model for dynamic monitoring of the assembly process using a synchronized combination of accelerometers, coherent ultrasonics and a light beam. This type of sensor fusion makes it possible to carry out both diagnosis of the separate operations and to support error recovery routines. In this paper we thus concentrate on some interesting uses of inertial sensing at impact in a typical insertion operation.The basic concept assumes a gripper, equipped with a light beam in the finger tips (to time-tag movement of an object into or out of the gripper), a coherent ultrasonic interferometer (to monitor the subsequent relative displacement between the gripper and the object) and inertial sensors (to monitor impulsive forces during gripping). The properties of the coherent ultrasonic sensor have been described elsewhere.

The article describes an extension to the well-established AI language Prolog. This allows Prolog to operate both an image processing system and a controller for a variety of electro-mechanical devices. The user can define his/her own pull-down menus and provides an interface to a speech synthesis package. The latter enables the user to follow the flow of a program, easily and in a natural way. The application of the software to food inspection is also discussed

Precision welding of aircraft engine assemblies requires accurate tracking of the weld joint under conditions of positional uncertainty. Real-time laser based joint tracking systems are becoming available but currently cost, accuracy, field-of-view, and applicability to specific joint types, weld processes and robot controllers limit their use. When tracking prior to welding is a feasible alternative to real-time tracking, these issues can be addressed. The software and hardware for such a system is described. The system can track weld joints in two dimensions and offset weld path programs to compensate for assembly and/or fixturing errors. The technique is applicable to both plasma arc welding (PAW) and tungsten inert gas (TIG) welding as the sensor is mounted ahead of the weld torch. The weld path tracking software refines a coarsely taught weld path. To enhance tracking accuracy, reliability and speed, the joint tracking algorithm searches for the joint along a line. Search direction is calculated as the normal to the coarse path trajectory. Search direction is accomplished by rotating a window of the image. A prototype system has been implemented. The system consists of a welding robot, a custom vision system, a CCD camera and fiber optic incandescent lighting. The system has been used to successfully weld production assemblies.

In this paper we present an AI-based technique to build an automatic robot programming system. This system uses assembly task descriptions as well as domain knowledge to produce a desired robot control program. This system can be divided into two functional parts-- (1) the task planner which produces an abstract level assembly plan, and (2) the control program generator which decomposes the assembly plan into a sequence of executable robot commands. The high-level task planner is a knowledge-based hierarchical system containing four types of system knowledge: (i) assembly parts description, (ii) workpiece structures, (iii) assembly operations, and (iv) assembly principles. An assembly plan is generated by constructing a task hierarchy which divides a main goal into subgoals. Orders of subgoals are suggested by assembly principles. Conflicts that arise from constructing the task hierarchy can be eliminated by posting constraints along each subgoal. Generating subgoals will continue until all subgoals are described by task primitives. Based on these primitives, the robot program generator checks the mounting specifications, and then determines the detailed spatial data pertaining to the location and orientation of each part. Finally, a robot control program is composed. A case study has been conducted by implementing the proposed method in OPS5 language. The implemented system can generate control programs for the IBM 7545 manufacturing system which includes a robot manipulator controlled by textual programs in AML (A Manufacturing Language). Planning efficiency is assured by a designed control structure to execute this automatic programming system. A user interface is also provided to accept assembly task description and update the knowledge base. Accuracy of the automatic programming is tested by the robot performing an assembly task planned by this system. A description of an automatically generated robot assembly task program is included to demonstrate the success of this research.

The paper describes a new robot tactile sensor, which, made of piezoresistance materials, simple in structure, easy in operation and Lower in cost, may increase greatly the positioning accuracy of robot assembly. With this sensor, any position and the assembly centre within the working range of the robot may be determined. In our experimental work, this sensor, working with a robot of ±0.1mm repeativie positioning accuracy, has been successful in help assembling a "peg-hole" which has just a clearance of 0.008--0.015mm. And the corresponding software has also been developed.

A machine vision system for real time inspection of moving web is discussed. The system is for use in detecting, locating, identifying, classifying and sizing defects on a moving sheet of web and, mapping the defects detected on the roll of web being inspected on a printer. On the latter, the precise location on the X and Y axes on the roll of web for each defect detected is recorded using a code whereby the type of defect is identified and it's size reported. The system's unique configuration of hardware components includes a novel Time Delay and Integration (TDI) type camera as the sensors, a novel image processing board directly coupled to the TDI camera forming a camera-image processing board subassembly and, a novel master timing and synchronization board subsystem. The latter being capable of synchronizing the triggering, timing and sampling time of multiple channels within the TDI sensor and multiple TDI camera-image processing board subassemblies, in parallel within the same time frame, by controlling a novel master clocking scheme within a TDI logic circuit, contained within each camera, wherein the clocks' speed for data output, in Megahertz (MHZ), remains constant regardless of web speed whereas, the imager clocks' speed and signal format varies with web speed. This allows a high level of stop motion at any web speed including high speeds, that is, motion within the defined pixel size on the web during sampling time being less than 0.00002" (0.02 mils) and, renders the system immune to process vibrations. Furthermore, through routines in firmware, the system's performance and abilities are not affected by web speed variations. The system is integrated on a 20 slots AT2 compatible backplane having a PC bus single board AT2 compatible computer system wherein is contained various routines to generate data by processing various types of defects on various types of webs which are downloaded on to each of the image processing boards' programmable chips at system start-up by a master program. Contained therein is an expert system master program especially conceived and designed for the purpose of this system having routines to operate the boards, render logical decisions based on the data generated by the image processing boards, and generate the required reports. The system also comprises a subsystem, controlled by routines contained in the master program, which maintains the intensity and uniformity of web illumination within 0.5%, according to the type and thickness of the web being inspected, and controls the various positions of the illumination source's components. The standard configuration of the system developed produces a pixel size on the moving web of 0.0025" (2.5 mils) but can be configured to have a smaller or larger pixel size, the latter being proportional to the number of camera-image processing board subassemblies used.

Robot control applications generally can be considered as a collection of real-time tasks executed in parallel. That's why multiprocessor machines seem to provide an efficient solution to the requirements of current robot control systems. Concurrent high-level languages used in such environments must contain primitives for parallel processes, synchronization, interrupt handlers, etc. We show how these concepts can be used in sensor controlled robot motions, in particular how asynchronous processes can be used in the implementation of compliant motion.

Our goal is to identify an inexpensive, accurate, and safe system for obtaining digitized range data for use in a variety of object recognition tasks. In this paper we present initial results on the use of a non-collimated ultrasound sensor in obtaining such data. Although ultrasound has long been used for other similar tasks, the applications for which the current system is intended require significant adaptation of existing systems and techniques.

The design and realization criteria of an integrated optic force sensor made via a X-cut titanium-diffused lithium niobate semi-asymmetric X-junction are presented. The X-cut crystal version whose force-perturbated permettivity tensor exhibits a longitudinal configuration, induces the propagation of guided modes with hybrid characteristics and substrate leaky modes. The field evolution is reported. This sensor configuration is less sensitive to the temperature changes and exhibits a better performance with respect to the Z-cut substrate geometry.The half-wave force is equal to 0.14 and 0.21 N for the TE-like and TM-like excitation, respectively, versus 0.32 and 0.23 N pertaining to a Z-cut waveguide.

Mushrooms are normally picked selectively as they reach an appropriate size and graded straight into boxes. For automation a robotic harvester would need to be guided to select the mature mushrooms and leave the immature mushrooms undamaged. This paper describes a vision processing algorithm developed to identify mushrooms in a growing bed and find their position and size. The algorithm uses knowledge of the shape of the mushroom and its expected appearance to give reliable performance. The edge of the mushroom is tracked in the grey level image, and an estimate of the centre of the mushroom is used to guide the tracking through confused areas. The algorithm can cope with the problems of touching mushrooms and variations in lighting levels. The algorithm appears to work well enough to guide a harvesting machine.

The paper describes a problem of fatigue strength of optical fibe under dynamic conditions in robotics systems and a method for measuring the fatigue strength based on measurements of changes in the radiation power transmitted in the examined fibers with increasing number of cycles of strains on the samples. Testing stand is described and some experimental results are also presented.

The algorithm presented in this paper constructs a geometric model of the environment using ultrasonic sensors. To do this in a reliable way, it has to take different error sources into account. Unlike other approaches, where a low-level, pixel based, probabilistic model is constructed to represent the uncertainty arising from false measurements, a high level, geometric, model is constructed. It is shown that a high level model, besides being faster to construct, is more appropriate for taking into account the typical characteristics of ultrasonic sensors. The algorithm detects and eliminates inconsistent measurements by combining evidence gathered from different points of view. This is made possible by extracting from the measurements not only information concerning the position of obstacles, but also information about regions that must be empty when seen from a certain angle. To conclude, some examples of the behaviour of this algorithm in real-world situations are presented.

The use of image projection to solve the coincidence problem for stereopsis is presented. A range map is obtained from image features obtained through a new image projection algorithm applied to a pair of stereo images. The use of image projection can significantly reduce the amount of computation and enable one to incorporate methods to solve the occlusion and noise problem present in image data. Preliminary simulation results are presented.

The lighting advisor is a Prolog program which is intended to suggest possible lighting and viewing configurations to an engineer faced with the task of designing a machine vision system for industrial inspection. The program asks a series of simple questions about the object that is to be inspected and the type of feature, or defect, which is to be highlighted. The program then displays a line drawing, which shows a sketch of the recommended optical configuration. Further questions are then posed and other diagrams are displayed. Each of the diagrams is accompanied by explanatory notes, but these will only be displayed if desired by the user. These notes describe the type of equipment required, how to set up the optical system and, where appropriate, a reference to the technical literature.

A method for tracking partially occluded two dimensional polygonal shapes undergoing unknown two dimensional translational and rotational motion has been developed based on Kalman filtering. Observation of a robotic workspace by a machine vision system presents many situations in which known objects may be occluded partially or completely by other objects, fixtures, or the robot itself. Tracking such objects using non-occluded, visible features is an important problem. The method assumes object corners, or some other feature set, can be identified to known accuracy by another technique, and that feature occlusion (absence) can also be detected or recognized. A linear, constant acceleration model is assumed for shape translational and rotation motion in which the shape centroid and angular orientation, as well as their velocities and accelerations, comprise the state. A nonlinear observation model is assumed where the corner or feature locations are measured. The proposed method is investigated under a variety of conditions, including non-constant acceleration, substantial, and total occlusion. Conditions under which tracking is lost are examined.

Robot servoing from video rate data inputs is now possible. Area Parameter Acceleration techniques allow computers to isolate and geometrically describe as many as 255 objects each 33 millisecond video frame time. From the parameters of each object, the host processor may determine size, location, orientation, and shape. It then may recognize appropriate objects and select the right target for robot action.

A benchmark was carried out to explore the differences in the speed-up behavior of two multiprocessors, a shared memory computer (Butterfly Parallel Processor, BPP) and a distributed memory computer (HATHI-2). Three vision problems were used in the tests: an algorithm for border tracking, a generalized Hough transform and a matching of attributed relational graphs. The results indicate that the HATHI-2 outperforms the BPP if the parallel program does not contain very intensive communication bursts.

An algorithm is described to detect corners from thinned edge maps of polygonal or polyhedral objects using a Kalman filter. Straight lines are modelled as constant slope segments. The length of a line is assumed to increase at a constant rate as more points are added to the segment. The state vector consists of the angle of inclination of the straight line, its length and the rate of change of the length with each new pixel. A linear state transition equation is developed based on this model. Plant noise is added to account for the fact that the straight lines are digital straight lines, and it is assumed to be zero mean, Gaussian and uncorrelated. The system measurement consists of the pixel coordinates of the edge. These coordinates can be expressed as a nonlinear function of the state vector. The measurement errors are also assumed to be zero mean, Gaussian and uncorrelated. Using this linear plant model and a nonlinear measurement model, an extended Kalman filter is used to track the line segments. The divergence of the filter from its steady state is detected to locate a corner. Tests done indicate that this technique works well. The results are compared to the results obtained by using a difference of low pass (DOLP) corner detector.