In this paper, we describe a system that automatically converts narratives into 3D scenes. The texts, written in Swedish, describe road accidents. One of the program’s key features is that it animates the generated scene using temporal relations between the events. We believe that this system is the first text-to-scene converter that is not restricted to invented narratives. The system consists of three modules: natural language interpretation based on information extraction (IE) methods, a planning module that produces a geometric description of the accident, and finally a visualization module that renders the geometric description as animated graphics. An evaluation of the system was carried out in two steps: First, we used standard IE scoring methods to evaluate the language interpretation. The results are on the same level as for similar systems tested previously. Secondly, we performed a small user study to evaluate the quality of the visualization. The results validate our choice of methods, and since this is the first evaluation of a text-to-scene conversion system, they also provide a baseline for further studies. 1

The algorithmic generation of textual descriptions of real world image sequences requires conceptual knowledge. The algorithmic generation of synthetic image sequences from textual descriptions requires conceptual knowledge, too. An explicit representation formalism for behavioral knowledge based on formal logic is presented which can be utilized in both tasks -- Understanding and Creation of video sequences.

The textual description of video sequences exploits conceptual knowledge about the behavior of depicted agents. An explicit representation of such behavioral knowledge facilitates not only the textual description of video evaluation results, but can also be used for the inverse task of generating synthetic image sequences from textual descriptions of dynamic scenes. Moreover, it is shown here that the behavioral knowledge representation within a cognitive vision system can be exploited even for prediction of movements of visible agents, thereby improving the overall performance of a cognitive vision system.

This paper describes a system to create animated 3D scenes of car accidents from reports written in Swedish. The system has been developed using news reports of varying size and complexity. The text-to-scene conversion process consists of two stages. An information extraction module creates a structured representation of the accident and a visual simulator generates and animates the scene.

Abstract We present a surveillance system, comprising wide field-of-view (FOV) passive cameras and pan/tilt/zoom (PTZ) active cameras, which automatically captures high-resolution videos of pedestrians as they move through a designated area. A wide-FOV static camera can track multiple pedestrians, while any PTZ active camera can capture high-quality videos of one pedestrian at a time. We formulate the multi-camera control strategy as an online scheduling problem and propose a solution that combines the information gathered by the wide-FOV cameras with weighted roundrobin scheduling to guide the available PTZ cameras, such that each pedestrian is observed by at least one PTZ camera while in the designated area. A centerpiece of our work is the development and testing of experimental surveillance systems within a visually and behaviorally realistic virtual environment simulator. The simulator is valuable as our research would be more or less infeasible in the real world given the impediments to deploying and experimenting with appropriately complex camera sensor networks in large public spaces. In particular, we demonstrate our surveillance system in a virtual train station environment populated by autonomous, lifelike virtual pedestrians, wherein easily reconfigurable virtual cameras generate synthetic video feeds. The video

Abstract We present a space robotic system capable of capturing a free-flying satellite for the purposes of on-orbit satellite servicing. Currently such operations are carried out either manually or through discrete-event scripted controllers. The manual approach is costly and exposes astronauts to danger, while the scripted approach is tedious and brittle. Consequently, there is substantial interest in performing these operations autonomously, and the work presented here is a step in this direction. To our knowledge, ours is the only satellite-capturing system that relies on vision and cognition to deal with an uncooperative satellite. Our innovative system combines visual perception (object identification, recognition, and tracking) with high-level reasoning in a hybrid deliberative/reactive computational framework. The reasoning module, which encodes a model of the environment, performs deliberation to control the perception pipeline— it guides the vision system, validates its performance, and suggests corrections when vision is performing poorly. Furthermore, it advises the behavioral controller to carry out its tasks. Reasoning and related elements, among them intention, context, and memory, are responsible for the robustness and reliability of the overall system. We demonstrate our prototype system controlling a robotic arm that autonomously captures a free-flying satellite in a realistic laboratory setting that faithfully mimics on-orbit conditions.

We present a cognitively-controlled vision system that combines low-level object recognition and tracking with high-level symbolic reasoning for the purpose of solving difficult space robotics problems---satellite rendezvous and docking. The reasoning module, which encodes a model of the environment, performs deliberation to 1) guide the vision system in a taskdirected manner, 2) activate vision modules depending on the progress of the task, 3) validate the performance of the vision system, and 4) suggest corrections to the vision system when the latter is performing poorly. Reasoning and related elements, among them intention, context, and memory, contribute to improve the performance (i.e., robustness, reliability, and usability). We demonstrate the vision system controlling a robotic arm that autonomously captures a free-flying satellite. Currently such operations are performed either manually or by constructing detailed control scripts. The manual approach is costly and exposes astronauts to danger, while the scripted approach is tedious and error-prone. Therefore, there is substantial interest in performing these operations autonomously, and the work presented here is a step in this direction. To our knowledge, this is the only satellite-capturing system that relies exclusively on vision to estimate the pose of the satellite and can deal with an uncooperative satellite.