We describe a real-time computer vision and machine learning system for modeling and recognizing human behaviors in a visual surveillance task [1]. The system is particularly concerned with detecting when interactions between people occur and classifying the type of interaction. Examples of interesting interaction behaviors include following another person, altering one's path to meet another, and so forth. Our system combines top-down with bottom-up information in a closed feedback loop, with both components employing a statistical Bayesian approach [2]. We propose and compare two different state-based learning architectures, namely, HMMs and CHMMs for modeling behaviors and interactions. The CHMM model is shown to work much more efficiently and accurately. Finally, to deal with the problem of limited training data, a synthetic ªAlife-styleº training system is used to develop flexible prior models for recognizing human interactions. We demonstrate the ability to use these a priori models to accurately classify real human behaviors and interactions with no additional tuning or training.

Abstract not found

Abstract not found

In this paper we describe our SmartCar testbed: a realtime data acquisition system and a machine learning framework for modeling and recognizing driver maneuvers at a tactical level, with special emphasis on how the context a#ects the driver's performance. The perceptual input is multi-modal: four video signals capturethecontextual tra#c, the driver's head and the driver's viewpoint; and a real-time data acquisition system records the car's brake, gear, steering wheel angle, speed and acceleration throttle signals. Over 70 drivers have driven the SmartCar for 1.25 hours in the greater Boston area. Graphical models, HMMs and potentially extensions #CHMMs#, have been trainedusing the experimental driving data to create models of seven di#erent driver maneuvers: passing, changing lanes right and left, turning right and left, starting and stopping. We show that, on average, the predictive power of our models is of 1 second before the maneuver starts taking place. Therefore, these models would be essential to facilitate operating mode transitions between driver and driver assistance systems, to prevent potential dangerous situations and to create morerealistic automatedcars in car simulators.

I propose a common framework that combines three different paradigms in machine learning: generative, discriminative and imitative learning. A generative probabilistic distribution is a principled way to model many machine learning and machine perception problems. Therein, one provides domain specific knowledge in terms of structure and parameter priors over the joint space of variables. Bayesian networks and Bayesian statistics provide a rich and flexible language for specifying this knowledge and subsequently refining it with data and observations. The final result is a distribution that is a good generator of novel exemplars.

This paper presents our SmartCar testbed platform: a real-time data acquisition and playback system and a machine learning --dynamical graphical models-- framework for modeling and recognizing driver maneuvers at a tactical level, with particular focus on how contextual information affects the driver's performance. The SmartCar's perceptual input is multi-modal: four video signals capture the surrounding traffic, the driver's head position and the driver's viewpoint; and a real-time data acquisition system records the car's brake, gear, steering wheel angle, speed and acceleration throttle signals. We have carried out driving experiments with the instrumented car over a period of 2 months. Over 70 drivers have driven the SmartCar for 1.25 hours in the greater Boston area. Dynamical Graphical models, HMMs and potentially extensions (CHMMs), have been trained using the experimental driving data to create models of seven different driver maneuvers: passing, changing lanes right and left, ...

This paper proposes a new family of Hidden Markov Models (HMMs) named Mutual Information Hidden Markov Models (MIHMMs) . MIHMMs have the same graphical structure as HMMs. However, the objective function being optimized is not the joint like- lihood of the observations and the hidden states. It is a convex combination of the mutual information between the hidden states and the observations, and the likelihood of the observations and the states. First, we present both theoretical and practical motivations for having such an objective function.

Modeling human behavior is important for the design of robots as well as humancomputer interfaces that use humanoid avatars. Constructive models have been built, but they have not captured all of the detailed structure of human behavior such as the moment-to-moment deployment and coordination of hand, head and eye gaze used in complex tasks. We show how this data from human subjects performing a task can be used to program a dynamic Bayes network (DBN) which in turn can be used to recognize new performance instances. As a specific demonstration we show that the steps in a complex activity such as sandwich making can be recognized by a DBN in real time.