## Facial Expression Recognition from Video Sequences: Temporal and Static Modelling (2003)

### Cached

### Download Links

Venue: | Computer Vision and Image Understanding |

Citations: | 121 - 19 self |

### BibTeX

@INPROCEEDINGS{Cohen03facialexpression,

author = {Ira Cohen and Nicu Sebe and Larry Chen and Ashutosh Garg and Thomas S. Huang},

title = {Facial Expression Recognition from Video Sequences: Temporal and Static Modelling},

booktitle = {Computer Vision and Image Understanding},

year = {2003},

pages = {160--187}

}

### Years of Citing Articles

### OpenURL

### Abstract

Human-computer intelligent interaction (HCII) is an emerging field of science aimed at providing natural ways for humans to use computers as aids. It is argued that for the computer to be able to interact with humans, it needs to have the communication skills of humans. One of these skills is the ability to understand the emotional state of the person. The most expressive way humans display emotions is through facial expressions. In this work we report on several advances we have made in building a system for classification of facial expressions from continuous video input. We introduce and test different architectures, focusing on changes in distribution assumptions and feature dependency structures. We also introduce a facial expression recognition from live video input using temporal cues. Methods for using temporal information have been extensively explored for speech recognition applications. Among these methods are template matching using dynamic programming methods and hidden Markov models (HMM). This work exploits existing methods and proposes a new architecture of HMMs for automatically segmenting and recognizing human facial expression from video sequences. The architecture performs both segmentation and recognition of the facial expressions automatically using an multi-level architecture composed of an HMM layer and a Markov model layer. We explore both person-dependent and person-independent recognition of expressions and compare the different methods.

### Citations

8936 |
Cache-oblivious algorithms
- Frigo, Leiserson, et al.
- 1999
(Show Context)
Citation Context ...ing the features such that the resultant graph is a tree and the sum of the weights of the arcs is maximized. There have been several algorithms proposed for building a maximum weighted spanning tree =-=[7]-=- and in our implementation we use the Kruskal’s algorithm described in Figure 4. The five steps of the TAN algorithm are described in Figure 5. This procedure ensures to find the TAN model ( that maxi... |

4553 | A tutorial on hidden markov models and selection applications in speech recognition
- Rabiner
- 1989
(Show Context)
Citation Context ...robability of observations given the model. This problem relates to the learning phase of the HMMs which describe each facial expression sequence. A comprehensive tutorial on HMMs is given by Rabiner =-=[34]-=-. 5.2 Expression Recognition Using Emotion-Specific HMMs Since the display of a certain facial expression in video is represented by a temporal sequence of facial motions it is natural to model each e... |

2238 | Principal component analysis - Jolliffe - 1986 |

1603 |
Fundamentals of Speech Recognition
- Rabiner, Juang
- 1993
(Show Context)
Citation Context ...t the same time. 5.1 Hidden Markov Models Hidden Markov models have been widely used for many classification and modeling problems. Perhaps the most common application of HMM is in speech recognition =-=[33]-=-. One of the main advantages of HMMs is their ability to model nonstationary signals or events. Dynamic programming methods allow one to align the signals so as to account for the non stationarity. Ho... |

851 | Text Classification from Labeled and Unlabeled Documents using EM
- Nigam, McCallum, et al.
- 2000
(Show Context)
Citation Context ...y classification problems, although the independence assumption is usually violated in practice. Examples of applications that use the Naive-Bayes classifiers are abundant such as text classification =-=[27]-=- and face pose estimation [1]. Although the Naive-Bayes model does not reflect in many cases the true underlying model generating the data, it is still observed to be successful as a classifier in pra... |

679 | Approximating Discrete Probability Distributions with Dependence Trees
- Chow, Liu
- 1968
(Show Context)
Citation Context ...AN structures. In general, searching for the best structure has no efficient solution, however, searching for the best TAN structure does have one. The method is using the modified Chow-Liu algorithm =-=[5]-=- for constructing tree augmented 10( Consider an undirected graph with vertices and edges, where each % edge the vertices and , has an associated positive weight . To construct the maximum weighted s... |

632 | Bayesian network classifiers
- Friedman, Geiger, et al.
- 1997
(Show Context)
Citation Context ...-Bayes classifier. More flexible learning methods allow Bayesian network classifiers to be selected from a small subset of possible structures — for example, the Tree-Augmented Naive-Bayes structures =-=[14]-=-. After a structure is selected, the parameters of the classifier are usually learned using maximum likelihood estimation. We propose using Bayesian network classifiers for recognizing facial expressi... |

454 | Emotional intelligence - Goleman - 1995 |

400 | Comprehensive database for facial expression analysis
- Kanade, Cohn, et al.
- 2000
(Show Context)
Citation Context ...nce. We perform experiments for all the described methods using two databases in Section 6. The first is our database of subjects displaying facial expressions. The second is the Cohn-Kanade database =-=[19]-=-. We have concluding remarks in Section 7. 2 Review of Facial Expression Recognition Since the early 1970s, Paul Ekman and his colleagues have performed extensive studies of human facial expressions [... |

318 | 2000b). Automatic analysis of facial expressions: the state of the art
- Pantic, Rothkrantz
(Show Context)
Citation Context ... to categorize different facial expressions. Recent work on facial expression analysis and recognition [2, 9, 12, 21, 23–26, 28, 30, 35, 40] has used these “basic expressions” or a subset of them. In =-=[32]-=-, Pantic and Rothkrantz provide an in depth review of many of the research done in automatic facial expression recognition in recent years. The work in computer-assisted quantification of facial expre... |

287 | Coding, analysis, interpretation, and recognition of facial expressions
- Essa, Pentland
- 1997
(Show Context)
Citation Context ...he six facial expressions. Rosenblum, Yacoob, and Davis [35] also computed optical flow of regions on the face, then applied a radial basis function network to classify expressions. Essa and Pentland =-=[12]-=- used an optical flow region-based method to recognize expressions. Donato et al. [9] tested different features for recognizing facial AUs and inferring the facial expression in the frame. Otsuka and ... |

273 | Classifying facial actions - Donato, Bartlett, et al. - 1999 |

239 | Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion
- Black, Yacoob
- 1995
(Show Context)
Citation Context ...essions. Lanitis et al. [21] used a flexible shape and appearance model for image coding, person identification, pose recovery, gender recognition, and facial expression recognition. Black and Yacoob =-=[2]-=- used local parameterized models of image motion to recover non-rigid motion. Once recovered, these parameters were used as inputs to a rule-based classifier to recognize the six basic facial expressi... |

204 | The emotion probe: Studies of motivation and attention - Lang - 1995 |

203 | On bias, variance, 0/1-loss, and the curse-of-dimensionality
- Friedman
- 1997
(Show Context)
Citation Context ...on for the Naive-Bayes model’s success as a classifier is attributed to the small number of parameters needed to be estimated, thus offsetting the large modeling bias with a small estimation variance =-=[13]-=-. Recently, Garg and Roth [15] showed using information theoretic arguments additional reasons for the success of Naive-Bayes classifiers. Thus, it is natural to explore the performance of the Naive-B... |

201 |
An Introduction to the Application of the Theory of Probabilistic Function of a Markov Process to Automatic Speech Recognition, Bell Syst
- Levinson, Rabiner, et al.
- 1983
(Show Context)
Citation Context ... and are the mean vector and full covariance matrix, respectively. The parameters of the model of emotion-expression specific HMM are learned using the well-known BaumWelch reestimation formulas (see =-=[22]-=- for details of the algorithm). For learning, hand labeled sequences of each of the facial expressions are used as ground truth sequences, and the Baum algorithm is used to derive the maximum likeliho... |

169 | Learning to Resolve Natural Language Ambiguities: A Unified Approach - Roth - 1998 |

139 |
Recognizing human facial expressions from log image sequences using optical flow
- Yacoob, Davis
- 1996
(Show Context)
Citation Context ...zed models of image motion to recover non-rigid motion. Once recovered, these parameters were used as inputs to a rule-based classifier to recognize the six basic facial expressions. Yacoob and Davis =-=[41]-=- computed optical flow and used similar rules to classify the six facial expressions. Rosenblum, Yacoob, and Davis [35] also computed optical flow of regions on the face, then applied a radial basis f... |

125 |
Recognition of Facial Expression from Optical Flow
- Mase
- 1991
(Show Context)
Citation Context ... review of many of the research done in automatic facial expression recognition in recent years. The work in computer-assisted quantification of facial expressions did not start until the 1990s. Mase =-=[25]-=- used optical flow (OF) to recognize facial expressions. He was one of the first to use image processing techniques to recognize facial expressions. Lanitis et al. [21] used a flexible shape and appea... |

113 |
Strong evidence for universals in facial expressions: A reply to Russell’s mistaken critique
- Ekman
- 1994
(Show Context)
Citation Context ...]. We have concluding remarks in Section 7. 2 Review of Facial Expression Recognition Since the early 1970s, Paul Ekman and his colleagues have performed extensive studies of human facial expressions =-=[10]-=-. They found evidence to support universality in facial expressions. These “universal facial expressions” are those representing happiness, sadness, anger, fear, surprise, and disgust. They studied fa... |

90 | Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face - Bassili - 1979 |

78 |
Facial action coding system: Investigator’s guide
- Ekman, Friesen
- 1978
(Show Context)
Citation Context ...ore reluctant to show their real expressions. Babies seem to exhibit a wide range of facial expressions without being taught, thus suggesting that these expressions are innate [18]. Ekman and Friesen =-=[11]-=- developed the Facial Action Coding System (FACS) to code facial expressions where 3movements on the face are described by a set of action units (AUs). Each AU has some related muscular basis. This s... |

78 |
Human Expression Recognition from Motion Using a Radial Basis Function Network Architecture
- Rosenblum, Yacoob, et al.
- 1996
(Show Context)
Citation Context ...based classifier to recognize the six basic facial expressions. Yacoob and Davis [41] computed optical flow and used similar rules to classify the six facial expressions. Rosenblum, Yacoob, and Davis =-=[35]-=- also computed optical flow of regions on the face, then applied a radial basis function network to classify expressions. Essa and Pentland [12] used an optical flow region-based method to recognize e... |

78 | A unified Approach to Coding and Interpreting Face Images
- Lantis
- 1995
(Show Context)
Citation Context ...not start until the 1990s. Mase [25] used optical flow (OF) to recognize facial expressions. He was one of the first to use image processing techniques to recognize facial expressions. Lanitis et al. =-=[21]-=- used a flexible shape and appearance model for image coding, person identification, pose recovery, gender recognition, and facial expression recognition. Black and Yacoob [2] used local parameterized... |

75 | Probability, Random processes, and Estimation Theory for Engineers - Stark, Woods - 1994 |

58 | Lafter: lips and face real time tracker
- Oliver, Pentland, et al.
- 1997
(Show Context)
Citation Context ...temporal pattern in displaying facial expression. We first describe the hidden Markov model (HMM) based classifiers for facial expression recognition which have been previously used 2in recent works =-=[23, 29, 30]-=-. We further advance this line of research and propose a multi-level HMM classifier, combining the temporal information which allows not only to perform the classification of a video segment to the co... |

55 | Inferring psychological significance from physiological signals
- Cacioppo, Tassinary
- 1990
(Show Context)
Citation Context ...nding of human emotional states. Voice and gestures are widely believed to play an important role as well [4, 8], and physiological states such as heart beat and skin conductivity are being suggested =-=[3]-=-. People also use context as an indicator of the emotional state of a person. This work is just another step on the way toward achieving the goal of building more effective computers that can serve us... |

51 |
Facial emotion recognition using multi-modal information
- Silva, Miyasato, et al.
- 1997
(Show Context)
Citation Context ...ld be expected to improve the recognition rates and eventually improve the computer’s understanding of human emotional states. Voice and gestures are widely believed to play an important role as well =-=[4, 8]-=-, and physiological states such as heart beat and skin conductivity are being suggested [3]. People also use context as an indicator of the emotional state of a person. This work is just another step ... |

51 |
Innate and universal facial expressions: Evidence from developmental and crosscultural research
- Izard
- 1994
(Show Context)
Citation Context ... Japanese viewers were more reluctant to show their real expressions. Babies seem to exhibit a wide range of facial expressions without being taught, thus suggesting that these expressions are innate =-=[18]-=-. Ekman and Friesen [11] developed the Facial Action Coding System (FACS) to code facial expressions where 3movements on the face are described by a set of action units (AUs). Each AU has some relate... |

49 | Probabilistic modeling for face orientation discrimination: Learning from labeled and unlabeled data
- Baluja
(Show Context)
Citation Context ...hough the independence assumption is usually violated in practice. Examples of applications that use the Naive-Bayes classifiers are abundant such as text classification [27] and face pose estimation =-=[1]-=-. Although the Naive-Bayes model does not reflect in many cases the true underlying model generating the data, it is still observed to be successful as a classifier in practice. The reason for the Nai... |

45 |
Emotional Intelligence”, Bantam
- Goleman
- 1995
(Show Context)
Citation Context ...play of emotions. Emotions are displayed by visual, vocal, and other physiological means. There is a growing amount of evidence showing that emotional skills are part of what is called “intelligence” =-=[16, 36]-=-. One of the important way humans display emotions is through facial expressions. This work describes our real time automatic facial expression recognition system using video input. Our work focuses o... |

45 | Toward improved Ranking Metrics
- Sebe, Lew, et al.
- 2000
(Show Context)
Citation Context ...n the class label. In practice, the common assumption is that we have a Gaussian distribution and the ML can be used to obtain the estimate of the parameters (mean and variance). However, Sebe et al. =-=[37]-=- have shown that the Gaussian assumption is often invalid and proposed the Cauchy distribution as an alternative model. Intuitively, this distribution can be thought of as being able to model the heav... |

44 | Automatic Lipreading by Optical-Flow Analysis - Mase, Pentland - 1991 |

41 | Connected vibrations: A modal analysis approach to non-rigid motion tracking, in
- Tao, Huang
- 1998
(Show Context)
Citation Context ...tors related to each frame such as the HMM based methods of [23, 28, 30]. 3 Face Tracking and Feature Extraction The face tracking we use in our system is based on a system developed by Tao and Huang =-=[39]-=- called the Piecewise Bézier Volume Deformation (PBVD) tracker. This face tracker uses a model-based approach where an explicit 3D wireframe model of the face is constructed. In the first frame of the... |

37 | Principal variables - McCabe - 1984 |

35 | Face Recognition Using an Embedded HMM
- Nefian, Hayes
- 1999
(Show Context)
Citation Context ...). Furthermore, they used the tracked motions to control the facial expression of an animated Kabuki system [31]. A similar approach, using different features, was used by Lien [23]. Nefian and Hayes =-=[26]-=- proposed an embedded HMM approach for face recognition that uses an efficient set of observation vectors based on the DCT coefficients. Martinez [24] introduced an indexing approach based on the iden... |

34 | ma - no hands!’ handsfree cursor control with real-time 3d face tracking - Toyama, “‘Look - 1998 |

34 | Joint Processing of Audio-Visual Information for the Recognition of Emotional Expressions
- Chen
(Show Context)
Citation Context ...e. The first is a database we collected of subjects that were instructed to display facial expressions corresponding to the six types of emotions. The data collection method is described in detail in =-=[4]-=-. All the tests of the algorithms 19are performed on a set of five people, each one displaying six sequences of each one of the six emotions, and always coming back to a neutral state between each em... |

32 | Automatic Recognition of Facial Expressions using Hidden Markov Models and Estimation of Expression Intensity,” Doctoral dissertation
- Lien
- 1998
(Show Context)
Citation Context ...temporal pattern in displaying facial expression. We first describe the hidden Markov model (HMM) based classifiers for facial expression recognition which have been previously used 2in recent works =-=[23, 29, 30]-=-. We further advance this line of research and propose a multi-level HMM classifier, combining the temporal information which allows not only to perform the classification of a video segment to the co... |

24 | Recognizing multiple persons facial expressions using HMM based on automatic extraction of significant frames from image sequences
- Otsuka, Ohya
(Show Context)
Citation Context ...temporal pattern in displaying facial expression. We first describe the hidden Markov model (HMM) based classifiers for facial expression recognition which have been previously used 2in recent works =-=[23, 29, 30]-=-. We further advance this line of research and propose a multi-level HMM classifier, combining the temporal information which allows not only to perform the classification of a video segment to the co... |

23 | Statistical methods of comparing different multivariate analyses of the same data - Gower - 1971 |

21 | Clustering and Classification - Arabie - 1996 |

16 | Developments in linear regression methodology - Hocking - 1983 |

16 | Selection of Variables to Preserve Multivariate Data Structure, Using Principal Components - Krzanowski - 1987 |

16 | Understanding probabilistic classifiers - Garg, Roth |

15 | Face Image Retrieval Using HMMs
- Martinez
- 1999
(Show Context)
Citation Context ...features, was used by Lien [23]. Nefian and Hayes [26] proposed an embedded HMM approach for face recognition that uses an efficient set of observation vectors based on the DCT coefficients. Martinez =-=[24]-=- introduced an indexing approach based on the identification of frontal face images under different illumination conditions, facial expressions, and occlusions. A Bayesian approach was used to find th... |

14 | Audio-visual speaker detection using dynamic bayesian networks - Garg, Pavlovic, et al. - 2000 |

14 | Emotion recognition from audiovisual information - Chen, Tao, et al. - 1998 |

11 |
Inference for the cauchy distribution based on maximum likelihood estimators
- Haas, Bain, et al.
- 1970
(Show Context)
Citation Context ...with the starting points given by the mean and the variance of the data. We were always able to find unique positive solutions for and which is in accordance with the conjecture stated by Hass et al. =-=[17]-=-. In certain cases, however, the Newton-Raphson iteration diverged, in which cases we selected new starting points. An interesting problem is determining when to use the Cauchy assumption versus the G... |

9 | Expression analysis/synthesis system based on emotion space constructed by multilayered neural network - Ueki, Morishima, et al. - 1995 |