Specific domains in video data contain rich temporal structures that help in classification process. In sports, the events that unfold are governed by the rules of the sport and hence, contain a recurring temporal structure. The rules of cinematographic production are also standardized. The classification of video data involves extracting patterns in the temporal behavior of each variable and also in dynamics of relationship between variables; and mapping these patterns to a high-level interpretation. We tackle this problem in a Dynamic Bayesian Network (DBN) framework. The temporal context of a segment of a video can be considered at different granularity depending on the desired application. Some algorithms on larger timeline are suggested (based on probability relaxation etc.) which modify the probabilities generated by DBN and can give more accurate classification. The novel applications of the system include classifying a group of shots called sequence and parsing a video program into individual segments by building a model of the video program. 1.