## Context-sensitive hidden Markov models for modeling long-range dependencies in symbol sequences (2006)

Venue: | IEEE Trans. Signal Processing |

Citations: | 13 - 9 self |

### BibTeX

@ARTICLE{Yoon06context-sensitivehidden,

author = {Byung-jun Yoon and Student Member and P. P. Vaidyanathan},

title = {Context-sensitive hidden Markov models for modeling long-range dependencies in symbol sequences},

journal = {IEEE Trans. Signal Processing},

year = {2006},

volume = {54},

pages = {4169--4184}

}

### OpenURL

### Abstract

The hidden Markov model (HMM) has been widely used in signal processing and digital communication applications. It is well-known for its efficiency in modeling short-term dependencies between adjacent symbols. However, it cannot be used for modeling long-range interactions between symbols that are distant from each other. In this paper, we introduce the concept of context-sensitive HMM. The proposed model is capable of modeling strong pairwise correlations between distant symbols. Based on this model, we propose dynamic programming algorithms that can be used for finding the optimal state sequence and for computing the probability of an observed symbol string. Furthermore, we also introduce a parameter re-estimation algorithm, which can be used for optimizing the model parameters based on the given training sequences. 1

### Citations

4597 | A tutorial on hidden Markov models and selected applications in speech processing
- Rabiner
- 1989
(Show Context)
Citation Context ...symbols, which made them popular in diverse areas. Traditionally, HMMs have been successfully applied to speech recognition, and many speech recognition systems are built upon HMMs and their variants =-=[1, 2]-=-. They have been also widely used in digital communications, and more recently, HMMs have become very popular in computational biology as well. They have been proved to be useful in various problems s... |

1279 |
Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm
- Viterbi
- 1967
(Show Context)
Citation Context ...e viewed as stochastic regular grammars (SRG), according to this hierarchy. Due to the restrictions on their production rules, regular grammars have efficient algorithms such as the Viterbi algorithm =-=[15]-=- for finding the optimal state sequence (popularly used in digital communication receivers), the forward algorithm [1, 2] for computing the probability of an observed symbol string, and the BaumWelch ... |

798 |
Statistical Methods for Speech Recognition”, The
- Jelinek
- 1997
(Show Context)
Citation Context ...symbols, which made them popular in diverse areas. Traditionally, HMMs have been successfully applied to speech recognition, and many speech recognition systems are built upon HMMs and their variants =-=[1, 2]-=-. They have been also widely used in digital communications, and more recently, HMMs have become very popular in computational biology as well. They have been proved to be useful in various problems s... |

650 |
Biological Sequence Analysis
- Durbin, Eddy, et al.
- 1998
(Show Context)
Citation Context ...y used in digital communications, and more recently, HMMs have become very popular in computational biology as well. They have been proved to be useful in various problems such as gene identification =-=[3, 4, 5]-=-, multiple sequence alignment [5, 6], and so forth. Due to its effectiveness in modeling symbol sequences, the HMM gave rise to a number of useful variants that extend and generalize the basic model [... |

550 |
An inequality and associated maximization technique in statistical estimation for probabilistic functions of Markov processes. Inequalities
- Baum
- 1972
(Show Context)
Citation Context ...ng the optimal state sequence (popularly used in digital communication receivers), the forward algorithm [1, 2] for computing the probability of an observed symbol string, and the BaumWelch algorithm =-=[16]-=- for re-estimation of the model parameters. Other transformational grammars that belong to a higher order class in the hierarchy have less restrictions on their production rules, and therefore they ha... |

382 |
The estimation of stochastic context-free grammars using the Inside-Outside algorithm, Computer Speech and Language
- Lari, Young
- 1990
(Show Context)
Citation Context ...al-time algorithm that solves the alignment problem in a recursive manner, similar to the Viterbi algorithm. The proposed algorithm is conceptually similar to the Cocke-Younger-Kasami (CYK) algorithm =-=[22, 23]-=- that can be used for parsing SCFGs. The main reason why the Viterbi algorithm cannot be used in context-sensitive HMMs is because the interactions between symbols are not sequential. Since the Viterb... |

336 |
Introduction to Formal Language Theory
- Harrison
- 1978
(Show Context)
Citation Context ...al-time algorithm that solves the alignment problem in a recursive manner, similar to the Viterbi algorithm. The proposed algorithm is conceptually similar to the Cocke-Younger-Kasami (CYK) algorithm =-=[22, 23]-=- that can be used for parsing SCFGs. The main reason why the Viterbi algorithm cannot be used in context-sensitive HMMs is because the interactions between symbols are not sequential. Since the Viterb... |

273 |
On certain formal properties of grammars
- Chomsky
- 1959
(Show Context)
Citation Context ...ore complex sequences with non-sequential dependencies cannot be effectively represented using these HMMs. In his work on transformational grammars, Chomsky categorized all grammars into four classes =-=[14]-=-. These include regular grammars, context-free grammars (CFG), context-sensitive grammars (CSG), and unrestricted grammars, in the order of decreasing restrictions on the production rules. The aforeme... |

186 | Microbial gene identification using interpolated Markov models. Nucleic Acids Res
- Salzberg, Delcher, et al.
- 1998
(Show Context)
Citation Context ...y used in digital communications, and more recently, HMMs have become very popular in computational biology as well. They have been proved to be useful in various problems such as gene identification =-=[3, 4, 5]-=-, multiple sequence alignment [5, 6], and so forth. Due to its effectiveness in modeling symbol sequences, the HMM gave rise to a number of useful variants that extend and generalize the basic model [... |

156 | Multiple alignment using hidden Markov models, Proceedings of the third international conference on Intelligent systems for molecular biology, Menlo
- Eddy
- 1995
(Show Context)
Citation Context ...ore recently, HMMs have become very popular in computational biology as well. They have been proved to be useful in various problems such as gene identification [3, 4, 5], multiple sequence alignment =-=[5, 6]-=-, and so forth. Due to its effectiveness in modeling symbol sequences, the HMM gave rise to a number of useful variants that extend and generalize the basic model [7]-[13]. Although HMMs have a number... |

106 |
Context-dependent phonetic hidden markov models for speaker-independent continuous speech recognition
- Lee
- 1990
(Show Context)
Citation Context ...concluded in Sec. 8. It has to be noted that the context-sensitive HMMs proposed in this paper are not related to the so-called context-dependent HMMs that have been widely used in speech recognition =-=[17, 18, 19]-=-. They are regular HMMs, whose basic building blocks are built by considering the phonetic context, hence called context-dependent HMMs. 3 A transformational grammar has two kinds of symbols, namely, ... |

86 |
control of vertebrate iron metabolism: mRNA-based regulatory circuits operated by iron, nitric oxide, and oxidative stress
- Hentze, Kuhn
- 1996
(Show Context)
Citation Context ... Fig. 13. This csHMM generates sequences with long-range correlations between distant symbols. Such pairwise dependencies are commonly found in the so-called “iron response elements” in RNA sequences =-=[24]-=-. The model in Fig. 13 has three single-emission states S1, S2 and S3, and two pairs of pairwise-emission states and context-sensitive states. Each pair (P1, C2) and (P2, C2) is associated with a sepa... |

51 | Pairwise Markov chains
- Pieczynski
- 2003
(Show Context)
Citation Context ...these models cannot explicitly model pairwise correlations between distant symbols as the csHMM does. There exists another interesting generalization of the HMM called the pairwise Markov chain (PMC) =-=[12]-=-. The PMC assumes that the pair of the random variables (xi, si) is a Markov chain. This model is mathematically more general than the HMM, which is a special case of the PMC, where the hidden state s... |

41 | A hidden Markov model that finds genes
- Krogh, Mian, et al.
- 1994
(Show Context)
Citation Context ...y used in digital communications, and more recently, HMMs have become very popular in computational biology as well. They have been proved to be useful in various problems such as gene identification =-=[3, 4, 5]-=-, multiple sequence alignment [5, 6], and so forth. Due to its effectiveness in modeling symbol sequences, the HMM gave rise to a number of useful variants that extend and generalize the basic model [... |

35 |
Estimating hidden semi-Markov chains from discrete sequences
- Guédon
- 2003
(Show Context)
Citation Context ...], multiple sequence alignment [5, 6], and so forth. Due to its effectiveness in modeling symbol sequences, the HMM gave rise to a number of useful variants that extend and generalize the basic model =-=[7]-=--[13]. Although HMMs have a number of advantages, the basic HMM [1, 2] and its variants in [7]-[13] have also inherent limitations. For example, they are capable of modeling sequences with strong corr... |

34 | An efficient forward-backward algorithm for an explicit duration hidden markov model - Yu, Kobashi |

22 |
Triplet Markov chains in hidden signal restoration
- Pieczynski, W, et al.
(Show Context)
Citation Context ...ultiple sequence alignment [5, 6], and so forth. Due to its effectiveness in modeling symbol sequences, the HMM gave rise to a number of useful variants that extend and generalize the basic model [7]-=-=[13]-=-. Although HMMs have a number of advantages, the basic HMM [1, 2] and its variants in [7]-[13] have also inherent limitations. For example, they are capable of modeling sequences with strong correlati... |

22 |
accuracy phone recognition using context clustering and quasitriphonec models
- Ljolje
- 1994
(Show Context)
Citation Context ...concluded in Sec. 8. It has to be noted that the context-sensitive HMMs proposed in this paper are not related to the so-called context-dependent HMMs that have been widely used in speech recognition =-=[17, 18, 19]-=-. They are regular HMMs, whose basic building blocks are built by considering the phonetic context, hence called context-dependent HMMs. 3 A transformational grammar has two kinds of symbols, namely, ... |

19 |
Improved hidden Markov modeling of phonemes for continuous speech recognition
- Schwartz, Klovstadt, et al.
- 1980
(Show Context)
Citation Context ...concluded in Sec. 8. It has to be noted that the context-sensitive HMMs proposed in this paper are not related to the so-called context-dependent HMMs that have been widely used in speech recognition =-=[17, 18, 19]-=-. They are regular HMMs, whose basic building blocks are built by considering the phonetic context, hence called context-dependent HMMs. 3 A transformational grammar has two kinds of symbols, namely, ... |

17 | Hidden hybrid Markov/semi-Markov chains - Guédon - 2005 |

17 | HMM with auxiliary memory: a new tool for modeling RNA structures
- Yoon, Vaidynathan
- 2004
(Show Context)
Citation Context ...that we observe. 3s2 Context-Sensitive Hidden Markov Models The context-sensitive HMM can be viewed as an extension of the traditional HMM, where some of the states are equipped with auxiliary memory =-=[20, 21]-=-. Symbols that are emitted at certain states are stored in the memory, and the stored data serves as the context which affects the emission probabilities and the transition probabilities of the model.... |

9 | Unsupervised learning and mapping of active brain functional MRI signals based on hidden semi-Markov event sequence models - Faisan, Thoraval, et al. - 2005 |

8 |
Speech reconstruction using a generalized HSMM
- Moore, Savic
(Show Context)
Citation Context ...tional HMM, which extend the basic model in various ways [7]-[13]. For example, the hidden semi-Markov model (HSMM) allows us to associate an explicit state occupancy distribution with each state [7]-=-=[11]-=-, instead of using the implicit geometric state occupancy distribution in the basic HMM. However, the hidden states in the HSMM are not context-sensitive, and the emission and transition probabilities... |

6 | An overview of the role of context-sensitive HMMs in the prediction of ncRNA genes
- Yoon, Vaidyanthan
(Show Context)
Citation Context ...that we observe. 3s2 Context-Sensitive Hidden Markov Models The context-sensitive HMM can be viewed as an extension of the traditional HMM, where some of the states are equipped with auxiliary memory =-=[20, 21]-=-. Symbols that are emitted at certain states are stored in the memory, and the stored data serves as the context which affects the emission probabilities and the transition probabilities of the model.... |