Citation Context

...ications and deserve further study. In this section we briefly mention just two. Conditional random fields can be trained using the exponential loss objective function used by the AdaBoost algorithm (=-=Freund & Schapire, 1997-=-). Typically, boosting is applied to classification problems with a small, fixed number of classes; applications of boosting to sequence labeling have treated each label as a separate classification p...

Citation Context

...tates can be traded off against each other. We can also think of a CRF as a finite state model with unnormalized transition probabilities. However, unlike some other weighted finite-state approaches (=-=LeCun et al., 1998-=-), CRFs assign a well-defined probability distribution over possible labelings, trained by maximum likelihood or MAP estimation. Furthermore, the loss function is convex,2 guaranteeing convergence to ...

Citation Context

...cations of exponential models in sequence modeling have either attempted to build generative models (Rosenfeld, 1997), which involve a hard normalization problem, or adopted local conditional models (=-=Berger et al., 1996-=-; Ratnaparkhi, 1996; McCallum et al., 2000) that may suffer from label bias. Non-probabilistic local decision models have also been widely used in segmentation and tagging (Brill, 1995; Roth, 1998; Ab...

Citation Context

...ational biology, HMMs and stochastic grammars have been successfully used to align biological sequences, find sequences homologous to a known evolutionary family, and analyze RNA secondary structure (=-=Durbin et al., 1998-=-). In computational linguistics and computer science, HMMs and stochastic grammars have been applied to a wide variety of problems in text and speech processing, including topic segmentation, part-ofs...

Citation Context

...models (Berger et al., 1996; Ratnaparkhi, 1996; McCallum et al., 2000) that may suffer from label bias. Non-probabilistic local decision models have also been widely used in segmentation and tagging (=-=Brill, 1995-=-; Roth, 1998; Abney et al., 1999). Because of the computational complexity of global training, these models are only trained to minimize the error of individual label decisions assuming that neighbori...

Citation Context

...entical state structure on a part-of-speech tagging task. 2. The Label Bias Problem Classical probabilistic automata (Paz, 1971), discriminative Markov models (Bottou, 1991), maximum entropy taggers (=-=Ratnaparkhi, 1996-=-), and MEMMs, as well as non-probabilistic sequence tagging and segmentation models with independently trained next-state classifiers (Punyakanok & Roth, 2001) are all potential victims of the label b...

Citation Context

...stance conditional independence given the labels, to achieve tractability. Maximum entropy Markov models (MEMMs) are conditional probabilistic sequence models that attain all of the above advantages (=-=McCallum et al., 2000-=-). In MEMMs, each source state1 has a exponential model that takes the observation features as input, and outputs a distribution over possible next states. These exponential models are trained by an a...

Citation Context

... structure of the model. In the above example we could collapse states 1 and 4, and delay the branching until we get a discriminating observation. This operation is a special case of determinization (=-=Mohri, 1997-=-), but determinization of weighted finite-state machines is not always possible, and even when possible, it may lead to combinatorial explosion. The other solution mentioned is to start with a fullyco...

Citation Context

... use a more global discriminative model to rerank those candidates. This approach is standard in large-vocabulary speech recognition (Schwartz & Austin, 1993), and has also been proposed for parsing (=-=Collins, 2000-=-). However, these methods fail when the correct output is pruned away in the first pass. Closest to our proposal are gradient-descent methods that adjust the parameters of all of the local classifiers...

Citation Context

...laimed advantages of conditional models by evaluating HMMs, MEMMs and CRFs with identical state structure on a part-of-speech tagging task. 2. The Label Bias Problem Classical probabilistic automata (=-=Paz, 1971-=-), discriminative Markov models (Bottou, 1991), maximum entropy taggers (Ratnaparkhi, 1996), and MEMMs, as well as non-probabilistic sequence tagging and segmentation models with independently trained...

Citation Context

...= (Y1,Y2, . . . ,Yn). If the graph G = (V,E) of Y is a tree (of which a chain is the simplest example), its cliques are the edges and vertices. Therefore, by the fundamental theorem of random fields (=-=Hammersley & Clifford, 1971-=-), the joint distribution over the label sequence Y given X has the form pθ(y |x) ∝ (1) exp ∑ e∈E,k λk fk(e,y|e,x) + ∑ v∈V,k µk gk(v,y|v,x) , where x is a data sequence, y a label sequence, and ...

Citation Context

...r et al., 1996; Ratnaparkhi, 1996; McCallum et al., 2000) that may suffer from label bias. Non-probabilistic local decision models have also been widely used in segmentation and tagging (Brill, 1995; =-=Roth, 1998-=-; Abney et al., 1999). Because of the computational complexity of global training, these models are only trained to minimize the error of individual label decisions assuming that neighboring labels ar...

Citation Context

...yconnected model and let the training procedure figure out a good structure. But that would preclude the use of prior structural knowledge that has proven so valuable in information extraction tasks (=-=Freitag & McCallum, 2000-=-). Proper solutions require models that account for whole state sequences at once by letting some transitions “vote” more strongly than others depending on the corresponding observations. This implies...

Citation Context

...ally, boosting is applied to classification problems with a small, fixed number of classes; applications of boosting to sequence labeling have treated each label as a separate classification problem (=-=Abney et al., 1999-=-). However, it is possible to apply the parallel update algorithm of Collins et al. (2000) to optimize the per-sequence exponential loss. This requires a forward-backward algorithm to compute efficien...

Citation Context

...e paper we tacitly assume that the graph G is fixed. In the simplest and most impor3Weighted determinization and minimization techniques shift transition weights while preserving overall path weight (=-=Mohri, 2000-=-); their connection to this discussion deserves further study. tant example for modeling sequences, G is a simple chain or line: G = (V = {1, 2, . . .m}, E = {(i, i + 1)}). X may also have a natural g...

Citation Context

...ng recall and doubling precision relative to HMMs in a FAQ segmentation task. MEMMs and other non-generative finite-state models based on next-state classifiers, such as discriminative Markov models (=-=Bottou, 1991-=-), share a weakness we call here the label bias problem: the transitions leaving a given state compete only against each other, rather than against all other transitions in the model. In probabilistic...

Citation Context

..., y) gy,x (v,y|v,x) = δ(yv, y) δ(xv, x) . The corresponding parameters λy′,y and µy,x play a similar role to the (logarithms of the) usual HMM parameters p(y′ | y) and p(x|y). Boltzmann chain models (=-=Saul & Jordan, 1996-=-; MacKay, 1996) have a similar form but use a single normalization constant to yield a joint distribution, whereas CRFs use the observation-dependent normalization Z(x) for conditional distributions. ...

Citation Context

...he benefits of conditional models with the global normalization of random field models. Other applications of exponential models in sequence modeling have either attempted to build generative models (=-=Rosenfeld, 1997-=-), which involve a hard normalization problem, or adopted local conditional models (Berger et al., 1996; Ratnaparkhi, 1996; McCallum et al., 2000) that may suffer from label bias. Non-probabilistic lo...

Citation Context

... δ(yv, y) δ(xv, x) . The corresponding parameters λy′,y and µy,x play a similar role to the (logarithms of the) usual HMM parameters p(y′ | y) and p(x|y). Boltzmann chain models (Saul & Jordan, 1996; =-=MacKay, 1996-=-) have a similar form but use a single normalization constant to yield a joint distribution, whereas CRFs use the observation-dependent normalization Z(x) for conditional distributions. Although it en...

Citation Context

...models (Bottou, 1991), maximum entropy taggers (Ratnaparkhi, 1996), and MEMMs, as well as non-probabilistic sequence tagging and segmentation models with independently trained next-state classifiers (=-=Punyakanok & Roth, 2001-=-) are all potential victims of the label bias problem. For example, Figure 1 represents a simple finite-state model designed to distinguish between the two words rib and rob. Suppose that the observat...