We review current statistical work on syntactic parsing and then consider part-of-speech tagging, which was the first syntactic problem to be successfully attacked by statistical techniques and also serves as a good warmup for the main topic, statistical parsing. Here we consider both the simplified case in which the input string is viewed as a string of parts of speech, and the more interesting case in which the parser is guided by statistical information about the particular words in the sentence. Finally we anticipate future research directions. 1 Introduction Syntactic parsing is the process of assigning a "phrase marker" to a sentence --- that is, the process that given a sentence like "The dog ate," produces a structure like that in Figure 1. In this example we adopt the standard abbreviations: np for "noun phrase," vp for "verb phrase," and det for "determiner." It is generally accepted that finding the sort of structure shown in Figure 1 is useful in determining the m...

There are two main methodologies for constructing the knowledge base of a natural language analyser: the linguis- tic and the data"driven. Recent state-of- the-art part-of-speech taggers are based on the data"driven approach. Because of the known feasibility of the linguistic rule-based approach at related levels of description, the success of the data" driven approach in part-of-speech analysis may appear surprising. In this paper, a case is made for the syntactic nature of part-of-speech tagging. A new tagger of English that uses only linguistic distributional rules is outlined and empirically evaluated. Tested against a benchmark corpus of 38,000 words of previously unseen text, this syntax-based system reaches an accuracy of above 99%. Compared to the 95-97% accuracy of its best competitors, this result suggests the feasibility of the linguistic approach also in part-of-speech analysis.

This report contains a sample text fragment annotated for part-of-speech, using a variety of taggers. It was used at CLUE to evaluate correctness of state-of-the-art taggers for English and for teaching purposes.