Abstract

We explain how to apply statistical techniques to solve several language-recognition problems that arise in cryptanalysis and other domains. Language recognition is important in cryptanalysis because, among other applications, an exhaustive key search of any cryptosystem from ciphertext alone requires a test that recognizes valid plaintext. Written for cryptanalysts, this guide should also be helpful to others as an introduction to statistical inference on Markov chains. Modeling language as a finite stationary Markov process, we adapt a statistical model of pattern recognition to language recognition. Within this framework we consider four welldefined language-recognition problems: 1) recognizing a known language, 2) distinguishing a known language from uniform noise, 3) distinguishing unknown 0th-order noise from unknown 1st-order language, and 4) detecting non-uniform unknown language. For the second problem we give a most powerful test based on the Neyman-Pearson Lemma. For the oth...

Citations

3116	A tutorial on hidden Markov models and selected applications in speech recognition - Rabiner - 1989
2262	Convergence of Probability Measures - Billingsley - 1968
1391	Robust Statistics - Huber - 1981
1105	Information theory and reliable communication - Gallager - 1968
596	Communication theory of secrecy systems - Shannon - 1949
473	An introduction to signal detection and estimation - Poor - 1994
462	Cryptographyand Data Security - Denning - 1982
339	Testing statistical hypotheses - Lehmann - 1959
279	Prediction and entropy of printed English - Shannon - 1951
187	Speakerindependent phone recognition using hidden Markov models - Lee, Hon - 1989
148	Discrete Multivariate Analysis - Bishop, Fienberg, et al. - 1975
135	Modulation Theory - Detection - 1968
130	Probability and Statistics with Reliability - Trivedi - 2001
125	The Estimation of Probability: An Essay on Modern Bayesian Methods - Good - 1965
121	Mathematical statistics - Wilks - 1962
78	Probability and The Weighing of Evidence - Good - 1950
76	Asymptototic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions - Self, Liang - 1987
73	A Probabilistic Distance Measure for Hidden Markov Models - Juang, Rabiner - 1985
67	Good Thinking: The Foundations of Probability and Its Applications - Good - 1983
62	A survey of exact inference for contingency tables - Agresti - 1992
61	The Codebreakers: The Story of Secret Writing - Kahn - 1996
57	Statistical Inference for Markov Processes - Billingsley - 1961
56	Testing for independence in a two-way table: New interpretations of the chi-square statistic - Diaconis, Efron - 1985
52	LA: Statistical inference about Markov chains - TW, Goodman - 1957
51	Introduction to Mathematical Statistics - Hoel - 1971
49	Seminumerical Algorithms, The Art of - Knuth - 1997
49	Pattern Analysis and Understanding - Niemann - 1990
42	Elements of Applied Stochastic Processes - Bhat - 1972
42	An Introduction to - Larsen, Marx - 1986
39	Cipher Systems: The Protection of Communications - Beker, Piper - 1982
36	Statistical methods in Markov chains - Billingsley - 1961
30	Effects of sample size in classifier design - Fukunaga, Hayes - 1989
23	Pseudo-random generators and complexity classes - Boppana, Hirschfeld - 1989
23	On the application of symmetric Dirichlet distributions and their mixtures to contingency tables - Good - 1976
21	On the Application of Mixture AR Hidden Markov Models to Text Independent Speaker Recognition - Tishby - 1991
17	Decision making in Markov chains applied to the problem of pattern recognition - Raviv - 1967
12	On the cryptanalysis of rotor machines and substitution-permutations networks - Andleman, Reeds - 1982
12	Positional frequency and versatility of bigrams for two- through nine-letter English words - Solso, Juel - 1980
11	Statistical Inference - Rohatgi - 1984
10	Elementary Cryptanalysis: A Mathematical Approach - Sinkov - 1980
9	The Bayes/non-Bayes compromise: A brief review - Good - 1992
9	Cryptography. In Handbook of Theoretical Computer Science (Volume A: Algorithms and Complexity - Rivest - 1990
8	How to Break Gifford's Cipher - Cain, Sherman - 1994
8	A Bayesian significance test for multinomial distributions - Good - 1967
8	Patterns in pattern recognition - Kanal - 1974
6	Tests for contingency tables and Markov chains - Kullback, Kupperman, et al. - 1962
6	Development of an acoustic–phonetic hidden Markov model for continuous speech recognition - Ljolje, Levinson - 1991
6	Adaptive, Learning and Pattern Recognition Systems: Theory and Applications - Fu, Mendel - 1970
4	The weighted likelihood ratio, sharp hypotheses about chances, the order of a Markov chain - Dickey, Lientz - 1970
4	A new spectral test for nonrandomness and the DES - Feldman - 1990