Abstract

Many applications need to solve the following problem of approximate string matching: from a collection of strings, how to find those similar to a given string, or the strings in another (possibly the same) collection of strings? Many algorithms are developed using fixed-length grams, which are substrings of a string used as signatures to identify similar strings. In this paper we develop a novel technique, called VGRAM, to improve the performance of these algorithms. Its main idea is to judiciously choose high-quality grams of variable lengths from a collection of strings to support queries on the collection. We give a full specification of this technique, including how to select high-quality grams from the collection, how to generate variable-length grams for a string based on the preselected grams, and what is the relationship between the similarity of the gram sets of two strings and their edit distance. A primary advantage of the technique is that it can be adopted by a plethora of approximate string algorithms without the need to modify them substantially. We present our extensive experiments on real data sets to evaluate the technique, and show the significant performance improvements on three existing algorithms. 1.

Citations

717	TC: Managing Gigabytes: compressing and indexing documents and images - IA, Moffat, et al. - 1999
306	A guided tour to approximate string matching - Navarro
193	Integration of heterogeneous databases without common domains using queries based on textual similarity - Cohen - 1998
130	Robust and efficient fuzzy match for online data cleaning - Chaudhuri, Ganjam, et al. - 2003
130	Approximate string-matching with q-grams and maximal matches, Theoretical Computer Science 92 - Ukkonen - 1992
129	Approximate string joins in a database (almost) for free - Gravano, Ipeirotis, et al. - 2001
78	E.: Sorting and searching - Knuth - 1973
76	Binary Codes Capable of Correcting Spurious Insertions and Deletions of Ones - Levenshtein - 1965
65	Efficient exact setsimilarity joins - Arasu, Ganti, et al.
61	Efficient set joins on similarity predicates - Sarawagi, Kirpal - 2004
56	A primitive operator for similarity joins in data cleaning - Chaudhuri, Ganti, et al. - 2006
49	Scaling up all pairs similarity search - Bayardo, Ma, et al. - 2007
49	Probabilistic linkage of large public health data file - Jaro - 1995
47	On using Q-Gram locations in approximate string matching - Sutinen, Tarhio - 1995
43	Record linkage: similarity measures and algorithms - Koudas, Sarawagi, et al. - 2006
42	A variable-length category-based n-gram language model - Niesler, Woodland - 1996
38	Filtration with q-samples in approximate string matching - Sutinen, Tarhio - 1996
33	Estimating alphanumeric selectivity in the presence of wildcards - Krishnan, Vitter, et al. - 1996
31	Comparative study of name disambiguation problem using a scalable blocking-based framework - On, Lee, et al.
29	Substring Selectivity Estimation - Jagadish, Ng, et al. - 1999
22	Selectivity Estimation for String Predicates: Overcoming the Underestimation Problem - Chaudhuri, Ganti, et al. - 2004
20	A Local Maxima Method and a Fair Dispersion Normalization for Extracting Multi-word Units - Silva, Lopes - 1999
19	n-gram/2l: A space and time efficient two-level n-gram inverted index structure - Kim, Whang, et al. - 2005
8	N-gram feature selection for authorship identification - Houvardas, Stamatatos - 2006
3	q-gram matching using tree models - Fogla, Lee
1	Application of variable length n-gram vectors to monolingual and bilingual information retrieval - Gayo-Avello, Gutiérrez, et al. - 2004