DMCA
Similarity Join Size Estimation using Locality Sensitive Hashing ∗
| Citations: | 5 - 0 self |
BibTeX
@MISC{Lee_similarityjoin,
author = {Hongrae Lee and Raymond T. Ng and Kyuseok Shim},
title = {Similarity Join Size Estimation using Locality Sensitive Hashing ∗},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Similarity joins are important operations with a broad range of applications. In this paper, we study the problem of vector similarity join size estimation (VSJ). It is a generalization of the previously studied set similarity join size estimation (SSJ) problem and can handle more interesting cases such as TF-IDF vectors. One of the key challenges in similarity join size estimation is that the join size can change dramatically depending on the input similarity threshold. We propose a sampling based algorithm that uses Locality-Sensitive-Hashing (LSH). The proposed algorithm LSH-SS uses an LSH index to enable effective sampling even at high thresholds. We compare the proposed technique with random sampling and the state-of-the-art technique for SSJ (adapted to VSJ) and demonstrate LSH-SS offers more accurate estimates throughout the similarity threshold range and small variance using real-world data sets. 1.
Keyphrases
similarity join size estimation locality sensitive hashing broad range tf-idf vector high threshold input similarity threshold important operation algorithm lsh-ss vector similarity join size estimation lsh index key challenge real-world data set join size small variance state-of-the-art technique similarity threshold range interesting case accurate estimate similarity join lsh-ss offer random sampling
