## Higher lower bounds for near-neighbor and further rich problems

Venue: | in Proc. 47th IEEE Symposium on Foundations of Computer Science (FOCS |

Citations: | 17 - 2 self |

### BibTeX

@INPROCEEDINGS{Thorup_higherlower,

author = {Mikkel Thorup},

title = {Higher lower bounds for near-neighbor and further rich problems},

booktitle = {in Proc. 47th IEEE Symposium on Foundations of Computer Science (FOCS},

year = {},

pages = {646--654}

}

### OpenURL

### Abstract

We convert cell-probe lower bounds for polynomial space into stronger lower bounds for near-linear space. Our technique applies to any lower bound proved through the richness method. For example, it applies to partial match, and to near-neighbor problems, either for randomized exact search, or for deterministic approximate search (which are thought to exhibit the curse of dimensionality). These problems are motivated by search in large databases, so near-linear space is the most relevant regime. Typically, richness has been used to imply Ω(d / lg n) lower bounds for polynomial-space data structures, where d is the number of bits of a query. This is the highest lower bound provable through the classic reduction to communication complexity. However, for space n lg O(1) n, we now obtain bounds of Ω(d / lg d). This is a significant improvement for natural values of d, such as lg O(1) n. In the most important case of d = Θ(lg n), we have the first superconstant lower bound. From a complexity theoretic perspective, our lower bounds are the highest known for any static data structure problem, significantly improving on previous records. 1

### Citations

759 | Approximate nearest neighbors: Towards removing the curse of dimensionality
- Indyk, Motwani
- 1998
(Show Context)
Citation Context ...arching for a γ-approximate nearest neighbor. If we allow both approximation and randomization, it is possible to avoid the curse of dimensionality, at least for constant approximation. If γ = 1 + ε, =-=[8]-=- and [10] provide data structures of size n O(1/ε2 ) which can solve the problem with O(1) cell probes. However, prohibiting either randomization or approximation seems to make the problem much harder... |

196 | Efficient search for approximate nearest neighbor in high dimensional spaces
- Kushilevitz, Ostrovsky, et al.
(Show Context)
Citation Context ...for a γ-approximate nearest neighbor. If we allow both approximation and randomization, it is possible to avoid the curse of dimensionality, at least for constant approximation. If γ = 1 + ε, [8] and =-=[10]-=- provide data structures of size n O(1/ε2 ) which can solve the problem with O(1) cell probes. However, prohibiting either randomization or approximation seems to make the problem much harder. Despite... |

54 | Dictionary matching and indexing with errors and don't cares
- Cole, Gottlieb, et al.
- 2004
(Show Context)
Citation Context ...ents in {0, 1} d , and a query in {0, 1, ⋆} d , the goal is to find a matching element in the database, where ⋆ matches anything. For a small number of ⋆ values, the problem can be solved efficiently =-=[7]-=-, but the bounds decay exponentially with the number of wild-cards. 1sPrevious work. Motivated by the conjectured curse of dimensionality, there has been much recent work on giving cell-probe lower bo... |

47 | Lower bounds for high dimensional nearest neighbor search and related problems
- Borodin, Ostrovsky, et al.
- 1999
(Show Context)
Citation Context ...ng either randomization or approximation seems to make the problem much harder. Despite extensive research, all known solutions have either space or time growing exponentially in d (see references in =-=[4]-=-). Another famous problem which seems to exhibit a curse of dimensionality is the partial match problem. Given n elements in {0, 1} d , and a query in {0, 1, ⋆} d , the goal is to find a matching elem... |

25 |
Shmuel Safra, and Avi Wigderson. On data structures and asymmetric communication complexity
- Miltersen, Nisan
- 1998
(Show Context)
Citation Context ...uery and the other holds the database. Lower bounds in asymmetric communication complexity have been shown through variants of only two techniques, richness and round elimination, both dating back to =-=[12]-=-. In general, richness is useful for “harder” problems, and can show significantly higher bounds than round elimination. Thus, it is more interesting when thinking about the curse of dimensionality. T... |

21 | On the optimality of the dimensionality reduction method
- ANDONI, INDYK, et al.
(Show Context)
Citation Context ... about data structures, instead of going through communication complexity. Define ρi : X × Y × {0, 1} → {0, 1} by ρi(x, y, z) = 1 if fi(x, y) �= z, and 0 otherwise. Also let ρ : Xk × Y k × {0, 1} k → =-=[0, 1]-=- be ρ(x, y, z) = 1 � k i ρi(xi, yi, zi). In other words, ρ measures the fraction of the outputs from z which are wrong. Lemma 8. Let ε > 99 k be arbitrary, and f1, . . . , fk be at least ε-dense. Assu... |

15 |
A strong lower bound for approximate nearest neighbor searching in the cell probe model. manuscript
- Liu
- 2003
(Show Context)
Citation Context ...am et al. [9] analyzed partial match directly and proved an almost maximal communication bound with a = Ω(d/ lg n) and b = Ω(n1−ε ). This yields a cell-probe lower bound of Ω(d/ lg 2 n). Finally, Liu =-=[11]-=- showed a tight communication lower bound for deterministic O(1)-approximate near neighbor search, giving a = Ω(d) and b = Ω(nd), hence a cell-probe lower bound of Ω(d/ lg n). We note that a parallel ... |

11 | Tighter lower bounds for nearest neighbor search and related problems in the cell probe model
- Barkol, Rabani
- 2000
(Show Context)
Citation Context ... bounds also apply to exact near-neighbor search. Thus, both partial match and exact near-neighbor search have cell-probe complexity Ω(lg d), for any w = O(n1−ε / lg d). In STOC’00, Barkol and Rabani =-=[2]-=- revisited randomized near-neighbor search and showed a lower bound with a = Ω(d) and b = Ω(n1/8−ε ). Note that this is optimal with regard to the querier’s communication. Furthermore, b is still larg... |

8 |
Alexey Lvov. A lower bound on the complexity of approximate nearest-neighbor searching on the hamming cube
- Chakrabarti, Chazelle, et al.
- 1999
(Show Context)
Citation Context ... n). We note that a parallel thread of research has been investigating the randomized approximate nearest neighbor problem, using variants of round elimination. Building on work of Chakrabarti et al. =-=[5]-=- from STOC’99, Chakrabarti and Regev [6] in FOCS’04 showed a tight cell-probe bound of Θ(lg lg d/ lg lg lg d). Remember that when both approximation and randomization are allowed, the near neighbor pr... |

4 |
Yuval Rabani. Cell-probe lower bounds for the partial match problem
- Jayram, Khot, et al.
- 2003
(Show Context)
Citation Context ...e, b is still large enough that it is irrelevant for reasonable word size, giving a cell-probe complexity of Ω(d/ lg n). This lower bound did not apply to partial match, but in STOC’03, Jayram et al. =-=[9]-=- analyzed partial match directly and proved an almost maximal communication bound with a = Ω(d/ lg n) and b = Ω(n1−ε ). This yields a cell-probe lower bound of Ω(d/ lg 2 n). Finally, Liu [11] showed a... |

3 |
Avi Wigderson. A direct sum theorem for corruption and a lower bound for the multiparty communication complexity of set disjointness
- Beame, Pitassi, et al.
- 2007
(Show Context)
Citation Context ...f these measures is relevant to asymmetric communication. In the symmetric case, corruption is the closest analog to our richness measure. A direct product result for corruption was recently shown by =-=[3]-=-. At a superficial level, the approach of our Lemma 9 resembles the approach of [3]. However, 1 We have learned that Paul Beame and Matthew Cary independently observed this property. 3sthe devil is in... |