Results 1  10
of
30
A Derandomized Sparse JohnsonLindenstrauss Transform
"... Recent work of [DasguptaKumarSarlós, STOC 2010] gave a sparse JohnsonLindenstrauss transform and left as a main open question whether their construction could be efficiently derandomized. We answer their question affirmatively by giving an alternative proof of their result requiring only bounded ..."
Abstract

Cited by 15 (4 self)
 Add to MetaCart
Recent work of [DasguptaKumarSarlós, STOC 2010] gave a sparse JohnsonLindenstrauss transform and left as a main open question whether their construction could be efficiently derandomized. We answer their question affirmatively by giving an alternative proof of their result requiring only bounded independence hash functions. Furthermore, the sparsity bound obtained in our proof is improved. The main ingredient in our proof is a spectral moment bound for quadratic forms that was recently used in [DiakonikolasKaneNelson, FOCS 2010].
Sparser JohnsonLindenstrauss Transforms
"... We give two different constructions for dimensionality reduction in ℓ2 via linear mappings that are sparse: only an O(ε)fraction of entries in each column of our embedding matrices are nonzero to achieve distortion 1+ε with high probability, while still achieving the asymptotically optimal number ..."
Abstract

Cited by 11 (5 self)
 Add to MetaCart
We give two different constructions for dimensionality reduction in ℓ2 via linear mappings that are sparse: only an O(ε)fraction of entries in each column of our embedding matrices are nonzero to achieve distortion 1+ε with high probability, while still achieving the asymptotically optimal number of rows. These are the first constructions to provide subconstant sparsity for all values of parameters. Both constructions are also very simple: a vector can be embedded in two for loops. Such distributions can be used to speed up applications where ℓ2 dimensionality reduction is used.
Optimal Bounds for JohnsonLindenstrauss Transforms and Streaming Problems with SubConstant Error
"... The JohnsonLindenstrauss transform is a dimensionality reduction technique with a wide range of applications to theoretical computer science. It is specified by a distribution over projection matrices from R n → R k where k ≪ d and states that k = O(ε −2 log 1/δ) dimensions suffice to approximate t ..."
Abstract

Cited by 11 (1 self)
 Add to MetaCart
The JohnsonLindenstrauss transform is a dimensionality reduction technique with a wide range of applications to theoretical computer science. It is specified by a distribution over projection matrices from R n → R k where k ≪ d and states that k = O(ε −2 log 1/δ) dimensions suffice to approximate the norm of any fixed vector in R d to within a factor of 1 ± ε with probability at least 1 − δ. In this paper we show that this bound on k is optimal up to a constant factor, improving upon a previous Ω((ε −2 log 1/δ) / log(1/ε)) dimension bound of Alon. Our techniques are based on lower bounding the information cost of a novel oneway communication game and yield the first space lower bounds in a data stream model that depend on the error probability δ. For many streaming problems, the most naïve way of achieving error probability δ is to first achieve constant probability, then take the median of O(log 1/δ) independent repetitions. Our techniques show that for a wide range of problems this is in fact optimal! As an example, we show that estimating the ℓpdistance for any p ∈ [0, 2] requires Ω(ε −2 log n log 1/δ) space, even for vectors in {0, 1} n. This is optimal in all parameters and closes a long line of work on this problem. We also show the number of distinct elements requires Ω(ε −2 log 1/δ + log n) space, which is optimal if ε −2 = Ω(log n). We also improve previous lower bounds for entropy in the strict turnstile and general turnstile models by a multiplicative factor of Ω(log 1/δ). Finally, we give an application to oneway communication complexity under product distributions, showing that unlike in the case of constant δ, the VCdimension does not characterize the complexity when δ = o(1).
IMPROVED ANALYSIS OF THE SUBSAMPLED RANDOMIZED HADAMARD TRANSFORM
"... Abstract. This paper presents an improved analysis of a structured dimensionreduction map called the subsampled randomized Hadamard transform. This argument demonstrates that the map preserves the Euclidean geometry of an entire subspace of vectors. The new proof is much simpler than previous appro ..."
Abstract

Cited by 11 (0 self)
 Add to MetaCart
Abstract. This paper presents an improved analysis of a structured dimensionreduction map called the subsampled randomized Hadamard transform. This argument demonstrates that the map preserves the Euclidean geometry of an entire subspace of vectors. The new proof is much simpler than previous approaches, and it offers—for the first time—optimal constants in the estimate on the number of dimensions required for the embedding. 1.
Efficient orthogonal matching pursuit using sparse random projections for scene and video classification
 In Proceedings of IEEE International Conference on Computer Vision
, 2011
"... Sparse projection has been shown to be highly effective in several domains, including image denoising and scene / object classification. However, practical application to large scale problems such as video analysis requires efficient versions of sparse projection algorithms such as Orthogonal Matchi ..."
Abstract

Cited by 5 (2 self)
 Add to MetaCart
Sparse projection has been shown to be highly effective in several domains, including image denoising and scene / object classification. However, practical application to large scale problems such as video analysis requires efficient versions of sparse projection algorithms such as Orthogonal Matching Pursuit (OMP). In particular, random projection based locality sensitive hashing (LSH) has been proposed for OMP. In this paper, we propose a novel technique called Comparison Hadamard random projection (CHRP) for further improving the efficiency of LSH within OMP. CHRP combines two techniques:(1) The Fast JohnsonLindenstrauss Transform (FJLT) which uses a randomized Hadamard transform and sparse projection matrix for LSH, and (2) Achlioptas ’ random projection that uses only addition and comparison operations. Our approach provides the robustness of FJLT while completely avoiding multiplications. We empirically validate CHRP’s efficacy by performing a suite of experiments for image denoising, scene classification, and video categorization. Our experiments indicate that CHRP significantly speedsup OMP with negligible loss in classification accuracy. 1.
Acceleration of Randomized Kaczmarz Method via the JohnsonLindenstrauss Lemma
, 2010
"... The Kaczmarz method is an algorithm for finding the solution to an overdetermined system of linear equations Ax = b by iteratively projecting onto the solution spaces. The randomized versionputforthbyStrohmerandVershyninyieldsprovablyexponentialconvergenceinexpectation, which for highly overdetermin ..."
Abstract

Cited by 4 (1 self)
 Add to MetaCart
The Kaczmarz method is an algorithm for finding the solution to an overdetermined system of linear equations Ax = b by iteratively projecting onto the solution spaces. The randomized versionputforthbyStrohmerandVershyninyieldsprovablyexponentialconvergenceinexpectation, which for highly overdetermined systems even outperforms the conjugate gradient method. In this article we present a modified version of the randomized Kaczmarz method which at each iteration selects the optimal projection from a randomly chosen set, which in most cases significantly improves the convergence rate. We utilize a JohnsonLindenstrauss dimension reduction technique to keep the runtime on the same order as the original randomized version, adding only extra preprocessing time. We present a series of empirical studies which demonstrate the remarkable acceleration in convergence to the solution using this modified approach. 1
LOWRANK MATRIX RECOVERY VIA ITERATIVELY REWEIGHTED LEAST SQUARES MINIMIZATION
"... Abstract. We present and analyze an efficient implementation of an iteratively reweighted least squares algorithm for recovering a matrix from a small number of linear measurements. The algorithm is designed for the simultaneous promotion of both a minimal nuclear norm and an approximatively lowran ..."
Abstract

Cited by 4 (1 self)
 Add to MetaCart
Abstract. We present and analyze an efficient implementation of an iteratively reweighted least squares algorithm for recovering a matrix from a small number of linear measurements. The algorithm is designed for the simultaneous promotion of both a minimal nuclear norm and an approximatively lowrank solution. Under the assumption that the linear measurements fulfill a suitable generalization of the Null Space Property known in the context of compressed sensing, the algorithm is guaranteed to recover iteratively any matrix with an error of the order of the best krank approximation. In certain relevant cases, for instance for the matrix completion problem, our version of this algorithm can take advantage of the Woodbury matrix identity, which allows to expedite the solution of the least squares problems required at each iteration. We present numerical experiments which confirm the robustness of the algorithm for the solution of matrix completion problems, and demonstrate its competitiveness with respect to other techniques proposed recently in the literature. AMS subject classification: 65J22, 65K10, 52A41, 49M30. Key Words: lowrank matrix recovery, iteratively reweighted least squares, matrix completion.
Subspace Embeddings for the L1norm with Applications
"... We show there is a distribution over linear mappings R: ℓ n O(d log d) 1 → ℓ1, such that with arbitrarily large constant probability, for any fixed ddimensional subspace L, for all x ∈ L we have ‖x‖1 ≤ ‖Rx‖1 = O(d log d)‖x‖1. This provides the first analogue of the ubiquitous subspace JohnsonLinde ..."
Abstract

Cited by 2 (1 self)
 Add to MetaCart
We show there is a distribution over linear mappings R: ℓ n O(d log d) 1 → ℓ1, such that with arbitrarily large constant probability, for any fixed ddimensional subspace L, for all x ∈ L we have ‖x‖1 ≤ ‖Rx‖1 = O(d log d)‖x‖1. This provides the first analogue of the ubiquitous subspace JohnsonLindenstrauss embedding for the ℓ1norm. Importantly, the target dimension and distortion are independent of the ambient dimension n. We give several applications of this result. First, we give a faster algorithm for computing wellconditioned bases. Our algorithm is simple, avoiding the linear programming machinery required of previous algorithms. We also give faster algorithms for least absolute deviation regression and ℓ1norm best fit hyperplane problems, as well as the first single pass streaming algorithms with low space for these problems. These results are motivated by practical problems in image analysis, spam detection, and statistics, where the ℓ1norm is used in studies where outliers may be safely and effectively ignored. This is because the ℓ1norm is more robust to outliers than the ℓ2norm.
A Randomized Approximate Nearest Neighbors Algorithm
, 2010
"... We present a randomized algorithm for the approximate nearest neighbor problem in ddimensional Euclidean space. Given N points {xj} in R d, the algorithm attempts to find k nearest neighbors for each of xj, where k is a userspecified integer parameter. The algorithm is iterative, and its CPU time ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
We present a randomized algorithm for the approximate nearest neighbor problem in ddimensional Euclidean space. Given N points {xj} in R d, the algorithm attempts to find k nearest neighbors for each of xj, where k is a userspecified integer parameter. The algorithm is iterative, and its CPU time requirements are proportional to T ·N ·(d·(log d)+ k · (log k) · (log N)) + N · k 2 · (d + log k), with T the number of iterations performed. The memory requirements of the procedure are of the order N · (d + k). A byproduct of the scheme is a data structure, permitting a rapid search for the k nearest neighbors among {xj} for an arbitrary point x ∈ R d. The cost of each such query is proportional to T · (d · (log d) + log(N/k) + k 2 · (d + log k)), and the memory requirements for the requisite data structure are of the order N · (d + k) + T · (d + N · k). The algorithm utilizes random rotations and a basic divideandconquer scheme, followed by a local graph search. We analyze the scheme’s behavior for certain types of distributions
New constructions of RIP matrices with fast multiplication and fewer rows
, 2012
"... In compressed sensing, the restricted isometry property (RIP) is a sufficient condition for the efficient reconstruction of a nearly ksparse vector x ∈ C d from m linear measurements Φx. It is desirable for m to be small, and for Φ to support fast matrixvector multiplication. In this work, we give ..."
Abstract

Cited by 2 (1 self)
 Add to MetaCart
In compressed sensing, the restricted isometry property (RIP) is a sufficient condition for the efficient reconstruction of a nearly ksparse vector x ∈ C d from m linear measurements Φx. It is desirable for m to be small, and for Φ to support fast matrixvector multiplication. In this work, we give a randomized construction of RIP matrices Φ ∈ C m×d, preserving the ℓ2 norms of all ksparse vectors with distortion 1 + ε, where the matrixvector multiply Φx can be computed in nearly linear time. The number of rows m is on the order of ε −2 k log d log 2 (k log d). Previous analyses of constructions of RIP matrices supporting fast matrixvector multiplies, such as the sampled discrete Fourier matrix, required m to be larger by roughly a log k factor. Supporting fast matrixvector multiplication is useful for iterative recovery algorithms which repeatedly multiply by Φ or Φ ∗. Furthermore, our construction, together with a connection between RIP matrices and the JohnsonLindenstrauss lemma in [KrahmerWard, SIAM. J. Math. Anal. 2011], implies fast JohnsonLindenstrauss embeddings with asymptotically fewer rows than previously known. Our approach is a simple twist on previous constructions. Rather than choosing the rows for the embedding matrix to be rows sampled from some larger structured matrix (such as the discrete Fourier transform or a random circulant matrix), we instead choose each row of the embedding matrix to be a linear combination of a small number of rows of the original matrix, with random sign flips as coefficients. The main tool in our analysis is a recent bound for the supremum of certain types of Rademacher chaos processes in [KrahmerMendelsonRauhut, arXiv abs/1207.0235]. 1