Citation Context

...t 2. By global convergence rate, we mean an upper bound on the maximum number of iterations that need to be taken by an algorithm to reach an -accurate first-order stationary point. This rate applies for any starting point of the algorithm. 3376 Matrix Completion by Fast ALS to the work of Candes and Tao (2009); Candes and Recht (2008). However, as pointed out by Jain et al. (2013), their alternating-minimization methods typically require |Ω |to be larger than than required by convex optimization based methods (Candes and Recht, 2008). We refer the interested reader to more recent work of Hardt (2014), analyzing the statistical properties of alternating minimization methods. The flavor of our present work is in spirit different from that described above (Jain et al., 2013; Hardt, 2014). Our main goal here is to develop non-convex algorithms for the optimization of Problem (6) for arbitrary λ and rank r. A special case of our framework corresponds to the case where Problem (6) is equivalent to Problem (1), for proper choices of r, λ. In this particular case, we study in Section 5 when our algorithm softImpute-ALS converges to a global minimizer of Problem (1)—this can be verified by a minor...

Citation Context

...SGD-like algorithm that has a simple update rule with a step size that is a simple function of the norm of the iterate Yk. We show that Alecton converges globally. We make the following contributions: • We establish the convergence rate to a global optimum of Alecton using a random initialization; in contrast, prior analyses (Candes et al., 2014; Jain et al., 2013) have required more expensive initialization methods, such as the singular value decomposition of an empirical average of the data. • In contrast to previous work that uses bounds on the magnitude of the noise (Hardt & Price, 2014; Hardt, 2014), our analysis depends only on the variance of the samples. As a result, we are able to be robust to different noise models, and we apply our technique to these problems, which did not previously have global convergence rates: – matrix completion, in which we observe entries of A one at a time (Jain et al., 2013; Keshavan et al., 2010) (Section 4.1), – phase retrieval, in which we observe tr(uTAv) for randomly selected u, v (Candes et al., 2014; Candes & Li, 2014) (Section 4.3), and – subspace tracking, in whichA is a projection matrix and we observe random entries of a random vector in its ...

Citation Context

...the input. In 7 comparison, the two standard algorithms for matrix completion, the iterative Singular Value Thresholding Algorithm [Cai et al., 2010] and alternating least-squares [Jain et al., 2013, =-=Hardt, 2013-=-], are significantly slower than Algorithm 1, not only due to their iterative nature, but also in per-iteration running time. 3.2 Necessary Conditions for Matrix Completion We now establish a lower bo...

Citation Context

... choices of initial conditions, providing a better initial condition for the optimization of (2). Remarkably, the performance achieved by MaCBetH with the inferred rank is essentially the same as the one achieved with an oracle rank. By contrast, estimating the correct rank from the (trimmed) SVD is more challenging. We note that for the choice of parameters we consider, trimming had a negligible effect. Along the same lines, OptSpace [3] uses a different minimization procedure, but from our tests we could not see any difference in performance due to that. When using Alternating Least Squares [23, 24] as optimization method, we also obtained a similar improvement in reconstruction by using the eigenvectors of the Bethe Hessian, instead of the singular vectors ofM, as initial condition. 7 10 20 30 40 50 P (R M S E < 10 − 1 ) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Rank 3 Macbeth OR Tr-SVD OR Random OR Macbeth IR Tr-SVD IR ϵ 10 20 30 40 50 P (R M S E < 10 − 8 ) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10 20 30 40 50 60 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Rank 10 ϵ 10 20 30 40 50 60 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Figure 4: RMSE as a function of the number of revealed entries per r...

Citation Context

...tes and use martingale convergence techniques to conclude that the iterates stay away from saddle surfaces. While [24] shows that SGD updates stay away from saddle surfaces, the stepsizes they can handle are 2 Table 1: Comparison of sample complexity and runtime of our algorithm with existing algorithms in order to obtain Frobenius norm error . O(·) hides log d factors. See Section 1.2 for more discussion. Algorithm Sample complexity Total runtime Online? Nuclear Norm [22] O(µdk) O(d3/ √ ) No Alternating minimization [14] O(µdkκ 8 log 1 ) O(µdk 2κ8 log 1 ) No Alternating minimization [8] O ( µdk2κ2 ( k + log 1 )) O ( µdk3κ2 ( k + log 1 )) No Projected gradient descent[12] O(µdk 5) O(µdk7 log 1 ) No SGD [24] O(µ2dk7κ6) poly(µ, d, k, κ) log 1 Yes Our result O ( µdkκ4 ( k + log 1 )) O ( µdk4κ4 log 1 ) Yes quite small (scaling as 1/poly(d1, d2)), leading to suboptimal computational complexity. Our framework makes it possible to establish the same statement for much larger step sizes, giving us near-optimal runtime. We believe these techniques may be applicable in other non-convex settings as well. 1.2 Related Work In this section we will mention some more related wor...

Citation Context

...e problem under an exact low-rank constraint. This leads a non-convex optimization problem, but has several computational advantages over most approaches to minimizing the nuclear norm and is widely used in large-scale applications (such as recommendation systems) [16]. In general, popular algorithms for solving the rank-constrained models – e.g., alternating minimization and alternating gradient descent – do not have as strong of convergence or recovery error guarantees due to the non-convexity of the rank constraint. However, there has been significant progress on this front in recent years [11, 10, 12, 13, 14, 23, 25], with many of these algorithms now having guarantees comparable to those for nuclear norm minimization. 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain. Nearly all of this existing work assumes that the underlying low-rank matrix X remains fixed throughout the measurement process. In many practical applications, this is a tremendous limitation. For example, users’ preferences for various items may change (sometimes quite dramatically) over time. Modelling such drift of user’s preference has been proposed in the context of both music and movies as a way t...

Citation Context

...tions to random sampling to the best of our knowledge are in (Heiman et al., 2014; Lee & Shraibman, 2013; Bhojanapalli & Jain, 2014). In this line the state-ofthe-art is (Bhojanapalli & Jain, 2014), where the support of the observation is a d-regular expander such that the weight matrix has a sufficiently large spectral gap. However, it only works for binary weights, and is for a nuclear norm convex relaxation and does not incorporate noise. Recently, there is an increasing interest in analyzing nonconvex optimization techniques for matrix completion. In two seminal papers (Jain et al., 2013; Hardt, 2014), it was shown that with an appropriate SVD-based initialization, the alternating minimization algorithm (with a few modifications) recovers the ground-truth. These results are for random binary weight matrix and crucially rely on resampling (i.e., using independent samples at each iteration), which is inherently not possible for the setting studied in this paper. More recently, Sun & Luo (2015) proved recovery guarantees for a family of algorithms including alternating minimization on matrix completion without resampling. However, the result is still for random binary weights and has not cons...

Citation Context

...unknown n1 × n2 matrix that has rank r but each of its entries has been corrupted by independent Gaussian noise with standard deviation δ. c© 2016 B. Barak & A. Moitra. BARAK MOITRA Then if we observe roughly m = (n1 + n2)r log(n1 + n2) of its entries, the locations of which are chosen uniformly at random, there is an algorithm that outputs a matrix X that with high probability satisfies err(X) = 1 n1n2 ∑ i,j ∣∣∣Xi,j −Mi,j∣∣∣ ≤ O(δ) . There are extensions to non-uniform sampling models (Lee and Shraibman, 2013; Chen et al., 2014), as well as various efficiency improvements (Jain et al., 2013; Hardt, 2014). What is particularly remarkable about these guarantees is that the number of observations needed is within a logarithmic factor of the number of parameters — (n1 + n2)r — that define the model. In fact, there are benefits to working with even higher-order structure but so far there has been little progress on natural extensions to the tensor setting. To motivate this problem, consider the Groupon Problem (which we introduce here to illustrate this point) where the goal is to predict user-activity ratings. The challenge is that which activities we should recommend (and how much a user liked a...