## Enhanced line search: A novel method to accelerate Parafac (2005)

### Cached

### Download Links

Venue: | in Eusipco’05 |

Citations: | 32 - 8 self |

### BibTeX

@INPROCEEDINGS{Comon05enhancedline,

author = {Myriam Rajih Pierre Comon},

title = {Enhanced line search: A novel method to accelerate Parafac},

booktitle = {in Eusipco’05},

year = {2005},

pages = {4--8}

}

### Years of Citing Articles

### OpenURL

### Abstract

Abstract. Several modifications have been proposed to speed up the alternating least squares (ALS) method of fitting the PARAFAC model. The most widely used is line search, which extrapolates from linear trends in the parameter changes over prior iterations to estimate the parameter values that would be obtained after many additional ALS iterations. We propose some extensions of this approach that incorporate a more sophisticated extrapolation, using information on nonlinear trends in the parameters and changing all the parameter sets simultaneously. The new method, called “enhanced line search (ELS), ” can be implemented at different levels of complexity, depending on how many different extrapolation parameters (for different modes) are jointly optimized during each iteration. We report some tests of the simplest parameter version, using simulated data. The performance of this lowest-level of ELS depends on the nature of the convergence difficulty. It significantly outperforms standard LS when there is a “convergence bottleneck, ” a situation where some modes have almost collinear factors but others do not, but is somewhat less effective in classic “swamp ” situations where factors are highly collinear in all modes. This is illustrated by examples. To demonstrate how ELS can be adapted to different N-way decompositions, we also apply it to a four-way array to perform a blind identification of an under-determined mixture (UDM). Since analysis of this dataset happens to involve a serious convergence “bottleneck ” (collinear factors in two of the four modes), it provides another example of a situation in which ELS dramatically outperforms standard line search. Key words. PARAFAC, alternating least squares (ALS), line search, enhanced line search (ELS), acceleration, swamps, bottlenecks, collinear factors, degeneracy AMS subject classifications. Authors must provide DOI. 10.1137/06065577 1. Introduction. PARAFAC

### Citations

363 |
Analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition
- Carroll, Chang
- 1970
(Show Context)
Citation Context ...lice weighting or “occasion weights” matrix C [13]. The trilinear decomposition used in the model is also known as CANDECOMP for CANonical DECOMPosition; it was introduced by Caroll and Chang in 1970 =-=[3]-=- to provide a basis for fitting INDSCAL, an important generalization of multidimensional scaling that provides unique dimensions and allows the estimation of dimension weights for individual subjects.... |

299 |
Foundations of the Parafac procedure: Models and conditions for an explanatory multimodal factor analysis
- Harshman
(Show Context)
Citation Context ...fications. Authors must provide DOI. 10.1137/06065577 1. Introduction. PARAFAC can be seen as a generalization of two-way factor analysis to multiway data. It was first introduced by Harshman in 1970 =-=[9]-=- based on the principle of parallel proportional profiles (PP) proposed by Cattell in 1944 [4]. The PP principle states that if two (or more) different two-way models are described by the same set of ... |

192 |
Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics, Linear Algebra and its
- Kruskal
- 1977
(Show Context)
Citation Context ... data situations. Uniqueness properties have been studied by multiple authors, with the most important general results found in [10] and its recent generalization [15], the Kruskal theorems in, e.g., =-=[16]-=-, and the extensions in [26], and elsewhere. Progress in this area is ongoing— for example, in the case of “tall” arrays, a significantly more relaxed condition has been derived in [19]. A relatively ... |

67 |
Signal processing based on multilinear algebra
- Lathauwer
- 1997
(Show Context)
Citation Context ...ding matrices. The three-way PARAFAC model, along with its extension to higher orders [9, 3], has most often been applied in psychometrics and chemometrics [26, 27], and in the signal processing area =-=[18, 6, 7]-=-. While the two-way factor model suffers a rotational indeterminacy that yields an infinite set of solutions, the PARAFAC model enjoys a uniqueness property under conditions that often can be met in r... |

61 | Multi-way analysis in the food industry: Models, algorithms and applications
- Bro
- 1998
(Show Context)
Citation Context ... of loading vectors but their relative proportions or weights change from one model to the other, then those loading vectors lead to a new model which is unambiguous with respect to (w.r.t.) rotation =-=[4, 5, 2]-=-. In other words, suppose that the matrix X1 can be modeled: X1 = a1b T 1 c11 + a2b T 2 c12 + ···+ aF b T F c1F + E1, where af and bf (1 ≤ f ≤ F ) are the columns of matrices A and B, respectively, an... |

59 |
The N-way toolbox for MATLAB
- Andersson, Bro
- 2000
(Show Context)
Citation Context ...are defined as : Dkn = ψ (3) n (∑Aqnuq[k]) (13) q where 1 ≤ k ≤ K and 1 ≤ n ≤ N. A is the channel matrix of size 2 × 3 to be identified. We use the ALS implementation proposed by Andersson and Bro in =-=[1]-=- and replace the Line Search procedure by the six optimizations of ELS shown in figure 3. The three sources are BPSK and we generate an ”infinite block” of data by taking all the 2 3 possible combinat... |

50 |
A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization
- Lathauwer
(Show Context)
Citation Context ...rems in, e.g., [16], and the extensions in [26], and elsewhere. Progress in this area is ongoing— for example, in the case of “tall” arrays, a significantly more relaxed condition has been derived in =-=[19]-=-. A relatively complete list of relevant articles up to 2006-07 can be found in [15]. A variety of algorithms have been used to fit the PARAFAC model (for a detailed summary and discussion; see, e.g.,... |

49 | On the uniqueness of multilinear decomposition of N−way arrays
- Sidiropoulos, Bro
(Show Context)
Citation Context ... Eijk. f Matrices A, B, and C are called loading matrices. The three-way PARAFAC model, along with its extension to higher orders [9, 3], has most often been applied in psychometrics and chemometrics =-=[26, 27]-=-, and in the signal processing area [18, 6, 7]. While the two-way factor model suffers a rotational indeterminacy that yields an infinite set of solutions, the PARAFAC model enjoys a uniqueness proper... |

44 | Computation of the canonical decomposition by means of a simultaneous generalized Schur decomposition
- Lathauwer, Moor, et al.
(Show Context)
Citation Context ... also requires that two of the loading matrices are linearly independent and that every pair of columns of the last loading matrix is linearly independent. Previous approaches are made more robust in =-=[20]-=- by taking all matrix slices into account, which leads to a simultaneous matrix decomposition. All of these methods require that the array rank (as defined in [17], for instance), F , is less than or ... |

39 |
The PARAFAC model for three-way factor analysis and multidimensional scaling. Research Methods for Multimode Data Analysis
- Harshman, Lundy
- 1984
(Show Context)
Citation Context ... across values of the third array index). This kind of variation in factor structure is not consistent with the PARAFAC model. However, PARAFAC can fit part of the “axis wobble” or “Tucker Variation” =-=[14]-=- by reweighing axes after the space has been sheared [11]. Thus, when too much axis “wobble” is present, the A, B, and C factor spaces become inversely sheared to better fit it, creating a “degenerate... |

35 |
Rank, decomposition, and uniqueness for 3-way and n-way arrays
- Kruskal
- 1989
(Show Context)
Citation Context ...s approaches are made more robust in [20] by taking all matrix slices into account, which leads to a simultaneous matrix decomposition. All of these methods require that the array rank (as defined in =-=[17]-=-, for instance), F , is less than or equal to two of the array dimensions. In [19] De Lathauwer generalizes the approach presented in [20] to the case where F is less than or equal to one of the array... |

30 | Blind identification of underdetermined mixtures based on the characteristic function
- COMON, RAJIH
- 2006
(Show Context)
Citation Context ...ding matrices. The three-way PARAFAC model, along with its extension to higher orders [9, 3], has most often been applied in psychometrics and chemometrics [26, 27], and in the signal processing area =-=[18, 6, 7]-=-. While the two-way factor model suffers a rotational indeterminacy that yields an infinite set of solutions, the PARAFAC model enjoys a uniqueness property under conditions that often can be met in r... |

27 |
Determination and proof of minimum uniqueness conditions for
- Harshman
- 1972
(Show Context)
Citation Context ...oys a uniqueness property under conditions that often can be met in real data situations. Uniqueness properties have been studied by multiple authors, with the most important general results found in =-=[10]-=- and its recent generalization [15], the Kruskal theorems in, e.g., [16], and the extensions in [26], and elsewhere. Progress in this area is ongoing— for example, in the case of “tall” arrays, a sign... |

27 |
Themultilinear engine—A table-driven, least squares program for solvingmultilinear problems, including the n-way parallel factor analysis model
- Paatero
- 1999
(Show Context)
Citation Context ...s,” characterized by high factor collinearity in all three modes. We will see that ELS (at least the simple version tested here) is most successful with a different kind of convergence difficulty. In =-=[22]-=-, Paatero proposed the multilinear engine (ME) program to accelerate the fit of the PARAFAC model. ME changes all of the sets of parameters at once, whereas ELS is based on ALS, and updates alternativ... |

26 |
A decomposition for three-way arrays
- Leurgans, Ross, et al.
- 1993
(Show Context)
Citation Context ...he loading matrices to be linearly independent.sENHANCED LINE SEARCH 3 Another closed-form solution for three-way arrays, and based on a single matrix eigen value decomposition (EVD) was presented in =-=[21]-=- by Leurgans, Ross, and Abel. It also requires that two of the loading matrices are linearly independent and that every pair of columns of the last loading matrix is linearly independent. Previous app... |

21 |
Parallel Proportional Profiles" and other principles for determining the choice of factors by rotation
- Cattell
- 1944
(Show Context)
Citation Context ...a generalization of two-way factor analysis to multiway data. It was first introduced by Harshman in 1970 [9] based on the principle of parallel proportional profiles (PP) proposed by Cattell in 1944 =-=[4]-=-. The PP principle states that if two (or more) different two-way models are described by the same set of loading vectors but their relative proportions or weights change from one model to the other, ... |

18 | channel identification and extraction of more sources than sensors
- Comon, “Blind
- 1998
(Show Context)
Citation Context ...ding matrices. The three-way PARAFAC model, along with its extension to higher orders [9, 3], has most often been applied in psychometrics and chemometrics [26, 27], and in the signal processing area =-=[18, 6, 7]-=-. While the two-way factor model suffers a rotational indeterminacy that yields an infinite set of solutions, the PARAFAC model enjoys a uniqueness property under conditions that often can be met in r... |

15 |
Tensorial resolution: A direct trilinear decomposition
- Sanchez, Kowalski
- 1990
(Show Context)
Citation Context ...C based on a gradient method. In fact, the loading matrices A, B, and C are updated using the gradient descent. A closed-form solution to fit the PARAFAC model was proposed by Sanchez and Kowalski in =-=[25]-=-. It reduces the problem to a rectangular eigenvalue-eigenvector equation, but it needs at least two of the loading matrices to be linearly independent.sENHANCED LINE SEARCH 3 Another closed-form solu... |

11 |
Practical and Computational Aspects in Chemometric Data Analysis
- Tomasi
- 2006
(Show Context)
Citation Context ... A relatively complete list of relevant articles up to 2006-07 can be found in [15]. A variety of algorithms have been used to fit the PARAFAC model (for a detailed summary and discussion; see, e.g., =-=[28]-=-. The most widely used is the alternating least squares (ALS) algorithm. The convergence of ALS was found to be very slow in some cases, typically when two factors are almost collinear. Line search [2... |

10 | Improving the speed of multiway algorithms: Part II: Compression - Bro, Andersson - 1998 |

9 |
Etude algebrique des multitableaux: apports de l’algebre tensorielle
- Franc
- 1992
(Show Context)
Citation Context ...ngine (ME) program to accelerate the fit of the PARAFAC model. ME changes all of the sets of parameters at once, whereas ELS is based on ALS, and updates alternatively each of the loading factors. In =-=[8]-=-, Franc proposed an acceleration to the convergence of PARAFAC based on a gradient method. In fact, the loading matrices A, B, and C are updated using the gradient descent. A closed-form solution to f... |

8 |
Multi-way Analysis with
- Bro, Geladi
- 2004
(Show Context)
Citation Context ... Eijk. f Matrices A, B, and C are called loading matrices. The three-way PARAFAC model, along with its extension to higher orders [9, 3], has most often been applied in psychometrics and chemometrics =-=[26, 27]-=-, and in the signal processing area [18, 6, 7]. While the two-way factor model suffers a rotational indeterminacy that yields an infinite set of solutions, the PARAFAC model enjoys a uniqueness proper... |

6 |
Factor rotation for proportional profiles: analytical solution and an example
- Cattell, Cattell
- 1955
(Show Context)
Citation Context ... of loading vectors but their relative proportions or weights change from one model to the other, then those loading vectors lead to a new model which is unambiguous with respect to (w.r.t.) rotation =-=[4, 5, 2]-=-. In other words, suppose that the matrix X1 can be modeled: X1 = a1b T 1 c11 + a2b T 2 c12 + ···+ aF b T F c1F + E1, where af and bf (1 ≤ f ≤ F ) are the columns of matrices A and B, respectively, an... |

5 |
Component resolution using multilinear models
- Ross, Leurgans
- 1995
(Show Context)
Citation Context ...8]. The most widely used is the alternating least squares (ALS) algorithm. The convergence of ALS was found to be very slow in some cases, typically when two factors are almost collinear. Line search =-=[2, 24]-=- is one of the most important solutions proposed to cope with the problem of slow convergence. We focus in this paper on the line search solution and present a generalization of this method for speedi... |

4 |
The N-way toolbox for
- Andersson, Bro
- 2000
(Show Context)
Citation Context ...d of the “new” ones. However, if Υ (new) < Υ (it−1) then acceleration is accomplished and we gain some iterations. The steps of the ALS algorithm with line search, as proposed by Andersson and Bro in =-=[1]-=-, are summarized in Figure 2. The dashed area corresponds to the line search part. Line search is executed after a few iterations of the ALS algorithm in order to wait for the system to stabilize. In ... |

3 |
Two-factor degeneracies and a stabilization of
- Rayens, Mitchell
- 1997
(Show Context)
Citation Context ... successfully in others—opening interesting directions for further exploration. We call this method enhanced line search (ELS). A regularized (ridge) regression was proposed by Rayens and Mitchell in =-=[23]-=- to speed up the ALS algorithm in case of ill-posed problems. While the estimates produced by ridge regression are biased, they suggested ways of dealing with this, including a switch back to regular ... |

2 |
Stretch” versus “slice” methods for representing three-way structure via matrix notation
- Harshman, Hong
(Show Context)
Citation Context ... model: (1) Xk = ACkB T + Ek,k =1, 2, where Ck is a diagonal matrix with the elements of vector ck, in its diagonal, where ck denotes the kth row of the slice weighting or “occasion weights” matrix C =-=[13]-=-. The trilinear decomposition used in the model is also known as CANDECOMP for CANonical DECOMPosition; it was introduced by Caroll and Chang in 1970 [3] to provide a basis for fitting INDSCAL, an imp... |

1 |
xx title to complete xx, 2004, unpublished. Slides available online at http://publish.uwo.ca/∼harshman
- Harshman
(Show Context)
Citation Context ...riation in factor structure is not consistent with the PARAFAC model. However, PARAFAC can fit part of the “axis wobble” or “Tucker Variation” [14] by reweighing axes after the space has been sheared =-=[11]-=-. Thus, when too much axis “wobble” is present, the A, B, and C factor spaces become inversely sheared to better fit it, creating a “degenerate solution,” which involves strong collinearities and seri... |

1 |
Factor and subspace uniqueness in N-way PARAFAC when two loading matrices have full column rank, submitted
- Harshman, Lundy
- 2007
(Show Context)
Citation Context ...ditions that often can be met in real data situations. Uniqueness properties have been studied by multiple authors, with the most important general results found in [10] and its recent generalization =-=[15]-=-, the Kruskal theorems in, e.g., [16], and the extensions in [26], and elsewhere. Progress in this area is ongoing— for example, in the case of “tall” arrays, a significantly more relaxed condition ha... |