## Regularization by truncated total least squares (1997)

Venue: | SIAM J. Sci. Comp |

Citations: | 39 - 4 self |

### BibTeX

@ARTICLE{Fierro97regularizationby,

author = {R. D. Fierro and G. H. Golub and P. C. Hansen and D. P. O’leary},

title = {Regularization by truncated total least squares},

journal = {SIAM J. Sci. Comp},

year = {1997},

pages = {124--1}

}

### Years of Citing Articles

### OpenURL

### Abstract

Abstract. The total least squares (TLS) method is a successful method for noise reduction in linear least squares problems in a number of applications. The TLS method is suited to problems in which both the coefficient matrix and the right-hand side are not precisely known. This paper focuses on the use of TLS for solving problems with very ill-conditioned coefficient matrices whose singular values decay gradually (so-called discrete ill-posed problems), where some regularization is necessary to stabilize the computed solution. We filter the solution by truncating the small singular values of the TLS matrix. We express our results in terms of the singular value decomposition (SVD) of the coefficient matrix rather than the augmented matrix. This leads to insight into the filtering properties of the truncated TLS method as compared to regularized least squares solutions. In addition, we propose and test an iterative algorithm based on Lanczos bidiagonalization for computing truncated TLS solutions.

### Citations

710 |
Method Of Conjugate Gradient for Solving Linear Equations
- Hestenes, E
- 1952
(Show Context)
Citation Context ...matically equivalent to applying the conjugate gradient method to the normal equations, the fact that the solution norm is monotonically increasing follows from equation (6:3) of Hestenes and Stiefel =-=[19]-=-. Notice that (22) is only guaranteed to hold in exact arithmetic while it fails to hold in inexact arithmetic when spurious singular values of (A,b) start to appear in (B k ,# 1 e 1 ). The cure is ei... |

545 |
Orthogonal Polynomials
- Szegö
- 1939
(Show Context)
Citation Context ... yields the relations so τ1w1 + γ2w2 = s 2 kw1 , τ1z1 + γ2z2 = s 2 k+1z1 , w2 = s2 k − τ1 γ2 , z2 = s2 k+1 − τ1 γ2 , 3This result can also be established as a consequence of equation (3.4.8) in Szegö =-=[22]-=- by noting that |¯v (k) 22 | and |¯v(k+1) 22 | are the square roots of the Christoffel numbers λ1k and λ1,k+1 [11], but we prefer a direct matrix algebra proof.1240 R. D. FIERRO, G. H. GOLUB, P. C. H... |

333 | LSQR: An algorithm for sparse linear equations and sparse least squares
- Paige
- 1982
(Show Context)
Citation Context ... A k V k y = U T k b k , or min #(B k ,# 1 e 1 )-( B k, e k )# F subject to B k y =e k , (19) where e 1 =(1,0,...,0) T ,and B k andse k are generally full. Our algorithm reduces to the LSQR algorithm =-=[21]-=- if we require B k = B k in each step. In each Lanczos step we can now compute an approximate truncated TLS solutionsx k by applying the Algorithm TLS to the small-size problem in (19). Hence, we comp... |

273 |
Generalized cross-validation as a method for choosing a good ridge parameter
- Golub, Heath, et al.
- 1979
(Show Context)
Citation Context ... had some success with this stopping rule; see section 6. Another popular method for choosing the regularization parameter is the method of generalized cross-validation due to Golub, Heath, and Wahba =-=[8]-=-. Currently, we do not have any experience with this method when applied to our algorithms. A third possible stopping criterion can be based on the L-curve criterion studied recently in [16, 18]. The ... |

203 | The Theory of Matrices', vol - Gantmacher - 1960 |

195 |
Calculating the singular values and pseudo-inverse of a matrix
- Golub, Kahan
- 1965
(Show Context)
Citation Context ...r the matrix A. It is well known that Lanczos bidiagonalization can be used to compute good approximations to the singular triplets associated with the largest singular values of a matrix; see, e.g., =-=[9, 20]-=-. We refer to the original papers and omit a discussion of the Lanczos bidiagonalization algorithm here. Again, we could choose some integer k max and perform k max Lanczos iterations applied to the a... |

192 | Regularization tools: a Matlab package for analysis and solution of discrete illposed problems
- Hansen
- 1994
(Show Context)
Citation Context ...om three classical methods for discrete ill-posed problems, namely, Tikhonov regularization, truncated SVD, and LSQR. Our experiments were carried out in MATLAB using the REGULARIZATION TOOLS package =-=[17]. Ou-=-r test problems were generated as follows. The matrix A is 64 �� 32 and comes from discretization of Phillips's test problem (cf. [17, phillips]). Two right-hand sides b [1] , b [2] were generated... |

191 | Lanczos Algorithms for Large Symmetric Eigenvalue Computations: Volume 1, Theory, volume 41 - Cullum, Willoughby - 2002 |

179 |
Analysis of Discrete Ill-Posed Problems by Means of the L-Curve
- Hansen
- 1992
(Show Context)
Citation Context ... and Wahba [8]. Currently, we do not have any experience with this method when applied to our algorithms. A third possible stopping criterion can be based on the L-curve criterion studied recently in =-=[16, 18]-=-. The idea in this method is to plot in log--log scale the solution norm versus the residual norm, in our case #x k # 2 versus #(A,b)-( A k , b k) # F,a nd choose as the optimal k the truncation param... |

167 |
An analysis of the total least squares problem
- Golub, Loan
- 1980
(Show Context)
Citation Context ...ions. It was independently derived in several bodies of work, and is known by statisticians as the errors in variables model. Numerical analysts came to know it through the work of Golub and Van Loan =-=[10]-=- and Van Hu#el and Vandewalle [24, 25, 26], and this literature has advanced the algorithmic and theoretical understanding of the method. 1.1. Motivation. The development of the TLS technique was moti... |

110 |
The use of the L-curve in the regularization of discrete ill-posed problems
- Hansen, O’Leary
- 1993
(Show Context)
Citation Context ... and Wahba [8]. Currently, we do not have any experience with this method when applied to our algorithms. A third possible stopping criterion can be based on the L-curve criterion studied recently in =-=[16, 18]-=-. The idea in this method is to plot in log--log scale the solution norm versus the residual norm, in our case #x k # 2 versus #(A,b)-( A k , b k) # F,a nd choose as the optimal k the truncation param... |

98 |
Calculation of Gauss quadrature rules
- Golub, Welsch
- 1969
(Show Context)
Citation Context ...3 This result can also be established as a consequence of equation (3.4.8) in Szego [22] by noting that |sv (k) 22 | and |sv (k+1) 22 | are the square roots of the Christo#el numbers # 1k and # 1,k+1 =-=[11]-=-, but we prefer a direct matrix algebra proof. 1240 R. D. FIERRO, G. H. GOLUB, P. C. HANSEN, AND D. P. O'LEARY so z 2s2sw 3 = (# 2 - s 2 k )(-w 2 ) - # 2 # 3 , z 3 = (# 2 - s 2 k+1 )(-z 2 ) - # 2 # 3 ... |

94 |
Regularization methods for large-scale problems
- Hanke, Hansen
- 1993
(Show Context)
Citation Context ...ften called discrete ill-posed problems, as they inherit many of the di#culties of the underlying ill-posed problem and therefore require a specialized treatment including some form of regularization =-=[13]-=- to suppress the e#ects of errors. Most regularization methods used today assume that the errors are confined to the right-hand side b. Although this is true in many applications there are also proble... |

55 |
Updating the singular value decomposition
- Bunch, Nielsen
- 1978
(Show Context)
Citation Context ...IZATION TOOLS package [17]. Our test problems were generated as follows. The matrix A is 64 �� 32 and comes from discretization of Phillips's test problem (cf. [17, phillips]). Two right-hand side=-=s b [1]-=- , b [2] were generated artificially by means of the SVD of A. The Fourier coe#cients # [1] i = u T i b [1] of the first satisfy 1236 R. D. FIERRO, G. H. GOLUB, P. C. HANSEN, AND D. P. O'LEARY # [1] 1... |

27 |
A bidiagonalization-regularization procedure for largescale discretizations of ill-posed problems
- O’Leary, Simmons
- 1981
(Show Context)
Citation Context ...r the matrix A. It is well known that Lanczos bidiagonalization can be used to compute good approximations to the singular triplets associated with the largest singular values of a matrix; see, e.g., =-=[9, 20]-=-. We refer to the original papers and omit a discussion of the Lanczos bidiagonalization algorithm here. Again, we could choose some integer k max and perform k max Lanczos iterations applied to the a... |

26 |
Some applications of the rank revealing QR-factorization
- Chan, Hansen
- 1992
(Show Context)
Citation Context ...anczos T-TLS 0.9 �� 10 5 1.7 �� 10 5 0.5 �� 10 5 of T-TLS, Tikhonov and truncated SVD, and the resemblance of Lanczos T-TLS and LSQR. Tes t 3. Our final test problem uses the second right-=-=hand side b [2] for which-=- all the Fourier coe#cients u T i b [2] decay, and the same "large" noise level as in Test 1. All five histograms (not shown here) are almost identical, illustrating that for this class of p... |

26 |
A weighted pseudoinverse, generalized singular values, and constrained least squares problems
- Eldén
- 1982
(Show Context)
Citation Context ...gularized solution xsf. Finally, transform xsf back to the general-form setting. There are several ways to transform a problem into standard form. The following transformation originally due to Eldén =-=[4]-=- is well suited. Let L † A = W diag(β−1 i ) ˘ V T denote the A-weighted generalized inverse of L; cf. [4] for a formal definition. Then Asf and bsf are given by (24) Asf = AL † −1 A =Ŭdiag(αiβi ) ˘ V ... |

15 |
The Total Least
- Huffel, Vanderwalle
- 1991
(Show Context)
Citation Context ... in several bodies of work, and is known by statisticians as the errors in variables model. Numerical analysts came to know it through the work of Golub and Van Loan [10] and Van Hu#el and Vandewalle =-=[24, 25, 26]-=-, and this literature has advanced the algorithmic and theoretical understanding of the method. 1.1. Motivation. The development of the TLS technique was motivated by linear models Ax#b in which both ... |

12 |
Truncated SVD solutions to discrete ill-posed problems with ill-determined numerical rank
- Hansen
- 1990
(Show Context)
Citation Context ...# 2 i # 2 i + # 2 u T i b # i v i , (5) showing that this approach suppresses (filters) the components of the solution corresponding to the small singular values of A; see, e.g., [12, Section 5.1] or =-=[15]-=-. In this paper we prove that the same is true for truncated TLS. Our paper is organized as follows. Section 2 summarizes the truncated TLS algorithm, and the filtering properties of this algorithm ar... |

10 |
Collinearity and total least squares
- Fierro, Bunch
- 1994
(Show Context)
Citation Context ...more small (nonzero) singular values well separated from the large ones. The idea is to simply treat the small singular values of (A,b) as zeros, reducing the problem to an exactly rank-deficient one =-=[5, 26]-=-. We shall call this technique truncated TLS. The technique is similar in spirit to truncated SVD, a natural generalization of the ordinary LS method for nearly rank-deficient problems that treats sma... |

9 |
An efficient Total Least Squares algorithm based on a rank-revealing two-sided orthogonal decomposition
- Huffel, Zha
- 1993
(Show Context)
Citation Context ...d approach to TLS, developed by Golub and Van Loan [10], is based on the SVD of (A,b). Recently, computationally cheaper techniques based on rankrevealing orthogonal decompositions have also appeared =-=[2, 28]-=-. For clarity, in this section we shall confine ourselves to the SVD-based approach and return to computational and algorithmic aspects in sections 4 and 5. 1226 R. D. FIERRO, G. H. GOLUB, P. C. HANSE... |

8 |
Regularization, GSVD and truncated
- Hansen
- 1989
(Show Context)
Citation Context ... regularized solution is expanded in terms of the columns w i of W , and the main contributions come from the vectors w i associated with the largest generalized singular values # i /# i ; see, e.g., =-=[14]-=-, [13, Section 4], or [17, Section 6] for details. In connection with our T-TLS algorithms it may also be convenient to implicitly use regularization in general form with L #= I. This is done in the s... |

8 |
An efficient and reliable algorithm for computing the singular subspace of a matrix, associated with its smallest singular values
- Huffel, Vandewalle
- 1987
(Show Context)
Citation Context ...ative formulasx k =(sV T 11 ) +sV T 21 . (18) 1232 R. D. FIERRO, G. H. GOLUB, P. C. HANSEN, AND D. P. O'LEARY The partial SVD can be computed by a technique similar to the PSVD algorithm described in =-=[27]-=- for computing the last few singular triplets. However, for large sparse or structured matrices (e.g., Toeplitz matrices, which arise in connection with discretization of many convolution problems) th... |

7 |
Analysis and solution of the nongeneric total least squares problems
- Huffel, Vandewalle
- 1988
(Show Context)
Citation Context ... in several bodies of work, and is known by statisticians as the errors in variables model. Numerical analysts came to know it through the work of Golub and Van Loan [10] and Van Hu#el and Vandewalle =-=[24, 25, 26]-=-, and this literature has advanced the algorithmic and theoretical understanding of the method. 1.1. Motivation. The development of the TLS technique was motivated by linear models Ax#b in which both ... |

7 |
The analysis for the total least squares problem with more than one solution
- Wei
- 1992
(Show Context)
Citation Context ...is well conditioned. A related analysis which also focuses on the similarities between the truncated SVD and truncated TLS solutions of problems with well-defined numerical rank has been given by Wei =-=[29, 30]-=-. Many ill-conditioned problems arising in practical applications do not have a well-determined numerical rank; instead the singular values decay gradually to zero. Typically, these problems arise in ... |

5 |
Perturbation theory for orthogonal projection methods with applications to least squares and total least squares. Linear Algebra and its Applications
- Fierro, Bunch
- 1996
(Show Context)
Citation Context ...#erence between the methods lies in the way that this is done: in truncated SVD the modification depends solely on A, while in truncated TLS the modification depends on both A and b. Fierro and Bunch =-=[5, 6]-=- made a sensitivity analysis for the truncated TLS technique applied to a nearly rank-deficient A and showed how subspace sensitivity translates to solution sensitivity. The conclusion from their anal... |

5 |
On the invariance of perturbed null vectors under column scaling
- Stewart
- 1984
(Show Context)
Citation Context ...out while the remaining, significant contributions are retained insx k . If k = n and the errors in A and b are small, then the di#erence #x LS -sx n # 2 between the LS and the TLS solutions is small =-=[23]-=-, and our experience is that the same is true for #x k -sx k # 2 when k x k can be very di#erent from x k , and the filter factors f i for i # k---especially f k ---can di#er considerably from one (we... |

4 |
Algebraic relations between the total least squares and least squares problems with more than one solution
- Wei
- 1992
(Show Context)
Citation Context ...is well conditioned. A related analysis which also focuses on the similarities between the truncated SVD and truncated TLS solutions of problems with well-defined numerical rank has been given by Wei =-=[29, 30]-=-. Many ill-conditioned problems arising in practical applications do not have a well-determined numerical rank; instead the singular values decay gradually to zero. Typically, these problems arise in ... |

2 |
en, A weighted pseudoinverse, generalized singular values, and constrained least squares problems
- Eld
- 1982
(Show Context)
Citation Context ...arized solution x sf . Finally, transform x sf back to the general-form setting. There are several ways to transform a problem into standard form. The following transformation originally due to Elden =-=[4]-=- is well suited. Let L + A = W diag(# -1 i )sV T denote the A-weighted generalized inverse of L; cf. [4] for a formal definition. Then A sf and b sf are given by A sf = AL + A = Udiag(# i # -1 i ) V T... |

1 |
Algebraic relationships between classical regression and total least-squares estimation
- HUFFEL, VANDEWALLE
- 1987
(Show Context)
Citation Context ... in several bodies of work, and is known by statisticians as the errors in variables model. Numerical analysts came to know it through the work of Golub and Van Loan [10] and Van Hu#el and Vandewalle =-=[24, 25, 26]-=-, and this literature has advanced the algorithmic and theoretical understanding of the method. 1.1. Motivation. The development of the TLS technique was motivated by linear models Ax#b in which both ... |