Results 11  20
of
36
Large Scale Variational Inference and Experimental Design for Sparse Generalized Linear Models
, 2008
"... Sparsity is a fundamental concept of modern statistics, and often the only general principle available at the moment to address novel learning applications with many more variables than observations. While much progress has been made recently in the theoretical understanding and algorithmics of spa ..."
Abstract

Cited by 9 (6 self)
 Add to MetaCart
Sparsity is a fundamental concept of modern statistics, and often the only general principle available at the moment to address novel learning applications with many more variables than observations. While much progress has been made recently in the theoretical understanding and algorithmics of sparse point estimation, higherorder problems such as covariance estimation or optimal data acquisition are seldomly addressed for sparsityfavouring models, and there are virtually no algorithms for large scale applications of these. We provide novel approximate Bayesian inference algorithms for sparse generalized linear models, that can be used with hundred thousands of variables, and run orders of magnitude faster than previous algorithms in domains where either apply. By analyzing our methods and establishing some novel convexity results, we settle a longstanding open question about variational Bayesian inference for continuous variable models: the Gaussian lower bound relaxation, which has been used previously for a range of models, is proved to be a convex optimization problem, if and only if the posterior mode is found by convex programming. Our algorithms reduce to the same computational primitives than commonly used sparse estimation methods do, but require Gaussian marginal variance estimation as well. We show how the Lanczos algorithm from numerical mathematics can be employed to compute the latter. We are interested in Bayesian experimental design here (which is mainly driven by efficient approximate inference), a powerful framework for optimizing measurement architectures of complex signals, such as natural images. Designs
Multiprocessor Sparse Svd Algorithms And Applications
, 1991
"... this memory is statically allocated, whereas on the Alliant FX/80 it is dynamically allocated as needed. On the Cray2S/4128, the vector z would be both retrieved from and written to core memory. However, on the Alliant FX/80, z may be fetched and held in the 512 kilobyte cache. Since memory accesse ..."
Abstract

Cited by 8 (3 self)
 Add to MetaCart
this memory is statically allocated, whereas on the Alliant FX/80 it is dynamically allocated as needed. On the Cray2S/4128, the vector z would be both retrieved from and written to core memory. However, on the Alliant FX/80, z may be fetched and held in the 512 kilobyte cache. Since memory accesses from the cache (fast local memory) can almost twice as fast as those from the larger globallyshared memory, we achieve an overall higher computational rate for multiplication by A
Arnoldi versus Nonsymmetric Lanczos Algorithms for Solving Nonsymmetric Matrix Eigenvalue Problems
 BIT
, 1996
"... We obtain several results which may be useful in determining the convergence behavior of eigenvalue algorithms based upo n Arnoldi and nonsymmetric Lanczos recursions. We derive a relationship between nonsymmetric Lanczos eigenvalue procedures and Arnoldi eigenvalue procedures. We demonstrate that t ..."
Abstract

Cited by 7 (1 self)
 Add to MetaCart
We obtain several results which may be useful in determining the convergence behavior of eigenvalue algorithms based upo n Arnoldi and nonsymmetric Lanczos recursions. We derive a relationship between nonsymmetric Lanczos eigenvalue procedures and Arnoldi eigenvalue procedures. We demonstrate that the Arnoldi recursions preserve a property which characterizes normal matrices, and that if we could determine the appropriate starting vectors, we could mimic the nonsymmetric Lanczos eigenvalue convergence on a general diagonalizable matrix by its convergence on related normal matrices. Using a unitary equivalence for each of these Krylov subspace methods, we define sets of test problems where we can easily vary certain spectral properties of the matrices. We use these and other test problems to examine the behavior of an Arnoldi and of a nonsymmetric Lanczos procedure. Mathematical Sciences Department, IBM Research Division, T.J. Watson Research Center, Yorktown Heights, NY 10598, USA, a...
Conjugate Gradient Algorithms with Reduced Synchronization Overhead on Distributed Memory Multiprocessors
, 1999
"... The standard formulation of the conjugate gradient algorithm involves two inner product computations. The results of these two inner products are needed to update the search direction and the computed solution. Since these inner products are mutually interdependent, in a distributed memory parallel ..."
Abstract

Cited by 7 (0 self)
 Add to MetaCart
The standard formulation of the conjugate gradient algorithm involves two inner product computations. The results of these two inner products are needed to update the search direction and the computed solution. Since these inner products are mutually interdependent, in a distributed memory parallel environment their computation and subsequent distribution requires two separate communication and synchronization phases. In this paper, we present three related mathematically equivalent rearrangements of the standard algorithm that reduce the number of communication phases. We present empirical evidence that two of these rearrangements are numerically stable. This claim is further substantiated by a proof that one of the empirically stable rearrangements arises naturally in the symmetric Lanczos method for linear systems, which is equivalent to the conjugate gradient method.
Numerical Stability Of GMRES
, 1995
"... . The Generalized Minimal Residual Method (GMRES) is one of the significant methods for solving linear algebraic systems with nonsymmetric matrices. It minimizes the norm of the residual on the linear variety determined by the initial residual and the nth Krylov residual subspace and is therefore ..."
Abstract

Cited by 5 (1 self)
 Add to MetaCart
. The Generalized Minimal Residual Method (GMRES) is one of the significant methods for solving linear algebraic systems with nonsymmetric matrices. It minimizes the norm of the residual on the linear variety determined by the initial residual and the nth Krylov residual subspace and is therefore optimal, with respect to the size of the residual, in the class of Krylov subspace methods. One possible way of computing the GMRES approximations is based on constructing the orthonormal basis of the Krylov subspaces (Arnoldi basis) and then solving the transformed least squares problem. This paper studies the numerical stability of such formulations of GMRES. Our approach is based on the Arnoldi recurrence for the actually, i.e. in finite precision arithmetic, computed quantities. We consider the Householder (HHA), iterated modified GramSchmidt (IMGSA), and iterated classical GramSchmidt (ICGSA) implementations. Under the obvious assumption on the numerical nonsingularity of the system m...
Influence of Orthogonality on the Backward Error and the Stopping Criterion for Krylov Methods
, 1995
"... Many algorithms for solving linear systems, least squares problems or eigenproblems need to compute an orthonormal basis. The computation is commonly performed using a QR factorization computed using the classical or the modified GramSchmidt algorithm, the Householder algorithm, the Givens algorith ..."
Abstract

Cited by 4 (2 self)
 Add to MetaCart
Many algorithms for solving linear systems, least squares problems or eigenproblems need to compute an orthonormal basis. The computation is commonly performed using a QR factorization computed using the classical or the modified GramSchmidt algorithm, the Householder algorithm, the Givens algorithm or the GramSchmidt algorithm with iterative reorthogonalization. For linear systems, it is wellknown that the backward stability of the process depends crucially on the algorithm used for the QR factorization. For the eigenproblem, although textbooks warn users about the possible instability of eigensolvers due to loss of orthogonality, few theoretical results exist. In this paper we show that the loss of orthogonality of the computed basis can affect the reliability of the computed eigenpair. We also show that the classical residual norm kAx \Gamma xk and the specific computed one using a Krylov method can differ because of the loss of orthogonality of the computed basis of the Krylov s...
Accurate Conjugate Gradient Methods for Families of Shifted Systems
, 2003
"... We present an e#cient and accurate variant of the conjugate gradient method for solving families of shifted systems. In particular we are interested in shifted systems that occur in Tikhonov regularization for inverse problems since these problems can be sensitive to roundo# errors. The success o ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
We present an e#cient and accurate variant of the conjugate gradient method for solving families of shifted systems. In particular we are interested in shifted systems that occur in Tikhonov regularization for inverse problems since these problems can be sensitive to roundo# errors. The success of our method in achieving accurate approximations is supported by theoretical arguments as well as several numerical experiments and we relate it to other implementations proposed in literature.
Tomographic Resolution Without Singular Value Decomposition
 Mathematical Methods in Geophysical Imaging II
, 1994
"... An explicit procedure is presented for computing both model and data resolution matrices within a PaigeSaunders LSQR algorithm for iterative inversion in seismic tomography. These methods are designed to avoid the need for an additional singular value decomposition of the raypath matrix. The techni ..."
Abstract

Cited by 3 (2 self)
 Add to MetaCart
An explicit procedure is presented for computing both model and data resolution matrices within a PaigeSaunders LSQR algorithm for iterative inversion in seismic tomography. These methods are designed to avoid the need for an additional singular value decomposition of the raypath matrix. The techniques discussed are completely general since they are based on the multiplicity of equivalent exact formulas that may be used to define the resolution matrices. Thus, resolution matrices may also be computed for a wide variety of iterative inversion algorithms using the same ideas. Keywords: seismic tomography, inversion, resolution matrices 1 INTRODUCTION Linear tomographic reconstruction schemes have welldefined resolution properties. 1 Yet, the resolution matrices summarizing these properties are not computed as often as they might be  at least in part  because of the common misconception that resolution matrices can only be found using singular value decomposition (SVD). Since ...
MINRESQLP: A Krylov subspace method for indefinite or singular symmetric systems, SIAMJ.Sci.Comput.,toappear
"... Abstract. CG, SYMMLQ, and MINRES are Krylov subspace methods for solving symmetric systems of linear equations. When these methods are applied to an incompatible system (that is, a singular symmetric leastsquares problem), CG could break down and SYMMLQ’s solution could explode, while MINRES would ..."
Abstract

Cited by 3 (2 self)
 Add to MetaCart
Abstract. CG, SYMMLQ, and MINRES are Krylov subspace methods for solving symmetric systems of linear equations. When these methods are applied to an incompatible system (that is, a singular symmetric leastsquares problem), CG could break down and SYMMLQ’s solution could explode, while MINRES would give a leastsquares solution but not necessarily the minimumlength (pseudoinverse) solution. This understanding motivates us to design a MINRESlike algorithm to compute minimumlength solutions to singular symmetric systems. MINRES uses QR factors of the tridiagonal matrix from the Lanczos process (where R is uppertridiagonal). MINRESQLP uses a QLP decomposition (where rotations on the right reduce R to lowertridiagonal form). On illconditioned systems (singular or not), MINRESQLP can give more accurate solutions than MINRES. We derive preconditioned MINRESQLP, new stopping rules, and better estimates of the solution and residual norms, the matrix norm, and the condition number.