## An affine scaling methodology for best basis selection (1999)

### Cached

### Download Links

- [dsp.rice.edu]
- [www.dsp.ece.rice.edu]
- [dsp.rice.edu]
- DBLP

### Other Repositories/Bibliography

Venue: | IEEE Trans. Signal Processing |

Citations: | 78 - 11 self |

### BibTeX

@ARTICLE{Rao99anaffine,

author = {Bhaskar D. Rao and Senior Member and Kenneth Kreutz-delgado and Senior Member},

title = {An affine scaling methodology for best basis selection},

journal = {IEEE Trans. Signal Processing},

year = {1999},

pages = {187--200}

}

### Years of Citing Articles

### OpenURL

### Abstract

Abstract — A methodology is developed to derive algorithms for optimal basis selection by minimizing diversity measures proposed by Wickerhauser and Donoho. These measures include the p-norm-like (`(p 1)) diversity measures and the Gaussian and Shannon entropies. The algorithm development methodology uses a factored representation for the gradient and involves successive relaxation of the Lagrangian necessary condition. This yields algorithms that are intimately related to the Affine Scaling Transformation (AST) based methods commonly employed by the interior point approach to nonlinear optimization. The algorithms minimizing the `(p 1) diversity measures are equivalent to a recently developed class of algorithms called FOCal Underdetermined System Solver (FOCUSS). The general nature of the methodology provides a systematic approach for deriving this class of algorithms and a natural mechanism for extending them. It also facilitates a better understanding of the convergence behavior and a strengthening of the convergence results. The Gaussian entropy minimization algorithm is shown to be equivalent to a well-behaved p =0norm-like optimization algorithm. Computer experiments demonstrate that the p-norm-like and the Gaussian entropy algorithms perform well, converging to sparse solutions. The Shannon entropy algorithm produces solutions that are concentrated but are shown to not converge to a fully sparse solution. I.

### Citations

8593 |
Elements of Information Theory
- Cover, Thomas
- 1991
(Show Context)
Citation Context ...thm, we need the following identities. Identity 1: Proof: Proof: The last inequality follows from the fact that with equality if and only if for all , we have , where here, and are probabilities [9], =-=[41]-=-. B. Modified Algorithm Using the above results, we consider the following form of an algorithm for minimizing the Shannon entropy (3) If is feasible, then it can be readily shown that is feasible. Th... |

1723 |
Ten lectures on wavelets
- Daubechies
- 1992
(Show Context)
Citation Context ...is the null space of . In this case, has a nontrivial nullspace of dimension . In many situations, a popular approach has been to set and to select as the desired solution, e.g., the method of frames =-=[4]-=-. However, the minimum 2-norm criteria favors solutions with many small nonzero entries, which is a property that is contrary to the goal of sparsity/concentration [11], [16]. Consequently, there is a... |

1672 | Atomic decomposition by basis pursuit
- Chen, Donoho, et al.
- 1998
(Show Context)
Citation Context ...ary using a carefully chosen set of redundant basis vectors that can represent a larger class of signals compactly. Popular dictionaries used are the Wavelet and Gabor dictionaries, among others [7], =-=[12]-=-. The problem of basis selection, i.e., choosing a proper and succinct subset of vectors from the dictionary, naturally arises in this case, and developing algorithms for optimal basis selection is a ... |

1351 |
Practical Optimization
- Gill, Murray, et al.
- 1987
(Show Context)
Citation Context ...icult nonlinear equation (5) is a standard occurrence in optimization problems. Several creative methods have been developed to address this problem in the nonlinear programming literature [29], [36]–=-=[38]-=-. Of particular interest to us in this context are the methods of finding iterative solutions by a relaxation approach where the Lagrangian is modified, and a related simpler problem is solved. In suc... |

1052 | Matching pursuits with time-frequency dictionaries
- Mallat, Zhang
- 1993
(Show Context)
Citation Context ...there has been a great deal of interest in finding efficient representations of signals [1]–[6]. Of particular interest to us is the approach of using an overcomplete dictionary to represent a signal =-=[7]-=-–[11]. The motivation for such an approach is that a minimal spanning set of basis vectors is usually only adequate to efficiently represent a small class of signals while forming an overcomplete dict... |

1045 |
Introduction to Linear and Nonlinear Programming
- Luenberger
- 1973
(Show Context)
Citation Context ...s. For example consider A. Algorithm Derivation To minimize the diversity measures subject to the equality constraints (1), we start with the standard method of Lagrange multipliers (see, e.g., [29], =-=[36]-=-, and [37]). Define the Lagrangian where is the vector of Lagrange multipliers. A necessary condition for a minimizing solution to exist is that be stationary points of the Lagrangian function, i.e., ... |

717 |
Nonlinear programming : theory and algorithms
- Bazaraa, Sherali, et al.
- 1993
(Show Context)
Citation Context ...mple consider A. Algorithm Derivation To minimize the diversity measures subject to the equality constraints (1), we start with the standard method of Lagrange multipliers (see, e.g., [29], [36], and =-=[37]-=-). Define the Lagrangian where is the vector of Lagrange multipliers. A necessary condition for a minimizing solution to exist is that be stationary points of the Lagrangian function, i.e., The gradie... |

491 | Entropy-based algorithms for best basis selection
- Coifman, Wickerhauser
- 1992
(Show Context)
Citation Context ...d.edu). Publisher Item Identifier S 1053-587X(99)00148-8. algorithm does not result in effective sparse representations [7], [13], [14]. Another effective approach to basis selection was developed in =-=[8]-=- and [9] in the context of special dictionaries, wavelet packets, and cosine packets. An entropy-based measure of sparsity was used to choose the optimal basis, and an efficient algorithm was develope... |

474 |
Wavelets and Subband Coding
- Vetterli, Kova˘cević
- 1995
(Show Context)
Citation Context ...tions that are concentrated but are shown to not converge to a fully sparse solution. I. INTRODUCTION RECENTLY, there has been a great deal of interest in finding efficient representations of signals =-=[1]-=-–[6]. Of particular interest to us is the approach of using an overcomplete dictionary to represent a signal [7]–[11]. The motivation for such an approach is that a minimal spanning set of basis vecto... |

357 | Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition
- Pati, Rezaiifar, et al.
- 1993
(Show Context)
Citation Context ...ego, La Jolla, CA 92093-0407 USA (e-mail: brao@ece.ucsd.edu; kreutz@ece.ucsd.edu). Publisher Item Identifier S 1053-587X(99)00148-8. algorithm does not result in effective sparse representations [7], =-=[13]-=-, [14]. Another effective approach to basis selection was developed in [8] and [9] in the context of special dictionaries, wavelet packets, and cosine packets. An entropy-based measure of sparsity was... |

318 |
Sparse approximate solutions to linear systems
- Natarajan
- 1995
(Show Context)
Citation Context ...se it is a direct measure of sparsity, providing a count of the number of nonzero elements of a vector Finding a global minimum to the numerosity measure requires an enumerative search and is NP hard =-=[26]-=-. Consequently, alternate diversity measures that are more amenable to optimization techniques are of interest. The measures for are useful candidate measures in this context and are indirectly relate... |

292 |
Adapted wavelet analysis from theory to software
- Wickerhauser
- 1994
(Show Context)
Citation Context ...Publisher Item Identifier S 1053-587X(99)00148-8. algorithm does not result in effective sparse representations [7], [13], [14]. Another effective approach to basis selection was developed in [8] and =-=[9]-=- in the context of special dictionaries, wavelet packets, and cosine packets. An entropy-based measure of sparsity was used to choose the optimal basis, and an efficient algorithm was developed, explo... |

218 | Sparse signal reconstruction from limited data using focuss: A re-weighted minimum norm algorithm
- Gorodnitsky, Rao
- 1997
(Show Context)
Citation Context ...ar inverse problems where the solution is known or required to be sparse, e.g., speech coding [21], bandlimited extrapolation and spectral estimation [22], [23], direction-of-arrival estimation [16], =-=[24]-=-, functional approximation [25]–[27], failure diagnosis [28], and pattern recognition for medical diagnosis [19]. We can exploit the advances in these disparate areas to develop effective solutions to... |

123 | On basis pursuit
- Chen, Donoho
- 1994
(Show Context)
Citation Context ... =20;n=30AND SPARSITY r =7. SUCCESS CORRESPONDS TO THE SEVEN COLUMNS OF THE TRUE SPARSE SOLUTION BEING INCLUDED IN THE SET OF COLUMNS CHOSEN BY THE ALGORITHM methods employed for norm minimization in =-=[11]-=-, [29], and [30] can be viewed as employing such an approach and are potentially extensible for minimizing the other diversity measures. To get a better understanding of the performance of the method,... |

88 |
Linear and Nonlinear Programming
- Nash, Sofer
- 1996
(Show Context)
Citation Context ...er is that the algorithms are shown to be equivalent to Affine Scaling Transformation (AST) algorithms, which have recently received attention in the literature on interior point optimization methods =-=[29]-=-–[31]. The outline of the paper is as follows. In Section II, we present the -norm-like ( , including negative) diversity measures and the Gaussian and Shannon entropy measures of sparsity proposed in... |

80 | Multiresolution Signal Decomposition. Transforms, Subbands, Wavelets - Akansu, Haddad - 1992 |

73 |
Neuromagnetic source imaging with focuss: a recursive weighted minimum norm algorithm, Electroencephalography and Clinical Neurophysiology 95
- Gorodnitsky, George, et al.
- 1995
(Show Context)
Citation Context ...s [33]. Note that minimizing diversity (antisparsity) is equivalent to maximizing concentration (sparsity). Our own interest in this problem was initially motivated by the biomagnetic imaging problem =-=[20]-=-. Basis selection has applications to linear inverse problems where the solution is known or required to be sparse, e.g., speech coding [21], bandlimited extrapolation and spectral estimation [22], [2... |

60 | Feature selection via mathematical programming
- Bradley, Mangasarian, et al.
(Show Context)
Citation Context ...selection arises in many other applications, and researchers in other areas have also attempted to define diversity measures and to compute sparse/concentrated solutions based on minimizing them [15]–=-=[19]-=-. The use of the term “diversity” in this paper refers to a measure of antisparsity and is consistent with the terminology used in several research areas [33]. Note that minimizing diversity (antispar... |

27 | Wavelets and timefrequency analysis - Hess-Nielsen, Wickerhauser - 1996 |

24 |
Wavelets, subband coding, and best bases
- Ramchandran, Vetterli, et al.
- 1996
(Show Context)
Citation Context ...s that are concentrated but are shown to not converge to a fully sparse solution. I. INTRODUCTION RECENTLY, there has been a great deal of interest in finding efficient representations of signals [1]–=-=[6]-=-. Of particular interest to us is the approach of using an overcomplete dictionary to represent a signal [7]–[11]. The motivation for such an approach is that a minimal spanning set of basis vectors i... |

23 | Fast orthogonal least squares algorithm for efficient subset model selection - Chen, Wigger - 1995 |

23 |
Hertog, Interior point approach to linear, quadratic and convex programming
- den
- 1994
(Show Context)
Citation Context ... that the algorithms are shown to be equivalent to Affine Scaling Transformation (AST) algorithms, which have recently received attention in the literature on interior point optimization methods [29]–=-=[31]-=-. The outline of the paper is as follows. In Section II, we present the -norm-like ( , including negative) diversity measures and the Gaussian and Shannon entropy measures of sparsity proposed in [9] ... |

21 |
Extrapolation and spectral estimation with iterative weighted norm modification
- Cabrera, Parks
- 1991
(Show Context)
Citation Context ...0]. Basis selection has applications to linear inverse problems where the solution is known or required to be sparse, e.g., speech coding [21], bandlimited extrapolation and spectral estimation [22], =-=[23]-=-, direction-of-arrival estimation [16], [24], functional approximation [25]–[27], failure diagnosis [28], and pattern recognition for medical diagnosis [19]. We can exploit the advances in these dispa... |

19 |
A new algorithm for computing sparse solutions to linear inverse problems
- Harikumar, Bresler
- 1996
(Show Context)
Citation Context ...d to sparse solutions. The question of good diversity measures has been studied in the past, and a good discussion can be found in [9] and [10], and in the literature on linear inverse problems [15], =-=[17]-=-, [18]. A popular diversity measure is , where We extend this class to include negative values of to the general class of diversity measures where sgn , leading (2) . The diversity measures for are th... |

16 | Comparison of basis selection methods
- Adler, Rao, et al.
- 1997
(Show Context)
Citation Context ...a Jolla, CA 92093-0407 USA (e-mail: brao@ece.ucsd.edu; kreutz@ece.ucsd.edu). Publisher Item Identifier S 1053-587X(99)00148-8. algorithm does not result in effective sparse representations [7], [13], =-=[14]-=-. Another effective approach to basis selection was developed in [8] and [9] in the context of special dictionaries, wavelet packets, and cosine packets. An entropy-based measure of sparsity was used ... |

14 | A globally convergent method for l problems - Li - 1993 |

13 |
On minimum entropy segmentation,” in Wavelets: Theory, Algorithms and Applications
- Donoho
- 1994
(Show Context)
Citation Context ...ersity measures” and often, more simply, as the “ -norm-like diversity measures.” It is well known that for , is not a true norm [15]. The diversity measure for , which is the numerosity discussed in =-=[10]-=-, is of special interest because it is a direct measure of sparsity, providing a count of the number of nonzero elements of a vector Finding a global minimum to the numerosity measure requires an enum... |

13 | Restoration of blurred star field images by maximally sparse optimization
- Jeffs, Gunsay
- 1993
(Show Context)
Citation Context ...asis selection arises in many other applications, and researchers in other areas have also attempted to define diversity measures and to compute sparse/concentrated solutions based on minimizing them =-=[15]-=-–[19]. The use of the term “diversity” in this paper refers to a measure of antisparsity and is consistent with the terminology used in several research areas [33]. Note that minimizing diversity (ant... |

13 | Satisficing search algorithms for selecting near-best bases in adaptive tree-structured wavelet transforms
- Taswell
- 1996
(Show Context)
Citation Context ...rsity measure element can be readily shown to be (5) with respect to will be minimized by making the entries of small, thereby encouraging sparsity. Many other diversity measures can be defined [33], =-=[35]-=-. We only examine here the Shannon entropy and Gaussian entropy, which are two other diversity measures described in [8]–[10]. The Shannon entropy diversity measure is defined as where (3) Substitutin... |

11 |
Automatic Test Generation Techniques for Analog Circuits and Systems: A Review
- Duhamel, Rault
- 1979
(Show Context)
Citation Context ...to be sparse, e.g., speech coding [21], bandlimited extrapolation and spectral estimation [22], [23], direction-of-arrival estimation [16], [24], functional approximation [25]–[27], failure diagnosis =-=[28]-=-, and pattern recognition for medical diagnosis [19]. We can exploit the advances in these disparate areas to develop effective solutions to the best basis selection problem. It is clear that an effec... |

10 |
Sparse approximate multiquadric interpolation
- Carlson, Natarajan
- 1994
(Show Context)
Citation Context ...on is known or required to be sparse, e.g., speech coding [21], bandlimited extrapolation and spectral estimation [22], [23], direction-of-arrival estimation [16], [24], functional approximation [25]–=-=[27]-=-, failure diagnosis [28], and pattern recognition for medical diagnosis [19]. We can exploit the advances in these disparate areas to develop effective solutions to the best basis selection problem. I... |

10 |
Analysis and extension of the FOCUSS algorithm
- Rao
- 1996
(Show Context)
Citation Context ...pplying interior point methods to large scale problems similar to the optimization problem described in [12]. Some additional details on initialization and computation can be found in [16], [24], and =-=[39]-=-. which is precisely the iterative procedure given by (11). A closer examination of the scaling matrix diag is worthwhile. In the standard AST methods, usually, the scaling matrix diag is used4 [29], ... |

8 |
Improvement of discrete band-limited signal extrapolation by iterative subspace modification
- Lee, Sullivan, et al.
- 1987
(Show Context)
Citation Context ...lem [20]. Basis selection has applications to linear inverse problems where the solution is known or required to be sparse, e.g., speech coding [21], bandlimited extrapolation and spectral estimation =-=[22]-=-, [23], direction-of-arrival estimation [16], [24], functional approximation [25]–[27], failure diagnosis [28], and pattern recognition for medical diagnosis [19]. We can exploit the advances in these... |

7 | A recursive weighted minimum norm algorithm: Analysis and applications - Gorodnitsky, Rao |

6 | A general approach to sparse basis selection: Majorization, concavity, and affine scaling
- Kreutz-Delgado, Rao
- 1997
(Show Context)
Citation Context ...d solutions based on minimizing them [15]–[19]. The use of the term “diversity” in this paper refers to a measure of antisparsity and is consistent with the terminology used in several research areas =-=[33]-=-. Note that minimizing diversity (antisparsity) is equivalent to maximizing concentration (sparsity). Our own interest in this problem was initially motivated by the biomagnetic imaging problem [20]. ... |

5 |
Sparse deconvolution using adaptive mixedGaussian models
- Santamaría-Caballero, Pantaleón-Prieto, et al.
- 1996
(Show Context)
Citation Context ...parse solutions. The question of good diversity measures has been studied in the past, and a good discussion can be found in [9] and [10], and in the literature on linear inverse problems [15], [17], =-=[18]-=-. A popular diversity measure is , where We extend this class to include negative values of to the general class of diversity measures where sgn , leading (2) . The diversity measures for are the gene... |

4 |
Linear Optimization and Extension
- Fang, Puthenpura
- 1993
(Show Context)
Citation Context .... which is precisely the iterative procedure given by (11). A closer examination of the scaling matrix diag is worthwhile. In the standard AST methods, usually, the scaling matrix diag is used4 [29], =-=[30]-=-. For the algorithm suggested here, naturally defines a scaling matrix via the relationship (12), which is more generally dependent on the choice of . For , which corresponds to the norm, (12) results... |

1 |
Digital Speech: Low Bit Rate Coding for Communication Systems
- Kondoz
- 1996
(Show Context)
Citation Context ... was initially motivated by the biomagnetic imaging problem [20]. Basis selection has applications to linear inverse problems where the solution is known or required to be sparse, e.g., speech coding =-=[21]-=-, bandlimited extrapolation and spectral estimation [22], [23], direction-of-arrival estimation [16], [24], functional approximation [25]–[27], failure diagnosis [28], and pattern recognition for medi... |

1 |
Convergence analysis of a class of adaptive weighted norm extrapolation algorithms
- Gorodnitsky, Rao
- 1993
(Show Context)
Citation Context ... choice of value for . In Section III-C and Appendix A, a detailed convergence analysis of the algorithm is performed, expanding the scope of the convergence results previously prescribed in [24] and =-=[32]-=-. In Section IV, we focus on the case . We show that the -norm-like algorithm obtained by setting and the algorithm obtained from minimizing the Gaussian entropy are identical and argue that this algo... |