Overcomplete representations are attracting interest in signal processing theory, particularly due to their potential to generate sparse representations of signals. However, in general, the problem of finding sparse representations must be unstable in the presence of noise. This paper establishes the possibility of stable recovery under a combination of sufficient sparsity and favorable structure of the overcomplete system. Considering an ideal underlying signal that has a sufficiently sparse representation, it is assumed that only a noisy version of it can be observed. Assuming further that the overcomplete system is incoherent, it is shown that the optimally sparse approximation to the noisy data differs from the optimally sparse decomposition of the ideal noiseless signal by at most a constant multiple of the noise level. As this optimal-sparsity method requires heavy (combinatorial) computational effort, approximation algorithms are considered. It is shown that similar stability is also available using the basis and the matching pursuit algorithms. Furthermore, it is shown that these methods result in sparse approximation of the noisy data that contains only terms also appearing in the unique sparsest representation of the ideal noiseless sparse signal.

Sparse coding in a redundant basis is of considerable interest in many areas of signal processing. The problem generally involves solving an under-determined system of equations under a sparsity constraint. Except for the exhaustive combinatorial approach, there is no known method to find the exact solution under general conditions on the dictionary. Among the various algorithms that find approximate solutions, pursuit algorithms are the most well-known. In this paper, we introduce the concept of a complementary matching pursuit (CMP). The algorithm is similar to the classical matching pursuit (MP), but performs the complementary action. Instead of selecting one atom to be included in the sparse approximation, it selects (N −1) atoms to be excluded from the approximation at each iteration. Though these two actions seem apparently the same, they are actually performed in two different spaces. On a conceptual level, the MP searches for ’the solution vector among sparse vectors ’ whereas the CMP searches for ’the sparse vector among the solution vectors’. We assume that the observations can be expressed as pure linear sums of atoms without any additive noise. As a consequence of the complementary action, the CMP does not minimize the residual error at each iteration, however it may converge faster yielding sparser solution vectors than the MP. We show that when the dictionary is a tight frame, the CMP is equivalent to the MP. We also present the orthogonal extensions of the CMP and show that they perform the complementary actions to those of their classical matching pursuit counterparts.

Signal approximation using a linear combination of basis from an overcomplete dictionary has been proven to be an NP-complete problem. By selecting a smaller number of basis than the span of the signal, we achieve lossy compression in exchange for a small reconstruction error. Several algorithms have been researched that reduce the complexity of the selection problem, sacrificing the optimality of the solution. The Matching Pursuit (MP) algorithm has been used for signal approximation for over a decade. Many variations have been proposed and implemented to enhance the performance of the algorithm. However, its greedy nature renders it sub-optimal. In this thesis, a survey of the different variations is provided. An enhancement for the MP algorithm is proposed that uses concepts from simulated annealing in improving the performance in terms of compression ratio and reconstructed quality. The algorithm is then applied to image signals. Results show superior