When constructing a classifier from labeled data, it is important not to assign too much weight to any single input feature, in order to increase the robustness of the classifier. This is particularly important in domains with nonstationary feature distributions or with input sensor failures. A common approach to achieving such robustness is to introduce regularization which spreads the weight more evenly between the features. However, this strategy is very generic, and cannot induce robustness specifically tailored to the classification task at hand. In this work, we introduce a new algorithm for avoiding single feature over-weighting by analyzing robustness using a game theoretic formalization. We develop classifiers which are optimally resilient to deletion of features in a minimax sense, and show how to construct such classifiers using quadratic programming. We illustrate the applicability of our methods on spam filtering and handwritten digit recognition tasks, where feature deletion is indeed a realistic noise model. 1. Building Robust Classifiers When constructing classifiers over high dimensional spaces such as texts or images, one is inherently faced with the problem of under-sampling of the true data distribution. Even so-called “discriminative ” methods which focus on minimizing classification error (or a bound on it) are exposed to this difficulty since the training objective will be calculated over the observed input vectors only, and thus may not be a good approximation of the average objective on the test data. This is especially important in settings such as document

In this paper we survey the primary research, both theoretical and applied, in the field of Robust Optimization (RO). Our focus will be on the computational attractiveness of RO approaches, as well as the modeling power and broad applicability of the methodology. In addition to surveying the most prominent theoretical results of RO over the past decade, we will also present some recent results linking RO to adaptable models for multi-stage decision-making problems. Finally, we will highlight successful applications of RO across a wide spectrum of domains, including, but not limited to, finance, statistics, learning, and engineering.

This paper concerns a fractional function of the form x T a / √ x T Bx, where B is positive definite. We consider the game of choosing x from a convex set, to maximize the function, and choosing (a,B) from a convex set, to minimize it. We prove the existence of a saddle point and describe an efficient method, based on convex optimization, for computing it. We describe applications in machine learning (robust Fisher linear discriminant analysis), signal processing (robust beamforming, robust matched filtering), and finance (robust portfolio selection). In these applications, x corresponds to some design variables to be chosen, and the pair (a,B) corresponds to the statistical model, which is uncertain. 1

Dimensionality reduction is often needed in many applications due to the high dimensionality of the data involved. In this paper, we first analyze the scatter measures used in the conventional linear discriminant analysis (LDA) model and note that the formulation is based on the average-case view. Based on this analysis, we then propose a new dimensionality reduction method called worst-case linear discriminant analysis (WLDA) by defining new between-class and within-class scatter measures. This new model adopts the worst-case view which arguably is more suitable for applications such as classification. When the number of training data points or the number of features is not very large, we relax the optimization problem involved and formulate it as a metric learning problem. Otherwise, we take a greedy approach by finding one direction of the transformation at a time. Moreover, we also analyze a special case of WLDA to show its relationship with conventional LDA. Experiments conducted on several benchmark datasets demonstrate the effectiveness of WLDA when compared with some related dimensionality reduction methods. 1

Bringing robustness into subspace methods is very important, for training as well as for recognition. In case of Linear Discriminant Analysis (LDA) the task of robust classification is already solved, therefore, we focus on treating pixel outliers and occlusions in the training stage. More precisely, in this work we consider the task of incremental learning. Based on an augmented LDA basis that incorporates a certain amount of reconstructive information we are able to achieve the desired robustness. The advantage of the enriched basis is that it contains enough reconstructive information to handle noisy data, while it still exploits its full discriminative property. 1

A Generalized Nonlinear Discriminant Analysis (GNDA) method is proposed, which implements Fisher discriminant analysis in a nonlinear mapping space. Linear discriminant analysis in the nonlinear mapping space corresponds to nonlinear discriminant analysis in an input space. GNDA suggests a unified framework of nonlinear discriminant analysis which includes the kernel Fisher discriminant analysis as a specific case. Experimental results on UCI data sets demonstrate the validity of our method. 1.

This paper is concerned with the problem of finding a vector from a given set in a Euclidean space that maximizes the worst-case correlation with vectors in this set, where ‘worst ’ means smallest. This problem, called the maximin correlation problem (MCP), comes up in several disciplines including pattern classification, portfolio selection, statistics, and signal processing. With a general infinite set, the associated MCP is a semi-infinite program and so difficult to solve exactly. As an important tractable case, we show that the MCP with an ellipsoid or, more generally, the union of finitely many ellipsoids can be solved efficiently using an iterative method which alternates between optimization and worst-case correlation analysis. The optimization step is to solve an MCP with a finite set of sampled points from the set, and the worst-case correlation analysis step is to find a point in the set that minimizes the correlation with the point found at the preceding optimization step. The optimization step and the worst-case correlation analysis step can be reformulated as a second-order cone program (SOCP) and a semidefinite program (SDP), respectively, each of which can be solved with great efficiency using interior-point methods. Combined with a technique that approximates a general set with the union of finitely many ellipsoids, the iterative method can approximately solve the MCP with a general non-ellipsoidal set. Key words: cosine similarity metric, convex optimization, pattern recognition, portfolio optimization, robust and regret optimization, signal detection. 1 1

Abstract not found

Abstract—Building on previous work in robust optimization, we present a formulation of robust logistic regression under bounded data uncertainties. The robust estimates are obtained using block coordinate gradient descent with iterative group thresholding, which zeros out highly uncertain variables. For high dimensional problems with uncertain measurements, we discuss the addition of regularization penalties such that both robustness and block sparsity are imposed in the parameter estimates. An empirical approach to estimate the uncertainty magnitude is presented through the use of quantiles. We compare the results of ℓ1-Logistic Regression against ℓ1-Robust Logistic Regression on a real gene expression data set and achieve reductions in the worstcase false alarm rate and probability of error by 10 % − 20%, thus illustrating the value added of using robust classifiers in risk sensitive domains when confronted with uncertain measurements.

If I can attain half of the success you have achieved in marriage and in life, I will have lived a full and purposeful life. A son could not ask for better parents. ii ACKNOWLEDGEMENTS I am extremely grateful to have been advised by a brilliantly creative human being. Professor Alfred Hero has allowed me to mature as an independent researcher capable of abstractly analyzing complex problems. Between day to day interactions, coursework, and discussions about research, I am forever grateful for the interactions I have had with my committee members Professors Burns, Burmeister, Shedden, and Zhu. I will always appreciate the hands on interaction and development of ideas with my post-doctoral researchers Mark Kliger and Ami Wiesel. I am so appreciative of the time and effort you both spent with me, especially early on in my graduate student career. I cannot thank Arvind Rao enough for his wisdom and insight into developing good research topics and sharing a few good laughs on our road trip to Madison, WI. I know he will be a very successful faculty member some day. For both academic collaboration and extracurricular mischief, I will never forget the moments spent with fellow graduate students and now life long friends Yongsheng Huang and Arnau Tibau Puig. Most importantly, I thank my beautiful wife and best friend Erica for her love and patience over these past four years while pursuing my Ph.D. iii TABLE OF CONTENTS