Abstract not found

ISBN 978-951-22-8929-5 (printed version)

Abstract: We show that with a class of penalty functions, numerical problems associated with the implementation of the penalized least square estimators are equivalent to the exact cover by 3-sets problem, which belongs to a class of NP-hard problems. We then extend this NP-hardness result to the cases of penalized least absolute deviation regression and penalized support vector machines. We discuss the practical implication of our results. In particular, we emphasize that the oracle property of a penalized likelihood estimator requires a local extremum, instead of a global extremum. Hence the penalized likelihood estimators are still favorable; however one should not attempt to find its global extremum(a)!

Since the penalized likelihood function of the smoothly clipped absolute deviation (SCAD) penalty is highly non-linear and has many local optima, finding a local solution to achieve the so-called oracle property is an open problem. We propose an iterative algorithm, called the OEM algorithm, to fill this gap. The development of the algorithm draws direct impetus from a missing-data problem arising in design of experiments with an orthogonal complete matrix. In each iteration, the algorithm imputes the missing data based on the current estimates of the parameters and updates a closed-form solution associated with the complete data. By introducing a procedure called active orthogonization, we make the algorithm broadly applicable to problems with arbitrary regression matrices. In addition to the SCAD penalty, the proposed algorithm works for other penalties like the MCP, lasso and nonnegative garrote. Convergence and convergence rate of the algorithm are examined. The algorithm has several unique theoretical properties. For the SCAD and MCP penalties, an OEM sequence can achieve the oracle property after sufficient iterations. For various penalties, an OEM sequence converges to a point having grouping coherence for fully aliased regression matrices. For computing the ordinary least squares estimator with a singular regression matrix, an OEM sequence converges to the Moore-Penrose generalized inverse-based least squares estimator.