Citation Context

...dle large problems and can also deal efficiently with sparse features. In comparative timings we find that the new algorithms are considerably faster than competing methods. 1 Introduction The lasso [=-=Tibshirani, 1996-=-] is a popular method for regression that uses an ℓ1 penalty to achieve a sparse solution. In the signal processing literature, the lasso is also known as basis pursuit [Chen et al., 1998]. This idea ...

Citation Context

...on The lasso [Tibshirani, 1996] is a popular method for regression that uses an ℓ1 penalty to achieve a sparse solution. In the signal processing literature, the lasso is also known as basis pursuit [=-=Chen et al., 1998-=-]. This idea has been broadly applied, for example to generalized linear models [Tibshirani, 1996] and Cox’s proportional hazard models for survival data [Tibshirani, 1997]. In recent years, there has...

Citation Context

...e sparsity of the solution to (1) (i.e. the number of coefficients equal to zero) increases monotonically from 0 to the sparsity of the lasso solution. Figure 1 shows an example. The dataset is from [=-=Golub et al., 1999-=-b], consisting of 72 observations on 3571 genes measured with DNA microarrays. The observations fall in two classes: we treat this as a regression problem for illustration. The coefficient profiles fr...

Citation Context

.... If ˜βj > 0, then ∂R ∂βj | β= ˜ β = − 1 N N∑ i=1 xij(yi − ˜ βo − x T i ˜ β) + λ(1 − α)βj + λα. (4) A similar expression exists if ˜ βj < 0, and ˜ βj = 0 is treated separately. Simple calculus shows [=-=Donoho and Johnstone, 1994-=-] that the coordinate4● ● ● ● ● ● ● ● ● ● 2 4 6 8 10 −0.02 0.00 0.02 0.04 0.06 Step Coefficients ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ...

Citation Context

...roportional hazard models for survival data (Tibshirani 1997). In recent years, there has been an enormous amount of research activity devoted to related regularization methods: 1. The grouped lasso (=-=Yuan and Lin 2007-=-; Meier et al. 2008), where variables are included or excluded in groups. 2. The Dantzig selector (Candes and Tao 2007, and discussion), a slightly modified version of the lasso. 3. The elastic net (Z...

Citation Context

...7; Meier et al. 2008), where variables are included or excluded in groups. 2. The Dantzig selector (Candes and Tao 2007, and discussion), a slightly modified version of the lasso. 3. The elastic net (=-=Zou and Hastie 2005-=-) for correlated variables, which uses a penalty that is part ℓ1, part ℓ2.2 Regularization Paths for GLMs via Coordinate Descent 4. ℓ1 regularization paths for generalized linear models (Park and Has...

Citation Context

...part ℓ2.2 Regularization Paths for GLMs via Coordinate Descent 4. ℓ1 regularization paths for generalized linear models (Park and Hastie 2007a). 5. Methods using non-concave penalties, such as SCAD (=-=Fan and Li 2005-=-) and Friedman’s generalized elastic net (Friedman 2008), enforce more severe variable selection than the lasso. 6. Regularization paths for the support-vector machine (Hastie et al. 2004). 7. The gra...

Citation Context

...nalized, and always enters the model unrestricted at the first step and remains in the model. Penalty rescaling would also allow, for example, our software to be used to implement the adaptive lasso (=-=Zou 2006-=-). Considerable speedup is obtained by organizing the iterations around the active set of features— those with nonzero coefficients. After a complete cycle through all the variables, we iterate on onl...

Citation Context

...ment classification problem with mostly binary features. The response is binary, and indicates whether the document is an advertisement. Only 1.2% nonzero values in the predictor matrix. • Newsgroup [=-=Lang, 1995-=-]: document classification problem. We used the training set cultured from these data by Koh et al. [2007]. The response is binary, and indicates a subclass of topics; the predictors are binary, and i...

Citation Context

...t (Friedman et al. 2009) implemented in the R programming system (R Development Core Team 2009). We do not revisit the wellestablished convergence properties of coordinate descent in convex problems (=-=Tseng 2001-=-) in this article. Lasso procedures are frequently used in domains with very large datasets, such as genomics and web analysis. Consequently a focus of our research has been algorithmic efficiency and...

Citation Context

..., such as SCAD (Fan and Li 2005) and Friedman’s generalized elastic net (Friedman 2008), enforce more severe variable selection than the lasso. 6. Regularization paths for the support-vector machine (=-=Hastie et al. 2004-=-). 7. The graphical lasso (Friedman et al. 2008) for sparse covariance estimation and undirected graphs. Efron et al. (2004) developed an efficient algorithm for computing the entire regularization pa...

Citation Context

...s outcome z with Pr(z = 1) = p, Pr(z = 0) = 1 − p. We compared the speed of glmnet to the interior point method l1logreg (Koh et al. 2007b,a), Bayesian binary regression (BBR, Madigan and Lewis 2007; =-=Genkin et al. 2007-=-), and the lasso penalized logistic program LPL supplied by Ken Lange (see Wu and Lange 2008). The latter two methods also use a coordinate descent approach. The BBR software automatically performs te...

Citation Context

...e number of coefficients equal to zero) increases monotonically from 0 to the sparsity of the lasso solution. Figure 1 shows an example that demonstrates the effect of varying α. The dataset is from (=-=Golub et al. 1999-=-), consisting of 72 observations on 3571 genes measured with DNA microarrays. The observations fall in two classes, so we use the penalties in conjunction with the 1 Zou and Hastie (2005) called this ...

Citation Context

...email: hastie@stanford.edu. Sequoia Hall, Stanford University, CA94305. 12. The Dantzig selector [Candes and Tao, 2007, and discussion], a slightly modified version of the lasso; 3. The elastic net [=-=Zhou and Hastie, 2005-=-] for correlated variables, which uses a penalty that is part ℓ1, part ℓ2; 4. ℓ1 regularization paths for generalized linear models [Park and Hastie, 2006]; 5. Regularization paths for the support-vec...

Citation Context

...fied version of the lasso; 3. The elastic net [Zou and Hastie, 2005] for correlated variables, which uses a penalty that is part ℓ1, part ℓ2; 4. ℓ1 regularization paths for generalized linear models [=-=Park and Hastie, 2007-=-]; 5. Regularization paths for the support-vector machine [Hastie et al., 2004]. 6. The graphical lasso [Friedman et al., 2008] for sparse covariance estimation and undirected graphs Efron et al. [200...

Citation Context

... also known as basis pursuit (Chen et al. 1998). This idea has been broadly applied, for example to generalized linear models (Tibshirani 1996) and Cox’s proportional hazard models for survival data (=-=Tibshirani 1997-=-). In recent years, there has been an enormous amount of research activity devoted to related regularization methods: 1. The grouped lasso (Yuan and Lin 2007; Meier et al. 2008), where variables are i...

Citation Context

... cancer classes. • Leukemia [Golub et al., 1999a]: gene-expression data with a binary response indicating type of leukemia—AML vs ALL. We used the preprocessed data of Dettling [2004]. • Internet-Ad [=-=Kushmerick, 1999-=-]: document classification problem with mostly binary features. The response is binary, and indicates whether the document is an advertisement. Only 1.2% nonzero values in the predictor matrix. • News...

Citation Context

...ied version of the lasso; 3. The elastic net [Zhou and Hastie, 2005] for correlated variables, which uses a penalty that is part ℓ1, part ℓ2; 4. ℓ1 regularization paths for generalized linear models [=-=Park and Hastie, 2006-=-]; 5. Regularization paths for the support-vector machine [Hastie et al., 2004]. 6. The graphical lasso [Friedman et al., 2007b] for sparse covariance estimation and undirected graphs Efron et al. [20...

Citation Context

...escent 4. ℓ1 regularization paths for generalized linear models (Park and Hastie 2007a). 5. Methods using non-concave penalties, such as SCAD (Fan and Li 2005) and Friedman’s generalized elastic net (=-=Friedman 2008-=-), enforce more severe variable selection than the lasso. 6. Regularization paths for the support-vector machine (Hastie et al. 2004). 7. The graphical lasso (Friedman et al. 2008) for sparse covarian...

Citation Context

...erior point method l1logreg (Koh et al. 2007b,a), Bayesian binary regression (BBR, Madigan and Lewis 2007; Genkin et al. 2007), and the lasso penalized logistic program LPL supplied by Ken Lange (see =-=Wu and Lange 2008-=-). The latter two methods also use a coordinate descent approach. The BBR software automatically performs ten-fold cross-validation when given a set of λ values. Hence we report the total time for ten...

Citation Context

... “one-standard-error” rule. The top of each plot is annotated with the size of the models.18 Regularization Paths for GLMs via Coordinate Descent Alternatively, they can use K-fold cross-validation (=-=Hastie et al. 2009-=-, for example), where the training data is used both for training and testing in an unbiased way. Figure 2 illustrates cross-validation on a simulated dataset. For logistic regression, we sometimes us...

Citation Context

...not much software available for elastic net. Comparisons of our glmnet code with the R package elasticnet will mimic the comparisons with lars (Hastie and Efron 2007) for the lasso, since elasticnet (=-=Zou and Hastie 2004-=-) is built on the lars package. 5.1. Regression with the lasso We generated Gaussian data with N observations and p predictors, with each pair of predictors Xj, Xj ′ having the same population correla...

Citation Context

...o-column matrix of counts, sometimes referred to as grouped data. We discuss this in more detail in Section 4.2. • The Newton algorithm is not guaranteed to converge without stepsize optimization [in =-=Lee et al., 2006-=-]. Our code does not implement any checks for divergence; this would slow it down, and when used as recommended we do not feel it is necessary.. We have a closed form expression for the starting solut...

Citation Context

...able under general public licence (GPL-2) from the Comprehensive R Archive Network at http://CRAN.R-project.org/package=glmnet. Sparse data inputs are handled by the Matrix package. MATLAB functions (=-=Jiang 2009-=-) are available from http://www-stat.stanford.edu/~tibs/glmnet-matlab/. Acknowledgments We would like to thank Holger Hoefling for helpful discussions, and Hui Jiang for writing the MATLAB interface t...

Citation Context

...s to generate a two-class outcome z with Pr(z = 1) = p, Pr(z = 0) = 1 − p. We compared the speed of glmnet to the interior point method l1logreg (Koh et al. 2007b,a), Bayesian binary regression (BBR, =-=Madigan and Lewis 2007-=-; Genkin et al. 2007), and the lasso penalized logistic program LPL supplied by Ken Lange (see Wu and Lange 2008). The latter two methods also use a coordinate descent approach. The BBR software autom...