The sigmoid kernel was quite popular for support vector machines due to its origin from neural networks. However, as the kernel matrix may not be positive semidefinite (PSD), it is not widely used and the behavior is unknown. In this paper, we analyze such non-PSD kernels through the point of view of separability. Based on the investigation of parameters in different ranges, we show that for some parameters, the kernel matrix is conditionally positive definite (CPD), a property which explains its practical viability. Experiments are given to illustrate our analysis. Finally, we discuss how to solve the non-convex dual problems by SMO-type decomposition methods. Suitable modifications for any symmetric non-PSD kernel matrices are proposed with convergence proofs.

In this paper, we show that decomposition methods with alpha seeding are extremely useful for solving a sequence of linear SVMs with more data than attributes. This strategy is motivated from (Keerthi and Lin 2003) which proved that for an SVM with data not linearly separable, after C is large enough, the dual solutions are at the same face. We explain why a direct use of decomposition methods for linear SVMs is sometimes very slow and then analyze why alpha seeding is much more effective for linear than nonlinear SVMs. We also conduct comparisons with other methods which are efficient for linear SVMs, and demonstrate the effectiveness of alpha seeding techniques for helping the model selection.