this paper. Let us place it within the neural network perspective, and particularly that of learning. The area of neural networks has greatly benefited from its unique position at the crossroads of several diverse scientific and engineering disciplines including statistics and probability theory, physics, biology, control and signal processing, information theory, complexity theory, and psychology (see [45]). Neural networks have provided a fertile soil for the infusion (and occasionally confusion) of ideas, as well as a meeting ground for comparing viewpoints, sharing tools, and renovating approaches. It is within the ill-defined boundaries of the field of neural networks that researchers in traditionally distant fields have come to the realization that they have been attacking fundamentally similar optimization problems.

Variable neighborhood search (VNS) is a recent metaheuristic for solving combinatorial and global optimization problems whose basic idea is systematic change of neighborhood within a local search. In this survey paper we present basic rules of VNS and some of its extensions. Moreover, applications are briefly summarized. They comprise heuristic solution of a variety of optimization problems, ways to accelerate exact algorithms and to analyze heuristic solution processes, as well as computer-assisted discovery of conjectures in graph theory.

An algorithm is presented for the fuzzy segmentation of two-dimensional (2-D) and three-dimensional (3-D) multispectral magnetic resonance (MR) images that have been corrupted by intensity inhomogeneities, also known as shading artifacts. The algorithm is an extension of the 2-D adaptive fuzzy C-means algorithm (2-D AFCM) presented in previous work by the authors. This algorithm models the intensity inhomogeneities as a gain field that causes image intensities to smoothly and slowly vary through the image space. It iteratively adapts to the intensity inhomogeneities and is completely automated. In this paper, we fully generalize 2-D AFCM to three-dimensional (3-D) multispectral images. Because of the potential size of 3-D image data, we also describe a new faster multigrid-based algorithm for its implementation. We show, using simulated MR data, that 3-D AFCM yields lower error rates than both the standard fuzzy C-means (FCM) algorithm and two other competing methods, when segmenting corrupted images. Its efficacy is further demonstrated using real 3-D scalar and multispectral MR brain images.

We introduce a method for validation of results obtained by clustering analysis of data. The method is based on resampling the available data. A figure of merit that measures the stability of clustering solutions against resampling is introduced. Clusters which are stable against resampling give rise to local maxima of this figure of merit. This is presented first for a one-dimensional data set, for which an analytic approximation for the figure of merit is derived and compared with numerical measurements. Next, the applicability of the method is demonstrated for higher dimensional data, including gene microarray expression data.

Vector quantization is a data compression method where a set of data points is encoded by a reduced set of reference vectors, the codebook. We discuss a vector quantization strategy which jointly optimizes distortion errors and the codebook complexity, thereby, determining the size of the codebook. A maximum entropy estimation of the cost function yields an optimal number of reference vectors, their positions and their assignment probabilities. The dependence of the codebook density on the data density for different complexity functions is investigated in the limit of asymptotic quantization levels. How different complexity measures influence the efficiency of vector quantizers is studied for the task of image compression, i.e., we quantize the wavelet coefficients of gray level images and measure the reconstruction error. Our approach establishes a unifying framework for different quantization methods like K-means clustering and its fuzzy version, entropy constrained vector quantizati...

Abstract—In this article, we evaluate the performance of three clustering algorithms, hard K-Means, single linkage, and a simulated annealing (SA) based technique, in conjunction with four cluster validity indices, namely Davies-Bouldin index, Dunn’s index, Calinski-Harabasz index, and a recently developed index I. Based on a relation between the index I and the Dunn’s index, a lower bound of the value of the former is theoretically estimated in order to get unique hard K-partition when the data set has distinct substructures. The effectiveness of the different validity indices and clustering methods in automatically evolving the appropriate number of clusters is demonstrated experimentally for both artificial and real-life data sets with the number of clusters varying from two to ten. Once the appropriate number of clusters is determined, the SA-based clustering technique is used for proper partitioning of the data into the said number of clusters.

Abstract-Our deterministic annealing approach to clustering is derived on the basis of the principle of maximum entropy, is independent of the initial state, and produces natural hier-archical clustering solutions by going through a sequence of phase transitions. This approach is modified here for a larger class of optimization problems by adding constraints to the free energy. The concept of constrained clustering is explained, and then, three examples are given in which it is used as means to introduce deterministic annealing. First, the previous clustering method is improved by adding cluster mass variables and a total mass constraint. Second, the traveling salesman problem (TSP) is reformulated as constrained clustering, yielding the elastic net (EN) approach to the problem. More insight is gained by identifying a second Lagrange multiplier that is related to the tour length add can also be used to control the annealing process. Finally, the “open path ” constraint formulation is shown to relate to dimensionality reduction by self-organization in unsupervised learning. A similar annealing procedure is applicable in this case as well. Index Terms-Annealing, clustering, maximum entropy, neural networks, nonconvex optimization, self-organization.

In this paper, a systematic optimization approach for clustering proximity or similarity data is developed. Starting from fundamental invariance and robustness properties, a set of axioms is proposed and discussed to distinguish different cluster compactness and separation criteria. The approach covers the case of sparse proximity matrices, and is extended to nested partitionings for hierarchical data clustering. To solve the associated optimization problems, a rigorous mathematical framework for deterministic annealing and mean--field approximation is presented. Efficient optimization heuristics are derived in a canonical way, which also clarifies the relation to stochastic optimization by Gibbs sampling. Similarity-based clustering techniques have a broad range of possible applications in computer vision, pattern recognition, and data analysis. As a major practical application we present a novel approach to the problem of unsupervised texture segmentation, which relies on statistical...

We present a novel algorithm for obtaining fuzzy segmentations of images that are subject to multiplicative intensity inhomogeneities, such as magnetic resonance images. The algorithm is formulated by modifying the objective function in the fuzzy C-means algorithm to include a multiplier field, which allows the centroids for each class to vary across the image. First and second order regularization terms ensure that the multiplier field is both slowly varying and smooth. An iterative algorithm that minimizes the objective function is described, and its efficacy is demonstrated on several test images. Key words: image segmentation, fuzzy c-means, intensity inhomogeneities, magnetic resonance imaging 1 Introduction Image segmentation plays an important role in a variety of applications such as robot vision, object recognition, and medical imaging. There has been considerable interest recently in the use of fuzzy segmentation methods, which retain more information from the original im...

This paper addresses the estimation of fuzzy Gaussian distribution mixture with applications to unsupervised statistical fuzzy image segmentation. In a general way, the fuzzy approach enriches the current statistical models by adding a fuzzy class, which has several interpretations in signal processing. One such interpretation in image segmentation is the simultaneous appearance of several thematic classes on the same site. We introduce a new procedure for estimating of fuzzy mixtures, which is an adaptation of the iterative conditional estimation (ICE) algorithm to the fuzzy framework. We first describe the blind estimation, i.e., without taking into account any spatial information, valid in any context of independent noisy observations. Then we introduce, in a manner analogous to classical hard segmentation, the spatial information by two different approaches: contextual segmentation and adaptive blind segmentation. In the first case, the spatial information is taken into account at the segmentation step level, and in the second case it is taken into account at the parameter estimation step level. The results obtained with the iterative conditional estimation algorithm are compared to those obtained with expectationmaximization (EM) and the stochastic EM (SEM) algorithms, on both parameter estimation and unsupervised segmentation levels, via simulations. The methods proposed appear as complementary to the fuzzy C-means algorithms.