A class of estimators of the Rényi and Tsallis entropies of an unknown distribution f in R m is presented. These estimators are based on the kth nearest-neighbor distances computed from a sample of N i.i.d. vectors with distribution f. We show that entropies of any order q, including Shannon’s entropy, can be estimated consistently with minimal assumptions on f.Moreover, we show that it is straightforward to extend the nearest-neighbor method to estimate the statistical distance between two distributions using one i.i.d. sample from each.

We present simple and computationally efficient nonparametric estimators of Rényi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over R d. The estimators are calculated as the sum of p-th powers of the Euclidean lengths of the edges of the ‘generalized nearest-neighbor ’ graph of the sample and the empirical copula of the sample respectively. For the first time, we prove the almost sure consistency of these estimators and upper bounds on their rates of convergence, the latter of which under the assumption that the density underlying the sample is Lipschitz continuous. Experiments demonstrate their usefulness in independent subspace analysis. 1

soft geometric constraint with radiometry

Abstract not found

Recently in [1], we proposed a way to bound how well any given image can be denoised. The bound was computed directly from the noise-free image that was assumed to be available. In this work we extend the formulation to the more practical case where no ground truth is available. We show that the parameters of the bounds, namely the cluster covariances and level of redundancy for patches in the image, can be estimated directly from the noise corrupted image. Further, we analyze the bounds formulation to show that these two parameters are interdependent and they, along with the bounds formulation as a whole, have a nice information-theoretic interpretation as well. The results are verified through a variety of well-motivated experiments.

We focus on motion estimation using a block matching approach and suggest using a minimum-entropy criterion. Many entropy-based estimation procedures exist, such as plug-in estimators based on Parzen windowing. We consider here an alternative that is applicable to data of any dimension and that circumvents the critical issues raised by kernel-based methods. To the best of our knowledge, this criterion has not yet been considered for image processing problems. The inherent robustness property of entropy is expected to provide a robust and efficient estimation of the motion vector of a block of a video sequence. In particular, the minimum-entropy estimator should be robust to occlusions and variations of luminance, for which standard approaches like SSD usually meet their limitations. Index Terms — image matching, minimum entropy methods, motion compensation, adaptive estimation, image processing, robustness 1.

In this paper we propose a novel clustering algorithm based on maximizing the mutual information between data points and clusters. Unlike previous methods, we neither assume the data are given in terms of distributions nor impose any parametric model on the within-cluster distribution. Instead, we utilize a non-parametric estimation of the average cluster entropies and search for a clustering that maximizes the estimated mutual information between data points and clusters. The improved performance of the proposed algorithm is demonstrated on several standard datasets. 1.

We propose new nonparametric, consistent Rényi-α and Tsallis-α divergence estimators for continuous distributions. Given two independent and identically distributed samples, a “naïve ” approach would be to simply estimate the underlying densities and plug the estimated densities into the corresponding formulas. Our proposed estimators, in contrast, avoid density estimation completely, estimating the divergences directly using only simple k-nearest-neighbor statistics. We are nonetheless able to prove that the estimators are consistent under certain conditions. We also describe how to apply these estimators to mutual information and demonstrate their efficiency via numerical experiments. 1

Fig. 1. Illustration of the kNN search problem in R 2 with k = 3 using the Euclidean distance. The k-nearest neighbor (kNN) search problem is widely used in domains and applications such as classification, statistics, and biology. In this paper, we propose two fast GPU-based implementations of the brute-force kNN search algorithm using the CUDA and CUBLAS APIs. We show that our CUDA and CUBLAS implementations are up to, respectively, 64X and 189X faster on synthetic data than the highly optimized ANN C++ library, and up to, respectively, 25X and 62X faster on high-dimensional SIFT matching. Index Terms — k-nearest neighbors, GPU, CUDA/CUBLAS, SIFT

A new universal estimator of divergence is presented for multidimensional continuous densities based on k-nearest-neighbor (k-NN) distances. Assuming independent and identically distributed (i.i.d.) samples, the new estimator is proved to be asymptotically unbiased and mean-square consistent. In experiments with high-dimensional data, the k-NN approach generally exhibits faster convergence than previous algorithms. It is also shown that the speed of convergence of the k-NN method can be further improved by an adaptive choice of k.