Because reinforcement learning suffers from a lack of scalability, online value (and Q-) function approximation has received increasing interest this last decade. This contribution introduces a novel approximation scheme, namely the Kalman Temporal Differences (KTD) framework, that exhibits the following features: sample-efficiency, non-linear approximation, non-stationarity handling and uncertainty management. A first KTD-based algorithm is provided for deterministic Markov Decision Processes (MDP) which produces biased estimates in the case of stochastic transitions. Than the eXtended KTD framework (XKTD), solving stochastic MDP, is described. Convergence is analyzed for special cases for both deterministic and stochastic transitions. Related algorithms are experimented on classical benchmarks. They compare favorably to the state of the art while exhibiting the announced features. 1.

Geometric active contours represented as the zero level sets of the graph of a surface have been used very successfully to segment static images. However, tracking involves estimating the global motion of the object and its local deformations as functions of time. Some attempts have been made to use geometric active contours for tracking, but most of these minimize the energy at each frame and do not utilize the temporal coherency of the motion or the deformation. Recently, particle filters for geometric active contours were used for tracking deforming objects. However, the method is computationally very expensive since it requires a large number of particles to approximate the state density. In the present work, we propose to use the unscented Kalman filter together with geometric active contours to track deformable objects in a computationally efficient manner.

Model compensation is a standard way of improving speech recognisers’ robustness to noise. Currently popular schemes are based on vector Taylor series (VTS) compensation. They often use the continuous time approximation to compensate dynamic parameters. In this paper, the accuracy of dynamic parameter compensation is improved by representing the dynamic features as a linear transformation of a window of static features. A modified version of VTS compensation is applied to the distribution of the window of static features and, importantly, their correlations. These compensated distributions are then transformed to standard static and dynamic distributions. The proposed scheme outperformed the standard VTS scheme by about 10 % relative. Index Terms — Speech recognition, acoustic noise, robustness 1.

This article considers the application of the unscented transform to optimal smoothing of non-linear state space models. In this article, a new Rauch-Tung-Striebel type form of the fixed-interval unscented Kalman smoother is derived. The new smoother differs from the previously proposed twofilter formulation based unscented Kalman smoother in the sense that it is not based on running two independent filters forward and backward in time. Instead, a separate backward smoothing pass is used, which recursively computes corrections to the forward filtering result. The smoother equations are derived as approximations to the formal Bayesian optimal smoothing equations. The performance of the new smoother is demonstrated with a simulation.

Abstract: The basic nonlinear ltering problem for dynamical systems is considered. Approximating the optimal lter estimate by particle lter methods has become perhaps the most common and useful method in recent years. Many variants of particle lters have been suggested, and there is an extensive literature on the theoretical aspects of the quality of the approximation. Still, a clear cut result that the approximate solution, for unbounded functions, converges to the true optimal estimate as the number of particles tends to in nity seems to be lacking. It is the purpose of this contribution to give such a basic convergence result.

This article considers the application of the unscented Kalman filter (UKF) to continuous-time filtering problems, where both the state and measurement processes are modeled as stochastic differential equations. The mean and covariance differential equations which result in the continuous-time limit of the UKF are derived. The continuous-discrete unscented Kalman filter is derived as a special case of the continuous-time filter, when the continuous-time prediction equations are combined with the update step of the discrete-time unscented Kalman filter. The filter equations are also transformed into sigma-point differential equations, which can be interpreted as matrix square root versions of the filter equations.

Visual and inertial sensors, in combination, are able to provide accurate motion estimates and are well-suited for use in many robot navigation tasks. However, correct data fusion, and hence overall performance, depends on careful calibration of the rigid body transform between the sensors. Obtaining this calibration information is typically difficult and time-consuming, and normally requires additional equipment. In this paper we describe an algorithm, based on the unscented Kalman filter, for self-calibration of the transform between a camera and an inertial measurement unit (IMU). Our formulation rests on a differential geometric analysis of the observability of the camera-IMU system; this analysis shows that the sensor-to-sensor transform, the IMU gyroscope and accelerometer biases, the local gravity vector, and the metric scene structure can be recovered from camera and IMU measurements alone. While calibrating the transform we simultaneously localize the IMU and build a map of the surroundings – all without additional hardware or prior knowledge about the environment in which a robot is To Appear in International Journal of Robotics Research operating. We present results from simulation studies and from experiments with a monocular camera and a low-cost IMU, which demonstrate accurate estimation of both the calibration parameters and the local scene structure. 1 1

Abstract — In nonlinear Bayesian estimation it is generally inevitable to incorporate approximate descriptions of the exact estimation algorithm. There are two possible ways to involve approximations: Approximating the nonlinear stochastic system model or approximating the prior probability density function. The key idea of the introduced novel estimator called Hybrid Density Filter relies on approximating the nonlinear system, thus approximating conditional densities. These densities nonlinearly relate the current system state to the future system state at predictions or to potential measurements at measurement updates. A hybrid density consisting of both Dirac delta functions and Gaussian densities is used for an optimal approximation. This paper addresses the optimization problem for treating the conditional density approximation. Furthermore, efficient estimation algorithms are derived based upon the special structure of the hybrid density, which yield a Gaussian mixture representation of the system state’s density.

The dilemma between exploration and exploitation is an important topic in reinforcement learning (RL). Most successful approaches in addressing this problem tend to use some uncertainty information about values estimated during learning. On another hand, scalability is known as being a lack of RL algorithms and value function approximation has become a major topic of research. Both problems arise in realworld applications, however few approaches allow approximating the value function while maintaining uncertainty information about estimates. Even fewer use this information in the purpose of addressing the exploration/exploitation dilemma. In this paper, we show how such an uncertainty information can be derived from a Kalman-based Temporal Differences (KTD) framework. An active learning scheme for a second-order value-iteration-like algorithm (named KTD-Q) is proposed. We also suggest adaptations of several existing exploration/exploitation dilemma schemes. This is a first step towards global handling of continuous state and action spaces and exploration/exploitation dilemma.

Abstract. We describe a technique to simultaneously estimate a local neural fiber model and trace out its path. Existing techniques estimate the local fiber orientation at each voxel independently so there is no running knowledge of confidence in the estimated fiber model. We formulate fiber tracking as recursive estimation: at each step of tracing the fiber, the current estimate is guided by the previous. To do this we model the signal as a mixture of Gaussian tensors and perform tractography within a filter framework. Starting from a seed point, each fiber is traced to its termination using an unscented Kalman filter to simultaneously fit the local model and propagate in the most consistent direction. Despite the presence of noise and uncertainty, this provides a causal estimate of the local structure at each point along the fiber. Synthetic experiments demonstrate that this approach reduces signal reconstruction error and significantly improves the angular resolution at crossings and branchings. In vivo experiments confirm the ability to trace out fibers in areas known to contain such crossing and branching while providing inherent path regularization. 1