Results 1  10
of
12
Solving factored MDPs with hybrid state and action variables
 J. Artif. Intell. Res. (JAIR
"... Efficient representations and solutions for large decision problems with continuous and discrete variables are among the most important challenges faced by the designers of automated decision support systems. In this paper, we describe a novel hybrid factored Markov decision process (MDP) model tha ..."
Abstract

Cited by 27 (4 self)
 Add to MetaCart
(Show Context)
Efficient representations and solutions for large decision problems with continuous and discrete variables are among the most important challenges faced by the designers of automated decision support systems. In this paper, we describe a novel hybrid factored Markov decision process (MDP) model that allows for a compact representation of these problems, and a new hybrid approximate linear programming (HALP) framework that permits their efficient solutions. The central idea of HALP is to approximate the optimal value function by a linear combination of basis functions and optimize its weights by linear programming. We analyze both theoretical and computational aspects of this approach, and demonstrate its scaleup potential on several hybrid optimization problems. 1.
Provably Efficient Learning with Typed Parametric Models
"... To quickly achieve good performance, reinforcementlearning algorithms for acting in large continuousvalued domains must use a representation that is both sufficiently powerful to capture important domain characteristics, and yet simultaneously allows generalization, or sharing, among experiences. ..."
Abstract

Cited by 12 (3 self)
 Add to MetaCart
(Show Context)
To quickly achieve good performance, reinforcementlearning algorithms for acting in large continuousvalued domains must use a representation that is both sufficiently powerful to capture important domain characteristics, and yet simultaneously allows generalization, or sharing, among experiences. Our algorithm balances this tradeoff by using a stochastic, switching, parametric dynamics representation. We argue that this model characterizes a number of significant, realworld domains, such as robot navigation across varying terrain. We prove that this representational assumption allows our algorithm to be probably approximately correct with a sample complexity that scales polynomially with all problemspecific quantities including the statespace dimension. We also explicitly incorporate the error introduced by approximate planning in our sample complexity bounds, in contrast to prior Probably Approximately Correct (PAC) Markov Decision Processes (MDP) approaches, which typically assume the estimated MDP can be solved exactly. Our experimental results on constructing plans for driving to work using real car trajectory data, as well as a small robot experiment on navigating varying terrain, demonstrate that our dynamics representation enables us to capture realworld dynamics in a sufficient manner to produce good performance.
Learning Basis Functions in Hybrid Domains
 In Proceedings of the 2006 National Conference on Artificial Intelligence (AAAI
, 2006
"... Markov decision processes (MDPs) with discrete and continuous state and action components can be solved efficiently by hybrid approximate linear programming (HALP). The main idea of the approach is to approximate the optimal value function by a set of basis functions and optimize their weights by li ..."
Abstract

Cited by 5 (2 self)
 Add to MetaCart
(Show Context)
Markov decision processes (MDPs) with discrete and continuous state and action components can be solved efficiently by hybrid approximate linear programming (HALP). The main idea of the approach is to approximate the optimal value function by a set of basis functions and optimize their weights by linear programming. The quality of this approximation naturally depends on its basis functions. However, basis functions leading to good approximations are rarely known in advance. In this paper, we propose a new approach that discovers these functions automatically. The method relies on a class of parametric basis function models, which are optimized using the dual formulation of a relaxed HALP. We demonstrate the performance of our method on two hybrid optimization problems and compare it to manually selected basis functions.
Clinical time series prediction with a hierarchical dynamical system
 In Artificial Intelligence in Medicine
, 2013
"... Abstract. In this work we develop and test a novel hierarchical framework for modeling and learning multivariate clinical time series data. Our framework combines two modeling approaches: Linear Dynamical Systems (LDS) and Gaussian Processes (GP), and is capable to model and work with time series o ..."
Abstract

Cited by 4 (3 self)
 Add to MetaCart
(Show Context)
Abstract. In this work we develop and test a novel hierarchical framework for modeling and learning multivariate clinical time series data. Our framework combines two modeling approaches: Linear Dynamical Systems (LDS) and Gaussian Processes (GP), and is capable to model and work with time series of varied length and with irregularly sampled observations. We test our framework on the problem of learning clinical time series data from the complete blood count panel, and show that our framework outperforms alternative time series models in terms of its predictive accuracy.
Modeling Clinical Time Series Using Gaussian Process Sequences
"... Development of accurate models of complex clinical time series data is critical for understanding the disease, its dynamics, and subsequently patient management and clinical decision making. Clinical time series differ from other time series applications mainly in that observations are often missing ..."
Abstract

Cited by 3 (2 self)
 Add to MetaCart
(Show Context)
Development of accurate models of complex clinical time series data is critical for understanding the disease, its dynamics, and subsequently patient management and clinical decision making. Clinical time series differ from other time series applications mainly in that observations are often missing and made at irregular time intervals. In this work, we propose and test a new probabilistic approach for modeling clinical time series data that is optimized to handle irregularly sampled observations. Our model is defined by a sequence of Gaussian processes (GPs), each restricted to a window of a finite size, where dependencies among two consecutive Gaussian processes are represented using a linear dynamical system. We develop algorithms supporting both model learning and inference. Experiments on realworld clinical time series data show that our model is better for modeling clinical time series and that it outperforms or is close to alternative time series prediction models. 1
Planning in Hybrid Structured Stochastic Domains
, 2006
"... Efficient representations and solutions for large structured decision problems with continuous and discrete variables are among the important challenges faced by the designers of automated decision support systems. In this work, we describe a novel hybrid factored Markov decision process (MDP) mod ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
Efficient representations and solutions for large structured decision problems with continuous and discrete variables are among the important challenges faced by the designers of automated decision support systems. In this work, we describe a novel hybrid factored Markov decision process (MDP) model that allows for a compact representation of these problems, and a hybrid approximate linear programming (HALP) framework that permits their efficient solutions. The central idea of HALP is to approximate the optimal value function of an MDP by a linear combination of basis functions and optimize its weights by linear programming. We study both theoretical and practical aspects of this approach, and demonstrate its scaleup potential on several hybrid optimization problems
Sparse Linear Dynamical System with Its Application in Multivariate Clinical Time Series
"... Linear Dynamical System (LDS) is an elegant mathematical framework for modeling and learning multivariate time series. However, in general, it is difficult to set the dimension of its hidden state space. A small number of hidden states may not be able to model the complexities of a time series, whi ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
Linear Dynamical System (LDS) is an elegant mathematical framework for modeling and learning multivariate time series. However, in general, it is difficult to set the dimension of its hidden state space. A small number of hidden states may not be able to model the complexities of a time series, while a large number of hidden states can lead to overfitting. In this paper, we study methods that impose an `1 regularization on the transition matrix of an LDS model to alleviate the problem of choosing the optimal number of hidden states. We incorporate a generalized gradient descent method into the Maximum a Posteriori (MAP) framework and use Expectation Maximization (EM) to iteratively achieve sparsity on the transition matrix of an LDS model. We show that our Sparse Linear Dynamical System (SLDS) improves the predictive performance when compared to ordinary LDS on a multivariate clinical time series dataset. 1
On the Smoothness of Linear Value Function Approximations
, 2006
"... Markov decision processes (MDPs) with discrete and continuous state and action components can be solved efficiently by hybrid approximate linear programming (HALP). The main idea of the approach is to approximate the optimal value function by a set of basis functions and optimize their weights b ..."
Abstract
 Add to MetaCart
Markov decision processes (MDPs) with discrete and continuous state and action components can be solved efficiently by hybrid approximate linear programming (HALP). The main idea of the approach is to approximate the optimal value function by a set of basis functions and optimize their weights by linear programming. It is known that the solution to this convex optimization problem minimizes the L1 norm distance in between the optimal value function and its approximation. In this paper, we relate this measure to the maxnorm error of the same value function. We believe that this theoretical analysis may help to understand the quality of HALP approximations in continuous domains.
Learning Basis Functions in Hybrid Domains
 IN PROCEEDINGS OF THE 21ST NATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE
, 2006
"... Markov decision processes (MDPs) with discrete and continuous state and action components can be solved efficiently by hybrid approximate linear programming (HALP). The main idea of the approach is to approximate the optimal value function by a set of basis functions and optimize their weights b ..."
Abstract
 Add to MetaCart
Markov decision processes (MDPs) with discrete and continuous state and action components can be solved efficiently by hybrid approximate linear programming (HALP). The main idea of the approach is to approximate the optimal value function by a set of basis functions and optimize their weights by linear programming. The quality of this approximation naturally depends on its basis functions. However, basis functions leading to good approximations are rarely known in advance. In this paper,
Clinical Time Series Prediction: Towards A Hierarchical Dynamical System Framework
, 2014
"... Developing machine learning and data mining algorithms for building temporal models of clinical time series is important for understanding of patient state, the dynamics of a disease, effect of various patient management interventions and clinical decision making. In this work, we propose and deve ..."
Abstract
 Add to MetaCart
(Show Context)
Developing machine learning and data mining algorithms for building temporal models of clinical time series is important for understanding of patient state, the dynamics of a disease, effect of various patient management interventions and clinical decision making. In this work, we propose and develop a novel hierarchical framework for modeling clinical time series that combines advantages of the two approaches: Linear Dynamical Systems (LDS) and Gaussian Processes (GP). The new framework is more flexible than the two approaches individually in that it can model and learn from time series data of varied length and with irregularly sampled observations. We test our framework on the problem of learning time series models for the complete blood count panel, and show that it outperforms the existing models in terms of its predictive accuracy.