Learning, planning, and representing knowledge at multiple levels of temporal abstraction are key, longstanding challenges for AI. In this paper we consider how these challenges can be addressed within the mathematical framework of reinforcement learning and Markov decision processes (MDPs). We extend the usual notion of action in this framework to include options---closed-loop policies for taking action over a period of time. Examples of options include picking up an object, going to lunch, and traveling to a distant city, as well as primitive actions such as muscle twitches and joint torques. Overall, we show that options enable temporally abstract knowledge and action to be included in the reinforcement learning framework in a natural and general way. In particular, we show that options may be used interchangeably with primitive actions in planning methods such as dynamic programming and in learning methods such as Q-learning.

Air Traffic Management (ATM) of the future allows for the possibility of free flight, in which aircraft choose their own optimal routes, altitudes, and velocities. The safe resolution of trajectory conflicts between aircraft is necessary to the success of such a distributed control system. In this paper, we present a method to synthesize provably safe conflict resolution maneuvers. The method models the aircraft and the maneuver as a hybrid control system and calculates the maximal set of safe initial conditions for each aircraft so that separation is assured in the presence of uncertainties in the actions of the other aircraft. Examples of maneuvers using both speed and heading changes are worked out in detail. Index Terms---Air traffic management, conflict resolution, hybrid systems, verification. I.

Complex natural and engineered systems typically possess a hierarchical structure, characterized by continuousvariable dynamics at the lowest level and logical decision-making at the highest. Virtually all control systems today---from flight control to the factory floor---perform computer-coded checks and issue logical as well as continuous-variable control commands. The interaction of these different types of dynamics and information leads to a challenging set of "hybrid" control problems. We propose a very general framework that systematizes the notion of a hybrid system, combining differential equations and automata, governed by a hybrid controller that issues continuous-variable commands and makes logical decisions. We first identify the phenomena that arise in real-world hybrid systems. Then, we introduce a mathematical model of hybrid systems as interacting collections of dynamical systems, evolving on continuous-variable state spaces and subject to continuous controls and discrete transitions. The model captures the identified phenomena, subsumes previous models, yet retains enough structure on which to pose and solve meaningful control problems. We develop a theory for synthesizing hybrid controllers for hybrid plants in an optimal control framework. In particular, we demonstrate the existence of optimal (relaxed) and near-optimal (precise) controls and derive "generalized quasi-variational inequalities" that the associated value function satisfies. We summarize algorithms for solving these inequalities based on a generalized Bellman equation, impulse control, and linear programming.

A simply-structured high-level controller called a `supervisor' has recently been proposed in [1] for the purpose of orchestrating the switching of a sequence of candidate set-point controllers into feedback with an imprecisely modeled siso process so as to cause the output of the process to approach and track a constant reference input. The process is assumed to be modeled by a siso linear system whose transfer function is in the union of a number of subclasses, each subclass being small enough so that one of the candidate controllers would solve the set-point tracking problem, were the process's transfer function to be one of the subclass's members. In [1] it is shown that in the absence of unmodelled process dynamics the proposed supervisor can successfully perform its function fi.e., achieve a zero steady state tracking errorg even if process disturbances are present, provided they are constant. This paper proves that without any further modification, the same supervisor can also perform this function in the face of norm-bounded unmodelled dynamics and moreover that none of the signals within the overall system can grow without bound in response to bounded disturbance and noise inputs, be they constant or not.

This paper addresses feedback stabilization problems for linear time-invariant control systems with saturating quantized measurements. We propose a new control design methodology, which relies on the possibility of changing the sensitivity of the quantizer while the system evolves. The equation that describes the evolution of the sensitivity with time (discrete rather than continuous in most cases) is interconnected with the given system (either continuous or discrete), resulting in a hybrid system. When applied to systems that are stabilizable by linear time-invariant feedback, this approach yields global asymptotic stability. Index Terms---Feedback stabilization, hybrid system, linear control system, quantized measurement. I. INTRODUCTION T HIS PAPER deals with quantized feedback stabilization problems for linear time-invariant control systems. A quantizer, as defined here, acts as a functional that maps a real-valued function into a piecewise constant function taking on a finite...

The problem of systematically synthesizing hybrid controllers which satisfy multiple control objectives is considered. We present a technique, based on the principles of optimal control, for determining the class of least restrictive controllers that satisfies the most important objective (which we refer to as safety). The system performance with respect to lower priority objectives (which we refer to as efficiency) can then be optimized within this class. We motivate our approach by showing how the proposed synthesis technique simplifies to well known results from supervisory control and pursuit evasion games when restricted to purely discrete and purely continuous systems respectively. We then illustrate the application of this technique to two examples, one hybrid (the steam boiler benchmark problem), and one primarily continuous (a flight vehicle management system with discrete flight modes). 1 Introduction Hybrid systems, or systems that involve the interaction of discrete and co...

This paper introduces the concept of a hybrid system and some of the challenges associated with the stability of such systems, including the issues of guaranteeing stability of switched stable systems and finding conditions for the existence of switched controllers for stabilizing switched unstable systems. In this endeavor, this paper surveys the major results in the (Lyapunov) stability of finite-dimensional hybrid systems and then discusses the stronger, more specialized results of switched linear (stable and unstable) systems. A section detailing how some of the results can be formulated as linear matrix inequalities is given. Stability analyses on the regulation of the angle of attack of an aircraft and on the PI control of a vehicle with an automatic transmission are given. Other examples are included to illustrate various results in this paper. Keywords---Hybrid systems, linear matrix inequalities, stability, stabilizability, switched systems. I.

Abstract — Hybrid systems which are capable of exhibiting simultaneously several kinds of dynamic behavior in different parts of a system (e.g., continuous-time dynamics, discrete-time dynamics, jump phenomena, switching and logic commands, and the like) are of great current interest. In the present paper we first formulate a model for hybrid dynamical systems which covers a very large class of systems and which is suitable for the qualitative analysis of such systems. Next, we introduce the notion of an invariant set (e.g., equilibrium) for hybrid dynamical systems and we define several types of (Lyapunov-like) stability concepts for an invariant set. We then establish sufficient conditions for uniform stability, uniform asymptotic stability, exponential stability, and instability of an invariant set of hybrid dynamical systems. Under some mild additional assumptions, we also establish necessary conditions for some of the above stability types (converse theorems). In addition to the above, we also establish sufficient conditions for the uniform boundedness of the motions of hybrid dynamical systems (Lagrange stability). To demonstrate the applicability of the developed theory, we present specific examples of hybrid dynamical systems and we conduct a stability analysis of some of these examples (a class of sampleddata feedback control systems with a nonlinear (continuous-time) plant and a linear (discrete-time) controller, and a class of systems with impulse effects). Index Terms — Asymptotic stability, boundedness, dynamical system, equilibrium, exponential stability, hybrid, hybrid dynamical

Abstract — In this paper, we investigate a state estimation problem involving finite communication capacity constraints. Unlike classical estimation problems where the observation is a continuous process corrupted by additive noises, there is a constraint that the observations must be coded and transmitted over a digital communication channel with finite capacity. This problem is formulated mathematically, and some convergence properties are defined. Moreover, the concept of a finitely recursive coder-estimator sequence is introduced. A new upper bound for the average estimation error is derived for a large class of random variables. Convergence properties of some coder-estimator algorithms are analyzed. Various conditions connecting the communication data rate with the rate of change of the underlying dynamics are established for the existence of stable and asymptotically convergent coder-estimator schemes. Index Terms—Finitely recursive coder-estimator sequence, hybrid systems, prefix code, state estimation. I.

this paper is to give a brief tutorial review of four different classes of hybrid systems of this type - each consists of a continuous-time process to be controlled, a parameterized family of candidate controllers, and an event driven switching logic. Three of the logics, called prerouted switching, hysteresis switching and dwelltime switching respectively, are simple strategies capable of determining in real time which candidate controller should be put in feedback with a process in order to achieve desired closed-loop performance. The fourth, called cyclic switching, has been devised to solve the long-standing stabilizability problem which arises in the synthesis of identifier-based adaptive controllers because of the existence of points in parameter space where the estimated model upon which certainty equivalence synthesis is based, loses stabilizability