## Dynamic System Evolution and Markov Chain Approximation (1998)

Venue: | Discrete Dynamics in NS, Gordon & Breach |

Citations: | 4 - 4 self |

### BibTeX

@INPROCEEDINGS{Melnik98dynamicsystem,

author = {Roderick V. Nicholas Melnik},

title = {Dynamic System Evolution and Markov Chain Approximation},

booktitle = {Discrete Dynamics in NS, Gordon & Breach},

year = {1998},

pages = {7--39}

}

### OpenURL

### Abstract

In this paper computational aspects of the mathematical modelling of dynamic system evolution have been considered as a problem in information theory. The construction of such models is treated as a decision making process with limited available information. The solution of the problem is associated with a computational model based on heuristics of a Markov Chain in a discrete space-time of events. A stable approximation of the chain has been derived and the limiting cases are discussed. An intrinsic interconnection of constructive, sequential, and evolutionary approaches in related optimization problems provides new challenges for future work. Key words: decision making with limited information, optimal control theory, hyperbolicity of dynamic rules, generalized dynamic systems, Markov Chain approximation. 1 Introduction Many mathematical problems in information theory and optimal control related to dynamic system studies can be formulated in the following generic form. A decision...

### Citations

6083 |
A mathematical theory of communication
- Shannon
- 1948
(Show Context)
Citation Context ...on. From an information theory perspective, this logical step, which in the end requires answering the question of system stability, is equivalent to a transformation from the classic Shannon entropy =-=[53, 49]-=- to the Boltzmann-Gibbs entropy [37]. Under such a transformation we formally identify a (thermo)dynamic system with a measure space [37]. If \Sigma is fixed and the measure is defined as a Lebesgue m... |

3738 |
Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images
- Geman, Geman
- 1984
(Show Context)
Citation Context ...dly, and the system may be trapped in a local energy minimum. However, the choice of Tn = c= log n with a sufficiently large c can theoretically guarantee the system's "escape" from the loca=-=l minimum [21]-=-. In practice, the algorithm works as follows. If for the time-index n x t n is given, then from the set N x t n we choose state t, calculate \Deltaf = f(t) \Gamma f(x t n ); and set x t n+1 = ae t; w... |

3559 | Optimization by simulated annealing
- Kirkpatrick, Gelatt, et al.
- 1983
(Show Context)
Citation Context ...lity, and evolution is thought as moving from a state to its neighbors uniformly. The computational implementation of the above idea is provided by the simulated annealing algorithm first proposed in =-=[31]-=-. For a real physical system, temperature may be lowered too rapidly, and the system may be trapped in a local energy minimum. However, the choice of Tn = c= log n with a sufficiently large c can theo... |

2626 |
Dynamic Programming
- Bellman
- 1957
(Show Context)
Citation Context ...urability on the interval (0; t). Under the above mentioned assumptions, associated theoretical issues are often addressed using the theory of Markov processes [19]. Starting from the work of Bellman =-=[5, 6]-=-, the theory has been extensively developed, and a number of 1 The functions f0 and g are called running and terminal costs respectively. If f0 j 0 we have Mayer's problem whereas for g j 0 the proble... |

1376 |
and Complex Analysis
- Rudin, Real
- 1976
(Show Context)
Citation Context ...dity of such a simplification ultimately depends on the Axiom of Choice excluding the logically possible case of incomparability of two arbitrary sets that correspond to two different times t and t 0 =-=[51, 32]-=-. However, on the other hand, such a simplification permits the development of a set-theoretic approach to dynamic system evolution, and simplifies the mathematical formalizations of complex optimizat... |

332 |
Viscosity Solutions of Hamilton-Jacobi Equations
- CRANDALL, LIONS
- 1983
(Show Context)
Citation Context ...ction F (l); then the associated optimal control problems can be studied through a nonlinear backward evolution PDE known as the Hamilton-Jacobi-Bellman equation with Cauchy-type terminal conditions (=-=[11, 19]-=- and references therin). If an algorithm for the numerical solution of the latter problem exists, it can be in principle represented in the form of the informational stream ((i xT ; s T ); (i xT \Gamm... |

290 |
Modern heuristic techniques for combinatorial problems
- Reeves
- 1995
(Show Context)
Citation Context ...he system evolution. In this sense an algorithm is always coupled to the problem specific information. In discrete optimization such algorithms can be conditionally divided into three main categories =-=[10, 50]-=-: ffl constructive algorithms (CAs) that require construction of decreasing and embedded in each other subsets of a given finite set of states \Sigma, ffl sequential algorithms (SAs) that attempt to c... |

274 |
Set-Valued Analysis
- Aubin, Frankowska
- 1990
(Show Context)
Citation Context ...turbed right-hand parts leads to the mathematical models in which dynamic rules are defined by differential inclusions. In fact, on the basis of the point-valued map f; we can define a set-valued map =-=[2, 19]-=- F(t; x t ) def = ff(t; x t ; s t )g; where s t is assumed to be defined by another set-valued map. Of course, the set-valued map for the definition of s t is coupled to the definition of the optimizi... |

259 |
Numerical Methods for Stochastic Control Problems in Continuous Time
- Kushner, Dupuis
- 1994
(Show Context)
Citation Context ... ideas have been essentially generalized for the continuous case during the past decades [18, 19], and many new results that appeared recently indicate the continued research interest in these topics =-=[19, 35]-=-. It should be noted, however, that many results in this area rely (explicitly or implicitly) on the assumption that a measurable function of strategies s t 2 U T may be effectively approximated using... |

232 |
Possibility Theory
- Dubois, Prade
- 1988
(Show Context)
Citation Context ...ysically reasonable, since a-priori information about the system can be given only as a certain possibilistic distribution which allows us to select a new distribution according to certain principles =-=[15, 49]. 6 Couple-=-d Mathematical Models of Macro and Micro evolution. The complexity in identifying a "hard boundary" interaction between system and its environment is eventually determined by the degree of r... |

182 |
Elementary principles in statistical mechanics: developed with especial reference to the rational foundation of thermodynamics
- Gibbs
- 1960
(Show Context)
Citation Context ...olution implied by the second law of thermodynamics and Eddington's time arrow. The complexity of the mathematical formalization of evolution irreversibility was well understood by J.Gibbs, who wrote =-=[22]-=-, it should not be forgotten when ensembles are chosen to illustrate the probabilities of events in the real world, that while the probabilities of subsequent events may often be determined from the p... |

161 |
High Resolution Schemes Using Flux Limiters for Hyperbolic Conservation Laws
- Sweby
- 1984
(Show Context)
Citation Context ...to switch from "observer" to "modeler" and vice versa whenever it is necessary. To construct a stable approximation of the model (7.1), (7.2) the idea of the upwind discrete scheme=-= with flux limiters [57]-=- is used. Without loss of generality for the numerical procedure, we assume that ~ f 0 = 0; which reduces the equation (7.1) to (6.6(b)). First, let us introduce in the cone of macroscopic events (7.1... |

99 |
Thermodynamics
- Müller
- 1985
(Show Context)
Citation Context ... the Hamilton-Jacobi-Bellman type for non-smooth and stochastic optimal control will be published separately [39, 40, 41]. Their connection with the principles of extended irreversible thermodynamics =-=[46, 30]-=- as well as computational algorithms shall be also discussed elsewhere. Acknowledgement This work has been supported by the School of Mathematics at the University of South Australia and by the PTRP U... |

99 |
Variational methods
- Struwe
- 1990
(Show Context)
Citation Context ...e is defined as a Lebesgue measure, then for any time-set (0; T ) (including the possibility of T !1) the validity of the above transformation requires an a-priori assumption of lower semi-continuity =-=[55]-=- of the recursive function i(!) = fn (f n\Gamma1 (:::f 1 (!):::)) (1.10) as a function of density, where a theoretical possibility of n ! 1 is permitted. If we assume that such a function exists, then... |

82 |
Extended Irreversible Thermodynamics
- Jou, Casas-Vázquez, et al.
- 1996
(Show Context)
Citation Context ...ise-deterministic stochastic systems. Such systems have been extensively studied during recent time by theoretical physicists [29], and indicate growing interest in hyperbolic dynamic rules of nature =-=[46, 30]. Mathem-=-atically speaking, we define a finite-state Markov Chain �� �� with the state space M. The chain is regarded as a parametric process for the dynamics of the system which is described by a stat... |

79 |
Fomin, Elements of the Theory of Functions and Functional Analysis
- Kolmogorov, V
- 1999
(Show Context)
Citation Context ...dity of such a simplification ultimately depends on the Axiom of Choice excluding the logically possible case of incomparability of two arbitrary sets that correspond to two different times t and t 0 =-=[51, 32]-=-. However, on the other hand, such a simplification permits the development of a set-theoretic approach to dynamic system evolution, and simplifies the mathematical formalizations of complex optimizat... |

66 |
Perturbation theory and finite Markov chains
- Schweitzer
- 1968
(Show Context)
Citation Context ...ions in the informational stream (1.6). Indeed, when perturbations of a Markov Chain change its ergodic structure, the stationary distribution of the perturbed system may not be a continuous function =-=[52, 1]-=-. Hence it is reasonable to assume that system dynamics depend on some parameters of the Markov Chain and due to the imprecision of available information we can study system dynamics using in general ... |

62 |
et al, “Equation of State Calculations by Fast Computing Machines,” The
- Metropolis
- 1953
(Show Context)
Citation Context ...espect to given ffl: The main advantage of SAs is based on the fact that they do not exclude the theoretical possibility of occasional acceptance of new states that may increase the energy functional =-=[43]. We also -=-assume that an "initial" solution x 0 2 \Sigma may be given (for example, obtained by a CA). Moving to a neighboring solution x 0 2 \Sigma; the structure of the neighborhood of the solution ... |

59 |
and Evgeny Lifshitz, Statistical physics
- Landau
- 1980
(Show Context)
Citation Context ...face. Now recall the definition of system entropy in statistical physics as a quantity that uncertain to an additive constant and is dependent on the choice of units, defined by the Liouville measure =-=[36] oe = \G-=-amma Z f log [(2����h) s f ] dpdq: (5.1) Here s is the degree of system freedom, p and q are momentum and position variables. If we assume that the whole system entropy can be defined through ... |

57 |
Piecewise-deterministic Markov processes: A general class of nondiffusion stochastic models
- Davis
- 1984
(Show Context)
Citation Context ...dy system dynamics using in general Singularly Perturbed Markov Chains. In this framework evolution of a system is coupled to its Markov Chain parameters. An example of this type DMDP was provided in =-=[13]-=- where non-diffusion stochastic models were studied. We assume that in general the parameter of the Markov Chain is allowed to jump, and the jumping rate may be dependent on the state function x t . T... |

57 |
A general theory of optimal algorithms
- Traub, Wo´zniakowski
- 1980
(Show Context)
Citation Context ...ch effective computational algorithm exists. In a search for a compromise between these two extremes induced by the "energetic"(combinatorial) and informational complexity of the underlying =-=algorithm [56, 58]-=- it is reasonable to include the decision maker as an intrinsic part of the constructed model using some learning rules. As a result, mathematical models become coupled to their computational associat... |

45 |
Lower semicontinuous solutions of Hamilton-Jacobi-Bellman equations
- Frankowska
- 1993
(Show Context)
Citation Context ... �� ; s �� )d�� + g(x(T )): (3.6) If we define the value function as V (t; x) def = inf s t 2UT J(t; x t ; s t ); (3.7) then using appropriate regularity assumptions and dynamic programmin=-=g principle [19, 20]-=-, the original optimal control problem can be studied through the Hamilton-Jacobi-Bellman (HJB) equationsV (t; x t ) +H(t;x t ; D x V (t; x t )) = 0; V (T ; \Delta) = g(\Delta); (3.8) where the Hamilt... |

34 |
Adaptive Markov Control Processes
- Hern'andez-Lerma
- 1989
(Show Context)
Citation Context ...DMDP is well-developed under the assumption of the possibility of complete information in (1.6). During recent years new challenging problems have stimulated further development in the theory of DMDP =-=[34, 25, 17]-=-. In brief, one of the most interesting problems in this field is induced by the question of data perturbations in the informational stream (1.6). Indeed, when perturbations of a Markov Chain change i... |

34 |
Time’s arrow: the origins of thermodynamic behavior
- Mackey
- 1992
(Show Context)
Citation Context ..., deterministic invariance of phase points along trajectories is replaced by the invariance of the density along trajectories. Physically, such a "conservation of extension in phase" (due to=-= J. Gibbs [37]) eventual-=-ly requires a construction of Gibbs distribution functions using a probabilistic description of states. Mathematically speaking, this problem can be seen as a problem of a "closure" of the r... |

28 | Universal schemes for sequential decision from individual sequences
- Merhav, Feder
- 1993
(Show Context)
Citation Context ...mated using past states x t 0 ; 0 ! t 0 ! t: If such an assumption is made, the attainability of the minimum in (1.3) becomes the subject of a corresponding smoothness assumption on the loss function =-=[42]-=-. On the other hand, regularity of this function is strongly dependent on complete information about the past states, and eventually on model data and parameters. Since the initial data for the model ... |

28 |
Associative Computing
- Potter
- 1992
(Show Context)
Citation Context ...ons [27] eventually remain the only algorithmic procedures that can be applied in theoretical generality. However, such procedures may encounter serious difficulties in non-equilibrium thermodynamics =-=[48]-=-. In a search for alternative approaches to the ensemble averaging, many useful ideas have been generated during recent years. The intrinsic ability of Markov Chains to form a canonical Gibbs ensemble... |

27 |
Space-time as a causal set, Phys
- Bombelli, Lee, et al.
- 1987
(Show Context)
Citation Context ...ls of these events defined by a topology base in the macroscopic frame of reference. A consideration of the space-time as a causal discrete set was the subject of many publications (see, for example, =-=[7, 9]-=- and references therein). Recently some new theoretical results on dynamic system discretizations on lattices have been obtained [14]. Below we formalize these ideas with respect to our models using t... |

25 |
Embedding a sequential procedure within an evolutionary algorithm for coloring problems in graphs
- Costa, Hertz, et al.
- 1995
(Show Context)
Citation Context ...he system evolution. In this sense an algorithm is always coupled to the problem specific information. In discrete optimization such algorithms can be conditionally divided into three main categories =-=[10, 50]-=-: ffl constructive algorithms (CAs) that require construction of decreasing and embedded in each other subsets of a given finite set of states \Sigma, ffl sequential algorithms (SAs) that attempt to c... |

17 |
From deterministic dynamics to probabilistic descriptions
- Misra, Prigogine, et al.
- 1979
(Show Context)
Citation Context ...unctional. The process of constructing mathematical models is always a competition between (i) an approximation of the system-environment boundary interface (which involves the system's internal time =-=[44]-=-), and (ii) the conservation laws for integral characteristics of the system (which involves modeler's time [39]). As a result of such a competition, the resulting mathematical models simulate couplin... |

15 |
Viscosity solutions of nonlinear second-order partial differential equations in Hilbert spaces. Comm. Partial Differential Equations
- Ishii
- 1993
(Show Context)
Citation Context ... set in which physical states of the system can be embedded. Mathematically, the problem is usually considered with respect to Euclidean spaces (either finite dimensional [19] or infinite dimensional =-=[28]-=-). It allows us to use the logical rule in the reverse order: provided g is specified in a topological space, the backward-evolution model can, in principle, recover the forward-evolution of the syste... |

15 | On Quantum Jumps, Events and Spontaneous Localization Models”, ESI-Wien Preprint
- Jadczyk
- 1994
(Show Context)
Citation Context .... The corresponding systems described by x �� at time �� are called piecewise-deterministic stochastic systems. Such systems have been extensively studied during recent time by theoretical phy=-=sicists [29], and in-=-dicate growing interest in hyperbolic dynamic rules of nature [46, 30]. Mathematically speaking, we define a finite-state Markov Chain �� �� with the state space M. The chain is regarded as a ... |

14 |
Partial differential equations
- Courant
- 1962
(Show Context)
Citation Context ...esult, in any such cell 9 the GDS velocity defined by (7.5) is always greater than or equal to 1. Hence, if the process is approximated in c ij ; the Courant-Friedrichs-Lewy (CFL) stability condition =-=[12] (��-=-��sh) is satisfied automatically, regardless of the actual values of the velocity function in c ij . Remark 7.1 In the limiting case h ! 0; definition 7.1 loses its meaning and a macroscopic system ... |

13 |
Structure of Random Discrete Spacetime,” Phys
- Brightwell, Gregory
- 1991
(Show Context)
Citation Context ...ls of these events defined by a topology base in the macroscopic frame of reference. A consideration of the space-time as a causal discrete set was the subject of many publications (see, for example, =-=[7, 9]-=- and references therein). Recently some new theoretical results on dynamic system discretizations on lattices have been obtained [14]. Below we formalize these ideas with respect to our models using t... |

12 |
Perturbation and stability theory for Markov control problems
- Abbad, Filar
- 1992
(Show Context)
Citation Context ...ions in the informational stream (1.6). Indeed, when perturbations of a Markov Chain change its ergodic structure, the stationary distribution of the perturbed system may not be a continuous function =-=[52, 1]-=-. Hence it is reasonable to assume that system dynamics depend on some parameters of the Markov Chain and due to the imprecision of available information we can study system dynamics using in general ... |

9 |
Sequential decisions in the control of a spaceship
- BATHER, CHERNOFF
- 1967
(Show Context)
Citation Context ...eam (1.6), even if ffl ! 0 + : It implies a consideration of singular stochastic problems in which the function x ffl t is allowed to be discontinuous (the first problems of this type were studied in =-=[3, 4]). In gene-=-ral, since a "transition" between s t and x ffl t (T 2 (0; 1)) may be discontinuous, we cannot use the principle of smooth fit (see [54] and references therein) to claim continuity of the re... |

8 |
Regularity of the value function for a twodimensional singular stochastic control problem
- Soner, Shreve
- 1989
(Show Context)
Citation Context ...us (the first problems of this type were studied in [3, 4]). In general, since a "transition" between s t and x ffl t (T 2 (0; 1)) may be discontinuous, we cannot use the principle of smooth=-= fit (see [54]-=- and references therein) to claim continuity of the recursive function of density i ffl x t when t ! T (possibly T !1). If our objective is a possibilistic attainability of the following limits lim ff... |

5 |
Sequential decisions in the control of a spaceship (finite fuel
- BATHER, CHERNOFF
- 1967
(Show Context)
Citation Context ...eam (1.6), even if ffl ! 0 + : It implies a consideration of singular stochastic problems in which the function x ffl t is allowed to be discontinuous (the first problems of this type were studied in =-=[3, 4]). In gene-=-ral, since a "transition" between s t and x ffl t (T 2 (0; 1)) may be discontinuous, we cannot use the principle of smooth fit (see [54] and references therein) to claim continuity of the re... |

5 |
Optimal continuous-parameter stochastic control
- Fleming
- 1969
(Show Context)
Citation Context ...m is referred to as the Lagrange problem. efficient algorithms have been proposed. Discrete dynamic programming ideas have been essentially generalized for the continuous case during the past decades =-=[18, 19]-=-, and many new results that appeared recently indicate the continued research interest in these topics [19, 35]. It should be noted, however, that many results in this area rely (explicitly or implici... |

5 |
Minimax Models in the Theory of Numerical Methods
- Sucharev
- 1992
(Show Context)
Citation Context ... the system-environment coupling in the mathematical model. If constructing a model is an art rather than a science, then the latter formally begins from the derivation of an algorithm from the model =-=[56]-=-. In concluding this section, it should be emphasized that the quality of a mathematical model for dynamic system evolution is decisively dependent on (i) the approximation of the initial conditions f... |

5 |
The Boundary Function Method for Singular Perturbation Problems
- Vasilieva, Butuzov, et al.
- 1995
(Show Context)
Citation Context ...ormally allows us to include the parameter of perturbation ffl into the model. We can assume, for example, that the dynamics of the system can be effectively described by "fast" and "sl=-=ow" components [59]-=-: ae fflsz = f 1 (z t ; y t ; s t ; t; ffl); z(0; ffl) = z 0 ;sy = f 2 (y t ; z t ; s t ; t; ffl); y(0; ffl) = y 0 : (3.24) If we choose a functional F in (1.3) as F (l) def = J ffl = g(y T ; z T ) + ... |

4 |
Nonlinear dynamical systems: coupling information and energy
- Melnik
- 1996
(Show Context)
Citation Context ... theory of dynamic system in continuous (space)time is a possibilistic assumption of vanishing error lim ffl!0 + % ffl 0 = 0: (8.1) Moreover, a concept of absolute or "external" to the syste=-=m DM-time [45, 38]-=- leads to the theoretical possibility of predicting a future event e n0 which is associated with the DM-time t = T (possibly T = 1) with the probability 1. This means that lim ffl!0 + % ffl n0 = 0; (8... |

3 |
Sufficient conditions to single out the Gibbs measure from other time-invariant measures
- Goldstein
- 1983
(Show Context)
Citation Context ...nomena. On the other hand, the main difficulty in effective applications of probabilistic models arises from the question of how common is the ergodicity of the Hamiltonian flow on the energy surface =-=[24]-=-. As was pointed out, perturbations can qualitatively change the ergodic structure of the underlying dynamic system. The examples of Markov Chains with discontinuities in the stationary distribution o... |

3 |
Nonconservation Law Equation
- Melnik
- 1996
(Show Context)
Citation Context ...of the system-environment boundary interface (which involves the system's internal time [44]), and (ii) the conservation laws for integral characteristics of the system (which involves modeler's time =-=[39]-=-). As a result of such a competition, the resulting mathematical models simulate coupling of the system to its environment, and can be considered as models of neither isolated nor closed systems. A fo... |

3 | A hierarchy of hyperbolic macrodynamic equations as a model for network training, in
- Melnik
(Show Context)
Citation Context ...ecursion rule have been discussed. The results on the derivation of hyperbolic equations of the Hamilton-Jacobi-Bellman type for non-smooth and stochastic optimal control will be published separately =-=[39, 40, 41]-=-. Their connection with the principles of extended irreversible thermodynamics [46, 30] as well as computational algorithms shall be also discussed elsewhere. Acknowledgement This work has been suppor... |

3 |
Optimal control for stochastic partial differential equations and viscosity solutions of Bellman equations
- Nisio
- 1991
(Show Context)
Citation Context ...ls for which the form of the Hamiltonian is specified. During the past years the theory has been extensively developed in this direction for deterministic and stochastic optimal control problems (see =-=[11, 47, 28, 19]-=- and references therein). Since the Hamiltonian of the system can be given only approximately, whereas regularity for the sought-for solution is not a-priori knowledge being the subject of our assumpt... |

2 |
A Markovian Decision Problem
- Bellman
- 1957
(Show Context)
Citation Context ...urability on the interval (0; t). Under the above mentioned assumptions, associated theoretical issues are often addressed using the theory of Markov processes [19]. Starting from the work of Bellman =-=[5, 6]-=-, the theory has been extensively developed, and a number of 1 The functions f0 and g are called running and terminal costs respectively. If f0 j 0 we have Mayer's problem whereas for g j 0 the proble... |

2 |
Degree theory on finite lattices: discretizations of dynamical systems
- Diamond, Kloeden, et al.
- 1995
(Show Context)
Citation Context ...y approximately. Such an approximation defines a degree n of the underlying recursion (1.10), and in turn defines a basic structure of a finite lattice on which the system dynamic can be approximated =-=[14]-=-. Hence, in general, information on an approximation of the same dynamic system can be provided in two possible ways: ffl using the sequence (1.1), and ffl using a subsequence of (1.11) that is (i xT ... |

2 | On Consistent Regularities of Control and Value - Melnik - 1997 |

1 |
Eds), Uncertainty in Knowledge Bases
- Bouchon-Meunier, Yager, et al.
- 1991
(Show Context)
Citation Context ...formation about the system, then the concept of a Markov Chain comes by itself. Because of uncertainty in knowledge base (1.1), such an accumulation cannot be understood in a purely deterministic way =-=[8]. The-=- origin of such uncertainty is induced by the strategy s 0 in the data stream (1.6). However, mathematically such uncertainty can be formalized if instead of constraints (1.2) we consider "relaxe... |

1 |
et al, Guranteed Accuracy in Numerical Linear Algebra
- Godunov
- 1992
(Show Context)
Citation Context ...is equivalent to the investigation of the ffl-spectrum of the original matrix A: The ffl-spectrum of a matrix is defined as the union of all spectra of perturbed matrices for a certain level of error =-=[23]-=-. In general, for any arbitrary matrix A there exists a special connection between its spectrum and its resolvent under ffl-perturbations. The problem consists of the fact that without restrictions on... |