This paper extends the framework of partially observable Markov decision processes (POMDPs) to multi-agent settings by incorporating the notion of agent models into the state space. Agents maintain beliefs over physical states of the environment and over models of other agents, and they use Bayesian update to maintain their beliefs over time. The solutions map belief states to actions. Models of other agents may include their belief states and are related to agent types considered in games of incomplete information. We express the agents ’ autonomy by postulating that their models are not directly manipulable or observable by other agents. We show that important properties of POMDPs, such as convergence of value iteration, the rate of convergence, and piece-wise linearity and convexity of the value functions carry over to our framework. Our approach complements a more traditional approach to interactive settings which uses Nash equilibria as a solution paradigm. We seek to avoid some of the drawbacks of equilibria which may be non-unique and are not able to capture off-equilibrium behaviors. We do so at the cost of having to represent, process and continually revise models of other agents. Since the agent’s beliefs may be arbitrarily nested the optimal solutions to decision making problems are only asymptotically computable. However, approximate belief updates and approximately optimal plans are computable. We illustrate our framework using a simple application domain, and we show examples of belief updates and value functions. 1.

Higher levels of the JDL model call for prediction of future development and awareness of the development of a situation. Since the situations handled by Command and Control systems develop by actions performed by opposing agents, pure probabilistic or evidential techniques are not quite sufficient tools for prediction. Game theoretic tools can give an improved appreciation of the real uncertainty in this prediction task, and also be a tool in the planning process. We review recent developments in game theory and apply them in a decision support tool for Command and Control situation awareness enhancements.

Analytical techniques are generally inadequate for dealing with causal interrelationships among a set of individual and social concepts. Usually, causal maps are used to cope with this type of interrelationships. However, the classical view of causal maps is based on an intuitive view with ad hoc rules and no precise semantics of the primitive concepts, nor a sound formal treatment of relations between concepts. In this paper, we solve this problem by proposing a formal model for causal maps with a precise semantics based on relation algebra and the software tool, CM-RELVIEW, in which it has been implemented. Then, we investigate the issue of using this tool in multiagent environments by explaining through different examples how and why this tool is useful for the following aspects: 1) the reasoning on agents' subjective views, 2) the qualitative distributed decision making, and 3) the organization of agents considered as a holistic approach. For each of these aspects, we focus on the computational mechanisms developed within CM-RELVIEW to support it.

In strategic multiagent decision making, it is often the case that a strategic reasoner must hold beliefs about other agents and use these beliefs to inform its decision making. The behavior thus produced by the reasoner involves an interaction between the reasoner’s beliefs about other agents and the reasoner’s own preferences. A significant challenge faced by model designers, therefore, is how to model such a reasoner’s behavior so that the reasoner’s preferences and beliefs can each be identified and distinguished from each other. In this paper, we introduce a model of strategic reasoning that allows us to distinguish between the reasoner’s utility function and the reasoner’s beliefs about another agent’s utility function as well as the reasoner’s beliefs about how that agent might interact with yet other agents. We show that our model is uniquely identifiable. That is, no two different parameter settings will cause the model to give the same behavior over all possible inputs. We then illustrate the performance of our model in a multiagent negotiation game played by human subjects. We find that our subjects have slightly incorrect beliefs about other agents in the game.

To perform rational decision-making, autonomous agents need considerable computational resources. In multi-agent settings, when other agents are present in the environment, these demands are even more severe. We investigate ways in which the agent's knowledge and the results of deliberative decision-making can be compiled to reduce the complexity of decision-making procedures and to save time in urgent situations. We use machine learning algorithms to compile decision-theoretic deliberations into condition-action rules on how to coordinate in a multi-agent environment. Using different learning algorithms, we endow a resource-bounded agent with a tapestry of decision making tools, ranging from purely reactive to fully deliberative ones. The agent can then select a method depending on the time constraints of the particular situation. We also propose combining the decision-making tools, so that, for example, more reactive methods serve as a pre-processing stage to the more accurate but sl...

Abstract. This paper aims at linking the AI notion of multi agent system with the notion of LEGAL RELATION [Allen & Saxon 95]. The paper is based on the idea that legal rules concern actions to accomplish or to exclude and states to achieve or to avoid or actions for changing existing legal relations. The body of the rule establishes who and under which conditions must respect the obligation. The notion of ‘obligation ’ has been defined elsewhere [Boella & Lesmo 01] and will be reviewed here. In this paper it is used as an ontological basis to define the LEGAL RELATIONS appearing in the A-Hohfeld language, and the concepts of ‘bearer of the obligation ’ (who undergoes the rule) and of ‘normative agent ’ (who watches on the rule) are connected to an ontology of legal entities.

Higher levels of the data fusion process call for prediction and awareness of the development of a situation. Since the situations handled by command and control systems develop by actions performed by opposing agents, pure probabilistic or evidential techniques are not fully sufficient tools for prediction. Game-theoretic tools can give an improved appreciation of the real uncertainty in this prediction task, and also be a tool in the planning process. Based on a combination of graphical inference models and game theory, we propose a decision support tool architecture for command and control situation awareness enhancements. This paper outlines a framework for command and control decision-making in multi-agent settings. Decision-makers represent beliefs over models incorporating other decision-makers and the state of the environment. When combined, the decision-makers’ equilibrium strategies of the game can be inserted into a representation of the state of the environment to achieve a joint probability distribution for the whole situation in the form of a Bayesian network representation.

The ability for a given agent to adapt on-line to better interact with another agent is a difficult and important problem. This problem becomes even more difficult when the agent to interact with is a human, since humans learn quickly and behave non-deterministically. In this paper we present a novel method whereby an agent can incrementally learn to predict the actions of another agent (even a human), and thereby can learn to better interact with that agent. We take a case-based approach, where the behavior of the other agent is learned in the form of state-action pairs. We generalize these cases either through continuous k-nearest neighbor, or a modified bounded minimax search. Through our case studies, our technique is empirically shown to require little storage, learn very quickly, and be fast and robust in practice. It can accurately predict actions several steps into the future. Our case studies include interactive virtual environments involving mixtures of synthetic agents and humans, with cooperative and/or competitive relationships. Key words: autonomous agents, user modeling, agent modeling, action prediction, plan recognition. 2

We develop a new graphical representation for interactive partially observable Markov decision processes (I-POMDPs) that is significantly more transparent and semantically clear than the previous representation. These graphical models called interactive dynamic influence diagrams (I-DIDs) seek to explicitly model the structure that is often present in real-world problems by decomposing the situation into chance and decision variables, and the dependencies between the variables. I-DIDs generalize DIDs, which may be viewed as graphical representations of POMDPs, to multiagent settings in the same way that I-POMDPs generalize POMDPs. I-DIDs may be used to compute the policy of an agent online as the agent acts and observes in a setting that is populated by other interacting agents. Using several examples, we show how I-DIDs may be applied and demonstrate their usefulness.