We study the process of multi-agent reinforcement learning in the context of load balancing in a distributed system, without use of either central coordination or explicit communication. We first define a precise framework in which to study adaptive load balancing, important features of which are its stochastic nature and the purely local information available to individual agents. Given this framework, we show illuminating results on the interplay between basic adaptive behavior parameters and their effect on system efficiency. We then investigate the properties of adaptive load balancing in heterogeneous populations, and address the issue of exploration vs. exploitation in that context. Finally, we show that naive use of communication may not improve, and might even harm system efficiency. 1. Introduction This article investigates multi-agent reinforcement learning in the context of a concrete problem of undisputed importance -- load balancing. Real life provides us with many exampl...

Abstract not found

I. Introduction Our interest is in large distributed systems in which de- cision makers or agents must make good, though not nec- essarily optimal, decisions. It is assumed that the cost of communication (e.g. transmitting, receiving, and inter- preting messages) does not permit continuous communica- tion between agents. As a result, each agent has its own view of the global state and is uncertain about the actual global state and the future actions of other agents [1]. Fre- quent communication decreases the uncertainty in global state but has a cost which can adversely affect the perfor- mance. The problem is to make good decisions with a low rate of communication of state information. The goal is to improve performance by understanding and addressing this communication problem in distributed decision making. This is relevant to coordination which is informally defined as "the activity of independent agents making harmonious, non-conflicting decisions " which is similar in spirit to the definition in [2]. The difficulty in co- ordination is exacerbated by uncertainty in the information regarding agents ' strategies, resources and constraints. As an example, an agent may apply a particular strategy when faced with limited resources and severe time constraints; a coordination problem arises if the view from another agent suggests that the first agent has ample resources or time.

Each job scheduling agent in large decentralized load balancing systems generally must consider whether it is advantageous to ooad jobs to remote hosts when the local load is too high. Although processing power may appear to be available at a very distant host, two problems arise due to the transmission delay between the two systems. Predictably, the response time of the job is adversely aected as the job spends valuable time in transit, but a more subtle problem involves the value, or reliability, of the state information regarding job queues. The longer the delay between systems, the less an agent should value the state information of the remote system. We examine the performance of agents in topologies with dierent proximity (in terms of the average number of hops between hosts) and show that adaptive agents prefer to work with agents in close proximity. That is, the agents prefer localized decision-making due to more valuable information and these agents perform better than non-...

In Automonous Decentralized Systems (ADS), there is a high degree of uncertainty, especially in global state information and the performance and reliability of various resoufces. We model a distributed computer system of local and shared global servers, each of which can process jobs. The local servers insure a degree of autonomy and the global servers provide added compu-tational power and redundancy. When a job arrives, an agent decides whether to schedule the job locally, or whether to ship it to a global server. The agents have access to state information concerning current loads on the global servers and the response times of com-pleted jobs, but because of decentmlization this infor-mation is delayed, and perhaps lost, along communica-tion channels. The agents must make good decisions under the constraint that service rates are unknown and dynamic. We evaluate a deterministic algorithm and two randomizing algorithms and show that the de-gree of uncertainty determines which one is the most successful. 1

The aim of this paper is to present an original model of dynamic load balancing inside groups using queueing theory. Communication speed is greater inside a group than from one group to another. An analytical formula of the average queue length is obtained if we take two groups each with two processors. Beyond two groups, we can compute the numeric values as a function of the processes' arrival rates, execution times and communication speeds. The values of these parameters can be used to define the conditions requiring migration. Keywords : Dynamic load balancing, modeling, queueing theory models, groups decomposition, performance evaluation. 1 Introduction 1.1 Load balancing The arrival of workstations networks and multicomputers has resulted in a considerable increase in available computing power, however it also results in wasted performance, due to unbalanced CPU loading. Resource allocation is an answer to this major problem, which must be solved if systems are to become even ...

We explore certain problems that inhibit coordination in distributed systems based on decentralized control. The most fundamental of these problems are those due to state-information uncertainty and mutually conflicting decisions. We present an integrated view of a series of experiments we carried out in previous studies, showing the common thread of how randomization can be used to directly attack the fundamental problems of decentralized control, and we show the resultant performance gains that are achieved. 1

At the Computer Systems Laboratory at UCSD, we conduct research in operating systems, communication networks, and performance evaluation. This document reviews the current research within the Computer Systems Laboratory. The final section contains descriptions of research projects undertaken by our students. For more information, contact the authors at the above address, or by FAX at (619) 534-7029. Joseph Pasquale may be contacted by e-mail at pasquale@ucsd.edu or by telephone at (619) 534-2673. George Polyzos may be contacted by e-mail at polyzos@ucsd.eduor by telephone at (619) 534-3508. 1 1 Introduction At the UCSD Computer Systems Laboratory (CSL), we conduct research in operating systems, communication networks, and performance evaluation. In operating systems, we are focusing on high performance I/O software support for a variety of I/O intensive applications, such as multimedia applications and scientific applications. In communication networks, we concentrate on problems a...