Traditional resource management techniques (resource allocation, admission control and scheduling) have been found to be inadequate for many shared Grid and distributed systems that face unpredictable and bursty workloads. They provide no incentive for users to request resources judiciously and appropriately, and they do not capture the true value and importance (the utility) of user jobs. Consequently, researchers and practitioners have been examining the appropriateness of ‘market-inspired ’ resource management techniques in ensuring that users are treated fairly, without unduly favouring one set of users over another. Such techniques aim to smooth out access patterns and reduce the chance of transient overload, by providing incentives for users to be flexible about their resource requirements and job deadlines. We examine the recent evolution of these systems, looking at the state of the art in price setting and negotiation, grid economy management and utilitydriven scheduling and resource allocation, and identify the advantages and limitations of these systems. We then look to the future of these systems, examining the emerging ‘Catallaxy ’ market paradigm and present Mercato, a decentralised, Catallaxy inspired architecture that encapsulates the future directions that need to be pursued to address the limitations of current generation of market oriented Grids and Utility Computing systems. 1

Abstract—Virtualization technologies like VMware and Xen provide features to specify the minimum and maximum amount of resources that can be allocated to a virtual machine (VM) and a shares based mechanism for the hypervisor to distribute spare resources among contending VMs. However much of the existing work on VM placement and power consolidation in data centers fails to take advantage of these features. One of our experiments on a real testbed shows that leveraging such features can improve the overall utility of the data center by 47 % or even higher. Motivated by these, we present a novel suite of techniques for placement and power consolidation of VMs in data centers taking advantage of the min-max and shares features inherent in virtualization technologies. Our techniques provide a smooth mechanism for power-performance tradeoffs in modern data centers running heterogeneous applications, wherein the amount of resources allocated to a VM can be adjusted based on available resources, power costs, and application utilities. We evaluate our techniques on a range of large synthetic data center setups and a small real data center testbed comprising of VMware ESX servers. Our experiments confirm the end-to-end validity of our approach and demonstrate that our final candidate algorithm, PowerExpandMinMax, consistently yields the best overall utility across a broad spectrum of inputs – varying VM sizes and utilities, varying server capacities and varying power costs – thus providing a practical solution for administrators. I.

We present a system for allocating resources in shared data and compute clusters that improves MapReduce job scheduling in three ways. First, the system uses regulated and user-assigned priorities to offer different service levels to jobs and users over time. Second, the system dynamically adjusts resource allocations to fit the requirements of different job stages. Finally, the system automatically detects and eliminates bottlenecks within a job. We show experimentally using real applications that users can optimize not only job execution time but also the cost-benefit ratio or prioritization efficiency of a job using these three strategies. Our approach relies on a proportional share mechanism that continuously allocates virtual machine resources. Our experimental results show a 11−31 % improvement in completion time and 4−187 % improvement in prioritization efficiency for different classes of MapReduce jobs. We further show that delay intolerant users gain even more from our system.

QoS parameters

Provisioning a storage system requires balancing the costs of the solution with the benefits that the solution will provide. Previous provisioning approaches have started with a fixed set of requirements and the goal of automatically finding minimum cost solutions to meet them. Such approaches neglect the cost-benefit analysis of the purchasing decision. Purchasing a storage system involves an extensive set of trade-offs between metrics such as purchase cost, performance, reliability, availability, power, etc. Increases in one metric have consequences for others, and failing to account for these trade-offs can lead to a poor return on the storage investment. Using a collection of storage acquisition and provisioning scenarios, we show that utility functions enable this cost-benefit structure to be conveyed to an automated provisioning tool, enabling the tool to make appropriate trade-offs between different system metrics including performance, data protection, and purchase cost.

Several companies offer computation on demand for a fee. More companies are expected to enter this business over the next decade, leading to a marketplace for computation resources. Resources will be allocated through economic mechanisms that establish the relative values of providers and customers. Society at large should benefit from discoveries obtained through the vast computing power that will become available. Given such a computation marketplace, can economics-based resource allocation provide benefits for providers, customers and society? To investigate this question, we simulate a grid economy where individual providers and customers pursue their own ends and we measure resulting effects on system welfare. In our experiments, customers attempt to maximize their individual utilities, while providers pursue strategies chosen from three classes: information-free, utilization-based and economics-based. We find that, during periods of excess demand, economics-based strategies yield overall resource allocation that benefits system welfare. Further, economics-based strategies respond well to sudden overloads caused by temporary provider failures. During periods of moderate demand, we find that economics-based strategies provide ample system welfare, comparable with that of utilization-based strategies. We also identify and discuss key factors that arise when using economic mechanisms to allocate resources in a computation marketplace.

Abstract. Emerging deadline-driven Grid applications require a number of computing resources to be available over a time frame, starting at a specific time in the future. To enable these applications, it is important to predict the resource availability and utilise this information during provisioning because it affects their performance. It is impractical to request the availability information upon the scheduling of every job due to communication overhead. However, existing work has not considered how the precision of availability information influences the provisioning. As a result, limitations exist in developing advanced resource provisioning and scheduling mechanisms. This work investigates how the precision of availability information affects resource provisioning in multiple site environments. Performance evaluation is conducted considering both multiple scheduling policies in resource providers and multiple provisioning policies in brokers, while varying the precision of availability information. Experimental results show that it is possible to avoid requesting availability information for every Grid job scheduled thus reducing the communication overhead. They also demonstrate that multiple resource partition policies improve the slowdown of Grid jobs. 1

Metaschedulers can distribute parts of a Bag-of-Tasks (BoT) application among various resource providers in order to speed up its execution. When providers cannot disclose private information such as their load and computing power, which are usually heterogeneous, the meta scheduler needs to make blind scheduling decisions. We propose three policies for composing resource offers to schedule deadline constrained BoT applications. Offers act as a mechanism in which resource providers expose their interest in executing an entire BoT or only part of it without revealing their load and total computing power. We also evaluate the amount of information resource providers need to expose to the metascheduler and its impact on the scheduling. Our main findings are: (i) offer-based scheduling produces less delay for jobs that cannot meet deadlines in comparison

power scheduling, data centers, autonomic computing Data centers contain heterogeneous sets of machines. Some machines are faster and some – often the same ones – consume more energy and cost more to operate. The data center coordinator must decide how to allocate these machines to multiple applications of potentially many customers, each of which has different requirements. Given a stream of customer requests for machines, how does the data center provider decide which machines to give to whom and when? We propose new algorithms for a cost-aware provider to maximize its profit as it makes admission and scheduling decisions for the customer requests. We show that it matters which machines are assigned to each customer, especially when the data center is undersaturated. (Most data centers are.) Our new algorithms do best when they try to anticipate the ”riskiness ” of their decisions, that is, the likelihood that even higher-value requests will arrive later. We also show that turning unused machines off, rather than leaving them idle, even using simple heuristics like “turn off a machine that has been idle for ten minutes, ” can save a lot of money. Finally, we show that having heterogeneity in the data center is, in fact, beneficial. We demonstrate that the same set of customers can be satisfied at a lower cost and a higher profit in a heterogeneous data center rather than in a data center comprised solely of the newest, fastest, machines.

Advances in server virtualization offer new mechanisms to provide resource management for shared server infrastructures. Resource sharing requires coordination across self-interested system participants (e.g., providers from different administrative domains or third-party brokering intermediaries). Assignments of the shared infrastructure must be fluid and adaptive to meet the dynamic demands of clients. This thesis addresses the hypothesis that a new, foundational layer for virtual computing is sufficiently powerful to support a diversity of resource management needs in a general and uniform manner. Incorporating resource management at a lower virtual computing layer provides the ability to dynamically share server infrastructure between multiple hosted software environments (e.g., grid computing middleware and job execution systems). Resource assignments within the virtual layer occur through a lease abstraction, and extensible policy modules define management functions. This research makes the following contributions: • Defines the foundation for resource management in a virtual computing layer. Defines protocols and extensible interfaces for formulating resource contracts