In recent years, overlay networks have become an effective alternative to IP multicast for efficient point to multipoint communication across the Internet. Typically, nodes self-organize with the goal of forming an efficient overlay tree, one that meets performance targets without placing undue burden on the underlying network. In this paper, we target high-bandwidth data distribution from a single source to a large number of receivers. Applications include large-file transfers and real-time multimedia streaming. For these applications, we argue that an overlay mesh, rather than a tree, can deliver fundamentally higher bandwidth and reliability relative to typical tree structures. This paper presents Bullet, a scalable and distributed algorithm that enables nodes spread across the Internet to self-organize into a high bandwidth overlay mesh. We construct Bullet around the insight that data should be distributed in a disjoint manner to strategic points in the network. Individual Bullet receivers are then responsible for locating and retrieving the data from multiple points in parallel. Key contributions of this work include: i) an algorithm that sends data to di#erent points in the overlay such that any data object is equally likely to appear at any node, ii) a scalable and decentralized algorithm that allows nodes to locate and recover missing data items, and iii) a complete implementation and evaluation of Bullet running across the Internet and in a large-scale emulation environment reveals up to a factor two bandwidth improvements under a variety of circumstances. In addition, we find that, relative to tree-based solutions, Bullet reduces the need to perform expensive bandwidth probing.

We propose a new scheme for content distribution of large files that is based on network coding. With network coding, each node of the distribution network is able to generate and transmit encoded blocks of information. The randomization introduced by the coding process eases the scheduling of block propagation, and, thus, makes the distribution more efficient. This is particularly important in large unstructured overlay networks, where the nodes need to make decisions based on local information only. We compare network coding to other schemes that transmit unencoded information (i.e. blocks of the original file) and, also, to schemes in which only the source is allowed to generate and transmit encoded packets. We study the performance of network coding in heterogeneous networks with dynamic node arrival and departure patterns, clustered topologies, and when incentive mechanisms to discourage free-riding are in place. We demonstrate through simulations of scenarios of practical interest that the expected file download time improves by more than 20-30 % with network coding compared to coding at the server only and, by more than 2-3 times compared to sending unencoded information. Moreover, we show that network coding improves the robustness of the system and is able to smoothly handle extreme situations where the server and nodes departure the system.

We propose a new approach for constructing P2P networks based on a dynamic decomposition of a continuous space into cells corresponding to processors. We demonstrate the power of these design rules by suggesting two new architectures, one for DHT (Distributed Hash Table) and the other for dynamic expander networks. The DHT network, which we call Distance Halving, allows logarithmic routing and load, while preserving constant degrees. Our second construction builds a network that is guaranteed to be an expander. The resulting topologies are simple to maintain and implement. Their simplicity makes it easy to modify and add protocols. We show it is possible to reduce the dilation and the load of the DHT with a small increase of the degree. We present a provably good protocol for relieving hot spots and a construction with high fault tolerance. Finally we show that, using our approach, it is possible to construct any family of constant degree graphs in a dynamic environment, though with worst parameters. Therefore we expect that more distributed data structures could be designed and implemented in a dynamic environment.

Although IP Multicast is an effective network primitive for best-effort, large-scale, multi-point communication, many multicast applications such as shared whiteboards, multi-player games and software distribution require reliable data delivery. Building services like reliable sequenced delivery on top of IP Multicast has proven to be a hard problem. The enormous extent of network and end-system heterogeneity in multipoint communication exacerbates the design of scalable end-to-end reliable multicast protocols. In this paper, we propose a radical departure from the traditional end-to-end model for reliable multicast and instead propose a hybrid approach that leverages the successes of unicast reliability protocols such as TCP while retaining the efficiency of IP multicast for multi-point data delivery. Our approach splits a large heterogeneous reliable multicast session into a number of multicast data groups of co-located homogeneous participants. A collection of application-aware agents--Reliable Multicast proxies (RMXs)--organizes these data groups into a spanning tree using an overlay network of TCP connections. Sources transmit data to their local group, and the RNLX in that group forwards the data towards the rest of the data groups. RMXs use detailed knowledge of application semantics to adapt to the effects of heterogeneity in the environment. To demonstrate the efficacy of our architecture, we have built a prototype implementation that can be customized for different kinds of applications.

The need to distribute large files across multiple wide-area sites is becoming increasingly common, for instance, in support of scientific computing, configuring distributed systems, distributing software updates such as open source ISOs or Windows patches, or disseminating multimedia content. Recently a number of techniques have been proposed for simultaneously retrieving portions of a file from multiple remote sites with the twin goals of filling the client’s pipe and overcoming any performance bottlenecks between the client and any individual server. While there are a number of interesting tradeoffs in locating appropriate download sites in the face of dynamically changing network conditions, to date there has been no systematic evaluation of the merits of different protocols. This paper explores the design space of file distribution protocols and conducts a detailed performance evaluation of a number of competing systems running in both controlled emulation environments and live across the Internet. Based on our experience with these systems under a variety of conditions, we propose, implement and evaluate Bullet ′ (Bullet prime), a mesh based high bandwidth data dissemination system that outperforms previous techniques under both static and dynamic conditions. 1

Advances in wireless and optical communication, as well as in Internet multicast protocols, make broadcast and multicast methods an effective solution to disseminate data. In particular, repetitive server-initiated broadcast is an effective technique in wireless systems and is a scalable solution to relieve Internet hot spots. A critical issue for the performance of multicast data dissemination is the multicast schedule. Previous work focused on a model where each data item is requested by clients with a certain probability that is independent of past accesses. In this paper, we consider the more complex scenario where a client accesses pages in blocks (e.g., a HTML file and all its embedded images), thereby introducing dependencies in the pattern of accesses to data. We present a sequence of heuristics that exploit page access dependencies. We measured the resulting client-perceived delay on multiple Web server traces, and observed an average speed-up over previous methods ranging from 8% to 91%. We conclude that scheduling for multi-item requests is a critical factor for the performance of repetitive broadcast.

Forward Error Correction (FEC) has been proposed as a technique for implementing efficient reliable multicast (RM). However, FEC incurs costs in encode/decode delay and implementation complexity. How much benefit is provided relative to these costs and how dependent is the benefit on the specific RM protocol? In this paper, we evaluate the benefits of FEC for RM, considering both proactive and reactive use with three RM recovery techniques: duplicate avoidance, limited scope multicast and subcast. Our simulationbased results indicate that FEC provides little benefit for an efficient RM protocol like OTERS and introduces extra delay for multi-point streaming data applications. 1

Server bandwidth has been identified as a major bottleneck in large Video-on-Demand (VoD) systems. Using multicast delivery to serve popular content helps increase scalability by making efficient use of server bandwidth. In addition, recent research has focused on proactive schemes in which the server periodically multicasts popular content without explicit requests from clients. Proactive schemes are attractive because they consume bounded server bandwidth irrespective of client arrival rate. In this work, we describe Fuzzycast, a scalable periodic multicast scheme that uses simple techniques to provide video on demand at reasonable client start-up times while consuming optimal server bandwidth. We present a theoretical analysis of its bandwidth and client buffer requirements and prove its optimality. We study the effect of variable bitrate (VBR) media on Fuzzycast performance and propose a simple extension to transmit VBR media over constant rate channels. Finally, we solve the problem of partitioning a transmission over multiple multicast groups by considering it as a specific instance of a more widely encountered resource trade-off.

Scalable Distributed Data Structures (SDDS) need scalable availability. This can be provided through replication, which is storage intensive, or through the use of Erasure Correcting Codes (ECC) to provide redundancy, which is more complicated. We calculate availability under both strategies and show that redundancy through use of an ECC implies significantly less overhead. We

This paper addresses the efficient multicast dissemination of bulk data from a single server to numerous clients. The challenge is complex: a client may commence reception at arbitrary times, should receive as Httle "extra" data as possible until it can reconstruct the entire content, and should have flexibility in choosing the data rate. From the network perspective, the data rate over any link should be as close as possible to the maximum singie-downstream-cHent subscription rate. Also, the solution should scale to huge ties and numerous subscribers, and should withstand changing network conditions and packet loss. Finally, it should be friendly to other traffic. For any base client-subscription rate and integer factors thereof, we jointly achieve all these goals in a near-optimal way while using standard ("any k of N") block erasure-correcting codes. Scalability in file size is attained by breaking the fie into equisized groups of equisized blocks and separately encoding each group. The other properties are attained by a unique openloop layered multicast transmission schedule. Each client merely subscribes to one or more standard multicast groups. The need to use special, non-standard and possibly proprietary codes that scale well is thus obviated.