Citation Context

...the shuffle rate and the output rate because of our locality optimization – most data is read from a local disk and bypasses our relatively bandwidth constrained network. The shuffle rate is higher than the output rate because the output phase writes two copies of the sorted data (we make two replicas of the output for reliability and availability reasons). We write two replicas because that is the mechanism for reliability and availability provided by our underlying file system. Network bandwidth requirements for writing data would be reduced if the underlying file system used erasure coding [14] rather than replication. To appear in OSDI 2004 9 5.4 Effect of Backup Tasks In Figure 3 (b), we show an execution of the sort program with backup tasks disabled. The execution flow is similar to that shown in Figure 3 (a), except that there is a very long tail where hardly any write activity occurs. After 960 seconds, all except 5 of the reduce tasks are completed. However these last few stragglers don’t finish until 300 seconds later. The entire computation takes 1283 seconds, an increase of 44% in elapsed time. 5.5 Machine Failures In Figure 3 (c), we show an execution of the sort program ...

Citation Context

... the block will be cached at the servers along the lookup path (the triangles). CFS could save space by storing coded pieces of blocks rather than whole-block replicas, using an algorithm such as IDA =-=[25]-=-. CFS doesn't use coding, because storage space is not expected to be a highly constrained resource. The placement of block replicas makes it easy for a client to select the replica likely to be faste...

Citation Context

...on is usually performed via likelihood checks [18], [15]. Approaches and schemes have been proposed for tolerating: . intrusions and physical faults, via information fragmentation and dispersal [20], =-=[56]-=-, . malicious logic, and more specifically to viruses, either via control flow checking [35], or via design diversity [36], . intrusions, malicious logic, vulnerabilities due to physical or developmen...

Citation Context

...server goes down, all pages that it has not distributed become inaccessible to any algorithm. This problem can be eliminated using standard techniques, such as Rabin's Information Dispersal Algorithm =-=[10]-=-. So we ignore server faults. Observe that a request is satisfied if and only if all the caches serving for the nodes of the tree path are not down. Since each node is mapped to a machine (k-wise) ind...

Citation Context

...y of streaming content, delivery of content to mobile clients in wireless networks, peer-to-peer applications [3], delivery of content along multiple paths to ensure resiliency to network disruptions =-=[24]-=-, and a number of other applications too numerous to detail in this paper. 2. LT Codes Design The length ℓ of the encoding symbols can be chosen as desired. The overall encoding and decoding is more e...

Citation Context

...t often unnecessarily and unsustainably high expense. (Indeed, a recent IDC report suggests that data-generation is outpacing storage availability [14].) Protocols like Rabin’s data-dispersion scheme =-=[33]-=- are more efficient: They share data across multiple repositories with minimum redundancy, and ensure the availability of the data given the integrity of a quorum (kout-of-n) of repositories. Such pro...