## Parallel sorting on a shared-nothing architecture using probabilistic splitting (1991)

Citations: | 81 - 1 self |

### BibTeX

@INPROCEEDINGS{DeWitt91parallelsorting,

author = {David J. DeWitt and Jeffrey F. Naughton and Donovan A. Schneider},

title = {Parallel sorting on a shared-nothing architecture using probabilistic splitting},

booktitle = {},

year = {1991},

pages = {280--291},

publisher = {}

}

### Years of Citing Articles

### OpenURL

### Abstract

We consider the problem of external sorting in a shared-nothing multiprocessor. A critical step in the algorithms we consider is to determine the range of sort keys to be handled by each processor. We consider two techniques for determining these ranges of sort keys: exact splitting, using a parallel version of the algorithm proposed by Iyer, Ricard, and Varman; and probabilistic splitting, which uses sampling to estimate quantiles. We present analytic results showing that probabilistic splitting performs better than exact splitting. Finally, we present experimental results from an implementation of sorting via probabilistic splitting in the Gamma parallel database machine.

### Citations

229 | The gamma database machine project - DeWitt, Ghandeharizadeh, et al. - 1990 |

173 | A comparison of sorting algorithm for the connection machine CM-2 - Blelloch, hiserson, et al. - 1991 |

148 |
The Case for Shared Nothing
- Stonebraker
- 1986
(Show Context)
Citation Context ... sorting algorithm based on probabilistic splitting, we implemented the algorithm using the Gamma Database Machine [DGS + 90] as our experimental vehicle. Gamma falls into the class of shared-nothing =-=[Sto86]-=- architectures. The hardware consists of a 32 processor Intel iPSC/2 hypercube. Each processor is configured with a 80386 CPU, 8 megabytes of memory, and a 330 megabyte MAXTOR 4380 (5 1/4 in.) disk dr... |

116 |
Multi-disk management algorithms
- Livny, Khoshafian, et al.
- 1987
(Show Context)
Citation Context ...e that contain a matching value. One indexed attribute may be designated to be a clustering attribute for the file. In Gamma, relations are horizontally partitioned [RE78] (also known as declustering =-=[LKB87]-=-) 14 across all disk drives in order to increase the aggregate I/O bandwidth provided by the hardware. The query language of Gamma provides the user with several alternative declustering methods. For ... |

71 |
Join and semijoin algorithms for a multiprocessor database machine
- Valduriez, Gardarin
- 1984
(Show Context)
Citation Context ...hrough the entire file. Beck et al. [BBW88] describes a similar sort. Other earlier algorithms that also used a final sequential merge phase include Bitton et al. [BBDW83], and Valduriez and Gardarin =-=[VG84]-=-. Graefe [Gra90] presents a thorough description of the implementation and evaluation of a multiple-input, multiple-output sorting algorithm for a shared-memory multiprocessor. This algorithm uses a s... |

49 |
Parallel sorting and data partitioning by sampling
- Chow
- 1983
(Show Context)
Citation Context ..., and run the recursive quicksorts in parallel. His experiments showed zero speedup beyond 10 processors, because of the skew in the sizes of the arguments to the recursive quicksorts. Huang and Chow =-=[HC83]-=- consider parallel external sorting using approximate partitioning based upon sampling. In their algorithm, the samples are used for two purposes: first, to compute the approximate splitting vector; s... |

48 |
Samplesort: A Sampling Approach to Minimal Storage Tree Sorting
- Frazer, McKellar
- 1977
(Show Context)
Citation Context ...sorted. As noted previously, the idea of sorting using an approximate splitting vector obtained by sampling has been invented and re-invented many times in the sorting literature. Frazer and McKellar =-=[FM70]-=- proposed a uniprocessor internal sort based upon sampling. They showed that asymptotically, given some restrictions on the distribution of the data being sorted, their algorithm runs in linear time. ... |

47 | Parallel database systems. The future of database processing or a passing fad? ACMSIGMODRecor
- DeWitt, Gray
- 1990
(Show Context)
Citation Context ...arches of the index for tuples stored on other processors. 6 Results Scaleup and speedup are useful metrics for evaluating the performance of a parallel algorithm on a multiprocessor database machine =-=[DG90]-=-. Scaleup is an interesting metric for multiprocessor database machines as it indicates whether a constant response time can be maintained as the workload is increased by adding a proportional number ... |

45 |
Evaluation of Distribution Criteria for Distributed Database Systems
- Ries, Epstein
- 1978
(Show Context)
Citation Context ...entifiers of the records in the file that contain a matching value. One indexed attribute may be designated to be a clustering attribute for the file. In Gamma, relations are horizontally partitioned =-=[RE78]-=- (also known as declustering [LKB87]) 14 across all disk drives in order to increase the aggregate I/O bandwidth provided by the hardware. The query language of Gamma provides the user with several al... |

44 |
K.: Parallel algorithms for the execution of relational database operations
- Bitton, Boral, et al.
- 1983
(Show Context)
Citation Context ...ished faster than a sequential pass through the entire file. Beck et al. [BBW88] describes a similar sort. Other earlier algorithms that also used a final sequential merge phase include Bitton et al. =-=[BBDW83]-=-, and Valduriez and Gardarin [VG84]. Graefe [Gra90] presents a thorough description of the implementation and evaluation of a multiple-input, multiple-output sorting algorithm for a shared-memory mult... |

35 |
The Convoy Phenomenon
- Blasgen, Gray, et al.
- 1979
(Show Context)
Citation Context ...m designed speci cally for supporting database management systems. NOSE provides multiple, lightweight processes with shared memory. A non-preemptive scheduling policy is used to help prevent convoys =-=[BGMP79]-=- from occurring. NOSE provides communications between NOSE processes using the reliable message passing hardware of the Intel iPSC/2 hypercube. File services in NOSE are based on the Wisconsin Storage... |

31 | A benchmark of NonStop SQL Release 2 demonstrating near-linear speedup and scaleup on large databases
- Englert, Gray, et al.
- 1990
(Show Context)
Citation Context ...is an interesting metric because it indicates whether additional processors and disks result in a corresponding decrease in the response time of a query. A similar set of experiments were reported in =-=[EGKS89]-=- for equi-join queries on Release 2 of Tandem's NonStop SQL system, in [DGS + 90] for equi-join queries in Gamma, and in [DNS91] for non-equijoin queries in Gamma. Scaleup and speedup results for seve... |

24 |
Sampling issues in parallel database systems
- Seshadri, Naughton
- 1992
(Show Context)
Citation Context ...n the file to be sorted, then the average number of records per processor is N=k. Then by definition of s, the maximum number of records at any site is sN=k. By Theorem 7.1 from Seshadri and Naughton =-=[SN91]-=-, if we take a total of kn samples, then the probability p that any processor contains more than sN=k keys is at most p = ke \Gamma(1\Gamma1=s) 2 sn=2 Solving this for n gives n = 2 ln(k=p) (1 \Gamma ... |

17 |
Sorting Large Files on a Backend Multiprocessor
- Beck, Bitton, et al.
- 1988
(Show Context)
Citation Context ...wever, sort-merge is probably not appropriate in a multiple-input multiple-output sort, since the final merge cannot be accomplished faster than a sequential pass through the entire file. Beck et al. =-=[BBW88]-=- describes a similar sort. Other earlier algorithms that also used a final sequential merge phase include Bitton et al. [BBDW83], and Valduriez and Gardarin [VG84]. Graefe [Gra90] presents a thorough ... |

15 | FastSort: a distributed single-input singleoutput external sort - Salzberg - 1990 |

14 |
Parallel external sorting in Volcano
- Graefe
- 1990
(Show Context)
Citation Context ...re file. Beck et al. [BBW88] describes a similar sort. Other earlier algorithms that also used a final sequential merge phase include Bitton et al. [BBDW83], and Valduriez and Gardarin [VG84]. Graefe =-=[Gra90]-=- presents a thorough description of the implementation and evaluation of a multiple-input, multiple-output sorting algorithm for a shared-memory multiprocessor. This algorithm uses a splitting vector.... |

14 |
A Low Communications Sort Algorithm for a Parallel Database Machine
- Lorie, Young
- 1989
(Show Context)
Citation Context ...vector is formed. Baugst and Greipsland do not discuss how to analyze the quality of the resulting partitioning vector or how to determine what fraction of the file should be sampled. Lorie and Young =-=[LY89]-=- describe an algorithm for multiple-input, single-output sorting. A novel feature of their algorithm is that the final output phase is heavily overlapped with the internal sorting phase, which reduces... |

11 |
Sorting by distributive partitioning
- Dobosiewicz
- 1978
(Show Context)
Citation Context ...ing sorted, their algorithm runs in linear time. Distributive Sorting, a recursive bucket sort based upon sampling, has some similarities with sorting via probabilistic splitting as well. Dobosiewicz =-=[Dob78]-=- originally proposed the algorithm and proved that if the data is uniformly distributed, it runs in time O(n). Later, 2 Janus and Lamagna [JL85] proposed a fix to distributive sorting that attempts to... |

11 |
Percentile finding algorithm for multiple sorted runs
- Iyer, Ricard, et al.
- 1989
(Show Context)
Citation Context ...vs. measured performance. 3 Exact Splitting In this section we consider the algorithm proposed by Iyer et al. for finding an exact splitting vector. The complete algorithm is rather complex; see Iyer =-=[IRV89]-=- for details. We have adapted some of the steps for a shared-nothing multiprocessor, since the original algorithm was designed with shared memory in mind. Our goal in this section is to present enough... |

9 |
Parallel sorting algorithms for tightly coupled multiprocessors
- Quinn
- 1988
(Show Context)
Citation Context ...thm's performance using 31 partitions. The results showed that if the distribution was smooth, the algorithm performed well. They did not investigate a parallel implementation of the algorithm. Quinn =-=[Qui88]-=- has suggested implementing a parallel quicksort as follows: to sort on k processors, choose k \Gamma 1 pivot elements at random, and run the recursive quicksorts in parallel. His experiments showed z... |

7 |
A comparison of non-equijoin algorithms
- DeWitt, Naughton, et al.
- 1997
(Show Context)
Citation Context ...esponse time of a query. A similar set of experiments were reported in [EGKS89] for equi-join queries on Release 2 of Tandem's NonStop SQL system, in [DGS + 90] for equi-join queries in Gamma, and in =-=[DNS91]-=- for non-equijoin queries in Gamma. Scaleup and speedup results for several parallel sorting algorithms are contained in [Gra90] and in [STG + 90]. A third interesting metric for algorithm performance... |

6 |
An adaptive method for unknown distributions in distributive partitioned sorting
- Janus, Lamagna
- 1985
(Show Context)
Citation Context ...rting via probabilistic splitting as well. Dobosiewicz [Dob78] originally proposed the algorithm and proved that if the data is uniformly distributed, it runs in time O(n). Later, 2 Janus and Lamagna =-=[JL85]-=- proposed a fix to distributive sorting that attempts to provide better performance for non-uniform data distributions. They use sampling to compute an approximate cumulative distribution function for... |

6 |
Parallel Partition Sort for Database Machines
- Yamane, Take
- 1988
(Show Context)
Citation Context ... 1. In this paper we consider two fundamentally different approaches to determining the splitting vector. The first approach we consider is due to Iyer et al. (An earlier paper due to Yamane and Take =-=[YT88]-=- proposes a similar algorithm with the same asymptotic running time as the algorithm of Iyer et al. However, since the description in Yamane and Take is not sufficiently detailed to estimate anything ... |

3 |
Baugst and Jarle Fredrik Greipsland. Parallel Sorting Methods for Large Data Volumes on a Hypercube Database Computer
- W
- 1989
(Show Context)
Citation Context ...pivot elements. Using combinatorial analysis they present analytic formulas showing good asymptotic performance, but did not implement their approach or compare it to any other. Baugst and Greipsland =-=[BG89]-=- also consider parallel external sorting using approximate partitioning based upon synchronized sampling of sorted subfiles. In more detail, in their algorithm, they implement sampling as follows: fir... |

3 |
The convoy phenomenon. Operating System Review
- Blasgen, Gray, et al.
- 1979
(Show Context)
Citation Context ... designed specifically for supporting database management systems. NOSE provides multiple, lightweight processes with shared memory. A non-preemptive scheduling policy is used to help prevent convoys =-=[BGMP79]-=- from occurring. NOSE provides communications between NOSE processes using the reliable message passing hardware of the Intel iPSC/2 hypercube. File services in NOSE are based on the Wisconsin Storage... |

1 |
Sorting large les on a backend multiprocessor
- Bitton
- 1988
(Show Context)
Citation Context ...� however, sort-merge is probably not appropriate in a multiple-input multiple-output sort, since the nal merge cannot be accomplished faster than a sequential pass through the entire le. Beck et al. =-=[BBW88]-=- describes a similar sort. Other earlier algorithms that also used a nal sequential merge phase include Bitton et al. [BBDW83], and Valduriez and Gardarin [VG84]. Graefe [Gra90] presents a thorough de... |

1 |
Percentile nding algorithm for multiple sorted runs
- Iyer, Ricard, et al.
- 1989
(Show Context)
Citation Context ...c model predictions vs. measured performance. In this section we consider the algorithm proposed by Iyer et al. for nding an exact splitting vector. The complete algorithm is rather complex� see Iyer =-=[IRV89]-=- for details. Wehave adapted some of the steps for a shared-nothing multiprocessor, since the original algorithm was designed with shared memory in mind. Our goal in this section is to present enough ... |

1 |
and Honesty C.Young. A low communication sort algorithm for a parallel database machine
- Lorie
- 1989
(Show Context)
Citation Context ...g vector is formed. Baugst and Greipsland do not discuss how to analyze the quality of the resulting partitioning vector or how to determine what fraction of the le should be sampled. Lorie and Young =-=[LY89]-=- describe an algorithm for multiple-input, single-output sorting. A novel feature of their algorithm is that the nal output phase is heavily overlapped with the internal sorting phase, which reduces t... |