Abstract

Early aggregation is a technique for speeding up the processing of GROUP BY queries by reducing the amount of intermediate data transferred between main memory and disk. It can also be applied to duplicate elimination because duplicate elimination is equivalent to grouping with no aggregation functions. This paper describes six different algorithms for grouping and aggregation, shows how to incorporate early aggregation in each of them, and analyzes the resulting reduction in intermediate data. In addition to the grouping algorithm used, the reduction depends on several factors: the number of groups, the skew in group size distribution, the input size, and the amount of main memory available. All six algorithms considered benefit from early aggregation with grouping by hash partitioning producing the least amount of intermediate data. If the group size distribution is skewed, the overall reduction can be very significant, even with a modest amount of additional main memory.

Citations

1813	The Art of Computer Programming - Knuth - 1968
592	Query evaluation techniques for large databases - Graefe - 1993
58	Duplicate Record Elimination in Large Data Files - Bitton, Dewitt - 1983
39	Adaptive Parallel Aggregation Algorithms - Shatdal, Naughton - 1995
18	Sorting and searching in multisets - Munro, Spira - 1976
11	L.: Quicksort for Equal Keys - Wegner - 1985
2	Computational complexity of sorting and joining relations with duplicates - Abdelguerfi, Sood - 1991
2	Minimal space, average linear time duplicate deletion - Teuhola, Wegner - 1991
2	Data reduction through early grouping - Yan, Larson - 1994