DMCA
Applying Streaming Algorithms to Data at Rest
BibTeX
@MISC{Ishaque_applyingstreaming,
author = {Mashhood Ishaque and Ligia Nistor and Kevin Backhouse},
title = {Applying Streaming Algorithms to Data at Rest},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Guided navigation [5] is fundamental for the Oracle Big Data Discovery product. Our customers need to know the most frequent elements in their datasets, and the number of elements of a partic-ular kind. In Figure 1, for the winetype property, we use a query that returns the top 6 kinds of wines (‘Red’, ‘Cabernet Sauvignon’, ‘Burgundy Cote de Beaune’, ‘Pinot Noir’, ‘Merlot’, ‘White’) and a query that counts the number of distinct types of wines (‘371 others’). These queries can become extremely expensive, both in terms of space and time. In this paper we propose to use stream-ing algorithms for speeding up queries and lowering the memory usage. We detail our experience in applying two known streaming algorithms: the SpaceSaving algorithm [4] that has favorable math-ematical guarantees [1], and HyperLogLog [2, 3], to data at rest. The first algorithm is normally used for finding the most frequent items in a data stream, while the second is used for counting the number of distinct elements in a data stream. Our innovation is to apply these algorithms to hundreds of gigabytes of data at rest. Figure 1. Guided Navigation (multiple properties) We have implemented streaming algorithms on gigabytes of data at rest, with dramatic performance improvements. Our solution has the following advantages: we only sort a constant number of values (a few thousand); we give accurate counts when there is a small number of distinct values; we return approximate counts when there is a large number of distinct values, but with a provably small error; we use well studied algorithms that have been shown to perform well in practice; and streaming algorithms are usually embarrassingly parallel. The assumption in the case of streaming algorithms is that the data cannot be stored, and hence that it cannot be sorted. Our insight is that we can avoid sorting by running these algorithms on our data, even though our data can be stored (hence the phrase ‘data at rest’).
Keyphrases
data stream distinct value cabernet sauvignon constant number approximate count frequent element partic-ular kind small error large number multiple property following advantage burgundy cote spacesaving algorithm dramatic performance improvement first algorithm oracle big data discovery product distinct element data cannot pinot noir winetype property frequent item stream-ing algorithm small number accurate count phrase data memory usage favorable math-ematical guarantee distinct type
