Hyperion: High Volume Stream Archival for Retrospective Querying

DMCA

Hyperion: High Volume Stream Archival for Retrospective Querying (2006)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Peter J. Desnoyers

Citations:

18 - 1 self

BibTeX

@TECHREPORT{Desnoyers06hyperion:high,
    author = {Peter J. Desnoyers},
    title = {Hyperion: High Volume Stream Archival for Retrospective Querying},
    institution = {},
    year = {2006}
}

OpenURL

Abstract

Network monitoring systems that support data archiving and after-the-fact (retrospective) queries are useful for a multitude of purposes, such as anomaly detection and network and security forensics. Data archiving for such systems, however, is complicated by (a) data arrival rates, which may be hundreds of thousands of packets per second on a single link, and (b) the need for online indexing of this data to support retrospective queries. At these data rates, both common database index structures and general-purpose file systems perform poorly. This paper describes Hyperion, a system for archiving, indexing, and on-line retrieval of high-volume data streams. We employ a write-optimized stream file system for high-speed storage of simultaneous data streams, and a novel use of signature file indexes in a distributed multi-level index. We implement Hyperion on commodity hardware and conduct a detailed evaluation using synthetic data and real network traces. Our streaming file system, StreamFS, is shown to be fast enough to archive traces at over a million packets per second. The index allows queries over hours of data to complete in as little as 10-20 seconds, and the entire system is able to index and archive over 200,000 packets/sec while processing simultaneous on-line queries. 1

Keyphrases

retrospective querying high volume stream archival write-optimized stream file system novel use 10-20 second simultaneous data stream high-volume data stream anomaly detection single link file system on-line retrieval detailed evaluation real network trace synthetic data general-purpose file system data arrival rate simultaneous on-line query implement hyperion entire system retrospective query distributed multi-level index high-speed storage common database index structure security forensics data rate signature file index online indexing commodity hardware

DMCA