Results 1  10
of
52
A nearoptimal algorithm for computing the entropy of a stream
 In ACMSIAM Symposium on Discrete Algorithms
, 2007
"... We describe a simple algorithm for approximating the empirical entropy of a stream of m values in a single pass, using O(ε −2 log(δ −1) log m) words of space. Our algorithm is based upon a novel extension of a method introduced by Alon, Matias, and Szegedy [1]. We show a space lower bound of Ω(ε −2 ..."
Abstract

Cited by 54 (20 self)
 Add to MetaCart
We describe a simple algorithm for approximating the empirical entropy of a stream of m values in a single pass, using O(ε −2 log(δ −1) log m) words of space. Our algorithm is based upon a novel extension of a method introduced by Alon, Matias, and Szegedy [1]. We show a space lower bound of Ω(ε −2 / log(ε −1)), meaning that our algorithm is near optimal in terms of its dependency on ε. This improves over previous work on this problem [8, 13, 17, 5]. We show that generalizing to kth order entropy requires close to linear space for all k ≥ 1, and give additive approximations using our algorithm. Lastly, we show how to compute a multiplicative approximation to the entropy of a random walk on an undirected graph. 1
Estimating entropy and entropy norm on data streams
 In Proceedings of the 23rd International Symposium on Theoretical Aspects of Computer Science (STACS
, 2006
"... Abstract. We consider the problem of computing information theoretic functions such as entropy on a data stream, using sublinear space. Our first result deals with a measure we call the “entropy norm ” of an input stream: it is closely related to entropy but is structurally similar to the wellstudi ..."
Abstract

Cited by 35 (4 self)
 Add to MetaCart
Abstract. We consider the problem of computing information theoretic functions such as entropy on a data stream, using sublinear space. Our first result deals with a measure we call the “entropy norm ” of an input stream: it is closely related to entropy but is structurally similar to the wellstudied notion of frequency moments. We give a polylogarithmic space onepass algorithm for estimating this norm under certain conditions on the input stream. We also prove a lower bound that rules out such an algorithm if these conditions do not hold. Our second group of results are for estimating the empirical entropy of an input stream. We first present a sublinear space onepass algorithm for this problem. For a stream of m items and a given real parameter α, our algorithm uses space�O(m 2α) and provides an approximation of 1/α in the worst case and (1 + ε) in “most ” cases. We then present a twopass polylogarithmic space (1+ε)approximation algorithm. All our algorithms are quite simple. 1
Estimating Entropy over Data Streams
 In ESA
, 2006
"... Abstract. We present an algorithm for estimating entropy of data streams consisting of insertion and deletion operations using Õ(1) space.1 1 ..."
Abstract

Cited by 25 (3 self)
 Add to MetaCart
Abstract. We present an algorithm for estimating entropy of data streams consisting of insertion and deletion operations using Õ(1) space.1 1
Sketching and Streaming Entropy via Approximation Theory
"... We conclude a sequence of work by giving nearoptimal sketching and streaming algorithms for estimating Shannon entropy in the most general streaming model, with arbitrary insertions and deletions. This improves on prior results that obtain suboptimal space bounds in the general model, and nearopti ..."
Abstract

Cited by 21 (0 self)
 Add to MetaCart
We conclude a sequence of work by giving nearoptimal sketching and streaming algorithms for estimating Shannon entropy in the most general streaming model, with arbitrary insertions and deletions. This improves on prior results that obtain suboptimal space bounds in the general model, and nearoptimal bounds in the insertiononly model without sketching. Our highlevel approach is simple: we give algorithms to estimate Rényi and Tsallis entropy, and use them to extrapolate an estimate of Shannon entropy. The accuracy of our estimates is proven using approximation theory arguments and extremal properties of Chebyshev polynomials, a technique which may be useful for other problems. Our work also yields the bestknown and nearoptimal additive approximations for entropy, and hence also for conditional entropy and mutual information.
Bayesian neural networks for internet traffic classification
 IEEE Transaction on Neural Networks
, 2007
"... Abstract—Internet traffic identification is an important tool for network management. It allows operators to better predict future traffic matrices and demands, security personnel to detect anomalous behavior, and researchers to develop more realistic traffic models. We present here a traffic classi ..."
Abstract

Cited by 19 (1 self)
 Add to MetaCart
Abstract—Internet traffic identification is an important tool for network management. It allows operators to better predict future traffic matrices and demands, security personnel to detect anomalous behavior, and researchers to develop more realistic traffic models. We present here a traffic classifier that can achieve a high accuracy across a range of application types without any source or destination hostaddress or port information. We use supervised machine learning based on a Bayesian trained neural network. Though our technique uses training data with categories derived from packet content, training and testing were done using features derived from packet streams consisting of one or more packet headers. By providing classification without access to the contents of packets, our technique offers wider application than methods that require full packet/payloads for classification. This is a powerful advantage, using samples of classified traffic to permit the categorization of traffic based only upon commonly available information. Index Terms—Internet traffic, network operations, neural network applications, pattern recognition, traffic identification.
Anomaly Extraction in Backbone Networks using Association Rules
, 2009
"... Anomaly extraction is an important problem essential to several applications ranging from root cause analysis, to attack mitigation, and testing anomaly detectors. Anomaly extraction is preceded by an anomaly detection step, which detects anomalous events and may identify a large set of possible ass ..."
Abstract

Cited by 13 (2 self)
 Add to MetaCart
Anomaly extraction is an important problem essential to several applications ranging from root cause analysis, to attack mitigation, and testing anomaly detectors. Anomaly extraction is preceded by an anomaly detection step, which detects anomalous events and may identify a large set of possible associated event flows. The goal of anomaly extraction is to find and summarize the set of flows that are effectively caused by the anomalous event. In this work, we use metadata provided by several histogrambased detectors to identify suspicious flows and then apply association rule mining to find and summarize the event flows. Using rich traffic data from a backbone network (SWITCH/AS559), we show that we can reduce the classification cost, in terms of items (flows or rules) that need to be classified, by several orders of magnitude. Further, we show that our techniques effectively isolate event flows in all analyzed cases and that on average trigger between 2 and 8.5 false positives, which can be trivially sorted out by an administrator.
ALADIN: Active Learning of Anomalies to Detect Intrusions, Microsoft Research
, 2008
"... i This page intentionally left blank. This paper proposes using active learning combined with rare class discovery and uncertainty identification to statistically train a network traffic classifier. For ingress traffic, a classifier can be trained for a network intrusion detection or prevention syst ..."
Abstract

Cited by 12 (1 self)
 Add to MetaCart
i This page intentionally left blank. This paper proposes using active learning combined with rare class discovery and uncertainty identification to statistically train a network traffic classifier. For ingress traffic, a classifier can be trained for a network intrusion detection or prevention system (IDS/IPS) while a classifier trained on egress traffic can detect malware on a corporate network. Active learning selects “interesting traffic ” to be shown to a security expert for labeling. Unlike previous statistical misuse or anomalydetectionbased approaches to training an IDS, active learning substantially reduces the number of labels required from an expert to reach an acceptable level of accuracy and coverage. Our system defines “interesting traffic ” in two ways, based on two goals for the system. The system is designed to discover new categories of traffic by showing examples of traffic for the analyst to label that do not fit a preexisting model of a known category of traffic. The system is also designed to accurately classify known categories of traffic by requesting labels for examples which it cannot classify with high certainty. Combining these two goals overcomes many problems associated with earlier anomalydetection based IDSs. Once trained, the system can be run as a fixed classifier with no further learning. Alternatively, it can continue to learn by labeling data on a particular network. In either case, the classifier is efficient enough to run in realtime for an IPS. We tested the system on the KDDCup99 Network Intrusion Detection dataset, where the algorithm identifies more rare classes with approximately half the number of labels required by previous active learning based systems. We have also used the algorithm to find previously unknown malware on a large corporate network from a set of firewall logs. 1
HistogramBased Traffic Anomaly Detection
"... Identifying network anomalies is essential in enterprise and provider networks for diagnosing events, like attacks or failures, that severely impact performance, security, and Service Level Agreements (SLAs). Featurebased anomaly detection models (ab)normal network traffic behavior by analyzing dif ..."
Abstract

Cited by 5 (1 self)
 Add to MetaCart
Identifying network anomalies is essential in enterprise and provider networks for diagnosing events, like attacks or failures, that severely impact performance, security, and Service Level Agreements (SLAs). Featurebased anomaly detection models (ab)normal network traffic behavior by analyzing different packet header features, like IP addresses and port numbers. In this work, we describe a new approach to featurebased anomaly detection that constructs histograms of different traffic features, models histogram patterns, and identifies deviations from the created models. We assess the strengths and weaknesses of many design options, like the utility of different features, the construction of feature histograms, the modeling and clustering algorithms, and the detection of deviations. Compared to previous featurebased anomaly detection approaches, our work differs by constructing detailed histogram models, rather than using coarse entropybased distribution approximations. We evaluate histogrambased anomaly detection and compare it to previous approaches using collected network traffic traces. Our results demonstrate the effectiveness of our technique in identifying a wide range of anomalies. The assessed technical details are generic and, therefore, we expect that the derived insights will be useful for similar future research efforts.
Hierarchical Sampling from Sketches: Estimating Functions over Data Streams
"... Abstract. We present a randomized procedure named Hierarchical Sampling from Sketches (HSS) that can be used for estimating a class of functions over the frequency vector f of update streams of the form Ψ(S) = ∑n i=1 ψ(fi). We illustrate this by applying the HSS technique to design nearly spaceo ..."
Abstract

Cited by 3 (1 self)
 Add to MetaCart
Abstract. We present a randomized procedure named Hierarchical Sampling from Sketches (HSS) that can be used for estimating a class of functions over the frequency vector f of update streams of the form Ψ(S) = ∑n i=1 ψ(fi). We illustrate this by applying the HSS technique to design nearly spaceoptimal algorithms for estimating the pth moment of the frequency vector, for real p ≥ 2 and for estimating the entropy of a data stream. 3 1
Entropy Based Adaptive Flow Aggregation
"... Abstract—Internet traffic flow measurement is vitally important for network management, accounting and performance studies. Cisco’s NetFlow is a widely deployed flow measurement solution that uses a configurable static sampling rate to control processor and memory usage on the router and the amount ..."
Abstract

Cited by 3 (0 self)
 Add to MetaCart
Abstract—Internet traffic flow measurement is vitally important for network management, accounting and performance studies. Cisco’s NetFlow is a widely deployed flow measurement solution that uses a configurable static sampling rate to control processor and memory usage on the router and the amount of reporting flow records generated. But during flooding attacks the memory and network bandwidth consumed by flow records can increase beyond what is available. Currently available countermeasures have their own problems: (1) reject new flows when the cache is full some legitimate new flows will not be counted; (2) export notterminated flows to make room for new onesthis will exhaust the export bandwidth; (3) adapt the sampling rate to traffic rate this will reduce the overall accuracy of accounting, including legitimate flows. In this paper, we propose an entropy based adaptive flow aggregation algorithm. Relying on informationtheoretic techniques, the algorithm efficiently identifies the clusters of attack flows in real time and aggregates those large number of short attack flows into a few metaflows. Compared to currently available solutions, our solution not only alleviates the problem in memory and export bandwidth, but also significantly improves the accuracy of legitimate flows. Finally, we evaluate our system using both synthetic trace file and real trace files from the Internet. Index Terms—Traffic measurement, Network monitoring, Data summarization, Information theory.