Results 1  10
of
57
Diagnosing NetworkWide Traffic Anomalies
 In ACM SIGCOMM
, 2004
"... Anomalies are unusual and significant changes in a network's traffic levels, which can often span multiple links. Diagnosing anomalies is critical for both network operators and end users. It is a difficult problem because one must extract and interpret anomalous patterns from large amounts of high ..."
Abstract

Cited by 256 (16 self)
 Add to MetaCart
Anomalies are unusual and significant changes in a network's traffic levels, which can often span multiple links. Diagnosing anomalies is critical for both network operators and end users. It is a difficult problem because one must extract and interpret anomalous patterns from large amounts of highdimensional, noisy data.
Mining anomalies using traffic feature distributions
 In ACM SIGCOMM
, 2005
"... The increasing practicality of largescale flow capture makes it possible to conceive of traffic analysis methods that detect and identify a large and diverse set of anomalies. However the challenge of effectively analyzing this massive data source for anomaly diagnosis is as yet unmet. We argue tha ..."
Abstract

Cited by 228 (9 self)
 Add to MetaCart
The increasing practicality of largescale flow capture makes it possible to conceive of traffic analysis methods that detect and identify a large and diverse set of anomalies. However the challenge of effectively analyzing this massive data source for anomaly diagnosis is as yet unmet. We argue that the distributions of packet features (IP addresses and ports) observed in flow traces reveals both the presence and the structure of a wide range of anomalies. Using entropy as a summarization tool, we show that the analysis of feature distributions leads to significant advances on two fronts: (1) it enables highly sensitive detection of a wide range of anomalies, augmenting detections by volumebased methods, and (2) it enables automatic classification of anomalies via unsupervised learning. We show that using feature distributions, anomalies naturally fall into distinct and meaningful clusters. These clusters can be used to automatically classify anomalies and to uncover new anomaly types. We validate our claims on data from two backbone networks (Abilene and Géant) and conclude that feature distributions show promise as a key element of a fairly general network anomaly diagnosis framework.
Profiling internet backbone traffic: Behavior models and applications
 In ACM Sigcomm
, 2005
"... Abstract — Recent spates of cyberattacks and frequent emergence of applications affecting Internet traffic dynamics have made it imperative to develop effective techniques that can extract, and make sense of, significant communication patterns from Internet traffic data for use in network operation ..."
Abstract

Cited by 121 (12 self)
 Add to MetaCart
Abstract — Recent spates of cyberattacks and frequent emergence of applications affecting Internet traffic dynamics have made it imperative to develop effective techniques that can extract, and make sense of, significant communication patterns from Internet traffic data for use in network operations and security management. In this paper, we present a general methodology for building comprehensive behavior profiles of Internet backbone traffic in terms of communication patterns of endhosts and services. Relying on data mining and informationtheoretic techniques, the methodology consists of significant cluster extraction, automatic behavior classification and structural modelling for indepth interpretive analyses. We validate our methodology using data sets from the core of the Internet. Our results demonstrate that it indeed can identify common traffic profiles as well as anomalous behavior patterns that are of interest to network operators and security analysts. I.
Combining filtering and statistical methods for anomaly detection
 In Proceedings of IMC
, 2005
"... In this work we develop an approach for anomaly detection for large scale networks such as that of an enterprize or an ISP. The traffic patterns we focus on for analysis are that of a networkwide view of the traffic state, called the traffic matrix. In the first step a Kalman filter is used to filt ..."
Abstract

Cited by 69 (14 self)
 Add to MetaCart
In this work we develop an approach for anomaly detection for large scale networks such as that of an enterprize or an ISP. The traffic patterns we focus on for analysis are that of a networkwide view of the traffic state, called the traffic matrix. In the first step a Kalman filter is used to filter out the “normal ” traffic. This is done by comparing our future predictions of the traffic matrix state to an inference of the actual traffic matrix that is made using more recent measurement data than those used for prediction. In the second step the residual filtered process is then examined for anomalies. We explain here how any anomaly detection method can be viewed as a problem in statistical hypothesis testing. We study and compare four different methods for analyzing residuals, two of which are new. These methods focus on different aspects of the traffic pattern change. One focuses on instantaneous behavior, another focuses on changes in the mean of the residual process, a third on changes in the variance behavior, and a fourth examines variance changes over multiple timescales. We evaluate and compare all of these methods using ROC curves that illustrate the full tradeoff between false positives and false negatives for the complete spectrum of decision thresholds. 1
Network anomography
 In IMC
, 2005
"... Anomaly detection is a first and important step needed to respond to unexpected problems and to assure high performance and security in IP networks. We introduce a framework and a powerful class of algorithms for network anomography, the problem of inferring networklevel anomalies from widely avail ..."
Abstract

Cited by 60 (13 self)
 Add to MetaCart
Anomaly detection is a first and important step needed to respond to unexpected problems and to assure high performance and security in IP networks. We introduce a framework and a powerful class of algorithms for network anomography, the problem of inferring networklevel anomalies from widely available data aggregates. The framework contains novel algorithms, as well as a recently published approach based on Principal Component Analysis (PCA). Moreover, owing to its clear separation of inference and anomaly detection, the framework opens the door to the creation of whole families of new algorithms. We introduce several such algorithms here, based on ARIMA modeling, the Fourier transform, Wavelets, and Principal Component Analysis. We introduce a new dynamic anomography algorithm, which effectively tracks routing and traffic change, so as to alert with high fidelity on intrinsic changes in networklevel traffic, yet not on internal routing changes. An additional benefit of dynamic anomography is that it is robust to missing data, an important operational reality. To the best of our knowledge, this is the first anomography algorithm that can handle routing changes and missing data. To evaluate these algorithms, we used several months of traffic data collected from the Abilene network and from a large Tier1 ISP network. To compare performance, we use the methodology put forward earlier for the Abilene data set. The findings are encouraging. Among the new algorithms introduced here, we see: high accuracy in detection (few false negatives and few false positives), and high robustness (little performance degradation in the presence of measurement noise, missing data and routing changes). 1.
Traffic Matrix Reloaded: Impact of Routing Changes
 in Proc. Passive and Active Measurement Workshop, March/April
, 2005
"... this paper, we investigate the causes of the traffic matrix variations. Identifying the reasons for these disruptions is an essential step toward predicting and planning for their occurrence, reacting to them more effectively, or avoiding them entirely. The traffic matrix is the composition of the t ..."
Abstract

Cited by 34 (12 self)
 Add to MetaCart
this paper, we investigate the causes of the traffic matrix variations. Identifying the reasons for these disruptions is an essential step toward predicting and planning for their occurrence, reacting to them more effectively, or avoiding them entirely. The traffic matrix is the composition of the traffic demands and the egress point selection. We represent the traffic demands during a time interval t as a matrix V , where each element V (i, p, t) represents the volume of traffic entering at ingress router i and headed toward a destination prefix p. Each ingress router selects the egress point for each destination prefix using the Border Gateway Protocol (BGP). We represent the BGP routing choice as a mapping # from a prefix to an egress point, where #(i, p, t) represents the egress router chosen by ingress router i for sending traffic toward destination p. At time t each element of the traffic matrix M is defined as: M(i, e, t) = # p#P :#(i,p,t)=e V (i, p, t). (1) where P is the set of all destination prefixes. Figure 1 presents a simple network with one ingress router i, two egress routers e and e , and two external destination prefixes p 1 and p 2 . Given traffic demands V (i, p 1 , t) and V (i, p 2 , t) and a prefixtoegress mapping #(i, p 1 , t) = #(i, p 2 , t) = e, the traffic matrix for this network is M(i, e, t) = V (i, p 1 , t) + V (i, p 2 , t) and M(i, e , t) = 0. e i e' p1 V(i,p2,t) V(i,p1,t) p2 V(i,p1,t) + V(i,p2,t) TM(i,e,t) = Fig. 1. Example of traffic matrix
Unveiling Core NetworkWide Communication Patterns through Application Traffic Activity Graph Decomposition
"... As Internet communications and applications become more complex, operating, managing and securing networks have become increasingly challenging tasks. There are urgent demands for more sophisticated techniques for understanding and analyzing the behavioral characteristics of network traffic. In this ..."
Abstract

Cited by 23 (7 self)
 Add to MetaCart
As Internet communications and applications become more complex, operating, managing and securing networks have become increasingly challenging tasks. There are urgent demands for more sophisticated techniques for understanding and analyzing the behavioral characteristics of network traffic. In this paper, we study the network traffic behaviors using traffic activity graphs (TAGs), which capture the interactions among hosts engaging in certain types of communications and their collective behavior. TAGs derived from real network traffic are large, sparse, yet seemingly complex and richly connected, therefore difficult to visualize and comprehend. In order to analyze and characterize these TAGs, we propose a novel statistical traffic graph decomposition technique based on orthogonal nonnegative matrix trifactorization (tNMF) to decompose and extract the core host interaction patterns and other structural properties. Using the real network traffic traces, we demonstrate that our tNMFbased graph decomposition technique produces meaningful and interpretable results. It enables us to characterize and quantify the key structural properties of large and sparse TAGs associated with various applications, and study their formation and evolution.
The many facets of Internet topology and traffic
 Networks and Heterogeneous Media
"... ABSTRACT. The Internet’s layered architecture and organizational structure give rise to a number of different topologies, with the lower layers defining more physical and the higher layers more virtual/logical types of connectivity structures. These structures are very different, and successful Inte ..."
Abstract

Cited by 18 (10 self)
 Add to MetaCart
ABSTRACT. The Internet’s layered architecture and organizational structure give rise to a number of different topologies, with the lower layers defining more physical and the higher layers more virtual/logical types of connectivity structures. These structures are very different, and successful Internet topology modeling requires annotating the nodes and edges of the corresponding graphs with information that reflects their networkintrinsic meaning. These structures also give rise to different representations of the traffic that traverses the heterogeneous Internet, and a traffic matrix is a compact and succinct description of the traffic exchanges between the nodes in a given connectivity structure. In this paper, we summarize recent advances in Internet research related to (i) inferring and modeling the routerlevel topologies of individual service providers (i.e., the physical connectivity structure of an ISP, where nodes are routers/switches and links represent physical connections), (ii) estimating the intraAS traffic matrix when the AS’s routerlevel topology and routing configuration are known, (iii) inferring and modeling the Internet’s ASlevel topology, and (iv) estimating the interAS traffic matrix. We will also discuss recent work on Internet connectivity structures that arise at the higher layers in the TCP/IP protocol stack and are more virtual and dynamic; e.g., overlay networks like the WWW graph, where nodes are web pages and edges represent existing hyperlinks, or P2P networks like Gnutella, where nodes represent peers and two peers are connected if they have an active network connection. 1. Introduction. The
Coresets and Sketches for High Dimensional Subspace Approximation Problems ∗
"... We consider the problem of approximating a set P of n points in R d by a jdimensional subspace under the ℓp measure, in which we wish to minimize the sum of ℓp distances from each point of P to this subspace. More generally, the Fq(ℓp)subspace approximation problem asks for a jsubspace that minim ..."
Abstract

Cited by 12 (7 self)
 Add to MetaCart
We consider the problem of approximating a set P of n points in R d by a jdimensional subspace under the ℓp measure, in which we wish to minimize the sum of ℓp distances from each point of P to this subspace. More generally, the Fq(ℓp)subspace approximation problem asks for a jsubspace that minimizes the sum of qth powers of ℓpdistances to this subspace, up to a multiplicative factor of (1 + ɛ). We develop techniques for subspace approximation, regression, and matrix approximation that can be used to deal with massive data sets in high dimensional spaces. In particular, we develop coresets and sketches, i.e. small space representations that approximate the input point set P with respect to the subspace approximation problem. Our results are: • A dimensionality reduction method that can be applied to Fq(ℓp)clustering and shape fitting problems, such as those in [8, 15]. • The first strong coreset for F1(ℓ2)subspace approximation in highdimensional spaces, i.e. of size polynomial in the dimension of the space. This coreset approximates the distances to any jsubspace (not just the optimal one). • A (1 + ɛ)approximation algorithm for the jdimensional F1(ℓ2)subspace approximation problem with running time nd(j/ɛ) O(1) + (n + d)2 poly(j/ɛ). • A streaming algorithm that maintains a coreset for the F1(ℓ2)subspace approximation problem and uses a space log n
ANTIDOTE: Understanding and Defending against Poisoning of Anomaly Detectors
"... Statistical machine learning techniques have recently garnered increased popularity as a means to improve network design and security. For intrusion detection, such methods build a model for normal behavior from training data and detect attacks as deviations from that model. This process invites adv ..."
Abstract

Cited by 12 (3 self)
 Add to MetaCart
Statistical machine learning techniques have recently garnered increased popularity as a means to improve network design and security. For intrusion detection, such methods build a model for normal behavior from training data and detect attacks as deviations from that model. This process invites adversaries to manipulate the training data so that the learned model fails to detect subsequent attacks. We evaluate poisoning techniques and develop a defense, in the context of a particular anomaly detector—namely the PCAsubspace method for detecting anomalies in backbone networks. For three poisoning schemes, we show how attackers can substantially increase their chance of successfully evading detection by only adding moderate amounts of poisoned data. Moreover such poisoning throws off the balance between false positives and false negatives thereby dramatically reducing the efficacy of the detector. To combat these poisoning activities, we propose an antidote based on techniques from robust statistics and present a new robust PCAbased detector. Poisoning has little effect on the robust model, whereas it significantly distorts the model produced by the original PCA method. Our technique substantially reduces the effectiveness of poisoning for a variety of scenarios and indeed maintains a significantly better balance between false positives and false negatives than the original method when under attack.