Network traffic arises from the superposition of Origin-Destination (OD) flows. Hence, a thorough understanding of OD flows is essential for modeling network traffic, and for addressing a wide variety of problems including traffic engineering, traffic matrix estimation, capacity planning, forecasting and anomaly detection. However, to date, OD flows have not been closely studied, and there is very little known about their properties. We present

Using entropy of traffic distributions has been shown to aid a wide variety of network monitoring applications such as anomaly detection, clustering to reveal interesting patterns, and traffic classification. However, realizing this potential benefit in practice requires accurate algorithms that can operate on high-speed links, with low CPU and memory requirements. In this paper, we investigate the problem of estimating the entropy in a streaming computation model. We give lower bounds for this problem, showing that neither approximation nor randomization alone will let us compute the entropy e#ciently. We present two algorithms for randomly approximating the entropy in a time and space e#- cient manner, applicable for use on very high speed (greater than OC-48) links. The first algorithm for entropy estimation is inspired by the structural similarity with the seminal work of Alon et al. for estimating frequency moments, and we provide strong theoretical guarantees on the error and resource usage. Our second algorithm utilizes the observation that the performance of the streaming algorithm can be enhanced by separating the high-frequency items (or elephants) from the low-frequency items (or mice). We evaluate our algorithms on traffic traces from different deployment scenarios.

Confronted with the generalization of monitoring in operational networks, researchers have proposed placement algorithms that can help ISPs deploy their monitoring infrastructure in a cost effective way, while maximizing the benefits of their infrastructure. However, a static placement of monitors cannot be optimal given the short-term and longterm variations in traffic due to re-routing events, anomalies and the normal network evolution. In addition, most ISPs already deploy router embedded monitoring functionalities. Despite some limitations (inherent to being part of a router), these monitoring tools give greater visibility on the network traffic but raise the question on how to configure a networkwide monitoring infrastructure that may contain hundreds of monitoring points. We reformulate the placement problem as follows. Given a network where all links can be monitored, which monitors should be activated and which sampling rate should be set on these monitors in order to achieve a given measurement task with high accuracy and low resource consumption? We provide a formulation of the problem, an optimal algorithm to solve it, and we study its performance on a real backbone network. 1.

Critical network management applications increasingly demand fine-grained flow level measurements. However, current flow monitoring solutions are inadequate for many of these applications. In this paper, we present the design, implementation, and evaluation of CSAMP, a system-wide approach for flow monitoring. The design of CSAMP derives from three key ideas: flow sampling as a router primitive instead of uniform packet sampling; hash-based packet selection to achieve coordination without explicit communication; and a framework for distributing responsibilities across routers to achieve network-wide monitoring goals while respecting router resource constraints. We show that CSAMP achieves much greater monitoring coverage, better use of router resources, and enhanced ability to satisfy network-wide flow monitoring goals compared to existing solutions. 1

Abstract — Inverting flow properties from sampled traffic is known to be complex and prone to errors. Previous work has mainly focused on inverting general traffic properties such as flow size distribution, average flow size, or total number of flows. In this work, we study the feasibility of the inversion of individual flow properties. We address this problem by analyzing the detection and ranking of the largest flows from sampled traffic. Surprisingly, our analytical analysis indicates that a high sampling rate (10 % and even more) is required. To reduce the sampling rate by an order of magnitude, the ranking must be limited to just a few large flows, or the traffic must consist of several millions of flows. The sampling rate can also be reduced if one is not interested in the relative sizes of the largest flows but just aims at detecting them. We verify our analytical result with trace-driven sampling simulations. I.

Abstract. Sampling has become an integral part of passive network measurement. This role is driven by the need to control the consumption of resources in the measurement infrastructure under increasing traffic rates and the demand for detailed measurements from applications and service providers. Classical sampling methods play an important role in the current practice of Internet measurement. The aims of this review are (i) to explain the classical sampling methodology in the context of the Internet to readers who are not necessarily acquainted with either, (ii) to give an account of newer applications and sampling methods for passive measurement and (iii) to identify emerging areas that are ripe for the application of statistical expertise. Key words and phrases: Traffic measurement, network management, sampling methods, estimation, packets, flows.

Abstract — In this paper, we consider the problem of nonparametric estimation of network flow characteristics, namely packet lengths and byte sizes, based on sampled flow data. We propose two different approaches to deal with the problem at hand. The first one is based on single stage Bernoulli sampling of packets and their corresponding byte sizes. Subsequently, the flow length distribution is estimated by an adaptive expectationmaximization (EM) algorithm that in addition provides an estimate for the number of active flows. The estimation of the flow sizes (in bytes) is accomplished through a random effects regression model that utilizes the flow length information previously obtained. A variation of this approach, particularly suited for mixture distributions that appear in real network traces, is also considered. The second approach relies on a two-stage sampling procedure, which in the first stage samples flows amongst the active ones, while in the second stage samples packets from the sampled flows. Subsequently, the flow length distribution is estimated using another EM algorithm and the flow byte sizes based on a regression model. The proposed approaches are illustrated and compared on a number of synthetic and real data sets. I.

Packet sampling is commonly deployed in highspeed backbone routers to minimize resources used for network monitoring. It is known that packet sampling distorts traffic statistics and its impact has been extensively studied for traffic engineering metrics such as flow size and mean rate. However, it is unclear how packet sampling impacts anomaly detection, which has become increasingly critical to network providers. This paper is the first attempt to address this question by focusing on one common class of non-volume based anomalies, portscans, which are associated with worm/virus propagation. Existing portscan detection algorithms fall into two general approaches: targetspecific and traffic profiling. We evaluated representative algorithms for each class, namely (a) TRWSYN that performs stateful traffic analysis, (b) TAPS that tracks connection pattern of scanners, and (c) Entropy-based traffic profiling. We applied these algorithms to detect portscans in both the original and sampled packet traces from a Tier-1 provider’s backbone network. Our results demonstrate that sampling introduces fundamental bias that degrades the effectiveness of these detection algorithms and dramatically increases false positives. Through both experiments and analysis, we identify the traffic features critical for anomaly detection that are affected by sampling. Finally, using insight gained from this study, we show how portscan algorithms can be enhanced to be more robust to sampling.

Statistical summaries of traffic in IP networks are at the heart of network operation and are used to recover information on arbitrary subpopulations of flows. It is therefore of great importance to collect the most accurate and informative summaries given the router’s resource constraints. Cisco’s sampled NetFlow, based on aggregating a sampled packet stream into flows, is the most widely deployed such system. We observe two sources of inefficiency in current methods. Firstly, a single parameter (the sampling rate) is used to control utilization of both memory and processing/access speed, which means that it has to be set according to the bottleneck resource. Secondly, the unbiased estimators are applicable to summaries that in effect are collected through uneven use of resources during the measurement period (information from the earlier part of the measurement period is either not collected at all and fewer counter are utilized or discarded when performing a sampling rate adaptation). We develop algorithms that collect more informative summaries through an even and more efficient use of available resources. The heart of our approach is a novel derivation of unbiased estimators that use these more informative counts. We show how to efficiently compute these estimators and prove analytically that they are superior (have smaller variance on all packet streams and subpopulations) to previous approaches. Simulations on Pareto distributions and IP flow data show that the new summaries provide significantly more accurate estimates. We provide an implementation design that can be efficiently deployed at routers.

Abstract—Sampling technology has been widely deployed in measurement systems to control memory consumption and processing overhead. However, most of the existing sampling methods suffer from large estimation errors in analyzing small-size flows. To address the problem, we propose a novel adaptive non-linear sampling (ANLS) method for passive measurement. Instead of statically configuring the sampling rate, ANLS dynamically adjusts the sampling rate for a flow depending on the number of packets having been counted. We provide the generic principles guiding the selection of sampling function for sampling rate adjustment. Moreover, we derive the unbiased flow size estimation, the bound of the relative error, and the bound of required counter size for ANLS. The performance of ANLS is thoroughly studied through theoretic analysis and experiments under synthetic/real network data traces, with comparison to several related sampling methods. The results demonstrate that the proposed ANLS can significantly improve the estimation accuracy, particularly for small-size flows, while maintain a memory and processing overhead comparable to existing methods. I.