## An Information-Theoretic Approach to Traffic Matrix Estimation (2003)

### Cached

### Download Links

Venue: | In Proc. ACM SIGCOMM |

Citations: | 119 - 13 self |

### BibTeX

@INPROCEEDINGS{Zhang03aninformation-theoretic,

author = {Yin Zhang and Matthew Roughan and Carsten Lund and David Donoho},

title = {An Information-Theoretic Approach to Traffic Matrix Estimation},

booktitle = {In Proc. ACM SIGCOMM},

year = {2003},

pages = {301--312}

}

### Years of Citing Articles

### OpenURL

### Abstract

Traffic matrices are required inputs for many IP network management

### Citations

1652 | Atomic decomposition by basis pursuit
- Chen, Donoho, et al.
- 2001
(Show Context)
Citation Context ...ossible if we can exploit one of the main properties of routing matrices: they are very sparse — the proportion of exact zero entries in each column and row is overwhelming. Accordingly, we use PDSCO =-=[5]-=-, a MATLAB package developed by Michael Saunders of Stanford University, which has been highly optimized to solve problems with sparse matrices A. PDSCO has been used (see e.g. [5]) to solve problems ... |

620 | Measuring ISP Topologies with Rocketfuel
- Spring, Mahajan, et al.
(Show Context)
Citation Context ...backbone ISP. The results show that the algorithm is fast, and accurate for point-to-point traffic matrix estimation. We also test the algorithm on topologies generated through the Rocketfuel project =-=[21, 14, 22]-=- to resemble alternative ISPs, providing useful insight into where the algorithm willwork well. One interesting side result is that there is a relationship between the network traffic and topology th... |

529 |
Theory of Point Estimation
- Lehmann, Casella
- 1998
(Show Context)
Citation Context ...e if we shrink towards some other fixed point, though it is better to shrink towards a point close to the x we are trying to estimate. For a fuller discussion of shrinkage estimation, see for example =-=[13, 6]-=-. For now, simply note that shrinkage of a very high-dimensional estimand towards a chosen point can be helpful. Note that no Bayesian assumption is being made here: whatever the underlying estimand m... |

426 |
Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion
- Hansen
- 1998
(Show Context)
Citation Context ... regularization functional matched the properties of the estimand. These are generally called strategies for regularization of ill-posed problems (for a more general description of regularization see =-=[11]-=-). A general approach to deriving such regularization ideas is the Bayesian approach (such as used in [23]), where we model the estimand x as being drawn at random from a so-called ‘prior’ probability... |

253 | Deriving traffic demands for operational IP networks: Methodology and experience
- Feldmann, Greenberg, et al.
- 2001
(Show Context)
Citation Context ...distribute to lists, requires prior specific permission and/or a fee. SIGCOMM’03, August 25–29, 2003, Karlsruhe, Germany. Copyright 2003 ACM 1-58113-735-4/03/0008 ...$5.00. links, or even IP prefixes =-=[8]-=-. The finer grained traffic matrices are generally more useful, for example, in the analysis of the reliability of a network under a component failure. During a failure, IP traffic is rerouted to find... |

208 | Network tomography: Estimating source-destination traffic intensities from link data - Vardi - 1996 |

192 | Regularization tools: a Matlab package for analysis and solution of discrete illposed problems
- Hansen
- 1994
(Show Context)
Citation Context ...formulation has been used in solving many other ill-posed problems, and so there exist publicly available software in Matlab (such as routine MaxEnt in Per Christian Hansen’s Inverse Problems Toolbox =-=[9, 10]-=-) to solve small-scale variants of such problems. Our problems are, however, large in scale and not suited to such basic implementations. The problem of solving such large-scale traffic matrices is on... |

179 | Ill-posed problems in early vision
- Bertero, Poggio, et al.
- 1987
(Show Context)
Citation Context ...— for large N the problem becomes massively underconstrained. There is extensive experience with ill-posed linear inverse problems from fields as diverse as seismology, astronomy, and medical imaging =-=[1, 2, 17, 18, 26]-=-, all leading to the conclusion that some sort of side information must be brought in, producing a result which may be good or bad depending on the quality of this information. All of the previous wor... |

172 | Traffic matrix estimation: Existing techniques and new directions
- Medina, Taft, et al.
- 2002
(Show Context)
Citation Context ...reliability analysis. However, it is difficult to measure these matrices directly, and so there is interest in inferring traffic matrices from link load statistics and other more easily measured data =-=[24, 23, 3, 16, 28]-=-. Traffic matrices may be estimated or measured at varying levels of detail [15]: between Points-of-Presence (PoPs) [16], routers [28], Permission to make digital or hard copies of all or part of this... |

146 | Fast accurate computation of large-scale IP traffic matrices from link loads
- Zhang, Roughan, et al.
- 2003
(Show Context)
Citation Context ...reliability analysis. However, it is difficult to measure these matrices directly, and so there is interest in inferring traffic matrices from link load statistics and other more easily measured data =-=[24, 23, 3, 16, 28]-=-. Traffic matrices may be estimated or measured at varying levels of detail [15]: between Points-of-Presence (PoPs) [16], routers [28], Permission to make digital or hard copies of all or part of this... |

129 | NetScope: Traffic engineering for IP networks
- Feldmann, Greenberg, et al.
- 2000
(Show Context)
Citation Context ...edict the results of these distributed computations from data gathered from router configuration files, or a route monitor such as [19]. In our investigation, we employ a routing simulator such as in =-=[7]-=- thatmakes use of this routing information to compute a routing matrix. We also simulate load balancing across multiple shortest paths. 2.2 Traffic Data In IP networks today, link load measurements a... |

111 |
L.,1977: "Understanding Inverse Theory
- Parker
(Show Context)
Citation Context ...— for large N the problem becomes massively underconstrained. There is extensive experience with ill-posed linear inverse problems from fields as diverse as seismology, astronomy, and medical imaging =-=[1, 2, 17, 18, 26]-=-, all leading to the conclusion that some sort of side information must be brought in, producing a result which may be good or bad depending on the quality of this information. All of the previous wor... |

107 | Inferring link weights using end-to-end measurements
- Mahajan, Spring, et al.
- 2002
(Show Context)
Citation Context ...backbone ISP. The results show that the algorithm is fast, and accurate for point-to-point traffic matrix estimation. We also test the algorithm on topologies generated through the Rocketfuel project =-=[21, 14, 22]-=- to resemble alternative ISPs, providing useful insight into where the algorithm willwork well. One interesting side result is that there is a relationship between the network traffic and topology th... |

87 |
The axioms of maximum entropy
- Skilling
- 1988
(Show Context)
Citation Context ...is, we choose the solution which maintains the most uncertainty while satisfying the constraints. The principle can also be derived directly from some simple axioms which we wish the solution to obey =-=[20]-=-. 2.4 Ill-Posed Linear Inverse Problems Many scientific and engineering problems have to solve inference problems which can be posed as follows. We observe data y which are thought to follow a system ... |

83 | Solving ill-conditioned and singular linear systems: A tutorial on regularization - Neumaier - 1998 |

82 | Bayesian inference on network traffic using link count data
- Tebaldi, West
- 1998
(Show Context)
Citation Context ...reliability analysis. However, it is difficult to measure these matrices directly, and so there is interest in inferring traffic matrices from link load statistics and other more easily measured data =-=[24, 23, 3, 16, 28]-=-. Traffic matrices may be estimated or measured at varying levels of detail [15]: between Points-of-Presence (PoPs) [16], routers [28], Permission to make digital or hard copies of all or part of this... |

61 | Pseudo Likelihood Estimation in Network Tomography
- Liang, Yu
(Show Context)
Citation Context ...er the comparison with maximum likelihood approaches ([24, 3, 16]) to future work, because scaling these methods to the size of problem described here requires additional techniques (for instance see =-=[4, 27]-=-) that have only recently been developed. The point of interest here is that the MMI principle above produces (an approximation of) the algorithm previously derived from an initial gravity model solut... |

60 |
B.Yu. Time-Varying network tomography
- Cao, Davis, et al.
- 2000
(Show Context)
Citation Context |

38 | A Case Study of OSPF Behavior in a Large Enterprise Network
- Shaikh, Isett, et al.
- 2002
(Show Context)
Citation Context ... build the forwarding tables within each router. It is possible to predict the results of these distributed computations from data gathered from router configuration files, or a route monitor such as =-=[19]-=-. In our investigation, we employ a routing simulator such as in [7] thatmakes use of this routing information to compute a routing matrix. We also simulate load balancing across multiple shortest pa... |

34 |
The measurement manifesto
- Varghese, Estan
- 2003
(Show Context)
Citation Context ...timates. We examine two possibilities: the first (suggested in [16]) is to make direct measurements of some rows of the traffic matrix, the second is to measure local traffic matrices as suggested in =-=[25]-=-. Both result in improvements in accuracy, however, we found in contrast to [16] that the order in which rows of the traffic matrix are included does matter — adding rows in order of the largest row s... |

30 |
Stein’s paradox in statistics
- Efron, Morris
- 1977
(Show Context)
Citation Context ...e if we shrink towards some other fixed point, though it is better to shrink towards a point close to the x we are trying to estimate. For a fuller discussion of shrinkage estimation, see for example =-=[13, 6]-=-. For now, simply note that shrinkage of a very high-dimensional estimand towards a chosen point can be helpful. Note that no Bayesian assumption is being made here: whatever the underlying estimand m... |

27 | A scalable method for estimating network traffic matrices," Bell Labs
- Cao, Wiel, et al.
- 2001
(Show Context)
Citation Context ...er the comparison with maximum likelihood approaches ([24, 3, 16]) to future work, because scaling these methods to the size of problem described here requires additional techniques (for instance see =-=[4, 27]-=-) that have only recently been developed. The point of interest here is that the MMI principle above produces (an approximation of) the algorithm previously derived from an initial gravity model solut... |

23 |
Relative Information
- Jumarie
- 1990
(Show Context)
Citation Context ...present limitations in vendor implementations prevent collection of this data from the entire network. 2.3 Information Theory Information theory is of course a standard tool in communications systems =-=[12]-=-, but a brief review will set up our terminology. We begin with basic probabilistic notation: we define pX(x) to mean the probability that a random variable X is equal to x. We shall typically abuse t... |

14 |
Inverse problems in astronomy: a guide to inversion strategies for remotely sensed data. Adam Hilger Ltd
- Craig, Brown
- 1986
(Show Context)
Citation Context ...— for large N the problem becomes massively underconstrained. There is extensive experience with ill-posed linear inverse problems from fields as diverse as seismology, astronomy, and medical imaging =-=[1, 2, 17, 18, 26]-=-, all leading to the conclusion that some sort of side information must be brought in, producing a result which may be good or bad depending on the quality of this information. All of the previous wor... |

14 |
A taxonomy of IP traffic matrices
- MEDINA, FRALEIGH, et al.
- 2002
(Show Context)
Citation Context ... is interest in inferring traffic matrices from link load statistics and other more easily measured data [24, 23, 3, 16, 28]. Traffic matrices may be estimated or measured at varying levels of detail =-=[15]-=-: between Points-of-Presence (PoPs) [16], routers [28], Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies ... |

7 |
Constrained regularization for ill posed linear operator equations, with applications in meteorology and medicine
- Wahba
- 1982
(Show Context)
Citation Context |

6 |
Rocketfuel: An ISP topology mapping engine. http://www.cs. washigton.edu/research/networking/rocketfuel
- Spring, Mahajan, et al.
(Show Context)
Citation Context ...backbone ISP. The results show that the algorithm is fast, and accurate for point-to-point traffic matrix estimation. We also test the algorithm on topologies generated through the Rocketfuel project =-=[21, 14, 22]-=- to resemble alternative ISPs, providing useful insight into where the algorithm willwork well. One interesting side result is that there is a relationship between the network traffic and topology th... |

2 | Regularization tools (for Matlab). http: //www.imm.dtu.dk/˜pch/Regutools/index.html - Hansen |

2 | Statistical Decision Theory and Related Topics III, volume 2, chapter Constrained regularization for ill posed linear operator equations, with applications in meteorology and medicine - Wahba - 1982 |

1 | Rocketfuel: An ISP topology mapping engine.http://www.cs. washigton.edu/research/networking/rocketfuel - Spring, Mahajan, et al. |