## Network tomography: recent developments (2004)

### Cached

### Download Links

Venue: | Statistical Science |

Citations: | 85 - 4 self |

### BibTeX

@ARTICLE{Castro04networktomography:,

author = {Rui Castro and Mark Coates and Gang Liang and Robert Nowak and Bin Yu},

title = {Network tomography: recent developments},

journal = {Statistical Science},

year = {2004},

volume = {19},

pages = {499--517}

}

### Years of Citing Articles

### OpenURL

### Abstract

Today's Int ernet is a massive, dist([/#][ net work which cont inuest o explode in size as ecommerce andrelatH actH]M/# grow. Thehet([H(/#]H( and largelyunregulatS stregula of t/ Int/HH3 renderstnde such as dynamicroutc/[ opt2]3fl/ service provision, service level verificatflH( and det(2][/ of anomalous/malicious behaviorext/[(22 challenging. The problem is compounded bytS fact tct onecannot rely ont[ cooperatH2 of individual servers and routSS t aid intS collect[3 of net workt/[S measurement vits fort/]3 t/]3] In many ways, net workmonit]/#[ and inference problems bear a st[fl[ resemblancet otnc "inverse problems" in which key aspect of asystfl are not direct/ observable. Familiar signal processing orst[]23/#[S problems such ast omographic imagereconst[/#[S] and phylogenet# tog identn/HH2[M have int erest3/ connect[HU t tonn arising in net working. This artflMM int/ ducesnet workt/H3]S]/ y, a new field which we believe will benefit greatU from tm wealt of stH2](/#S( ttH2 andalgorit#S( It focuses especially on recent development s int2 field includingtl applicat[fl of pseudolikelihoodmetfl ds andt reeestfl3](/# formulat]M23 Keyw ords:Net workt/HflS33/ y, pseudo-likelihood,t opology identn/]H22(/ tn est/]H tst 1 Introducti6 Nonet work is an island, ent/S ofitS[S] everynet work is a piece of an int/]SS work, a part of t/ main . Alt[]][ administHSHSS of small-scale net works can monit( localt ra#ccondit][/ and ident ify congest/# point s and performance botU((2/ ks, very few net works are complet/# # Rui Castroan Robert Nowak are with theDepartmen t of Electricalan ComputerEnterX Rice Unc ersity,Houston TX; Mark Coates is with the Departmen t of Electricalan ComputerEnterX McGill UnG ersity,Mon treal, Quebec,Can Gan Lian an Bin Yu are with theDepartmen t of Statistics,...

### Citations

1767 | On the self-similar nature of Ethernet traffic - Leland, Taqqu, et al. - 1993 |

1224 |
Monte Carlo sampling methods using Markov chains and their applications
- Hastings
- 1970
(Show Context)
Citation Context ...robability zero. Denote by {si : i ∈ IN} the Markov chain constructed. It can be easily shown that the chain {si} is irreducible. 33sUsing a generalization of the Metropolis algorithm due to Hasting=-=s [48] w-=-e construct another Markov Chain in order to obtain a chain whose unique limit distribution (and also unique stationary distribution) is precisely L ′ (x|T ). Thus if we sample from this chain for a... |

1142 |
Spatial interaction and the statistical analysis of lattice systems
- BESAG
- 1974
(Show Context)
Citation Context ...the computational burden but maintains good statistical efficiency. The idea of modifying likelihood is not new, and many modified likelihood models have been proposed. For example, pseudo-likelihood =-=[25, 26]-=- for Markov Random Field (MRF) by Besag (1974), partial likelihood [27] for hazards regression by Cox (1973), and quasi-maximum likelihood [28] for finance models by White (1994). In this section, we ... |

909 | Monte Carlo Statistical Methods
- Robert, Casella
- 1999
(Show Context)
Citation Context ...ples [49] motivate a penalty that is dependent on the size of the network (in terms of the number of receivers). However, other model selection techniques lead to the selection of different penalties =-=[50]-=-. 5 Experimental Results We have implemented a software tool called nettomo that performs sandwich probing measurements and estimates the topology of a tree-structured network. We conducted Internet e... |

902 |
R: A language for data analysis and graphics
- Ihaka, Gentleman
- 1996
(Show Context)
Citation Context ...direct measurements. The performances of MPLE and MLE are comparable in this case, but the computation of the MPLE is faster than MLE. For this example, the computations are carried out using R 1.5.0 =-=[33]-=- on a 1G Hz laptop: it takes about 12 seconds for computing the MPLE, and about 49 seconds for the MLE in producing Figure 6. Second, in order to assess the performance of MPLE more thoroughly, simula... |

574 |
Multivariate density estimation: theory, practice, and visualization
- Scott
- 1992
(Show Context)
Citation Context ...fference between probability estimates and the true probabilities. As a common measure of the performance of density estimates, the L1 error norm enjoys several theoretical advantages as discussed in =-=[29]-=-. The plot shows that MLE and MPLE have comparable estimation performance for tracking link delay distributions, while the recursive algorithm has much larger L1 errors on all links. Meanwhile, we can... |

504 |
H.: Hierarchical grouping to optimize an objective function
- Ward
- 1963
(Show Context)
Citation Context ...evise simple and effective bottom-up merging algorithm that identifies the complete, logical topology [13, 39, 40, 41]. These same techniques are used in agglomerative hierarchical clustering methods =-=[42, 43, 44]. For a given tr-=-ee T , the set of all metrics satisfying the monotonicity property is defined as G(T )= � γ ∈ Γ(T ):γ f(k) <γk, ∀k ∈ W (T ) � , W(T )=V \{0, 1,R} . (9) The set W (T ) ∈ V is the set of... |

499 |
Stochastic Complexity
- RISSANEN
- 1989
(Show Context)
Citation Context ...hree devices inside the circle can be identified as one unique device. favors simpler trees with fewer links. The choice of the penalty λ must also be addressed. Minimum description length principles=-= [49]-=- motivate a penalty that is dependent on the size of the network (in terms of the number of receivers). However, other model selection techniques lead to the selection of different penalties [50]. 5 E... |

271 |
Statistical analysis of non-lattice data
- Besag
- 1975
(Show Context)
Citation Context ...the computational burden but maintains good statistical efficiency. The idea of modifying likelihood is not new, and many modified likelihood models have been proposed. For example, pseudo-likelihood =-=[25, 26]-=- for Markov Random Field (MRF) by Besag (1974), partial likelihood [27] for hazards regression by Cox (1973), and quasi-maximum likelihood [28] for finance models by White (1994). In this section, we ... |

260 |
divergence geometry of probability distributions and minimization problems
- Csiszár
- 1975
(Show Context)
Citation Context ...ameters, and the positivity constraints on the OD traffic, are used as the initial estimates of the OD traffic. The linear equation y = Ax is enforced via the iterative proportional fitting algorithm =-=[23, 32]-=- to obtain the final estimates of the OD traffic. The positivity and the linear constraints are very important final steps to get reliable estimates of the OD traffic, in addition to the implicit regu... |

252 | Multicast-based Inference of Network-internal Loss Characteristics
- Caceres, Duffield, et al.
- 1999
(Show Context)
Citation Context ...inference and medical tomography. Two forms of network tomography have been addressed in the recent literature: i) link-level parameter estimation based on end-to-end, path-level traffic measurements =-=[6, 7, 8, 9, 10, 11, 12, 13, 14, 15]-=- and ii) sender-receiver path-level traffic intensity estimation based on link-level traffic measurements [5, 16, 17, 18, 19, 20]. In link-level parameter estimation, the traffic measurements typicall... |

214 |
an e cient data clustering method for very large databases
- BIRCH
- 1996
(Show Context)
Citation Context ... that share certain properties. These clusters can be represented by a dendritic tree. Hierarchical clustering has been used in a variety of areas and are particularly popular for document clustering =-=[35, 36, 37, 38]-=-. 4.1 Problem Statement We formulate the topology identification problem as a tree estimation exercise. Let T =(V,L) denote a rooted tree with nodes V and directed links L (we consider a strongly acyc... |

209 |
Network Tomography: Estimating source-destination traffic intensities from link data
- Vardi
- 1996
(Show Context)
Citation Context ...rk performance parameters based on traffic measurements at a limited subset of the nodes. Y. Vardi was one of the first to rigorously study this sort of problem and coined the term network tomography =-=[5]-=- due to the similarity between network inference and medical tomography. Two forms of network tomography have been addressed in the recent literature: i) link-level parameter estimation based on end-t... |

129 |
Partial likelihood
- Cox
- 1975
(Show Context)
Citation Context ...a of modifying likelihood is not new, and many modified likelihood models have been proposed. For example, pseudo-likelihood [25, 26] for Markov Random Field (MRF) by Besag (1974), partial likelihood =-=[27]-=- for hazards regression by Cox (1973), and quasi-maximum likelihood [28] for finance models by White (1994). In this section, we decribe the pseudo-likelihood approach. We explore two concrete example... |

122 | Inferring link loss using striped unicast probes
- Duffield, Presti, et al.
- 2001
(Show Context)
Citation Context ...inference and medical tomography. Two forms of network tomography have been addressed in the recent literature: i) link-level parameter estimation based on end-to-end, path-level traffic measurements =-=[6, 7, 8, 9, 10, 11, 12, 13, 14, 15]-=- and ii) sender-receiver path-level traffic intensity estimation based on link-level traffic measurements [5, 16, 17, 18, 19, 20]. In link-level parameter estimation, the traffic measurements typicall... |

116 |
Estimation, inference and specification analysis
- White
- 1994
(Show Context)
Citation Context ...s have been proposed. For example, pseudo-likelihood [25, 26] for Markov Random Field (MRF) by Besag (1974), partial likelihood [27] for hazards regression by Cox (1973), and quasi-maximum likelihood =-=[28]-=- for finance models by White (1994). In this section, we decribe the pseudo-likelihood approach. We explore two concrete examples : i) internal link delay distribution inference through multicast end-... |

110 | Internet tomography
- Coates, Nowak, et al.
(Show Context)
Citation Context ...f inferential network monitoring or network tomography, highlighting challenges and open problems, and identifying key issues that must be addressed. It builds upon the signal processing survey paper =-=[1]-=- and focuses on recent developments in the 2sfield. The task of inferential network monitoring demands the estimation of a potentially very large number of spatially distributed parameters. To success... |

107 |
A survey of recent advances in hierarchical clustering algorithms. The Computer Journal, 26(4):354–359
- Murtagh
- 1983
(Show Context)
Citation Context ...evise simple and effective bottom-up merging algorithm that identifies the complete, logical topology [13, 39, 40, 41]. These same techniques are used in agglomerative hierarchical clustering methods =-=[42, 43, 44]. For a given tr-=-ee T , the set of all metrics satisfying the monotonicity property is defined as G(T )= � γ ∈ Γ(T ):γ f(k) <γk, ∀k ∈ W (T ) � , W(T )=V \{0, 1,R} . (9) The set W (T ) ∈ V is the set of... |

105 |
A Statistical Perspective on Ill-Posed Inverse Problems A
- O’Sullivan
- 1986
(Show Context)
Citation Context ...rameters and measurements sites in the Internet. The associated high dimensional problems of estimating Xt are specific examples of inverse problems. Inverse problems have a very extensive literature =-=[21].-=- Solution methods for such inverse problems depend on the nature of the noise ɛ and the A matrix and typically require iterative algorithms since they cannot be solved directly. In general, A is not ... |

91 | Inference of multicast routing trees and bottleneck bandwidths using end-to-end measurements
- Ratnasamy, McCanne
- 1999
(Show Context)
Citation Context ...nd 19 have a common parent in the logical tree. This property can be exploited in this manner to devise simple and effective bottom-up merging algorithm that identifies the complete, logical topology =-=[13, 39, 40, 41]. -=-These same techniques are used in agglomerative hierarchical clustering methods [42, 43, 44]. For a given tree T , the set of all metrics satisfying the monotonicity property is defined as G(T )= � ... |

91 | Time-varying network tomography: Router link data
- Cao, Davis, et al.
(Show Context)
Citation Context ...tion based on end-to-end, path-level traffic measurements [6, 7, 8, 9, 10, 11, 12, 13, 14, 15] and ii) sender-receiver path-level traffic intensity estimation based on link-level traffic measurements =-=[5, 16, 17, 18, 19, 20]-=-. In link-level parameter estimation, the traffic measurements typically consist of counts of packets transmitted and/or received between nodes or time delays between packet transmissions and receptio... |

82 | Network loss inference using unicast end-to-end measurement
- Coates, Nowak
- 2000
(Show Context)
Citation Context ...inference and medical tomography. Two forms of network tomography have been addressed in the recent literature: i) link-level parameter estimation based on end-to-end, path-level traffic measurements =-=[6, 7, 8, 9, 10, 11, 12, 13, 14, 15]-=- and ii) sender-receiver path-level traffic intensity estimation based on link-level traffic measurements [5, 16, 17, 18, 19, 20]. In link-level parameter estimation, the traffic measurements typicall... |

82 | Bayesian inference on network traffic using link count data
- Tebaldi, West
- 1998
(Show Context)
Citation Context ...tion based on end-to-end, path-level traffic measurements [6, 7, 8, 9, 10, 11, 12, 13, 14, 15] and ii) sender-receiver path-level traffic intensity estimation based on link-level traffic measurements =-=[5, 16, 17, 18, 19, 20]-=-. In link-level parameter estimation, the traffic measurements typically consist of counts of packets transmitted and/or received between nodes or time delays between packet transmissions and receptio... |

80 |
Implementing agglomerative hierarchical clustering algorithms for use in document retrieval. Information Processing and Management
- Voorhees
- 1986
(Show Context)
Citation Context ... that share certain properties. These clusters can be represented by a dendritic tree. Hierarchical clustering has been used in a variety of areas and are particularly popular for document clustering =-=[35, 36, 37, 38]-=-. 4.1 Problem Statement We formulate the topology identification problem as a tree estimation exercise. Let T =(V,L) denote a rooted tree with nodes V and directed links L (we consider a strongly acyc... |

75 |
Recent trends in hierarchic document clustering: A critical review
- WILLET
- 1988
(Show Context)
Citation Context ... that share certain properties. These clusters can be represented by a dendritic tree. Hierarchical clustering has been used in a variety of areas and are particularly popular for document clustering =-=[35, 36, 37, 38]-=-. 4.1 Problem Statement We formulate the topology identification problem as a tree estimation exercise. Let T =(V,L) denote a rooted tree with nodes V and directed links L (we consider a strongly acyc... |

74 | An Analysis of Recent Work on Clustering Algorithms
- Fasulo
- 1999
(Show Context)
Citation Context ...evise simple and effective bottom-up merging algorithm that identifies the complete, logical topology [13, 39, 40, 41]. These same techniques are used in agglomerative hierarchical clustering methods =-=[42, 43, 44]. For a given tr-=-ee T , the set of all metrics satisfying the monotonicity property is defined as G(T )= � γ ∈ Γ(T ):γ f(k) <γk, ∀k ∈ W (T ) � , W(T )=V \{0, 1,R} . (9) The set W (T ) ∈ V is the set of... |

74 | Maximum Likelihood Network Topology Identification from Edge-Based Unicast Measurements
- Coates, Castro, et al.
(Show Context)
Citation Context ...g Techniques and Modeling log p(xij|γij) . (12) To illustrate our approach we will focus on one type of metric. In earlier work we proposed a topology identification method based on delay differences=-= [46]-=-. The method relies on a measure24sd p2 q p1 0 1 2 3 Figure 9: An example of sandwich probe measurement. The large packet is destined for node 2, the small packets for node 3. The black circles on the... |

67 | Robust identification of shared losses using end-to-end unicast probes
- Harfoush, Bestavros, et al.
- 2000
(Show Context)
Citation Context |

63 | Maximum pseudo likelihood estimation in network tomography
- Liang, Yu
(Show Context)
Citation Context ...tion based on end-to-end, path-level traffic measurements [6, 7, 8, 9, 10, 11, 12, 13, 14, 15] and ii) sender-receiver path-level traffic intensity estimation based on link-level traffic measurements =-=[5, 16, 17, 18, 19, 20]-=-. In link-level parameter estimation, the traffic measurements typically consist of counts of packets transmitted and/or received between nodes or time delays between packet transmissions and receptio... |

60 | Multicast topology inference from measured end-to-end loss
- Duffield, Horowitz, et al.
- 2002
(Show Context)
Citation Context ...nd 19 have a common parent in the logical tree. This property can be exploited in this manner to devise simple and effective bottom-up merging algorithm that identifies the complete, logical topology =-=[13, 39, 40, 41]. -=-These same techniques are used in agglomerative hierarchical clustering methods [42, 43, 44]. For a given tree T , the set of all metrics satisfying the monotonicity property is defined as G(T )= � ... |

50 | Pc based precision timing without gps - Pasztor, Veitch - 2002 |

49 |
Integrated likelihood methods for eliminating nuisance parameters. Stat. Sci
- BERGER, LISEO, et al.
- 1999
(Show Context)
Citation Context ... we are not interested in �γ(x), an estimate of γ from the measurements. Hence we can regard γ as nuisance parameters. In that case (10) can be interpreted as a maximization of the profile likeli=-=hood [45] L(x|T -=-) ≡ sup p(x|γ) (11) γ∈G(T ) The solution of (10) is referred to as the Maximum Likelihood Tree (MLT). Consider now some more structure in the log-likelihood log p(x|γ): Assume the random variab... |

43 | Inference and labeling of metricinduced network topologies
- Byers, Bestavros, et al.
- 2005
(Show Context)
Citation Context ...nd 19 have a common parent in the logical tree. This property can be exploited in this manner to devise simple and effective bottom-up merging algorithm that identifies the complete, logical topology =-=[13, 39, 40, 41]. -=-These same techniques are used in agglomerative hierarchical clustering methods [42, 43, 44]. For a given tree T , the set of all metrics satisfying the monotonicity property is defined as G(T )= � ... |

40 | Variance of Aggregated Web Traffic - Morris, Lin - 2000 |

39 | Unicast Network Tomography using EM. Algorithms
- Coates, Nowak
- 2000
(Show Context)
Citation Context ...rk transport protocols required by many current probing techniques. This has prompted investigations into more passive traffic monitoring techniques, for example based on sampling TCP traffic streams =-=[52]-=-. Furthermore, the ultimate goal of carrying out network tomography on a massive scale poses a significant computational challenge. Decentralized processing and data fusion will probably play an impor... |

38 | Network delay tomography - Tsang, Coates, et al. |

27 | A scalable method for estimating network traffic matrices," Bell Labs
- Cao, Wiel, et al.
- 2001
(Show Context)
Citation Context |

25 |
Queueing Networks: Customers, Signals, and Product Form Solutions
- Chao, Miyazawa, et al.
- 1999
(Show Context)
Citation Context ...cs and fine time-scale traffic behaviour, but they are often sufficient for inference of performance characteristics. The approach shifts the focus from detailed queuing analysis and traffic modeling =-=[2, 3]-=- to careful design of measurement techniques and large-scale inference strategies. Mesurement may be passive (monitoring traffic flows and sampling extant traffic) or active (generating probe traffic)... |

25 | Sequential monte carlo inference of internal delays in nonstationary data networks
- Coates, Nowak
(Show Context)
Citation Context ...or. In addition to the estimation of time-varying OD traffic matrices discussed in Section 3.2, others have adopted a dynamical systems approach to handle nonstationary link-level tomography problems =-=[51]-=-. Sequential Monte Carlo inference techniques are employed in [51] to track time-varying link delay distributions in nonstationary networks. One common source of temporal variability in link-level per... |

22 | Adaptive multicast topology inference - Duffield, Horowitz, et al. - 2001 |

21 |
Network delay distribution inference from end-to-end unicast measurement
- Coates, Nowak
- 2001
(Show Context)
Citation Context |

21 |
Hierarchic Document Clustering Using Ward's Method
- El-Hamdouchi, Willet
- 1986
(Show Context)
Citation Context |

21 | Multicast topology inference from end-to-end measurements
- Duffield, Horowitz, et al.
- 2000
(Show Context)
Citation Context |

19 | Passive network tomography using bayesian inference - Padmanabhan, Qiu, et al. |

18 | Unicast inference of network link delay distributions from edge measurements
- Shih, Hero
- 2001
(Show Context)
Citation Context |

18 | An em approach to od matrix estimation
- Vanderbei, Iannone
- 1994
(Show Context)
Citation Context |

17 | Estimation, Inference and Specification Analysis, Cambridge Univ - White - 1994 |

14 | Likelihood based hierarchical clustering - Castro, Coates, et al. - 2004 |

8 |
Estimation of network link loss rates via chaining in multicast trees
- Ziotopolous, Hero, et al.
- 2001
(Show Context)
Citation Context |

6 |
Approximate normality of large products
- Blackwell
- 1985
(Show Context)
Citation Context ... is more robust than MLE. This is because the pseudo likelihood function, which is a product of less complex likelihood functions on subproblems, has a nicer surface than the full likelihood function =-=[30]. -=-13sL−1 Error Norm 0.0 0.5 1.0 1.5 1 2 3 4 Links 5 6 7 Figure 4: Link L1 error norm averaged over 30 simulations: solid line is MPLE, dashed line is MLE, and dotted line is recursive algorithm. For e... |