Abstract

Recent experiments and analysis suggest that there are about 800 million publicly-indexable web pages. However, unlike books in a traditional library, web pages continue to change even after they are initially published by their authors and indexed by search engines. This paper describes preliminary data on and statistical analysis of the frequency and nature of web page modications. Using empirical models and a novel analytic metric of \up-to-dateness", we estimate the rate at which web search engines must re-index the web to remain current. Keywords: web dynamics, monitoring, document management 1 Introduction Since its inception scarcely a decade ago, the World Wide Web has become a popular vehicle for disseminating scientic, commercial and personal information. The web consists of individual pages linked to and from other pages through Hyper Text Markup Language (HTML) constructs. The web is patently decentralized. Web pages are created, maintained and modied at random t...

Citations