Abstract

The design of a Web spider entails many things, including a concern for reasonable behavior, as well as more technical concerns. The RBSE Spider is a mechanism for exploring World Wide Web structure and indexing useful material thereby discovered. We relate our experience in constructing and operating this spider.

Citations

29	A Standard for Robot Exclusion - Koster - 1994
17	Guidelines for robot writers - Koster - 1994
13	Integrating Structured Databases Into the Web: The MORE System - Eichmann, McGregor, et al.
8	The World Wide Web Initiative - Berners-Lee - 1993
4	World Wide Web Wanderers, Spiders and Robots - Koster
3	A Beginner's Guide to HTML - Andreessen
2	HyperText Mark-up Language," CERN, http://info.cern.ch/hypertext /WWW/MarkUp/MarkUp.html - Berners-Lee
2	The World Wide Web Initiative," CERN, http://info.cern.ch/hypertext /WWW/TheProject.html - Berners-Lee
2	List of Robots - Koster
2	WWWW - the World Wide Web Worm - McBryan
1	EINet Galaxy - EIT - 1994
1	JumpStation Front Page," http://www.stir.ac.uk/jf1bin/js - Fletcher - 1994
1	Growth of the World Wide Web," http://www.mit.edu:8001/aft/sipb/user/ mkgray/ht/web-growth.html - Gray - 1993