:
|
Wow, just like real spiders! 
Sounds interesting. Apparently we're popular enough to get four or five of these spiders crawling the Forums - that's how many I saw online the other day. Are spiders how the Wayback Machine was compiled? |
Yes, the waybackspider is the one that gets wayback entries. It only starts doing it after it finds a site that has a certain number of hits. So OWF will probably have quite a few wayback entries.
I wouldn't worry about the spiders on this website. They do suck up a bit of bandwidth, yes, but not loads. They download the entire site (or most of it) using several spiders at once, in order to reduce the bandwidth leeching effects, and to get the site downloaded faster. It's an intermittant nuisance at best.