Theres another thread on this furthur down the list..
anyway, they're search engine bots which save pages into the Google/Lycos etc database so they can link to it if people search for keywords on that page...
Or, as Death aptly put it:
"Spiders are not quite "Bots" as such. Spiders do exactly what spiders do in real life. They crawl the web, downloading things as they go. Spiders often download and save entire websites just to provide faster searches and word matches. They only do it on popular sites though, so this site must be quite popular.
Another feature of a spider is to gather information such as statistics, and hits. This is to figure out whether the website the spider is visiting is popular or unpopular. If it's popular, a second spider will be called in, and the two spiders will systematically download the website, and add it to the search info database.
The reason spider bots are assosciated with websearches, is because a spider bot gets sent around the web on most searches, looking for content that the database does not yet have. The spiders' first role though, is to check the database to see if any matches can be found. This all happens in a matter of miliseconds, and gets faster and faster as time passes.
A searchbot, is quite similar to a spider, only that it just seeks out content matches, then generates a link using php. The searchbots' role is really to take snippets of relevant data, that can be processed through php to appear on a results page. The spidered database pretty much "hints" the searchbot on where to look, and what to look for. For instance, if you searched for the word "Oddworld", the searchbot would ask the spiderbot to check the database for any websites that are listed to contain that word, then the spiderbot would tell the searchbot which ones contain that word, where they contain it, and what the URL is - once it does that, it puts those links onto the results page ready to go into php - then it searches the net accompanied by a spider, looking for content matches. Once those are found, links are created, and the list of results are put in order of best match to worst match.
Again, that all happens in a matter of miliseconds. The difference between a spider and a searchbot, is that the spider deals with content, and the searchbot deals with searches. Originally, there were only searchbots, and searches used to take ages, but now you can get to the most popular site in much less time."
|
you really cleared that up...
It was like 50, before the spiders arrived that one time, not so long ago.
It must be late, I'm acting weirder than usual. 
I can go without being psycho until 3 am (that's usually when the psycho parties are).
