|
Theres another thread on this furthur down the list..
anyway, they're search engine bots which save pages into the Google/Lycos etc database so they can link to it if people search for keywords on that page...
Or, as Death aptly put it:
"Spiders are not quite "Bots" as such. Spiders do exactly what spiders do in real life. They crawl the web, downloading things as they go. Spiders often download and save entire websites just to provide faster searches and word matches. They only do it on popular sites though, so this site must be quite popular.
Another feature of a spider is to gather information such as statistics, and hits. This is to figure out whether the website the spider is visiting is popular or unpopular. If it's popular, a second spider will be called in, and the two spiders will systematically download the website, and add it to the search info database.
The reason spider bots are assosciated with websearches, is because a spider bot gets sent around the web on most searches, looking for content that the database does not yet have. The spiders' first role though, is to check the database to see if any matches can be found. This all happens in a matter of miliseconds, and gets faster and faster as time passes.
A searchbot, is quite similar to a spider, only that it just seeks out content matches, then generates a link using php. The searchbots' role is really to take snippets of relevant data, that can be processed through php to appear on a results page. The spidered database pretty much "hints" the searchbot on where to look, and what to look for. For instance, if you searched for the word "Oddworld", the searchbot would ask the spiderbot to check the database for any websites that are listed to contain that word, then the spiderbot would tell the searchbot which ones contain that word, where they contain it, and what the URL is - once it does that, it puts those links onto the results page ready to go into php - then it searches the net accompanied by a spider, looking for content matches. Once those are found, links are created, and the list of results are put in order of best match to worst match.
Again, that all happens in a matter of miliseconds. The difference between a spider and a searchbot, is that the spider deals with content, and the searchbot deals with searches. Originally, there were only searchbots, and searches used to take ages, but now you can get to the most popular site in much less time."
|
People who aren't logged in but online....
.
. Okay i'll stop bragging now. I just felt like having a moment of power there. Okay I now bow down to zee spiders for making me populah! (Well not really..). Okay zee topic can be closed now. Everything is fine and dandy!