Spider / Bots / Robots / Crawlers - You hear then everywhere when you start looking into the world of Search Engine Optimization. Don't worry they are no 8 legged creatures of Nickel headed cyborgs from future that are here to ruin your present. These are specialized software that automatically scans the World Wide Web for pages to read, classify and index so that their engines can process them latter to deliver some output. In our case the output are search results on Google / Yahoo / Bing or some similar search engine.

How does a Spider Work?

Matt Cutts from Google will give tell you how web search works. Notice what he has to say after video timeline 00:25. Its about how spiders work.

Certain things about bots/spiders you must understand

• They index your page. That is they keep a copy (HTML output) of your page on their own databases. That need not be what your site/page looks right now. It could be a month long copy too.
• Spiders do can only find a page if it is are connected from other places on the worldwide web (linked from other places). Though certain search engines try to index newly registered domains via popular domain registrars too.
• They have a priority set for you. For example the Google Bot will visit your pages if your domain has a high reputation and high PR and is frequently updated.
• You can request a faster or a slower revisit rate but that is just a request. The spider has a mind and it will decide what the best thing to do is.
• You actually use up bandwidth when a spider crawls your page / image/ video / document ets. So make sure you control the indexing if you are on a tight bandwidth hosting package.
• If you have content on your site that you do not want the public to access please password protect the pages or use a robot.txt file to instruct the crawlers not to access them anymore. Particular subdirectories can be blocked form crawler access. But do not expect every crawler to honor this robots.txt file, especially the ones with malicious intensions. (email harvesting bots)

For more information on Spiders and Bots refr to this wikipeida article.