How to keep robots/spiders out of your web site

Article Author: Roberto Bonomi. Posted: 28-03-2006.

THE ROBOTS.TXT FILE
You probably know that search engines were created to help folks find information quickly on the Internet and the information we get is through the robots, which are also known as spiders or crawlers. They search all web pages. The spiders or crawlers robots explore the web looking for and recording all the information. They usually start with a URL submitted by users, or from links they find on the web sites, the sitemap files or the top level of any site. Once the robot gets the home page, it then recursively (self-repeats) gains access to all the pages that are linked from that page. And, the robot also checks out all the pages that it finds on any server. After the robot finds a web page, it gets to work indexing the title, the keywords, the text, the meta tags, etc. But sometimes you might want to stop the search engines from indexing some of your web pages, ones like news postings and specially marked web pages, for example: an affiliate page, but whether individual robots comply to these conventions is purely voluntary.

ROBOTS EXCLUSION PROTOCOL
So, when and if you want robots to keep out of some of your web pages, you can ask robots to ignore the web pages that you don´t want indexed. To do that, you place a robots.txt file on the local root server of your web site. For example, if you have a directory called 'e-books' that you want to ask robots to keep out of, your robots.txt file will read:
User-agent: * Disallow: e-books/
When you don´t have enough control over your server to set up a robots.txt file, you can try adding a META tag to the head section of any HTML document. For example, a tag like the following, tells robots not to index and not to follow links on that particular page:
meta name="ROBOTS" content="NOINDEX, NOFOLLOW"
Support for the META tag among robots is not so frequent as the Robots Exclusion Protocol, but most of the major web indexes currently support it.

NEWS POSTINGS
If you want to keep the search engines out of your news postings, you can create an "X-no-archive" line in your postings' headers, like this: X-no-archive: yes
Although common news clients allow you to add a 'X-no-archive' line to the header of your news postings, some of them won’t let you. The problem is that most search engines assume that all the information they find is public, unless marked otherwise. So be careful, because although the robot and archive exclusion standards may help to keep your material out of major search engines there are some others that just have no respect for such rules. If you're highly concerned about the privacy of your e-mail and Usenet postings, you can use some anonymous remailers and PGP. Find out about them here:
http://www.well.com/user/abacard/remail.html http://www.io.com/~combs/htmls/crypto.html
http://world.std.com/~franl/pgp/
Even if you’re not particularly concerned about privacy, remember that everything you write is indexed and archived somewhere for eternity, so use the robots.txt file as much as you need & want it.

-----------------------------------------------------------------
Check out Bette's home based business ideas and opportunities, just for you.
Copy & paste each of these web-dresses into your browser:
http://www.MakeALotOfMoneyHelpingOthers.com
http://tny.se/ft;
http://tny.se/fu;
http://tny.se/fw;
http://tny.se/fy.
-----------------------------------------------------------------