# # Robots.txt # # Original File # ------------- # http://www.Linux-Consulting.com/robots.txt # # # Other Docs # ---------- # http://www.robotstxt.org/wc/faq.html # http://www.searchtools.com/robots/robots-txt.html # http://www.searchengineworld.com/robots/robots_tutorial.htm # http://www.searchengineworld.com/misc/robots_txt_crawl.htm # # # robots.txt Syntax Checkers # -------------------------- # http://www.searchtools.com/robots/robots-txt.html#checkers # http://www.searchengineworld.com/cgi-bin/robotcheck.cgi # # # Use httpd authentication to prevent users reading this file # http://httpd.apache.org/docs/howto/auth.html # # # # 01-Nov-02 amo Date-of-Birth # 06-Jan-03 amo Added Plan.* # 29-Aug-04 amo Synchronize all files # 11-Dec-05 amo Use(force) symlinks html.lc/{robots.txt,favicon.ico} # 10-Mar-07 amo Added .Pricing # # # # For all robots, disallow them seeing these 3 directories # User-agent: * # # # Disallow scanning these directories # Disallow: /cgi-bin/ Disallow: /gif/ Disallow: /STATS/ # Disallow: /Plan*/ Disallow: /.Pricing/ # # # End of file