Search Engine: User-Agent
AltaVista: Scooter
Infoseek: Infoseek
Hotbot: Slurp
AOL: Slurp
Excite: ArchitextSpider
Google: Googlebot
Goto: Slurp:
Lycos: Lycos
MSN: Slurp
Netscape: Googlebot
NorthernLight: Gulliver
WebCrawler: ArchitextSpider
Iwon: Slurp
Fast: Fast
DirectHit: Grabber
Yahoo Web Pages: Googlebot
Looksmart Web Pages: Slurp
User-agent: *
Disallow: admin.php
Disallow: /templates/
Disallow: /admin/
Disallow: /images/
Disallow: /include/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/
Disallow: /scripts/
Disallow: /assets/
Disallow: /cgi/
Disallow: /cgi-bin/
Disallow: connect.php
Disallow: add*.php
Disallow: edit*.php
Disallow: del*.php
Disallow: delete.php
Disallow: edit.php
Disallow: /comdev/
Disallow: /photo/
Disallow: /assets/
Disallow: /templates/
Disallow: readme.txt
Disallow: *.sql
Disallow: install.*
Disallow: setup.*
Disallow: /install/
Disallow: /Connections/
Disallow: /menu/