Official Google Webmaster Central Blog: Improving on Robots Exclusion Protocol

googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-pro…

Saved 181 times on del.icio.us, most recently in 2012.

Tags: google robots.txt seo robots search webdev tips reference web crawler

More links tagged google

More links tagged robots.txt

More links tagged seo