Get yourself a smart robots.txt

sebastians-pamphlets.com/smart-robots-txt

Saved 32 times on del.icio.us, most recently in 2013.

Tags: robots.txt seo crawling robots search bots tips spam webdesign architecture

More links tagged robots.txt

More links tagged seo

More links tagged crawling