Already a member? Log in

Sign up with your...

or

Sign Up with your email address

Add Tags

Duplicate Tags

Rename Tags

Share It With Others!

Save Link

Sign in

Sign Up with your email address

Sign up

By clicking the button, you agree to the Terms & Conditions.

Forgot Password?

Please enter your username below and press the send button.
A password reset link will be sent to you.

If you are unable to access the email address originally associated with your Delicious account, we recommend creating a new account.

ADVERTISEMENT
ADVERTISEMENT

Links 1 through 10 of 18 by J. Albert Bowden II tagged bots

Share It With Others!

Share It With Others!

Share It With Others!

Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

Note, if you already have directives for MSNBot in your robots.txt file, BingBot will obey those directives. It is important to note that if you have two different directives, one for MSNBot and one for BingBot and they say two different things, Microsoft will listen to BingBot over MSNBot.

Share It With Others!

Share It With Others!

Share It With Others!

To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index.

The noindex meta standard is described at http://www.robotstxt.org/meta.html. This method is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis.

To prevent all robots from indexing a page on your site, place the following meta tag into the <head> section of your page:

<meta name="robots" content="noindex">

To allow other robots to index the page on your site, preventing only Google's robots from indexing the page:

<meta name="googlebot" content="noindex">

Share It With Others!

Share It With Others!

Share It With Others!

Share It With Others!

ADVERTISEMENT
ADVERTISEMENT
ADVERTISEMENT