SEO

Hi ive been told there are dirty bots that use up your bandwidth and its best to block some.

Can the experts of vbSEO tell me or advise me rather on the most popular bots "not to block" i.e Google, Yahoo, Msn etc... what are the other top SEO bots not to block?

Could you also give me the correct string for the bots that i should enable server wide? As we know there are two different types of google and yahoo bots. Yahoo, Yahoo Slurp etc.. and also we have Google, Google Adsense bots etc.

In otherwords should we be even blocking these bots or the more the merrier?

Any advise would be appreciated.

Here is an interesting resource with a list of suggestions for bots to block:
Block bad bots from your site

__________________
Joe Ward / Crawlability Inc.
vBSEO Styles: Version 1.0 PRO For Everyone Version 2.0 For vBSEO Customers

vBSEO 3.3.0 Gold Released with New "Virtual HTML Display" Feature Available for Download Now

6X Traffic - $1400 in One Day with vBSEO! Max Out Your SEO

Im assuming there are more bots to block as oppose to allow, would you be able to tell me the number other bots to have other than the usuall ones?

How can we distinguish from what is a bad/good bot and whether to allow it or not...?

Cheers

The obvious ones are Google, Yahoo, & MSN. However, you might want to simply check your referrers. Find all search engines that have sent you any meaningful traffic, and then consider adding them to your approve list.

Here is an interesting resource you should check out:
What Do SEO/SEM People Put In Robots.txt Files? | Hobo

__________________
Joe Ward / Crawlability Inc.
vBSEO Styles: Version 1.0 PRO For Everyone Version 2.0 For vBSEO Customers

vBSEO 3.3.0 Gold Released with New "Virtual HTML Display" Feature Available for Download Now

6X Traffic - $1400 in One Day with vBSEO! Max Out Your SEO

My last query...

I understand this is via Robots.txt, my robots.txt is open to all.. however what can be done if my host is blocking robots via server? Their excuse is that to save me bandwidth and to eliminate rogue bots which are "dirty"?

Can they even do this via server?

Well - there are two things your host could do in this regard:

a) They could actually place a robots.txt in client accounts, or revise yours.
b) They can simply block traffic from the bots via IP at the server software level.

While this may actually be beneficial to you in some cases, it could be a serious problem in other cases, such as if the host decides to block Googlebot from all clients hosted! It has happened:

I would ask them what method they are using the block the bots, and I would ask for a full list of everything blocked.

If the list is unacceptable to you, such as the blocking of a major crawler like Googlebot, I would change hosting providers. The reason I say "change providers" and NOT "ask them to stop doing it" is because:

They already did you a disservice (in the case of Googlebot blocking), and since there are so many other providers out there, you do not need to feel compelled to give them a 2nd chance.
Highly Traffic Drop

Some hosting info:
vBSEO Web Hosting Survey - Demand Blazing Speed!

__________________
Joe Ward / Crawlability Inc.
vBSEO Styles: Version 1.0 PRO For Everyone Version 2.0 For vBSEO Customers

vBSEO 3.3.0 Gold Released with New "Virtual HTML Display" Feature Available for Download Now

6X Traffic - $1400 in One Day with vBSEO! Max Out Your SEO

Cheers for that, yes they never told me about this. Because i noticed a huge drop in bots and had a few htaccess issues i thought it was htaccess. So i queried with them and i explained htaccess could be a problem as i may of done the url rewrite incorrect and told them that ive noticed my bot count go down drastically... they then told me that they have blocked some bots, however they told me that they left the major good bots in place.

Ive asked them several times when this change took place, but they were not able to tell me several times... (thus assuming they must of done this long time ago and cannot remember...) i wanted to see whether it coincides with when i installed vbseo... as if it dosent then its not a server side issue, but vbseo taking its time? I didnt know whether it was vbseo or the host at fault.. im just hoping its not a change they made recently.

Ive asked them to tell me what bots are on the allowed list and i will also ask them to give me a list of those that they have denied and possibly get back to you with the result.


Cheers

Yes - definitely ask them to give you the full list of disallowed bots. Ask them for ballpark estimates for when that was setup. They should be able to give you any idea.

If they do not give you a list, I would have a hard time trusting them. They should be able to provide you with that.

Which host are you using?

__________________
Joe Ward / Crawlability Inc.
vBSEO Styles: Version 1.0 PRO For Everyone Version 2.0 For vBSEO Customers

vBSEO 3.3.0 Gold Released with New "Virtual HTML Display" Feature Available for Download Now

6X Traffic - $1400 in One Day with vBSEO! Max Out Your SEO

I was wondering if i can use the exact robots.txt provided on the link posted earlier for my forums:
http://toomanysecrets.com/robots.txt
thank you

Well, chances are you don't have the same directories, so i would say no.


IMO, unless your breaking your bandwidth bill, or your site is incredibly slow due to huge bot usage, you shouldn't disallow any particular robot.

I've outlined a suggested robots file in my Ultimate Guide.