Confused about Spiders...

I am confused about these spiders...

Even though my site has been up for like 3 months...301 twice
to its new and final domain. The 301 indexing seems to be
fairly quick, even with the brand new domain, and spiders seem
to follow the 301 to where the domain is, np...but it all seems
to be unstable...1 min I will have 5 spiders, the next 20,25,30,
a few min later, 0 spiders, then it starts all over again, nothing
appears to be really stable at all...

Then I have the baidu..I know this spider is from china, but
what is its purpose on my site if people in china can not
visit the site since they are not allowed open access...so
it does not make sense on why it is even there....

It would seem the other spiders are just as worthless, they could
be hackers, spammers, scrapers, etc...junk spiders...

and google/yahoo seem to show up randomly throughout the day...

ideas?

Spiders will come and go as they please. There's no controlling them. Since they don't really create a session, it can often appear that more than one is on your site. In reality, it's usually just one spider loading X pages and appearing as X spiders online. Sometimes there are more than one, but most times not.

All spiders will visit all sites they want to unless told otherwise. You can block baidu the same way you blocked china, via htaccess and/or robots.txt directives.

All in all, I find blocking spiders to be generally a waste of time. Unless you're peaking resources and every little bit counts, I wouldn't waste any time trying to tame spiders. There's easily 100,000 known bots/spiders for various items and sites and trying to exclude the 'bad' ones is a daunting effort for effectively 0 gain. And finally, if you're at this point of peak resources, your time will be better spent migrating to better/newer/faster hardware.

You can use robots.txt to allow only google yahoo and bign spiders and disallow the rest. You can implement this solution to htaccess but either ways will cost you spend some resources.

Then I would recommend you check cloudflare system. They have a pretty good and free solution for harmful spiders and bots. They use general blacklist for such bad intentioned spiders and keep them away from reaching your site at dns level. It maybe little bit complicated plase check my final blog post.

Originally Posted by Brian Cummiskey

Spiders will come and go as they please. There's no controlling them. Since they don't really create a session, it can often appear that more than one is on your site. In reality, it's usually just one spider loading X pages and appearing as X spiders online. Sometimes there are more than one, but most times not.

All spiders will visit all sites they want to unless told otherwise. You can block baidu the same way you blocked china, via htaccess and/or robots.txt directives.

All in all, I find blocking spiders to be generally a waste of time. Unless you're peaking resources and every little bit counts, I wouldn't waste any time trying to tame spiders. There's easily 100,000 known bots/spiders for various items and sites and trying to exclude the 'bad' ones is a daunting effort for effectively 0 gain. And finally, if you're at this point of peak resources, your time will be better spent migrating to better/newer/faster hardware.

ok thats good and all...but I do not understand why they visit old forum content like 3 months old,
rather then bombarding the new daily content, and blog content...I see them mostly visiting
parts of the forum, rarely do i see any on the VB Blog...

whadaya know, I see a vBSEO spider...is that really a vBSEO spider?

The vBSEO spider is used for the linkback/trackback/refback system. If you're seeing one on your site, another vbseo-powered forum likely has linked to you.

I cannot comment on how or why spiders do what they do. Those questions are probably better asked to the spider owners themselves as I/we have no control over what or why they do.