Major Spidering issue

Hi guys, our forums had always had over 100 spiders visiting, however for some reason we are only seeing 3 spiders and the most 4 spiders at any one time.

Could somebody please advise as to what part of vbSEO is blocking the spider flow please?

Cheers

do you have a robots.txt file?

Originally Posted by briansol

do you have a robots.txt file?

Hi Brian i have all default SEO practicse utlised on the domain. What i am saying is. Before i installed vbseo our normal spider rank was 100+, but after installing vbseo i am only getting 3 which was a major major cut down?

We also had other search engines visit us, but now all i am seeing is Google, Yahoo! Slurp (2), Google AdSense thats it?

It is still early for you. However, you should verify your sitemap:
Verifying your vBSEO Google/Yahoo Sitemap Generator is Installed Correctly

Also you should check out our other improvement resources:
Your Site Review - Stuff to Do First

Briansol's Ultimate Guide to vBSEO

A few targeted suggestions for you:

• I would make the top 10 stats on your forumhome, not visible for guests/search crawlers.
• I would get rid of your footer archive links. I think 50 is too many.
• I would use our replace showpost with permalink feature on your thread pages.
• You should convert your welcome message to an image & use our relevant replacements feature.

You can find details or links on how to do the above in the resources provided above.

Originally Posted by Joe Ward

It is still early for you. However, you should verify your sitemap:
Verifying your vBSEO Google/Yahoo Sitemap Generator is Installed Correctly

Also you should check out our other improvement resources:
Your Site Review - Stuff to Do First

Briansol's Ultimate Guide to vBSEO

A few targeted suggestions for you:

• I would make the top 10 stats on your forumhome, not visible for guests/search crawlers.
• I would get rid of your footer archive links. I think 50 is too many.
• I would use our replace showpost with permalink feature on your thread pages.
• You should convert your welcome message to an image & use our relevant replacements feature.

You can find details or links on how to do the above in the resources provided above.

Much appreciated for that, i will make the changes. May i ask what do you mean by "I would use our replace showpost with permalink feature on your thread pages." and how can i go about doing this?


Coming back to my initial post, i understand its early stages... the thing is ive had over 100+ spiders before i installed vbseo.. the moment i installed vbseo the spiders gradually went down...? Thats whats puzzling me..

Check out the 2nd item in the list of vBSEO Tweaks on this page:
Your Site Review - Stuff to Do First

Crawler activity will have highs and lows, sometimes highly active and sometimes a bit slow. If they were crawling heavily at the time you installed vBSEO, it may have temporarily stopped since it noticed the URL changes and did not want to continue until it was sure your URL structure was stable and non-changing.

With the help of our sitemap you should see this activity return soon.

Please keep an eye on your long tail search traffic as well to see how that grows:
Measuring Your Long Tail - vBulletin SEO Forums