Results 1 to 6 of 6

Major Spidering issue

This is a discussion on Major Spidering issue within the General Discussion forums, part of the vBulletin SEO Discussion category; Hi guys, our forums had always had over 100 spiders visiting, however for some reason we are only seeing 3 ...

  1. #1
    Member
    Real Name
    maksam
    Join Date
    Apr 2008
    Posts
    80
    Liked
    0 times

    Major Spidering issue

    Hi guys, our forums had always had over 100 spiders visiting, however for some reason we are only seeing 3 spiders and the most 4 spiders at any one time.

    Could somebody please advise as to what part of vbSEO is blocking the spider flow please?

    Cheers

  2. #2
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    do you have a robots.txt file?

  3. #3
    Member
    Real Name
    maksam
    Join Date
    Apr 2008
    Posts
    80
    Liked
    0 times
    Quote Originally Posted by briansol View Post
    do you have a robots.txt file?
    Hi Brian i have all default SEO practicse utlised on the domain. What i am saying is. Before i installed vbseo our normal spider rank was 100+, but after installing vbseo i am only getting 3 which was a major major cut down?

    We also had other search engines visit us, but now all i am seeing is Google, Yahoo! Slurp (2), Google AdSense thats it?

  4. #4
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    36 times
    Blog Entries
    9
    It is still early for you. However, you should verify your sitemap:
    Verifying your vBSEO Google/Yahoo Sitemap Generator is Installed Correctly

    Also you should check out our other improvement resources:
    Your Site Review - Stuff to Do First

    Briansol's Ultimate Guide to vBSEO

    A few targeted suggestions for you:

    • I would make the top 10 stats on your forumhome, not visible for guests/search crawlers.
    • I would get rid of your footer archive links. I think 50 is too many.
    • I would use our replace showpost with permalink feature on your thread pages.
    • You should convert your welcome message to an image & use our relevant replacements feature.

    You can find details or links on how to do the above in the resources provided above.

  5. #5
    Member
    Real Name
    maksam
    Join Date
    Apr 2008
    Posts
    80
    Liked
    0 times
    Quote Originally Posted by Joe Ward View Post
    It is still early for you. However, you should verify your sitemap:
    Verifying your vBSEO Google/Yahoo Sitemap Generator is Installed Correctly

    Also you should check out our other improvement resources:
    Your Site Review - Stuff to Do First

    Briansol's Ultimate Guide to vBSEO

    A few targeted suggestions for you:

    • I would make the top 10 stats on your forumhome, not visible for guests/search crawlers.
    • I would get rid of your footer archive links. I think 50 is too many.
    • I would use our replace showpost with permalink feature on your thread pages.
    • You should convert your welcome message to an image & use our relevant replacements feature.
    You can find details or links on how to do the above in the resources provided above.
    Much appreciated for that, i will make the changes. May i ask what do you mean by "I would use our replace showpost with permalink feature on your thread pages." and how can i go about doing this?


    Coming back to my initial post, i understand its early stages... the thing is ive had over 100+ spiders before i installed vbseo.. the moment i installed vbseo the spiders gradually went down...? Thats whats puzzling me..

  6. #6
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    36 times
    Blog Entries
    9
    Check out the 2nd item in the list of vBSEO Tweaks on this page:
    Your Site Review - Stuff to Do First

    Crawler activity will have highs and lows, sometimes highly active and sometimes a bit slow. If they were crawling heavily at the time you installed vBSEO, it may have temporarily stopped since it noticed the URL changes and did not want to continue until it was sure your URL structure was stable and non-changing.

    With the help of our sitemap you should see this activity return soon.

    Please keep an eye on your long tail search traffic as well to see how that grows:
    Measuring Your Long Tail - vBulletin SEO Forums

Similar Threads

  1. Anyone else seeing a huge amount of Google Spidering?
    By BamaStangGuy in forum General Discussion
    Replies: 8
    Last Post: 07-11-2006, 09:22 PM
  2. Spidering members forums
    By JohnnyBGood in forum General Discussion
    Replies: 8
    Last Post: 06-12-2006, 05:26 AM
  3. Major Issue
    By Michael in forum Troubleshooting
    Replies: 12
    Last Post: 12-12-2005, 06:25 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •