Results 1 to 9 of 9

sitemap & spiders

This is a discussion on sitemap & spiders within the General Discussion forums, part of the vBulletin SEO Discussion category; Can somebody please tell me what this setting should be set at? Max URLs in Sitemap Files - I currently ...

  1. #1
    Member
    Real Name
    Chat Football
    Join Date
    Jun 2008
    Posts
    71
    Liked
    0 times

    sitemap & spiders

    Can somebody please tell me what this setting should be set at?

    Max URLs in Sitemap Files - I currently have it set at 40000 although default is set at 50000 (Google limit)

    Also I set this up just under a week ago but I havent seen vast amount of spiders being on the site, don't get me wrong they are coming on but not in silly numbers like some people mention.

    Is this still ok?

    Oh also can you let me know if this url looks ok to you? I just want to make sure that the urls are set correctly.

    http://www.chatfootballforum.com/pre...html#post11162

  2. #2
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    36 times
    Blog Entries
    9
    We break up your sitemap, because of the huge amount of pages forums have, into smaller files. Google and the others read a file at a time. You should be fine with the default setting.

    You should verify your sitemap:
    Verifying your vBSEO Google/Yahoo Sitemap Generator is Installed Correctly

    Also check your Google Webmaster Tools to make sure there are no errors:
    Google Webmaster Tools - Recommended for All vBSEO Customers

    Beyond that, you need to give it more time so that we can assess your growth. Stick with it.

  3. #3
    Member
    Real Name
    Chat Football
    Join Date
    Jun 2008
    Posts
    71
    Liked
    0 times
    I upped mine by 10000 from 30000 to 40000, should I go back to default or is it ok at 40?

    Looks like it's ok? http://www.chatfootballforum.com/sitemap_index.xml.gz

    Google webmaster last downloaded 8hrs ago status was ok and 30844 urls submitted so looks good I think.

  4. #4
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    36 times
    Blog Entries
    9
    It should be fine. Leave it and see how it goes for about 2 weeks.

  5. #5
    Member
    Real Name
    Chat Football
    Join Date
    Jun 2008
    Posts
    71
    Liked
    0 times
    Ok thanks Joe, is there any reason as to why my threads are .html and yours aren't?

  6. #6
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    36 times
    Blog Entries
    9
    We selected another option that tests threads as virtual directories instead.

    Yours is the default.

  7. #7
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    i run 10,000 with a 30 sec pause between files.

    i get sql locks/high load if i just let it go.

  8. #8
    Member
    Real Name
    Chat Football
    Join Date
    Jun 2008
    Posts
    71
    Liked
    0 times
    Perhaps I should change it back to 30,000 then Brian. I was thinking the more the better but guess it isn't the case.

  9. #9
    Junior Member
    Real Name
    David
    Join Date
    Oct 2006
    Posts
    21
    Liked
    0 times
    Quote Originally Posted by briansol View Post
    i run 10,000 with a 30 sec pause between files.

    i get sql locks/high load if i just let it go.
    Brian,

    Have you done any testing to discover if Google will index more URLs when the sitemap files are larger or smaller? The reason I ask is that I originally set my file size at 48k. That was a couple years ago and let everything run on auto pilot. Now I'm starting to hyper-focus on getting better indexing. I submit about 2.5-million pages and Google seems to grab about 1200 to 1500 URLs from each sitemap file. If I reduced the file size to 24,000 URLs would Google still grab 1200 from each file? I'm going to give it a try and see what happens.

    ~ David

Similar Threads

  1. Vbseo Sitemap and Spiders...
    By gotlinks in forum General Discussion
    Replies: 3
    Last Post: 09-22-2007, 04:14 PM
  2. where are the spiders?
    By obx11 in forum General Discussion
    Replies: 8
    Last Post: 03-03-2007, 11:17 PM
  3. Spiders!
    By sohan in forum Analysis: Traffic & SERPS
    Replies: 5
    Last Post: 11-11-2006, 06:40 PM
  4. Understanding Spiders, vBSEO, and Sitemap
    By Sonnie in forum General Discussion
    Replies: 13
    Last Post: 05-19-2006, 04:14 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •