Results 1 to 9 of 9

Am I doing something wrong?

This is a discussion on Am I doing something wrong? within the General Discussion forums, part of the vBSEO Google/Yahoo Sitemap category; According to my sitemap spider logs, Google seems to only crawl 50-100 (if I am lucky) pages per day. Yahoo ...

  1. #1
    vBSEO Staff Ace Shattock's Avatar
    Real Name
    Ace Shattock
    Join Date
    Jul 2005
    Location
    Auckland, New Zealand, New Zealand
    Posts
    4,012
    Liked
    13 times

    Am I doing something wrong?

    According to my sitemap spider logs, Google seems to only crawl 50-100 (if I am lucky) pages per day.

    Yahoo does several thousand.

    Is there maybe something I am doing wrong to cause G to not show me the love? Main page PR5, a few PR4 main pages...that should mean they come more often than they are.

  2. #2
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,744
    Liked
    169 times
    Hello,

    just in case, you can check the googlebot crawl stats for your site in Google Webmaster Account: http://www.google.com/webmasters/sitemaps/siteoverview.

    At least you will see "official" crawl stats and the crawl errors:
    HTTP errors
    Unreachable URLs
    URLs restricted by robots.txt
    URLs not followed
    URLs timed out

  3. #3
    vBSEO Staff Ace Shattock's Avatar
    Real Name
    Ace Shattock
    Join Date
    Jul 2005
    Location
    Auckland, New Zealand, New Zealand
    Posts
    4,012
    Liked
    13 times
    Hmm... errors:
    Quote Originally Posted by Google Sitemaps
    Below are pages that we tried to crawl (found either through links from your Sitemaps or from other pages) but couldn't access.
    http://www.nzboards.com/forums/membe...rklechick.html Sitemap Web General HTTP error Jan 4
    http://www.nzboards.com/forums/members/amystery.html Sitemap Web General HTTP error Jan 3
    http://www.nzboards.com/forums/members/covosat.html Sitemap Web General HTTP error Jan 3
    http://www.nzboards.com/forums/members/fatcash.html Web General HTTP error Jan 3
    http://www.nzboards.com/forums/members/latina17.html Web General HTTP error Jan 3
    http://www.nzboards.com/forums/members/nitefenix.html Sitemap Web General HTTP error Jan 3
    http://www.nzboards.com/forums/21010-post3.html Web 404 not found Jan 1
    Where is it getting those URLs from? :(

  4. #4
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,744
    Liked
    169 times
    Perhaps these URLs were catched by googlebot while you have tried different URL formats in vbseocp? Anyway, having a few errors like these is not a big problem, I believe.
    Or do you have a lot of them?

  5. #5
    vBSEO Staff Ace Shattock's Avatar
    Real Name
    Ace Shattock
    Join Date
    Jul 2005
    Location
    Auckland, New Zealand, New Zealand
    Posts
    4,012
    Liked
    13 times
    There are fewer errors than a page full .. and yes, they do appear to be related to changes in URL format I have made.

    I'm feeling a lot more secure now. Cheers Oleg.

    Gotta now wonder why Google doesn't crawl it as much as I would like.

  6. #6
    vBSEO Staff Ace Shattock's Avatar
    Real Name
    Ace Shattock
    Join Date
    Jul 2005
    Location
    Auckland, New Zealand, New Zealand
    Posts
    4,012
    Liked
    13 times
    Dammit. Those URLs are still showing up in my sitemap errors page, and all between Jan 15 and today. :(

  7. #7
    vBSEO Staff Ace Shattock's Avatar
    Real Name
    Ace Shattock
    Join Date
    Jul 2005
    Location
    Auckland, New Zealand, New Zealand
    Posts
    4,012
    Liked
    13 times
    Heres my sitemap_index.xml:
    Code:
    <?xml version='1.0' encoding='UTF-8'?>
    <sitemapindex xmlns="http://www.google.com/schemas/sitemap/0.84"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84
    http://http://www.google.com/schemas/sitemap/0.84/siteindex.xsd">
    <sitemap>http://www.nzboards.com/sitemap_1.xml.gz</sitemap>
    
    </sitemapindex>
    Notice the http://http://www.google.com part? Will that make a difference?

    Plus, here is something very scary from sitemap_2.xml...
    <loc>http://www.nzboards.com/forums/members/amystery.html</loc>
    . That is NOT following the rewrite rule I have for members. It is
    Code:
    members/[user_name]/
    .

  8. #8
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,744
    Liked
    169 times
    Notice the http://http://www.google.com part? Will that make a difference?
    That's fine - this is a part of xml structure definition.

    Plus, here is something very scary from sitemap_2.xml...
    sitemap_2? But in sitemap index you have only one sitemap file - sitemap_1.xml.gz! Perhaps you have sitemap_2.xml.gz from one of the previously generated sitemaps with smalled number of pages per file? You should remove it if that is the case.

  9. #9
    vBSEO Staff Ace Shattock's Avatar
    Real Name
    Ace Shattock
    Join Date
    Jul 2005
    Location
    Auckland, New Zealand, New Zealand
    Posts
    4,012
    Liked
    13 times
    Nice spotting! I have removed it. We will see what happens now.

Similar Threads

  1. Rewriting index.php?page=home in cmps
    By BamaStangGuy in forum Custom Rewrite Rules
    Replies: 10
    Last Post: 01-05-2006, 07:28 AM
  2. Sitemap format + wrong mime.
    By rob in forum Troubleshooting
    Replies: 8
    Last Post: 12-21-2005, 11:24 AM
  3. adding my vbadvanced pages....
    By BamaStangGuy in forum General Discussion
    Replies: 8
    Last Post: 12-12-2005, 10:18 AM
  4. All gone wrong ...
    By Michael in forum Troubleshooting
    Replies: 4
    Last Post: 09-26-2005, 10:40 PM
  5. Replies: 2
    Last Post: 08-11-2005, 07:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •