Results 1 to 4 of 4

Sitemap crawling issue

This is a discussion on Sitemap crawling issue within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; Hello, I think that my sitemaps are not being crawled regularly by SE, although there are reports of sitemaps being ...

  1. #1
    Member Array samel's Avatar
    Real Name
    Val
    Join Date
    Jul 2007
    Posts
    57
    Liked
    7 times

    Sitemap crawling issue

    Hello,

    I think that my sitemaps are not being crawled regularly by SE, although there are reports of sitemaps being downloaded.

    Just for your info I have the ioncube version running on vBulletin 3.6.7 PL1 with several modifications from vBulletin.org. I have my vbseo_sitemap folder chmodded to 777 and the data dolder is also set to 777.

    Here is my .htaccess

    Code:
    # Comment the following line (add '#' at the beginning)
    # to disable mod_rewrite functions.
    # Please note: you still need to disable the hack in
    # the vBSEO control panel to stop url rewrites.
    RewriteEngine On
    
    # Some servers require the Rewritebase directive to be
    # enabled (remove '#' at the beginning to activate)
    # Please note: when enabled, you must include the path
    # to your root vBulletin folder (i.e. RewriteBase /forums)
    #RewriteBase /
    
    #RewriteCond %{HTTP_HOST} !^www.yourdomain.com
    #RewriteRule (.*) http://www.yourdomain.com/forums/$1 [L,R=301]
    
    RewriteRule ^((urllist|sitemap_).*.(xml|txt)(.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]
    
    RewriteCond %{QUERY_STRING} !vbseourl=
    RewriteCond %{REQUEST_URI} !(admincp/|modcp/|chat|cron)
    RewriteRule ^(.*.php)$ vbseo.php?vbseourl=$1 [L,QSA]
    
    RewriteCond %{REQUEST_FILENAME} !.(jpg|gif)$
    RewriteRule ^(archive/.*)$ vbseo.php?vbseourl=$1 [L,QSA]
    
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
    RewriteRule ^(.+)$ vbseo.php?vbseourl=$1 [L,QSA]
    
    RewriteEngine On
    RewriteRule ^((urllist|sitemap).*.(xml|txt)(.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]
    I will also forward you 4 screenshots of my issue..

    1. This one is the VBSEO Forum rewrite settings
    http://img165.imageshack.us/img165/2858/16091232uf1.jpg

    2. Here are my vBulletin sitemap options
    http://img227.imageshack.us/img227/1772/93668342lf0.jpg

    3. The Sitemap Report
    http://img165.imageshack.us/img165/5682/16626642xh5.jpg

    4. The sitemaps downloads log
    http://img227.imageshack.us/img227/7366/47712529lm8.jpg

    5. And the Sitemaps SE activity log.
    http://img227.imageshack.us/img227/5379/15012815jf5.jpg

    I also have the vbseo conditional signatures installed.

    I have submitted the sitemaps to Google, Yahoo etc.. and although they are processed it does not state the number of URL's. I have submitted all of three sitemap files (sitemap_index_xml.gz, sitemap_1.xml.gz, urllist.txt.gz) files to Google and Yahoo.


    Please can you give me a thorough explanation of what might be the issue here..

    Also feel free to browse my forum and let me know what might be the issue, or I am just panicking while everything is OK, and it just takes time for SE to start crawling the forum with the new URL's. Before vBSEO, my site was regularly crawled by Google and Yahoo...
    Insurance Agent Forum

    Regards.

  2. #2
    vBSEO Staff Array Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,818
    Liked
    192 times
    Hello,

    as long as you see that sitemaps are downloaded by SE bots, it must be working correctly. The crawling of your content pages may be slower though and is not guaranteed to start right away.
    I have submitted the sitemaps to Google, Yahoo etc.. and although they are processed it does not state the number of URL's. I have submitted all of three sitemap files (sitemap_index_xml.gz, sitemap_1.xml.gz, urllist.txt.gz) files to Google and Yahoo.
    When did you submit it?
    Oleg Ignatiuk / Crawlability Inc.
    Security bulletin - Patch Level for all supported versions released

    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  3. #3
    Member Array samel's Avatar
    Real Name
    Val
    Join Date
    Jul 2007
    Posts
    57
    Liked
    7 times
    Quote Originally Posted by Oleg Ignatiuk View Post
    Hello,

    as long as you see that sitemaps are downloaded by SE bots, it must be working correctly. The crawling of your content pages may be slower though and is not guaranteed to start right away.When did you submit it?
    I re-submitted the sitemaps yesterday. Google has already downloaded and the sitemap status is OK, while the number of URL's is still not being displayed..

    Are my custom rewrites maybe causing the sitemaps to be analyzed longer than usual?

    Currently my subforum URL's look like this
    General Insurance Agent Discussions - Insurance Agent Forum
    Code:
    http://www.insurance-forums.net/forum/general-insurance-agent-discussions/
    but I took off the subforum from the thread URL and this is how they look now
    New zero premium life insurance policy - Insurance Agent Forum
    Code:
    http://www.insurance-forums.net/forum/new-zero-premium-life-insurance-policy-thread1074.html
    Thanks for your time.

  4. #4
    vBSEO Staff Array Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,818
    Liked
    192 times
    I re-submitted the sitemaps yesterday. Google has already downloaded and the sitemap status is OK, while the number of URL's is still not being displayed..
    Perhaps there is some issue with Google webmaster central at the moment, so there may be delays with sitemaps processing. There are also reports related to Google Analytics being down last days: Google Analytics Down?



    Are my custom rewrites maybe causing the sitemaps to be analyzed longer than usual?
    That should not be related.
    Oleg Ignatiuk / Crawlability Inc.
    Security bulletin - Patch Level for all supported versions released

    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


Similar Threads

  1. vBSEO Sitemap Generator 2.0 Released
    By Juan Muriente in forum vBSEO Announcements
    Replies: 18
    Last Post: 12-11-2007, 01:44 PM
  2. Google Crawling - Sitemap.
    By Lasher in forum General Discussion
    Replies: 5
    Last Post: 07-29-2007, 09:15 AM
  3. Replies: 0
    Last Post: 12-17-2005, 01:11 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •