Sitemap crawling issue

samel · #1 07-30-2007, 11:12 AM

Hello,

I think that my sitemaps are not being crawled regularly by SE, although there are reports of sitemaps being downloaded.

Just for your info I have the ioncube version running on vBulletin 3.6.7 PL1 with several modifications from vBulletin.org. I have my vbseo_sitemap folder chmodded to 777 and the data dolder is also set to 777.

Here is my .htaccess

Code:

# Comment the following line (add '#' at the beginning)
# to disable mod_rewrite functions.
# Please note: you still need to disable the hack in
# the vBSEO control panel to stop url rewrites.
RewriteEngine On

# Some servers require the Rewritebase directive to be
# enabled (remove '#' at the beginning to activate)
# Please note: when enabled, you must include the path
# to your root vBulletin folder (i.e. RewriteBase /forums)
#RewriteBase /

#RewriteCond %{HTTP_HOST} !^www.yourdomain.com
#RewriteRule (.*) http://www.yourdomain.com/forums/$1 [L,R=301]

RewriteRule ^((urllist|sitemap_).*.(xml|txt)(.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

RewriteCond %{QUERY_STRING} !vbseourl=
RewriteCond %{REQUEST_URI} !(admincp/|modcp/|chat|cron)
RewriteRule ^(.*.php)$ vbseo.php?vbseourl=$1 [L,QSA]

RewriteCond %{REQUEST_FILENAME} !.(jpg|gif)$
RewriteRule ^(archive/.*)$ vbseo.php?vbseourl=$1 [L,QSA]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
RewriteRule ^(.+)$ vbseo.php?vbseourl=$1 [L,QSA]

RewriteEngine On
RewriteRule ^((urllist|sitemap).*.(xml|txt)(.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

I will also forward you 4 screenshots of my issue..

1. This one is the VBSEO Forum rewrite settings
http://img165.imageshack.us/img165/2858/16091232uf1.jpg

2. Here are my vBulletin sitemap options
http://img227.imageshack.us/img227/1772/93668342lf0.jpg

3. The Sitemap Report
http://img165.imageshack.us/img165/5682/16626642xh5.jpg

4. The sitemaps downloads log
http://img227.imageshack.us/img227/7366/47712529lm8.jpg

5. And the Sitemaps SE activity log.
http://img227.imageshack.us/img227/5379/15012815jf5.jpg

I also have the vbseo conditional signatures installed.

I have submitted the sitemaps to Google, Yahoo etc.. and although they are processed it does not state the number of URL's. I have submitted all of three sitemap files (sitemap_index_xml.gz, sitemap_1.xml.gz, urllist.txt.gz) files to Google and Yahoo.

Please can you give me a thorough explanation of what might be the issue here..

Also feel free to browse my forum and let me know what might be the issue, or I am just panicking while everything is OK, and it just takes time for SE to start crawling the forum with the new URL's. Before vBSEO, my site was regularly crawled by Google and Yahoo...
Insurance Agent Forum

Regards.

Oleg Ignatiuk · #2 07-30-2007, 12:26 PM

Hello,

as long as you see that sitemaps are downloaded by SE bots, it must be working correctly. The crawling of your content pages may be slower though and is not guaranteed to start right away.

Quote:

I have submitted the sitemaps to Google, Yahoo etc.. and although they are processed it does not state the number of URL's. I have submitted all of three sitemap files (sitemap_index_xml.gz, sitemap_1.xml.gz, urllist.txt.gz) files to Google and Yahoo.

When did you submit it?

samel · #3 07-30-2007, 12:59 PM

Quote:

Originally Posted by Oleg Ignatiuk

Hello,

as long as you see that sitemaps are downloaded by SE bots, it must be working correctly. The crawling of your content pages may be slower though and is not guaranteed to start right away.When did you submit it?

I re-submitted the sitemaps yesterday. Google has already downloaded and the sitemap status is OK, while the number of URL's is still not being displayed..

Are my custom rewrites maybe causing the sitemaps to be analyzed longer than usual?

Currently my subforum URL's look like this
General Insurance Agent Discussions - Insurance Agent Forum

Code:

http://www.insurance-forums.net/forum/general-insurance-agent-discussions/

but I took off the subforum from the thread URL and this is how they look now
New zero premium life insurance policy - Insurance Agent Forum

Code:

http://www.insurance-forums.net/forum/new-zero-premium-life-insurance-policy-thread1074.html

Thanks for your time.

Oleg Ignatiuk · #4 07-31-2007, 06:10 AM

Quote:

I re-submitted the sitemaps yesterday. Google has already downloaded and the sitemap status is OK, while the number of URL's is still not being displayed..

Perhaps there is some issue with Google webmaster central at the moment, so there may be delays with sitemaps processing. There are also reports related to Google Analytics being down last days: Google Analytics Down?

Quote:

Are my custom rewrites maybe causing the sitemaps to be analyzed longer than usual?

That should not be related.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
vBSEO Sitemap Generator 2.0 Released	Juan Muriente	vBSEO Announcements	18	12-11-2007 02:44 PM
Google Crawling - Sitemap.	Lasher	General Discussion	5	07-29-2007 10:15 AM
vBSEO Google/Yahoo Sitemap Generator for vBulletin 3.6.x, vBulletin 3.5.x & vBulletin 3.0.x	Joe Ward	General Discussion	0	12-17-2005 02:11 PM

SEO

Sitemap crawling issue

Sitemap crawling issue

Similar Threads