vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
vBSEO Total Support Team Launches DeskPro New vBSEO Discount Level for Network Builders vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

Sitemap crawling issue

This is a discussion on Sitemap crawling issue within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; Hello, I think that my sitemaps are not being crawled regularly by SE, although there are reports of sitemaps being ...

Go Back   vBulletin SEO Forums > vBSEO Google/Yahoo Sitemap > Troubleshooting

Enhancing 80 million pages.

Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read
  #1  
Old 07-30-2007, 11:12 AM
samel's Avatar
Member
 
Real Name: Sam
Join Date: Jul 2007
Posts: 32
Sitemap crawling issue

Hello,

I think that my sitemaps are not being crawled regularly by SE, although there are reports of sitemaps being downloaded.

Just for your info I have the ioncube version running on vBulletin 3.6.7 PL1 with several modifications from vBulletin.org. I have my vbseo_sitemap folder chmodded to 777 and the data dolder is also set to 777.

Here is my .htaccess

Code:
# Comment the following line (add '#' at the beginning)
# to disable mod_rewrite functions.
# Please note: you still need to disable the hack in
# the vBSEO control panel to stop url rewrites.
RewriteEngine On

# Some servers require the Rewritebase directive to be
# enabled (remove '#' at the beginning to activate)
# Please note: when enabled, you must include the path
# to your root vBulletin folder (i.e. RewriteBase /forums)
#RewriteBase /

#RewriteCond %{HTTP_HOST} !^www.yourdomain.com
#RewriteRule (.*) http://www.yourdomain.com/forums/$1 [L,R=301]

RewriteRule ^((urllist|sitemap_).*.(xml|txt)(.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

RewriteCond %{QUERY_STRING} !vbseourl=
RewriteCond %{REQUEST_URI} !(admincp/|modcp/|chat|cron)
RewriteRule ^(.*.php)$ vbseo.php?vbseourl=$1 [L,QSA]

RewriteCond %{REQUEST_FILENAME} !.(jpg|gif)$
RewriteRule ^(archive/.*)$ vbseo.php?vbseourl=$1 [L,QSA]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
RewriteRule ^(.+)$ vbseo.php?vbseourl=$1 [L,QSA]

RewriteEngine On
RewriteRule ^((urllist|sitemap).*.(xml|txt)(.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]
I will also forward you 4 screenshots of my issue..

1. This one is the VBSEO Forum rewrite settings
http://img165.imageshack.us/img165/2858/16091232uf1.jpg

2. Here are my vBulletin sitemap options
http://img227.imageshack.us/img227/1772/93668342lf0.jpg

3. The Sitemap Report
http://img165.imageshack.us/img165/5682/16626642xh5.jpg

4. The sitemaps downloads log
http://img227.imageshack.us/img227/7366/47712529lm8.jpg

5. And the Sitemaps SE activity log.
http://img227.imageshack.us/img227/5379/15012815jf5.jpg

I also have the vbseo conditional signatures installed.

I have submitted the sitemaps to Google, Yahoo etc.. and although they are processed it does not state the number of URL's. I have submitted all of three sitemap files (sitemap_index_xml.gz, sitemap_1.xml.gz, urllist.txt.gz) files to Google and Yahoo.


Please can you give me a thorough explanation of what might be the issue here..

Also feel free to browse my forum and let me know what might be the issue, or I am just panicking while everything is OK, and it just takes time for SE to start crawling the forum with the new URL's. Before vBSEO, my site was regularly crawled by Google and Yahoo...
Insurance Agent Forum

Regards.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #2  
Old 07-30-2007, 12:26 PM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 21,923
Hello,

as long as you see that sitemaps are downloaded by SE bots, it must be working correctly. The crawling of your content pages may be slower though and is not guaranteed to start right away.
Quote:
I have submitted the sitemaps to Google, Yahoo etc.. and although they are processed it does not state the number of URL's. I have submitted all of three sitemap files (sitemap_index_xml.gz, sitemap_1.xml.gz, urllist.txt.gz) files to Google and Yahoo.
When did you submit it?
__________________
Oleg Ignatiuk / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #3  
Old 07-30-2007, 12:59 PM
samel's Avatar
Member
 
Real Name: Sam
Join Date: Jul 2007
Posts: 32
Quote:
Originally Posted by Oleg Ignatiuk View Post
Hello,

as long as you see that sitemaps are downloaded by SE bots, it must be working correctly. The crawling of your content pages may be slower though and is not guaranteed to start right away.When did you submit it?
I re-submitted the sitemaps yesterday. Google has already downloaded and the sitemap status is OK, while the number of URL's is still not being displayed..

Are my custom rewrites maybe causing the sitemaps to be analyzed longer than usual?

Currently my subforum URL's look like this
General Insurance Agent Discussions - Insurance Agent Forum
Code:
http://www.insurance-forums.net/forum/general-insurance-agent-discussions/
but I took off the subforum from the thread URL and this is how they look now
New zero premium life insurance policy - Insurance Agent Forum
Code:
http://www.insurance-forums.net/forum/new-zero-premium-life-insurance-policy-thread1074.html
Thanks for your time.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #4  
Old 07-31-2007, 06:10 AM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 21,923
Quote:
I re-submitted the sitemaps yesterday. Google has already downloaded and the sitemap status is OK, while the number of URL's is still not being displayed..
Perhaps there is some issue with Google webmaster central at the moment, so there may be delays with sitemaps processing. There are also reports related to Google Analytics being down last days: Google Analytics Down?



Quote:
Are my custom rewrites maybe causing the sitemaps to be analyzed longer than usual?
That should not be related.
__________________
Oleg Ignatiuk / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads

Thread Thread Starter Forum Replies Last Post
vBSEO Sitemap Generator 2.0 Released Juan Muriente vBSEO Announcements 18 12-11-2007 02:44 PM
Google Crawling - Sitemap. Lasher General Discussion 5 07-29-2007 10:15 AM
vBSEO Google/Yahoo Sitemap Generator for vBulletin 3.6.x, vBulletin 3.5.x & vBulletin 3.0.x Joe Ward General Discussion 0 12-17-2005 02:11 PM


All times are GMT -4. The time now is 10:11 PM.


Powered by vBulletin Version 3.8.0 Release Candidate 2
Copyright ©2000 - 2009, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.