vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now!
vBSEO 2.0 Style Released vBSEO 3.3.0 GOLD Launched vBSEO's "LiveStats" for Google Analytics vB Sitemap Generator, Version 2.5 Success with vBSEO = 600ore Web Visitors + $1400 in a Day!

Google-Sitemap not loaded (403)

This is a discussion on Google-Sitemap not loaded (403) within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; My file is here: http://www.symptome.ch/vbboard/sitemap_index.xml.gz It loaded some months without problems. But since 21 of august, there seemes to be ...

Go Back   vBulletin SEO Forums > vBSEO Google/Yahoo Sitemap > Troubleshooting

Enhancing 80 million pages.

  #1  
Old 09-06-2007, 01:38 AM
Senior Member
 
Real Name: Admin
Join Date: Nov 2006
Posts: 165
Google-Sitemap not loaded (403)

My file is here: http://www.symptome.ch/vbboard/sitemap_index.xml.gz

It loaded some months without problems. But since 21 of august, there seemes to be a problem, because SE-Bot-Activity shows NO activity.

But the download-log looks fine.

What can it be that causes that 403?

Thanks a lot!

Marcel
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #2  
Old 09-06-2007, 02:18 AM
Mert Gökçeimam's Avatar
vBSEO.com Webmaster
vBSEO Total Customer SupportDesign for SEOBig Board Administrator
 
Real Name: Lizard King
Join Date: Oct 2005
Location: Istanbul
Posts: 13,216
Blog Entries: 4
Send a message via MSN to Mert Gökçeimam
Hello ,

It loads fine for me

Code:
<sitemapindex xsi:schemaLocation="             http://www.sitemaps.org/schemas/sitemap/0.9             http://www.sitemaps.org/schemas/sitemap/09/siteindex.xsd">
−
	<sitemap>
<loc>http://www.symptome.ch/vbboard/sitemap_1.xml.gz</loc>
<lastmod>2007-09-06T02:07:46+00:00</lastmod>
</sitemap>
</sitemapindex>
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #3  
Old 09-06-2007, 03:31 AM
Senior Member
 
Real Name: Admin
Join Date: Nov 2006
Posts: 165
But on https://www.google.com/webmasters/tools/siteoverview

it shows this
Quote:

sitemap_index.xml.gz Web Vor 3 Stunden Vor 3 Stunden Fehler

Allgemeiner HTTP-Fehler: Fehler 4xx
Bei dem Versuch, auf Ihre Sitemap zuzugreifen, ist ein Fehler aufgetreten. Stellen Sie sicher, dass Ihre Sitemap unseren Richtlinien entspricht und dass der Zugriff an der von Ihnen angegebenen Position möglich ist. Reichen Sie anschließend die Daten erneut ein. [?] HTTP-Fehler:
Datum des Fehlers: 403 (Verboten) [?]
05.09.2007
What can I do?
Is there a possibility to give you the password to google-sitemap? Would that help solving the problem?

The strange thing is, that this damn thing worked for months. And I'm not aware of having changed something these days.

Marcel
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #4  
Old 09-06-2007, 04:27 AM
Senior Member
 
Real Name: Admin
Join Date: Nov 2006
Posts: 165
Sorry for the german:
It's a 403 (forbidden):
It is a fault to access the sitemap and I should put the sitemap on the correct place and be sure that the sitemaps content is correct.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #5  
Old 09-06-2007, 11:39 AM
Senior Member
 
Real Name: Admin
Join Date: Nov 2006
Posts: 165
anyone?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #6  
Old 09-06-2007, 11:47 AM
Mert Gökçeimam's Avatar
vBSEO.com Webmaster
vBSEO Total Customer SupportDesign for SEOBig Board Administrator
 
Real Name: Lizard King
Join Date: Oct 2005
Location: Istanbul
Posts: 13,216
Blog Entries: 4
Send a message via MSN to Mert Gökçeimam
Hello ,

Did you submit your board url to your sitemaps accound as Das Ende der Symptombekämpfung - Das Gesundheitsforum or did you submit it as Das Ende der Symptombekämpfung - Das Gesundheitsforum
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #7  
Old 09-06-2007, 02:20 PM
Senior Member
 
Real Name: Admin
Join Date: Nov 2006
Posts: 165
The first one without "vbboard".
I justed deleted the sitemap and created a new one unter www.symptome.ch/sitemap_index.xml.gz
I could enter that but after 15 Minutes there was the 403.
Now I deleted the sitemap and tried to enter www.symptome.ch/vbboard/sitemap_index.xml.gz. THEN: the 403 came while entering!!
I went back to enter without vbboard and NOW: also there is the 403.
Now I am not able to load my sitemap again.
Strange!!


I have a .htaccess under Das Ende der Symptombekämpfung - Das Gesundheitsforum like this:
Quote:
RewriteEngine On
RewriteRule ^((urllist|sitemap).*\.(xml|txt)(\.gz)?)$ /vbboard/vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]
This is the .htaccess in Das Ende der Symptombekämpfung - Das Gesundheitsforum:
Quote:
# Comment the following line (add '#' at the beginning)
# to disable mod_rewrite functions.
# Please note: you still need to disable the hack in
# the vBSEO control panel to stop url rewrites.
RewriteEngine On
# Some servers require the Rewritebase directive to be
# enabled (remove '#' at the beginning to activate)
# Please note: when enabled, you must include the path
# to your root vB folder (i.e. RewriteBase /forums)
#RewriteBase /
#RewriteCond %{HTTP_HOST} !^www\.yourdomain\.com
#RewriteRule (.*) http://www.yourdomain.com/forums/$1 [L,R=301]
RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]
RewriteCond %{QUERY_STRING} !vbseourl=
RewriteCond %{REQUEST_URI} !(admincp/|modcp/|chat|cron)
RewriteRule ^(.*\.php)$ vbseo.php?vbseourl=$1 [L,QSA]
RewriteCond %{REQUEST_FILENAME} !\.(jpg|gif)$
RewriteRule ^(archive/.*)$ vbseo.php?vbseourl=$1 [L,QSA]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
RewriteRule ^(.+)$ vbseo.php?vbseourl=$1 [L,QSA]

# Das musste eingefügt werden, um beim Direkt-Antworten keine Probleme zu kriegen
<IfModule mod_security.c>
SecFilterEngine Off
SecFilterScanPOST Off
</IfModule>
In vBSEO Google / Yahoo Sitemap for vBulletin there ist no .htaccess.
And in www.symptome.ch/vbboard/vbseo_sitemap/data there is a .htaccess with:
Quote:
deny from all
This is the only .htaccess with rwx-rwx-rwx. Others just have rw-r-r.

In the google-sitemap-tool, I see that google loaded down the sitemap. But at the end, there is the 403.

Thanks a lot!
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #8  
Old 09-06-2007, 02:26 PM
Senior Member
 
Real Name: Admin
Join Date: Nov 2006
Posts: 165
When I look in google-webmaster-tools in web-crawl, then I see a 403-result on Das Ende der Symptombekämpfung - Das Gesundheitsforum since 27.8.07.

Ist this a result of the sitemap-problem or does it cause it?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #9  
Old 09-06-2007, 02:36 PM
Senior Member
 
Real Name: Admin
Join Date: Nov 2006
Posts: 165
This is driving me mad:
Now I can enter www.symptome.ch/sitemap_index.xml.gz again without any errors ...
It loads it and after about 15 minutes ... 403.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #10  
Old 09-06-2007, 06:11 PM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer Support
 
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 23,637
Hello,

I've investigated this issue a bit and it seems to be similar to the case described in:
Debugging The Network Unreachable / Robots. txt Unreachable Error - HomeWithAndrew.com

The problem was related to the host/datacenter:
Quote:
Our firewall has an automated mechanism which will block IP addresses deemed to be making too many concurrent connections to our server in a short time. Our security department has whitelisted the google network range that is noticed to make these connections. On top of that we have made the firewall less stringent in the sense we will allow a higher threshold of concurrent connections compared to previously. Based on your feedback, the configuration is just right.

It is not the server that has the problem but the datacenter network that is not reachable from certain locations. We have not change any settings at the time. However, it is possible that there are more users who use Google Sitemap, causing increased concurrent connections to the server. For the current issue, it appears that our firewall’s stringent policy has temporary block the bot.
__________________
Oleg Ignatiuk / Crawlability Inc.
vBSEO Styles: Version 1.0 PRO For Everyone Version 2.0 For vBSEO Customers

vBSEO 3.3.0 Gold Released with New "Virtual HTML Display" Feature Available for Download Now

6X Traffic - $1400 in One Day with vBSEO! Max Out Your SEO

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #11  
Old 09-07-2007, 03:45 AM
Senior Member
 
Real Name: Admin
Join Date: Nov 2006
Posts: 165
Thank you very much!
I asked my provider, if he limited Google-Bot ... and yes, he did.
Because there was too much traffic from google-Bot, they had to limit it.

Now they reactivated it and i've got no problems anymore to send my sitemap!!!
I've got 24'000 Pages with a webcrawl-error. And my sitemap just submitted 14'000 Pages.

- Can I do anything more, so that Google fixes those webcrawl-errors?
- How much time do I have to calculate, until the situation is like before (where I had no webcrawl-errors and plenty of visitors from google? I have to say, that I now know, how good vbseo/vbsitemap is for my site, because I lost 65% of my traffic an my adsense-income in just 2 weeks.
- My provider said something about "limiting google-bot". I saw a page in webmaster-tools, where I can set the webcrawl-rate. is that really a good idea to limit that? Or does this has an impact on how good my site is crawled and my position in google (and so the number of visitors from google)?
- and last question: if I would change my shared-hosting into an own server (which would cost me a minimum of 300.- Dollars a month) ... could it be that the extra-performance with that server would have a great impact on how google is rating me?

Thanks again!!!

Marcel
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #12  
Old 04-06-2008, 10:26 PM
Joe Ward's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Joseph Ward
Join Date: Jun 2005
Location: Puerto Rico
Posts: 24,060
Blog Entries: 9
You should not limit Googlebot unless it was causing server issues for you directly.

Your webhost should *NEVER* be limiting GoogleBot period. As long as you are under your limit of bandwidth made available to you, they should not be making decisions about what traffic to block, especially something as vital to a site's success as GoogleBot.
__________________
Joe Ward / Crawlability Inc.
vBSEO Styles: Version 1.0 PRO For Everyone Version 2.0 For vBSEO Customers

vBSEO 3.3.0 Gold Released with New "Virtual HTML Display" Feature Available for Download Now

6X Traffic - $1400 in One Day with vBSEO! Max Out Your SEO

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads

Thread Thread Starter Forum Replies Last Post
Google Now Reporting Anchor Text Phrases Eros SEO Buzz 5 02-11-2009 03:42 PM
Interview with Google's Matt Cutts about Next-Generation Search Eros SEO Buzz 2 03-16-2007 12:58 PM
Google Sitemap Error Exitilus Troubleshooting 3 01-18-2007 04:23 PM
vBSEO Google/Yahoo Sitemap Generator for vBulletin 3.6.x, vBulletin 3.5.x & vBulletin 3.0.x Joe Ward General Discussion 0 12-17-2005 01:11 PM


All times are GMT -4. The time now is 09:44 AM.


Powered by vBulletin Version 3.8.3
Copyright ©2000 - 2009, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.3.0 ©2009, Crawlability, Inc.