Google-Sitemap not loaded (403)

symptome · #1 09-06-2007, 01:38 AM

My file is here: http://www.symptome.ch/vbboard/sitemap_index.xml.gz

It loaded some months without problems. But since 21 of august, there seemes to be a problem, because SE-Bot-Activity shows NO activity.

But the download-log looks fine.

What can it be that causes that 403?

Thanks a lot!

Marcel

Mert Gökçeimam · #2 09-06-2007, 02:18 AM

Hello ,

It loads fine for me

Code:

<sitemapindex xsi:schemaLocation="             http://www.sitemaps.org/schemas/sitemap/0.9             http://www.sitemaps.org/schemas/sitemap/09/siteindex.xsd">
−
	<sitemap>
<loc>http://www.symptome.ch/vbboard/sitemap_1.xml.gz</loc>
<lastmod>2007-09-06T02:07:46+00:00</lastmod>
</sitemap>
</sitemapindex>

symptome · #3 09-06-2007, 03:31 AM

But on https://www.google.com/webmasters/tools/siteoverview

it shows this

Quote:

sitemap_index.xml.gz Web Vor 3 Stunden Vor 3 Stunden Fehler

Allgemeiner HTTP-Fehler: Fehler 4xx
Bei dem Versuch, auf Ihre Sitemap zuzugreifen, ist ein Fehler aufgetreten. Stellen Sie sicher, dass Ihre Sitemap unseren Richtlinien entspricht und dass der Zugriff an der von Ihnen angegebenen Position möglich ist. Reichen Sie anschließend die Daten erneut ein. [?] HTTP-Fehler:
Datum des Fehlers: 403 (Verboten) [?]
05.09.2007

What can I do?
Is there a possibility to give you the password to google-sitemap? Would that help solving the problem?

The strange thing is, that this damn thing worked for months. And I'm not aware of having changed something these days.

Marcel

symptome · #4 09-06-2007, 04:27 AM

Sorry for the german:
It's a 403 (forbidden):
It is a fault to access the sitemap and I should put the sitemap on the correct place and be sure that the sitemaps content is correct.

symptome · #5 09-06-2007, 11:39 AM

anyone?

Mert Gökçeimam · #6 09-06-2007, 11:47 AM

Hello ,

Did you submit your board url to your sitemaps accound as Das Ende der Symptombekämpfung - Das Gesundheitsforum or did you submit it as Das Ende der Symptombekämpfung - Das Gesundheitsforum

symptome · #7 09-06-2007, 02:20 PM

The first one without "vbboard".
I justed deleted the sitemap and created a new one unter www.symptome.ch/sitemap_index.xml.gz
I could enter that but after 15 Minutes there was the 403.
Now I deleted the sitemap and tried to enter www.symptome.ch/vbboard/sitemap_index.xml.gz. THEN: the 403 came while entering!!
I went back to enter without vbboard and NOW: also there is the 403.
Now I am not able to load my sitemap again.
Strange!!

I have a .htaccess under Das Ende der Symptombekämpfung - Das Gesundheitsforum like this:

Quote:

RewriteEngine On
RewriteRule ^((urllist|sitemap).*\.(xml|txt)(\.gz)?)$ /vbboard/vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

This is the .htaccess in Das Ende der Symptombekämpfung - Das Gesundheitsforum:

Quote:

# Comment the following line (add '#' at the beginning)
# to disable mod_rewrite functions.
# Please note: you still need to disable the hack in
# the vBSEO control panel to stop url rewrites.
RewriteEngine On
# Some servers require the Rewritebase directive to be
# enabled (remove '#' at the beginning to activate)
# Please note: when enabled, you must include the path
# to your root vB folder (i.e. RewriteBase /forums)
#RewriteBase /
#RewriteCond %{HTTP_HOST} !^www\.yourdomain\.com
#RewriteRule (.*) http://www.yourdomain.com/forums/$1 [L,R=301]
RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]
RewriteCond %{QUERY_STRING} !vbseourl=
RewriteCond %{REQUEST_URI} !(admincp/|modcp/|chat|cron)
RewriteRule ^(.*\.php)$ vbseo.php?vbseourl=$1 [L,QSA]
RewriteCond %{REQUEST_FILENAME} !\.(jpg|gif)$
RewriteRule ^(archive/.*)$ vbseo.php?vbseourl=$1 [L,QSA]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
RewriteRule ^(.+)$ vbseo.php?vbseourl=$1 [L,QSA]

# Das musste eingefügt werden, um beim Direkt-Antworten keine Probleme zu kriegen
<IfModule mod_security.c>
SecFilterEngine Off
SecFilterScanPOST Off
</IfModule>

In vBSEO Google / Yahoo Sitemap for vBulletin there ist no .htaccess.
And in www.symptome.ch/vbboard/vbseo_sitemap/data there is a .htaccess with:

Quote:

deny from all

This is the only .htaccess with rwx-rwx-rwx. Others just have rw-r-r.

In the google-sitemap-tool, I see that google loaded down the sitemap. But at the end, there is the 403.

Thanks a lot!

symptome · #8 09-06-2007, 02:26 PM

When I look in google-webmaster-tools in web-crawl, then I see a 403-result on Das Ende der Symptombekämpfung - Das Gesundheitsforum since 27.8.07.

Ist this a result of the sitemap-problem or does it cause it?

symptome · #9 09-06-2007, 02:36 PM

This is driving me mad:
Now I can enter www.symptome.ch/sitemap_index.xml.gz again without any errors ...
It loads it and after about 15 minutes ... 403.

Oleg Ignatiuk · #10 09-06-2007, 06:11 PM

Hello,

I've investigated this issue a bit and it seems to be similar to the case described in:
Debugging The Network Unreachable / Robots. txt Unreachable Error - HomeWithAndrew.com

The problem was related to the host/datacenter:

Quote:

Our firewall has an automated mechanism which will block IP addresses deemed to be making too many concurrent connections to our server in a short time. Our security department has whitelisted the google network range that is noticed to make these connections. On top of that we have made the firewall less stringent in the sense we will allow a higher threshold of concurrent connections compared to previously. Based on your feedback, the configuration is just right.

It is not the server that has the problem but the datacenter network that is not reachable from certain locations. We have not change any settings at the time. However, it is possible that there are more users who use Google Sitemap, causing increased concurrent connections to the server. For the current issue, it appears that our firewall’s stringent policy has temporary block the bot.

symptome · #11 09-07-2007, 03:45 AM

Thank you very much!
I asked my provider, if he limited Google-Bot ... and yes, he did.
Because there was too much traffic from google-Bot, they had to limit it.

Now they reactivated it and i've got no problems anymore to send my sitemap!!!
I've got 24'000 Pages with a webcrawl-error. And my sitemap just submitted 14'000 Pages.

- Can I do anything more, so that Google fixes those webcrawl-errors?
- How much time do I have to calculate, until the situation is like before (where I had no webcrawl-errors and plenty of visitors from google? I have to say, that I now know, how good vbseo/vbsitemap is for my site, because I lost 65% of my traffic an my adsense-income in just 2 weeks.
- My provider said something about "limiting google-bot". I saw a page in webmaster-tools, where I can set the webcrawl-rate. is that really a good idea to limit that? Or does this has an impact on how good my site is crawled and my position in google (and so the number of visitors from google)?
- and last question: if I would change my shared-hosting into an own server (which would cost me a minimum of 300.- Dollars a month) ... could it be that the extra-performance with that server would have a great impact on how google is rating me?

Thanks again!!!

Marcel

Joe Ward · #12 04-06-2008, 10:26 PM

You should not limit Googlebot unless it was causing server issues for you directly.

Your webhost should *NEVER* be limiting GoogleBot period. As long as you are under your limit of bandwidth made available to you, they should not be making decisions about what traffic to block, especially something as vital to a site's success as GoogleBot.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Google Now Reporting Anchor Text Phrases	Eros	SEO Buzz	5	02-11-2009 03:42 PM
Interview with Google's Matt Cutts about Next-Generation Search	Eros	SEO Buzz	2	03-16-2007 12:58 PM
Google Sitemap Error	Exitilus	Troubleshooting	3	01-18-2007 04:23 PM
vBSEO Google/Yahoo Sitemap Generator for vBulletin 3.6.x, vBulletin 3.5.x & vBulletin 3.0.x	Joe Ward	General Discussion	0	12-17-2005 01:11 PM

SEO

Google-Sitemap not loaded (403)

Google-Sitemap not loaded (403)

Similar Threads