vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.

Results 1 to 12 of 12

Google-Sitemap not loaded (403)

This is a discussion on Google-Sitemap not loaded (403) within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; My file is here: http://www.symptome.ch/vbboard/sitemap_index.xml.gz It loaded some months without problems. But since 21 of august, there seemes to be ...

  1. #1
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    172

    Google-Sitemap not loaded (403)

    My file is here: http://www.symptome.ch/vbboard/sitemap_index.xml.gz

    It loaded some months without problems. But since 21 of august, there seemes to be a problem, because SE-Bot-Activity shows NO activity.

    But the download-log looks fine.

    What can it be that causes that 403?

    Thanks a lot!

    Marcel

  2. #2
    vBSEO.com Webmaster Mert Gökçeimam's Avatar
    Real Name
    Lizard King
    Join Date
    Oct 2005
    Location
    Istanbul
    Posts
    16,444
    Blog Entries
    4
    Hello ,

    It loads fine for me

    Code:
    <sitemapindex xsi:schemaLocation="             http://www.sitemaps.org/schemas/sitemap/0.9             http://www.sitemaps.org/schemas/sitemap/09/siteindex.xsd">
    −
    	<sitemap>
    <loc>http://www.symptome.ch/vbboard/sitemap_1.xml.gz</loc>
    <lastmod>2007-09-06T02:07:46+00:00</lastmod>
    </sitemap>
    </sitemapindex>
    
    Mert Gökçeimam / Crawlability Inc.

    Yeni vBSEO Kontrol Paneli ve vBSEO 3.5 Yayın Tarihleri

    vBSEO 3.5 Beta 2 Yayınlandı - vBulletin 4 Uyumlu!



    Kişisel Sitem : Felsefe - Kültür merkeziniz

  3. #3
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    172
    But on https://www.google.com/webmasters/tools/siteoverview

    it shows this

    sitemap_index.xml.gz Web Vor 3 Stunden Vor 3 Stunden Fehler

    Allgemeiner HTTP-Fehler: Fehler 4xx
    Bei dem Versuch, auf Ihre Sitemap zuzugreifen, ist ein Fehler aufgetreten. Stellen Sie sicher, dass Ihre Sitemap unseren Richtlinien entspricht und dass der Zugriff an der von Ihnen angegebenen Position möglich ist. Reichen Sie anschließend die Daten erneut ein. [?] HTTP-Fehler:
    Datum des Fehlers: 403 (Verboten) [?]
    05.09.2007
    What can I do?
    Is there a possibility to give you the password to google-sitemap? Would that help solving the problem?

    The strange thing is, that this damn thing worked for months. And I'm not aware of having changed something these days.

    Marcel

  4. #4
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    172
    Sorry for the german:
    It's a 403 (forbidden):
    It is a fault to access the sitemap and I should put the sitemap on the correct place and be sure that the sitemaps content is correct.

  5. #5
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    172
    anyone?

  6. #6
    vBSEO.com Webmaster Mert Gökçeimam's Avatar
    Real Name
    Lizard King
    Join Date
    Oct 2005
    Location
    Istanbul
    Posts
    16,444
    Blog Entries
    4
    Hello ,

    Did you submit your board url to your sitemaps accound as Das Ende der Symptombekämpfung - Das Gesundheitsforum or did you submit it as Das Ende der Symptombekämpfung - Das Gesundheitsforum
    Mert Gökçeimam / Crawlability Inc.

    Yeni vBSEO Kontrol Paneli ve vBSEO 3.5 Yayın Tarihleri

    vBSEO 3.5 Beta 2 Yayınlandı - vBulletin 4 Uyumlu!



    Kişisel Sitem : Felsefe - Kültür merkeziniz

  7. #7
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    172
    The first one without "vbboard".
    I justed deleted the sitemap and created a new one unter www.symptome.ch/sitemap_index.xml.gz
    I could enter that but after 15 Minutes there was the 403.
    Now I deleted the sitemap and tried to enter www.symptome.ch/vbboard/sitemap_index.xml.gz. THEN: the 403 came while entering!!
    I went back to enter without vbboard and NOW: also there is the 403.
    Now I am not able to load my sitemap again.
    Strange!!


    I have a .htaccess under Das Ende der Symptombekämpfung - Das Gesundheitsforum like this:
    RewriteEngine On
    RewriteRule ^((urllist|sitemap).*\.(xml|txt)(\.gz)?)$ /vbboard/vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]
    This is the .htaccess in Das Ende der Symptombekämpfung - Das Gesundheitsforum:
    # Comment the following line (add '#' at the beginning)
    # to disable mod_rewrite functions.
    # Please note: you still need to disable the hack in
    # the vBSEO control panel to stop url rewrites.
    RewriteEngine On
    # Some servers require the Rewritebase directive to be
    # enabled (remove '#' at the beginning to activate)
    # Please note: when enabled, you must include the path
    # to your root vB folder (i.e. RewriteBase /forums)
    #RewriteBase /
    #RewriteCond %{HTTP_HOST} !^www\.yourdomain\.com
    #RewriteRule (.*) http://www.yourdomain.com/forums/$1 [L,R=301]
    RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]
    RewriteCond %{QUERY_STRING} !vbseourl=
    RewriteCond %{REQUEST_URI} !(admincp/|modcp/|chat|cron)
    RewriteRule ^(.*\.php)$ vbseo.php?vbseourl=$1 [L,QSA]
    RewriteCond %{REQUEST_FILENAME} !\.(jpg|gif)$
    RewriteRule ^(archive/.*)$ vbseo.php?vbseourl=$1 [L,QSA]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
    RewriteRule ^(.+)$ vbseo.php?vbseourl=$1 [L,QSA]

    # Das musste eingefügt werden, um beim Direkt-Antworten keine Probleme zu kriegen
    <IfModule mod_security.c>
    SecFilterEngine Off
    SecFilterScanPOST Off
    </IfModule>
    In vBSEO Google / Yahoo Sitemap for vBulletin there ist no .htaccess.
    And in www.symptome.ch/vbboard/vbseo_sitemap/data there is a .htaccess with:
    deny from all
    This is the only .htaccess with rwx-rwx-rwx. Others just have rw-r-r.

    In the google-sitemap-tool, I see that google loaded down the sitemap. But at the end, there is the 403.

    Thanks a lot!

  8. #8
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    172
    When I look in google-webmaster-tools in web-crawl, then I see a 403-result on Das Ende der Symptombekämpfung - Das Gesundheitsforum since 27.8.07.

    Ist this a result of the sitemap-problem or does it cause it?

  9. #9
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    172
    This is driving me mad:
    Now I can enter www.symptome.ch/sitemap_index.xml.gz again without any errors ...
    It loads it and after about 15 minutes ... 403.

  10. #10
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    24,367
    Hello,

    I've investigated this issue a bit and it seems to be similar to the case described in:
    Debugging The Network Unreachable / Robots. txt Unreachable Error - HomeWithAndrew.com

    The problem was related to the host/datacenter:
    Our firewall has an automated mechanism which will block IP addresses deemed to be making too many concurrent connections to our server in a short time. Our security department has whitelisted the google network range that is noticed to make these connections. On top of that we have made the firewall less stringent in the sense we will allow a higher threshold of concurrent connections compared to previously. Based on your feedback, the configuration is just right.

    It is not the server that has the problem but the datacenter network that is not reachable from certain locations. We have not change any settings at the time. However, it is possible that there are more users who use Google Sitemap, causing increased concurrent connections to the server. For the current issue, it appears that our firewall’s stringent policy has temporary block the bot.
    Oleg Ignatiuk / Crawlability Inc.
    Sneek Preview Video of the new Control Panel

    vBSEO 3.5 RC2 (Pre-Release)- Released for your Evaluation


  11. #11
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    172
    Thank you very much!
    I asked my provider, if he limited Google-Bot ... and yes, he did.
    Because there was too much traffic from google-Bot, they had to limit it.

    Now they reactivated it and i've got no problems anymore to send my sitemap!!!
    I've got 24'000 Pages with a webcrawl-error. And my sitemap just submitted 14'000 Pages.

    - Can I do anything more, so that Google fixes those webcrawl-errors?
    - How much time do I have to calculate, until the situation is like before (where I had no webcrawl-errors and plenty of visitors from google? I have to say, that I now know, how good vbseo/vbsitemap is for my site, because I lost 65% of my traffic an my adsense-income in just 2 weeks.
    - My provider said something about "limiting google-bot". I saw a page in webmaster-tools, where I can set the webcrawl-rate. is that really a good idea to limit that? Or does this has an impact on how good my site is crawled and my position in google (and so the number of visitors from google)?
    - and last question: if I would change my shared-hosting into an own server (which would cost me a minimum of 300.- Dollars a month) ... could it be that the extra-performance with that server would have a great impact on how google is rating me?

    Thanks again!!!

    Marcel

  12. #12
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,849
    Blog Entries
    9
    You should not limit Googlebot unless it was causing server issues for you directly.

    Your webhost should *NEVER* be limiting GoogleBot period. As long as you are under your limit of bandwidth made available to you, they should not be making decisions about what traffic to block, especially something as vital to a site's success as GoogleBot.

Similar Threads

  1. Google Now Reporting Anchor Text Phrases
    By Eros in forum General Discussion
    Replies: 5
    Last Post: 02-11-2009, 04:42 PM
  2. Replies: 2
    Last Post: 03-16-2007, 01:58 PM
  3. Google Sitemap Error
    By Exitilus in forum Troubleshooting
    Replies: 3
    Last Post: 01-18-2007, 05:23 PM
  4. Replies: 0
    Last Post: 12-17-2005, 02:11 PM