Page 1 of 2 1 2 LastLast
Results 1 to 15 of 18

robots.txt timeout?

This is a discussion on robots.txt timeout? within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; For some reason, my sitemap stopped working. It's not vbseo 's fault because joomla's sitemap is too reported as broken ...

  1. #1
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    168
    Liked
    0 times

    robots.txt timeout?

    For some reason, my sitemap stopped working. It's not vbseo's fault because joomla's sitemap is too reported as broken in google webmaster's tools.

    URL timeout: robots.txt timeout
    We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
    Here's my robots.txt
    http://www.chilehardware.com/robots.txt

    These are my sitemaps:
    VBSEO's: http://www.chilehardware.com/sitemap_index.xml.gz
    Joomla's: http://www.chilehardware.com/index.p...=xml&no_html=1


    how can a robots.txt be wrong? I edited it two days ago to add a delay for yahoo slurp. After noticing that google was avoiding my site, I removed that line. Now it doesn't work anymore

  2. #2
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    168
    Liked
    0 times
    update: I temporarily removed robots.txt to see what happens.

  3. #3
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    36 times
    Blog Entries
    9
    I think someone else recently reported strange issues after updating their robots.txt file.

  4. #4
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    168
    Liked
    0 times
    I thought that perhaps I uploaded it in binary mode, *shrug* anyway I have no robots.txt file right now, let's see what happens.

  5. #5
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    168
    Liked
    0 times
    Ok, I removed robots.txt and now I'm getting this message:

    Network unreachable: Network unreachable
    We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
    I guess somehow I'm restricting googlebot from accessing my page, but where to look for that?

  6. #6
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,744
    Liked
    169 times
    Please check this similar case: Google-Sitemap not loaded (403)

  7. #7
    Senior Member
    Real Name
    Seleno
    Join Date
    Mar 2007
    Posts
    255
    Liked
    0 times
    Blog Entries
    1
    Hi There
    i have the same problem
    how can i solve this problem?
    is it really from the server?
    my sitemaps loading and opening normally
    since 1 week every thing was okey with them and i saw all urls
    but since 2 days i see this message:
    Network unreachable: robots.txt unreachable
    We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

  8. #8
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,744
    Liked
    169 times
    You should contact your host to find out whether they apply the same filtering.

  9. #9
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    168
    Liked
    0 times
    I discarded the four most common reasons:

    1.- It's not robots.txt, because I validated, edited and removed it, with no changes whatsoever.
    2.- It's not my firewall (double checked, and the logs are showing header 200 for googlebot when he tries to crawl).
    3.- It's not my host. My server isn't behind any filter or external firewall but mine.
    4.- It's not the DNS, because I moved my domain's DNS to another nameserver and the problem is still there.

    It's amazing. It took me five years to get there, and google took a week to destroy it.

  10. #10
    Senior Member
    Real Name
    Seleno
    Join Date
    Mar 2007
    Posts
    255
    Liked
    0 times
    Blog Entries
    1
    omg
    what should we do then?
    google remove your site?

  11. #11
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    168
    Liked
    0 times
    nope, it's not a google ban, nor have we done anything that can fall into "black hat SEO techniques". It's just that google's DNS cannot resolve my URL.

    I've tried making google crawl my site directly by its IP number, and it does, but of course it retrieves a sitemap full of my urls, which in turn it cannot crawl.

  12. #12
    Senior Member
    Real Name
    Seleno
    Join Date
    Mar 2007
    Posts
    255
    Liked
    0 times
    Blog Entries
    1
    my support told me the same first time
    but i keep asking them, then they found that they block googlebot ip

  13. #13
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    168
    Liked
    0 times
    It's not the case. I'm using an external DNS, and my server is directly attached to the internet.

    To make sure, I used a spare domain I have, pointed to my machine, installed a copy of my site in another folder, created the vhost, then I went to google webmasters and added that domain: guess what, it passed with flying colors.

    I insist: google cannot resolve my URL.

  14. #14
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    168
    Liked
    0 times
    Edit: Finally, on saturday, the problem solved out of the blue. Google is back.

    I lost a lot of pageviews but I hope I'll catch up soon to what we had.

  15. #15
    Junior Member grandepuntotr's Avatar
    Real Name
    Erdi YILMAZ
    Join Date
    Jun 2007
    Location
    Ankara
    Posts
    24
    Liked
    0 times
    how did you solve this problem?

    my robots.txt is also timeouts, thats why; my google rank fell down. my forum was the first in search results but last 1 week it fell down to 8th. :(

    sometime googlebot indexing, another day when i look it is not indexed. i can reach my robots.txt via int. exp. or firefox normally (forum.myforumsite.com/robots.txt) but google webmaster tools say; robots.txt timeout detected and then not indexing my sitemaps :(
    Last edited by grandepuntotr; 03-16-2009 at 07:13 PM.

Page 1 of 2 1 2 LastLast

Similar Threads

  1. robots.txt
    By Zenith in forum General Discussion
    Replies: 64
    Last Post: 12-01-2010, 07:52 PM
  2. Cookies TimeOut probelem?
    By aycan555 in forum Pre-Sales Questions
    Replies: 1
    Last Post: 05-09-2007, 09:45 PM
  3. Timeout error when submitting to Yahoo
    By BamaStangGuy in forum Troubleshooting
    Replies: 12
    Last Post: 01-23-2006, 11:59 AM
  4. Replies: 17
    Last Post: 01-15-2006, 05:38 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •